Skip to content

Add task UUID label to Kubernetes pods#5544

Merged
xoxys merged 2 commits into
woodpecker-ci:mainfrom
henkka:k8s-label-task-uuid
Sep 23, 2025
Merged

Add task UUID label to Kubernetes pods#5544
xoxys merged 2 commits into
woodpecker-ci:mainfrom
henkka:k8s-label-task-uuid

Conversation

@henkka

@henkka henkka commented Sep 23, 2025

Copy link
Copy Markdown
Contributor

Slightly relates to #5402 and #3344

Problem

When using the Kubernetes backend with ReadWriteOnce PersistentVolumes, volumes can only be attached to one Kubernetes worker node at a time. This creates scheduling conflicts when workflow steps are scheduled to run on different nodes (due to worker node size constraints for example).

Background

The somewhat ideal solution would be to implement Kubernetes PodAffinity and PodAntiAffinity rules directly in Woodpecker to ensure:

  • All pods in the same workflow run on the same node (to share PVs)
  • Pods from different workflows avoid the same node (to prevent conflicts)

However, a previous attempt to add this functionality (PR #3345) was declined as there's different views how the root problem should be resolved. In our opinion, #3366 sounds like a good approach, but at this moment we cannot afford to wait for implementing such a major change.

Proposed Solution

Instead of modifying Woodpecker's core scheduling logic, this PR enables external solutions by adding a woodpecker-ci.org/task-uuid label to each pod. This label:

  • Contains the workflow/task UUID (identical for all pods in the same workflow)
  • Allows external tools like Kubernetes MutatingAdmissionWebhooks to inject appropriate affinity rules to the pods scheduled by Woodpecker Agent
  • Provides a clean separation of concerns between Woodpecker and cluster-specific scheduling policies

TL;DR: We figured this approach is the path of least resistance that allows Woodpecker CI users to overcome the background issue without big changes to Woodpecker CI project itself.

@henkka henkka changed the title Add task UUID label to Kubernetes pods for workflow affinity support Add task UUID label to Kubernetes pods Sep 23, 2025
@woodpecker-bot

woodpecker-bot commented Sep 23, 2025

Copy link
Copy Markdown
Contributor

Surge PR preview deployment succeeded. View it at https://woodpecker-ci-woodpecker-pr-5544.surge.sh

@codecov

codecov Bot commented Sep 23, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.72727% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 26.33%. Comparing base (7707e84) to head (b053436).

Files with missing lines Patch % Lines
pipeline/backend/kubernetes/pod.go 80.00% 2 Missing ⚠️
pipeline/backend/kubernetes/kubernetes.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5544   +/-   ##
=======================================
  Coverage   26.32%   26.33%           
=======================================
  Files         405      405           
  Lines       29028    29031    +3     
=======================================
+ Hits         7641     7644    +3     
  Misses      20688    20688           
  Partials      699      699           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xoxys xoxys added feature add new functionality backend/kubernetes labels Sep 23, 2025

@xoxys xoxys left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I like this approach as a pragmatic way to solve the current issue.

@xoxys xoxys merged commit 0bd69e8 into woodpecker-ci:main Sep 23, 2025
9 checks passed
@woodpecker-bot woodpecker-bot mentioned this pull request Sep 23, 2025
1 task
@henkka

henkka commented Sep 24, 2025

Copy link
Copy Markdown
Contributor Author

Thank you for the really quick turnaround on this pull request! We're amazed by how well this project is run and how easily we can contribute to it.

@henkka henkka deleted the k8s-label-task-uuid branch September 24, 2025 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend/kubernetes feature add new functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants