KEP-5055: DRA device taints: document state in 1.35 and plan beta in 1.36 by pohly · Pull Request #5716 · kubernetes/enhancements

pohly · 2025-12-03T13:21:58Z

One-line PR description: DRA device taints: document state in 1.35 and plan beta in 1.36
Issue link: DRA: device taints and tolerations #5055
Other comments:

The actual implementation in 1.35 was a bit different than planned. This gets reflected here, plus the changes for beta.

k8s-ci-robot · 2025-12-03T13:22:01Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

pohly · 2025-12-11T07:38:12Z

keps/sig-scheduling/5055-dra-device-taints-and-tolerations/README.md

+Automated upgrade/downgrade testing verifies that:
+- A DeviceTaintRule created before a downgrade prevents pod scheduling after a downgrade.
+- A pod which gets scheduled because of a toleration after the downgrade
+  is kept running after an upgrade.


I've written this in present tense ("verifies") because I am assuming that it will be true soon, maybe even before this PR gets merged.

However, it is not true yet right now. I'm currently working on rewriting the upgrade/downgrade tests in kubernetes/kubernetes#135664 and will add the test case described here soon.

I have it implemented and noticed one small difference: the KEP asks about "upgrade->downgrade->upgrade" (three cluster changes). The automated test only does "upgrade->downgrade" (two). Does this really matter?

Maybe, but as I cannot image what else can go wrong for "upgrade->downgrade->upgrade" that doesn't already go wrong for just "upgrade" I am not sure what I should test for.

My simplified test does:

A pod which gets scheduled on the previous release because of a toleration is kept running after an upgrade.

A DeviceTaintRule created to evict the pod before a downgrade prevents pod scheduling after a downgrade.

Has this been completed? Looks that way since kubernetes/kubernetes#135664 has merged. Asking both to check whether this comment should be updated to reflect the current state and if this has been updated so we can use the pattern for other features.

kubernetes/kubernetes#135664 is just preparation, and it had to be reverted. New attempt in kubernetes/kubernetes#136156.

The test described here is pending in kubernetes/kubernetes#135732, so not fully completed yet.

Asking both to check whether this comment should be updated to reflect the current state and if this has been updated so we can use the pattern for other features.

I'm struggling with how to handle this in KEPs. If we write this in future tense during the KEP planning period before KEP freeze, then we express the intent. I'm skeptical about such a KEP being updated later after the work has been done (low priority work never gets done, plus it needs another review round...).

If we write this in present tense, it's not quite correct yet, but the same is true for "stage: beta", which we also need to merge now. I'm leaning towards present tense and then during the final beta promotion PR checking that everything that was meant to be done really is done.

The same problem occurs further down when it asks for proof that tests are executed. That only works for tests which were already implemented in the previous release.

Ultimately the problem is that the KEP review process is trying to verify that a feature is ready for promotion at a time when that verification cannot be done completely yet - IMHO.

At this point in time (given that kubernetes/kubernetes#136156 merged, and kubernetes/kubernetes#135732 is still WIP) it's ok to leave it as is. But after the other merges it's probably best to link to these tests.

So do another update of the KEP README.md in a future PR?

liggitt

updates reflecting API decisions made during 1.35 lgtm

liggitt · 2025-12-11T15:52:32Z

keps/sig-scheduling/5055-dra-device-taints-and-tolerations/README.md


+Usage of DeviceTaintRules can be seen in the apiserver's
+`apiserver_resource_objects` metric with labels `group=resource.k8s.io` and
+`resource=deviceTaintRules`.


Suggested change

`resource=deviceTaintRules`.

`resource=devicetaintrules`.

Nit, but it would be good to fix it.

Agreed, I was merely waiting for more change requests.

Updated now:

https://github.com/kubernetes/enhancements/compare/725970b17f9857d2c69544435746aefdfd024bbf..6c3748487dbd66144a6e3b1d3a6e7bd1bdf01465

macsko · 2025-12-18T10:15:13Z

/approve
SIG Scheduling

thisisvaishnav · 2025-12-23T14:57:20Z

/assign

mortent · 2026-01-06T15:32:55Z

/wg device-management

pohly · 2026-01-13T07:19:56Z

/assign @soltysh

For PRR review/approval.

soltysh

/lgtm
/approve
it was previously approved by sig-scheduling here

/hold
in case you want to address the SLO question, the typo in the metric and implementation history.

soltysh · 2026-01-21T13:02:02Z

keps/sig-scheduling/5055-dra-device-taints-and-tolerations/README.md

+Automated upgrade/downgrade testing verifies that:
+- A DeviceTaintRule created before a downgrade prevents pod scheduling after a downgrade.
+- A pod which gets scheduled because of a toleration after the downgrade
+  is kept running after an upgrade.


At this point in time (given that kubernetes/kubernetes#136156 merged, and kubernetes/kubernetes#135732 is still WIP) it's ok to leave it as is. But after the other merges it's probably best to link to these tests.

soltysh · 2026-01-21T13:03:30Z

keps/sig-scheduling/5055-dra-device-taints-and-tolerations/README.md


+Usage of DeviceTaintRules can be seen in the apiserver's
+`apiserver_resource_objects` metric with labels `group=resource.k8s.io` and
+`resource=deviceTaintRules`.


Nit, but it would be good to fix it.

soltysh · 2026-01-21T13:05:22Z

keps/sig-scheduling/5055-dra-device-taints-and-tolerations/README.md


-###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
+As for normal pod scheduling of pods using ResourceClaims there is no SLO for
+scheduling with taints.


Is it much different from the default scheduling SLOs?

We only have SLOs for "stateless pods" - in particular, no volumes, no extra devices. As soon as those enter the picture, it becomes impossible to predict what additional latency might be introduced by the additional, non-Kubernetes components.

One could argue that pure scheduling should have an SLO, but even that then depends on the complexity of the ResourceSlices published by driver(s).

soltysh · 2026-01-21T13:06:52Z

keps/sig-scheduling/5055-dra-device-taints-and-tolerations/README.md


 - 1.33: first KEP revision and implementation
+- 1.35: revised alpha with `effect: None` and DeviceTaintRule status



Nit: missing entry for 1.36 update.

1.36 isn't "history" yet?

I can add "- 1.36: graduation to beta (tentative)" if you want something.

I always treat it as history when you start working on something within given timeline 😉

soltysh

/lgtm

soltysh · 2026-01-21T15:29:13Z

keps/sig-scheduling/5055-dra-device-taints-and-tolerations/README.md


 - 1.33: first KEP revision and implementation
+- 1.35: revised alpha with `effect: None` and DeviceTaintRule status



I always treat it as history when you start working on something within given timeline 😉

k8s-ci-robot · 2026-01-21T15:30:02Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: macsko, pohly, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~keps/prod-readiness/OWNERS~~ [soltysh]
~~keps/sig-scheduling/OWNERS~~ [macsko]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

soltysh · 2026-01-21T15:30:08Z

/hold cancel

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 3, 2025

k8s-ci-robot requested review from dom4ha and kikisdeliveryservice December 3, 2025 13:22

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. labels Dec 3, 2025

github-project-automation bot added this to SIG Scheduling Dec 3, 2025

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 3, 2025

pohly mentioned this pull request Dec 3, 2025

DRA: device taints and tolerations #5055

Open

15 tasks

pohly force-pushed the dra-device-taints-1.36 branch from e92153d to 725970b Compare December 11, 2025 07:35

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Dec 11, 2025

pohly changed the title ~~WIP: KEP-5055: DRA device taints: document state in 1.35 and plan beta in 1.36~~ KEP-5055: DRA device taints: document state in 1.35 and plan beta in 1.36 Dec 11, 2025

pohly marked this pull request as ready for review December 11, 2025 07:35

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 11, 2025

k8s-ci-robot requested review from macsko and palnabarun December 11, 2025 07:35

pohly commented Dec 11, 2025

View reviewed changes

liggitt reviewed Dec 11, 2025

View reviewed changes

k8s-ci-robot assigned thisisvaishnav Dec 23, 2025

k8s-ci-robot added the wg/device-management Categorizes an issue or PR as relevant to WG Device Management. label Jan 6, 2026

github-project-automation bot added this to Dynamic Resource Allocation Jan 6, 2026

github-project-automation bot moved this to 🆕 New in Dynamic Resource Allocation Jan 6, 2026

mortent moved this from 🆕 New to 👀 In review in Dynamic Resource Allocation Jan 6, 2026

k8s-ci-robot assigned soltysh Jan 13, 2026

soltysh approved these changes Jan 21, 2026

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 21, 2026

github-project-automation bot moved this to Needs Final Approver in SIG Scheduling Jan 21, 2026

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jan 21, 2026

DRA device taints: document state in 1.35 and plan beta in 1.36

6c37484

pohly force-pushed the dra-device-taints-1.36 branch from 725970b to 6c37484 Compare January 21, 2026 13:38

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 21, 2026

soltysh approved these changes Jan 21, 2026

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 21, 2026

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 21, 2026

k8s-ci-robot merged commit 42a5f3a into kubernetes:master Jan 21, 2026
4 checks passed

k8s-ci-robot added this to the v1.36 milestone Jan 21, 2026

github-project-automation bot moved this from Needs Final Approver to Done in SIG Scheduling Jan 21, 2026

mortent moved this from 👀 In review to ✅ Done in Dynamic Resource Allocation Jan 21, 2026

pohly mentioned this pull request Feb 20, 2026

DRA device taints: automatically bump TimeAdded when changing effect kubernetes/kubernetes#137167

Merged


		- 1.33: first KEP revision and implementation
		- 1.35: revised alpha with `effect: None` and DeviceTaintRule status

Conversation

pohly commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Dec 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mortent Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liggitt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

macsko commented Dec 18, 2025

Uh oh!

thisisvaishnav commented Dec 23, 2025

Uh oh!

mortent commented Jan 6, 2026

Uh oh!

pohly commented Jan 13, 2026

Uh oh!

soltysh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

soltysh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jan 21, 2026

Uh oh!

soltysh commented Jan 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

pohly commented Dec 3, 2025 •

edited

Loading

mortent Jan 12, 2026 •

edited

Loading