Document KEP-5963: DRADeviceCompatibilityGroups for K8s 1.37 by omeryahud · Pull Request #56288 · kubernetes/website

omeryahud · 2026-06-28T10:20:02Z

Docs PR for:

netlify · 2026-06-28T10:20:06Z

👷 Deploy Preview for kubernetes-io-vnext-staging processing.

Name	Link
🔨 Latest commit	`add1e14`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-io-vnext-staging/deploys/6a6a25fb74099c0008eb9ca8

kubernetes-prow · 2026-06-28T10:20:11Z

Welcome @omeryahud!

It looks like this is your first PR to kubernetes/website 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/website has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

netlify · 2026-06-28T10:22:25Z

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`add1e14`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-io-main-staging/deploys/6a6a25fbd3a6b90008a7a384
😎 Deploy Preview	https://deploy-preview-56288--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

lmktfy · 2026-06-28T13:23:03Z

When you know what feature you want to document, try this Prow command:
/retitle Document $NAME_OF_FEATURE for K8s 1.37

As it's marked as draft, you don't need to say that the PR is a placeholder.

omeryahud · 2026-06-29T07:35:31Z

/retitle Document KEP-5963: DRADeviceCompatibilityGroups for K8s 1.37

kubernetes-prow · 2026-06-29T07:35:40Z

@omeryahud: Re-titling can only be requested by trusted users, like repository collaborators.

Details

In response to this:

/retitle Document KEP-5963: DRADeviceCompatibilityGroups for K8s 1.37

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

omeryahud · 2026-06-29T07:35:40Z

When you know what feature you want to document, try this Prow command: /retitle Document $NAME_OF_FEATURE for K8s 1.37

As it's marked as draft, you don't need to say that the PR is a placeholder.

Done, thank you!

omeryahud · 2026-06-29T07:36:18Z

I guess I can't do that yet, so I manually updated the title

kernel-kun · 2026-07-21T17:19:36Z

Hi @omeryahud 👋! v1.37 Docs team here

We noticed that this Pull Request is currently in the Draft state.
If you haven't already, please go ahead and add the required documentation changes and move this PR from a Draft to Open state.

Important

Upcoming Docs Deadlines:

PR Ready for Review: Tuesday 28th July 2026
Docs Freeze: Wednesday 5th August 2026 (AoE) / Thursday 6th August 2026, 12:00 UTC

Thanks!

Add a new section under DRA alpha features describing the DRADeviceCompatibilityGroups feature gate, which lets DRA drivers declare opaque compatibility groups on each consumesCounters[] entry of a ResourceSlice to prevent the scheduler from co-allocating incompatible partitioned devices (e.g. MIG vs vGPU on the same GPU). Includes: motivation, how-it-works, a MIG/vGPU YAML example, API constraints, and version-skew safety notes.

johnbelamaric

/approve

for DRA content

Can you take a look at the #56352 feature and make a plan there for 1.38?

nojnhuh · 2026-07-28T16:22:35Z

/wg device-management

nojnhuh · 2026-07-28T17:27:12Z

+If a Pod or PodGroup requests two devices from this pool, the scheduler checks
+whether the two chosen devices share a common compatibility group on the
+`gpu-0-memory` counter set. Since `{"mig"} ∩ {"vgpu"} = ∅`, the pair is


In this example, are the two different partitions already not co-allocateable given that the counter set defines 8Gi memory and each partition consumes 6Gi of that? The example would be a bit more compelling if the partitions consumed <=4Gi of memory where the compatibility groups are the only limiting factor.

I agree, I'll change that

nojnhuh · 2026-07-28T17:30:02Z

+
+A driver adds a `compatibilityGroups` list to each
+`device.consumesCounters[]` entry in a `ResourceSlice`. The list contains
+one or two opaque string names that represent the operating mode or partition


"One or two" implies "not zero" which actually is valid as explained in a later paragraph. Could we align this with the "at most 2" language used in the "Constraints" section below?

nojnhuh · 2026-07-28T17:38:01Z

+When the `DRADeviceCompatibilityGroups` feature gate is disabled (the default
+for alpha), the kube-apiserver strips the `compatibilityGroups` field from any
+new or updated `ResourceSlice` — unless the old object already had the field
+set. The scheduler then treats devices in any pool that previously had grouped
+devices as belonging to an incomplete pool and skips them entirely.


Since the "How it works" section mentions "an unset, nil, or empty list", I think it's worth mentioning which of those constitute the field being "set". e.g. If a ResourceSlice is created with compatibilityGroups: nil and the DRADeviceCompatibilityGroups feature enabled, then the feature is disabled, can that device still be allocated, or does the scheduler consider the pool incomplete?

Hi @nojnhuh !

In the case you describe, the device can be allocated (only with other devices that leave the field unset OR declare ANY of nil, []) and the pool is considered complete.

The only case where a pool is considered incomplete is when both of the following are true:

DRADeviceCompatibilityGroups is disabled

At least one device in the pool specifies compatibilityGroups (len(compatibilityGroups) > 0)

So the field is being "set" only when len(compatibilityGroups) > 0

I'll make sure to add this somewhere in the doc

Co-authored-by: Jon Huhn <nojnhuh@users.noreply.github.com>

…the limiting factor

Signed-off-by: Omer Yahud <oyahud@nvidia.com>

omeryahud · 2026-07-28T22:21:13Z

I've addressd everything @nojnhuh , thanks!

omeryahud · 2026-07-28T22:22:38Z

/approve

for DRA content

Can you take a look at the #56352 feature and make a plan there for 1.38?

Thanks for the approval @johnbelamaric !

I've had a look, and it is not obvious to me how copmat groups would affect it.
I see that there is already an attribute that drivers should include in order to specify partition type, regardless of groups.

Have you had something else in mind?

nojnhuh

Looks good, thanks! Just a couple other tiny comments.

Co-authored-by: Jon Huhn <nojnhuh@users.noreply.github.com>

omeryahud · 2026-07-29T16:10:51Z

Looks good, thanks! Just a couple other tiny comments.

Done! Thanks!

nojnhuh

/lgtm

Thanks! Could you please squash to one commit?

kubernetes-prow · 2026-07-29T16:20:48Z

LGTM label has been added.

Details

Git tree hash: f333c1ae9f014fbcc636a9ed07f92af27cd7449c

nojnhuh · 2026-07-29T17:22:59Z

Thanks! Could you please squash to one commit?

I keep forgetting we can do this the easy way:

/label tide/merge-method-squash

Manually squashing is preferred, but we don't need to lose sleep over it if you run into issues.

nojnhuh · 2026-07-29T17:23:43Z

/assign @lmktfy
for approval

katcosgrove · 2026-08-04T18:43:32Z

/approve

kubernetes-prow · 2026-08-04T18:44:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnbelamaric, katcosgrove, nojnhuh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~content/en/docs/OWNERS~~ [katcosgrove]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubernetes-prow Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 28, 2026

kubernetes-prow Bot added this to the 1.37 milestone Jun 28, 2026

kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 28, 2026

omeryahud changed the title ~~Placeholder for dev-1.37~~ Document KEP-5963: DRADeviceCompatibilityGroups for K8s 1.37 Jun 29, 2026

omeryahud mentioned this pull request Jun 29, 2026

DRA Device Compatibility Groups kubernetes/enhancements#5963

Open

4 tasks

omeryahud force-pushed the placeholder-dev-1.37 branch from 634d09f to 63240fe Compare July 26, 2026 11:06

kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 26, 2026

omeryahud force-pushed the placeholder-dev-1.37 branch from b5221cf to 7320b68 Compare July 26, 2026 11:28

Add DRADeviceCompatibilityGroups feature gate description file

940c79c

omeryahud marked this pull request as ready for review July 26, 2026 11:37

kubernetes-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 26, 2026

kubernetes-prow Bot requested review from natalisucks and pohly July 26, 2026 11:38

johnbelamaric reviewed Jul 27, 2026

View reviewed changes

kubernetes-prow Bot added the wg/device-management Categorizes an issue or PR as relevant to WG Device Management. label Jul 28, 2026

github-project-automation Bot added this to Dynamic Resource Allocation Jul 28, 2026

github-project-automation Bot moved this to 🆕 New in Dynamic Resource Allocation Jul 28, 2026

nojnhuh reviewed Jul 28, 2026

View reviewed changes

omeryahud and others added 5 commits July 28, 2026 20:48

Apply suggestions from code review

ebdff73

Co-authored-by: Jon Huhn <nojnhuh@users.noreply.github.com>

Address review: align group-count wording, fix example so groups are …

1b113a9

…the limiting factor

Clarify what counts as the compatibilityGroups field being set

b1c4207

Reword: treated identically rather than stored identically

a75d9de

update

a6382f5

Signed-off-by: Omer Yahud <oyahud@nvidia.com>

nojnhuh reviewed Jul 29, 2026

View reviewed changes

Comment thread content/en/docs/concepts/resource-management/dynamic-resource-allocation/dra-features.md Outdated

Comment thread content/en/docs/concepts/resource-management/dynamic-resource-allocation/dra-features.md Outdated

omeryahud and others added 2 commits July 29, 2026 19:10

Update dra-features.md

88d1506

Co-authored-by: Jon Huhn <nojnhuh@users.noreply.github.com>

Update dra-features.md

add1e14

Co-authored-by: Jon Huhn <nojnhuh@users.noreply.github.com>

nojnhuh approved these changes Jul 29, 2026

View reviewed changes

kubernetes-prow Bot assigned nojnhuh Jul 29, 2026

kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 29, 2026

kubernetes-prow Bot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Jul 29, 2026

kubernetes-prow Bot assigned lmktfy Jul 29, 2026

nojnhuh moved this from 🆕 New to 👀 In review in Dynamic Resource Allocation Jul 29, 2026

kubernetes-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 4, 2026

kubernetes-prow Bot merged commit 54c8385 into kubernetes:dev-1.37 Aug 4, 2026
6 checks passed

nojnhuh moved this from 👀 In review to ✅ Done in Dynamic Resource Allocation Aug 7, 2026

Uh oh!

Conversation

omeryahud commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify Bot commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

👷 Deploy Preview for kubernetes-io-vnext-staging processing.

Uh oh!

kubernetes-prow Bot commented Jun 28, 2026

Uh oh!

netlify Bot commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Pull request preview available for checking

Uh oh!

lmktfy commented Jun 28, 2026

Uh oh!

omeryahud commented Jun 29, 2026

Uh oh!

kubernetes-prow Bot commented Jun 29, 2026

Uh oh!

omeryahud commented Jun 29, 2026

Uh oh!

omeryahud commented Jun 29, 2026

Uh oh!

kernel-kun commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johnbelamaric left a comment

Choose a reason for hiding this comment

Uh oh!

nojnhuh commented Jul 28, 2026

Uh oh!

Uh oh!

Uh oh!

nojnhuh Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

omeryahud Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

nojnhuh Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

nojnhuh Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

omeryahud Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

omeryahud Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

omeryahud commented Jul 28, 2026

Uh oh!

omeryahud commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nojnhuh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

omeryahud commented Jul 29, 2026

Uh oh!

nojnhuh left a comment

Choose a reason for hiding this comment

Uh oh!

kubernetes-prow Bot commented Jul 29, 2026

Uh oh!

nojnhuh commented Jul 29, 2026

Uh oh!

nojnhuh commented Jul 29, 2026

Uh oh!

katcosgrove commented Aug 4, 2026

Uh oh!

kubernetes-prow Bot commented Aug 4, 2026

Uh oh!

Uh oh!

Reviewers

omeryahud commented Jun 28, 2026 •

edited

Loading

netlify Bot commented Jun 28, 2026 •

edited

Loading

netlify Bot commented Jun 28, 2026 •

edited

Loading

kernel-kun commented Jul 21, 2026 •

edited

Loading

omeryahud commented Jul 28, 2026 •

edited

Loading