DRA: control plane controller ("classic DRA")

[pohly]

Enhancement Description

• One-line enhancement description: dynamic resource allocation

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3063-dynamic-resource-allocation

• Discussion Link: CNCF TAG Runtime Container Device Interface (COD) Working Group meeting(s)

• Primary contact (assignee): @pohly

• Responsible SIGs: SIG Node

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.27
  • Beta release target (x.y): 1.29
  • Stable release target (x.y): 1.31

• Alpha (1.26)

  • KEP (k/enhancements) update PR(s):
    • KEP-3063: dynamic resource allocation #3064
    • KEP-3063: dynamic resource allocation updates for 1.26 #3502
    • KEP-3063 Update kep.yaml #3868
  • Code (k/k) update PR(s): dynamic resource allocation kubernetes#111023
  • Docs (k/website) update PR(s):
    • blog: add dynamic resource allocation feature blog post website#37434
    • dynamic resource allocation concepts website#37766

• Alpha (1.27)

  • KEP (k/enhancements) update PR(s):
    • KEP-3063 Update kep.yaml #3868
    • KEP 3063: Update to reflect changes merged into v1.27 #3802
  • Code (k/k) update PR(s):
    • api: resource.k8s.io v1alpha1 -> v1alpha2 kubernetes#116299
    • Update resource.AllocationResult with a slice of ResourceHandlers kubernetes#116332
    • dra: PodScheduling -> PodSchedulingContext kubernetes#116556
    • Update kubeletplugin API for DRA to v1alpha2 kubernetes#116558
    • optional: api: extend validation of dynamic resource allocation fields in PodSpec kubernetes#116576
  • Docs (k/website) update PR(s):
    • dynamic resource allocation: update for 1.27 website#40086

• Alpha (1.28)

  • KEP (k/enhancements) update PR(s):
    • KEP-3063: dra: pre-scheduled pods #4063
    • KEP-3063: dra: continue as alpha in 1.28 #4096
  • Code (k/k) update PR(s):
    • node authorizer: limit kubelet access to ResourceClaim objects kubernetes#116254
    • kubelet: support batched prepare/unprepare in v1alpha3 DRA plugin API kubernetes#119012
    • dra: pre-scheduled pods kubernetes#118209
    • DRA: generated resource claim names kubernetes#117351
    • dra: scheduler queueing hints kubernetes#119078
    • DRA controller: batch resource claims for Allocate kubernetes#118862
    • DRA: improve handling of completed pods kubernetes#118817
    • DRA: for delayed allocation, deallocate when no longer used kubernetes#118936
    • dynamic resource allocation: Improve code coverage of state checkpoint kubernetes#119058
    • DRA: don't query claims from API server kubernetes#119307
    • DRA: report NodePrepareResource errors kubernetes#118578
    • add unit test for dra/manager.go kubernetes#118711
    • Increase timeout for DRA kubelet plugin client kubernetes#119414
    • feature(scheduler): implement ClusterEventWithHint to filter out useless events kubernetes#118551
    • feature(scheduling_queue): track events per Pods kubernetes#118438
    • dynamic resource allocation: reuse gRPC connection kubernetes#118619 (v1.29)
  • Docs (k/website) update PR(s): dra: update for Kubernetes 1.28 website#41856

• Alpha (1.29)

  • KEP (k/enhancements) update PR(s):
    • DRA: handle non graceful node shutdowns #4260
    • KEP-3063: dynamic resource allocation: update kubelet plugin RPC versions supported. #4164
  • Code (k/k) update PR(s):
    • DRA: notify ResourceClaim or ResourceClass of params error kubernetes#121065
    • dra scheduler: fall back to SSA for PodSchedulingContext updates kubernetes#120534
    • dynamic resource allocation: reuse gRPC connection kubernetes#118619
    • dra e2e: fix "reallocation works" flake kubernetes#119195
    • DRA: Pass CDI devices with a new CRI field kubernetes#115891
    • DRA: call plugins for claims even if exist in cache kubernetes#120550
  • Docs (k/website) update PR(s):
    • Document impact of DRA on scheduling website#43907

• Alpha (1.30)

  • KEP (k/enhancements) update PR(s):
    • KEP-3063: DRA: 1.30 update #4181
  • Code (k/k) update PR(s):
    • dra: reserve + publish during pod binding kubernetes#121876
    • DRA: resource quotas kubernetes#120611
  • Docs (k/website) update PR(s):
    • Add Resource Drivers section to extending Kubernetes documentation website#43742
    • DRA in Kubernetes 1.30: adds structured parameters website#45287

• Alpha (1.31)

  • KEP (k/enhancements) update PR(s): KEP-4381: DRA update for 1.31 #4709
  • Code (k/k) update PR(s): DRA for 1.31 kubernetes#125488
  • Docs (k/website) update PR(s): DRA documentation for 1.31 website#46816

• Withdrawn (1.32)

  • KEP (k/enhancements) update PR(s): KEP-3063: Classic DRA: withdraw the KEP #4904
  • Code (k/k) update PR(s): DRA: remove "classic DRA" kubernetes#128003
  • Docs (k/website) update PR(s): DRA: remove "classic DRA" website#48289

[pohly]

/assign @pohly
/sig node

[ahg-g]

do we have a discussion issue on this enhancement?

[pohly]

@ahg-g: with discussion issue you mean a separate issue in some repo (where?) in which arbitrary comments are welcome?

No, not at the moment. I've also not seen that done elsewhere before. IMHO at this point the open KEP PR is a good place to collect feedback and questions. I also intend to come to the next SIG-Scheduling meeting.

[ahg-g]

@ahg-g: with discussion issue you mean a separate issue in some repo (where?) in which arbitrary comments are welcome?

Yeah, this is what I was looking for, the issue would be under k/k repo.

No, not at the moment. I've also not seen that done elsewhere before.

That is actually the common practice, one starts a feature request issue where the community discusses initial ideas and the merits of the request (look for issues with label kind/feature). That is what I would expect in the discussion link.

IMHO at this point the open KEP PR is a good place to collect feedback and questions. I also intend to come to the next SIG-Scheduling meeting.

But the community have no idea what this is about yet, so better to have an issue discusses "What would you like to be added?" and "Why is this needed" beforehand. Also, meetings are attended by fairly small groups of contributors, having an issue tracking the discussion is important IMO.

[pohly]

In my work in SIG-Storage I've not seen much use of such a discussion issue. Instead I had the impression that the usage of "kind/feature" is discouraged nowadays.

https://github.com/kubernetes/kubernetes/issues/new?assignees=&labels=kind%2Ffeature&template=enhancement.yaml explicitly says

Feature requests are unlikely to make progress as issues. Please consider engaging with SIGs on slack and mailing lists, instead. A proposal that works through the design along with the implications of the change can be opened as a KEP.

This proposal was discussed with various people beforehand, now we are in the formal KEP phase. But I agree, it is hard to provide a good link to those prior discussions.

[ahg-g]

We use that in sig-scheduling, and it does serve as a very good place for initial rounds of discussions, discussions on slack and meetings are hard to reference as you pointed out.

I still have no idea what this is proposing, and I may not attend the next sig meeting for example...

[gracenng]

Hi @ ! 1.24 Enhancements team here.
Checking in as we approach enhancements freeze in less than a week on 18:00pm PT on Thursday Feb 3rd
Here’s where this enhancement currently stands:

• Updated KEP file using the latest template has been merged into the k/enhancements repo. KEP-3063: dynamic resource allocation #3064
• KEP status is marked as implementable for this release with latest-milestone: 1.24
• KEP has a test plan section filled out.
• KEP has up to date graduation criteria.
• KEP has a production readiness review that has been completed and merged into k/enhancements.

The status of this enhancement is track as at risk.
Thanks!

[gracenng]

The Enhancements Freeze is now in effect and this enhancement is removed from the release.
Please feel free to file an exception.

/milestone clear

[gracenng]

Hi Patrick, `tracked/yes` will be applied when the KEP is merged and all requirements for Enhancement Freeze are met. We are definitely keeping an eye on this! Feel free to ping me once it's merged

…

On Tue, Mar 1, 2022 at 11:17 AM Patrick Ohly ***@***.***> wrote: @gracenng <https://github.com/gracenng> : an exception was requested and granted for this enhancement to move to GA in 1.24: https://groups.google.com/g/kubernetes-sig-release/c/sUpd2H1wxnk/m/lL_I6GT-BwAJ Can you label it again as "tracked/yes"? — Reply to this email directly, view it on GitHub <#3063 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCRLO46IYCBRTXXZE6EHGTU5ZUM3ANCNFSM5JCIMTQA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kerthcet]

/remove-lifecycle stale

[marosset]

Hello @pohly 👋, 1.25 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PST on Thursday June 16, 2022.

For note, This enhancement is targeting for stage alpha for 1.25 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP file using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable
• KEP has a updated detailed test plan section filled out
• KEP has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

It looks like #3064 will address everything in this list.

For note, the status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[marosset]

Hello @pohly 👋, just a quick check-in again, as we approach the 1.25 enhancements freeze.

Please plan to get #3064 reviewed and merged before enhancements freeze on Thursday, June 23, 2022 at 18:00 PM PT.

For note, the current status of the enhancement is atat-risk. Thank you!

[johnbelamaric]

The two are independent since Kubernetes 1.31, with separate feature gates. Keeping #3063 as alpha does not block #4381 as beta. It also does not cause extra work (that was all already done for 1.31).

@klueska (or was it @SergeyKanzhelev?) mentioned that there are round tripping implications, is that accurate?

[pohly]

Probably Jordan.

If we don't remove it now, the following fields remain reserved forever:

• ResourceClaimSpec.Controller
• ResourceClaimStatus.DeallocationRequested
• AllocationResult.Controller

They don't get set, but the names are "burned" and cannot be used for something else in the future. I think that's okay and won't block future extensions.

[cyclinder]

We're still using Classic DRA at the moment, and if it's removed, it's a big breaking change for us, so before we move to the Structure Parameter, we want the Classic DRA to be here for a while, thanks

[sftim]

• We can rename the existing fields now and then keep them forever. The earlier such a rename happens, the fewer people who need to update their code / integrations.
• We can have certain fields as alpha (behind their own gate) in APIs that are otherwise beta.

[alculquicondor]

We're still using Classic DRA at the moment, and if it's removed, it's a big breaking change for us, so before we move to the Structure Parameter, we want the Classic DRA to be here for a while, thanks

Alpha features don't have backwards compatibility guarantees. I suggest you start the migration process to structured DRA or elaborate on why it's not possible for you to migrate.

[catblade]

So we can wait another cycle, but perhaps expect a rename?

[aojea]

We can rename the existing fields now and then keep them forever. The earlier such a rename happens, the fewer people who need to update their code / integrations.

I don't think we need to find a technical solution to perpetuate an alpha feature specially when we are developing the alternative that solves the problem with the original one, also if there is a bug with classic DRA it will not be backported and most probably not fixed

We're still using Classic DRA at the moment, and if it's removed, it's a big breaking change for us, so before we move to the Structure Parameter, we want the Classic DRA to be here for a while, thanks

Request for leaving this here a little longer, @klueska . We would like some time to go evaluate what is best, from the scheduling side. I

@catblade @cyclinder it seems you are working with old versions of Kubernetes, can you explain which versions are you using now and how far are you from current development? we need more information than "please don't remove it" to objectively evaluate the cost of maintaining code that should not be used ... also is important to describe the exact problems and why you can not use the new one, is a custom DRA driver you already have? or one you are using from a third party?

[pohly]

I had a call with @catblade. She cannot share in public yet what she is working on, but I found it interesting and worth supporting by keeping classic DRA as alpha for another release. It's important to note that it's not about supporting some existing solution. Instead, she is currently exploring both classic DRA and structured parameters and wants to have all options available until she reaches a conclusion of that exploration.

@cyclinder already said on Slack that they will use classic DRA only with older Kubernetes and want to migrate to structured parameters for Kubernetes >= 1.31.

[cyclinder]

We haven't delved into whether the structured parameter will be able to meet our needs, and it will take some time. I can give you feedback on the results. 1.31 has removed the ResourceClass resource, so we had to make some changes to get ClassicDRA to run at 1.31, so we're considering moving directly to structured parameters.

[aojea]

It's important to note that it's not about supporting some existing solution. Instead, she is currently exploring both classic DRA and structured parameters and wants to have all options available until she reaches a conclusion of that exploration.

@pohly but this is still confusing, classic DRA has some limitations and we invested and decided to move with structured DRA, what is the point of exploring classic DRA?
what happens if the result of the exploration is to use classic DRA? are we going to open the debate again?

[thockin]

I agree. I do not see any possible future where classic DRA is revived. "Exploring" can be done on 1.31 or 1.30 or ... - why do we need to keep it in 1.32?

[jenshu]

@pohly (enhancements team here) can you confirm if this is slated for deprecation in 1.32?

[pohly]

classic DRA has some limitations and we invested and decided to move with structured DRA, what is the point of exploring classic DRA?

Structured parameters has other limitations. That's why we are working on additional KEPs for it, and that is likely to continue for a while.

"Exploring" can be done on 1.31 or 1.30 or ... - why do we need to keep it in 1.32?

Perhaps because it's easier to install one version of Kubernetes and then try out different approaches? Just a thought.

can you confirm if this is slated for deprecation in 1.32?

I am not sure whether we have reached a consensus. Deadline for a decision is probably soon enough before KEP freeze so that we can still record a decision to remove it. If we keep it, no updates will be needed.

[jenshu]

@pohly ok thank you, I will mark this at risk for enhancement freeze for now, pending your decision.

Please keep in mind that the PRR freeze is coming up on Thursday 3rd October 2024 and the enhancements freeze is on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024

[alculquicondor]

Perhaps because it's easier to install one version of Kubernetes and then try out different approaches? Just a thought.

That's not a strong argument. Unless there is a compelling argument defending the need for classic DRA to stay one more release, I prefer we remove it ASAP. The more we keep it, the more vendors will depend on it and make it harder and harder to remove every passing release.

[thockin]

I agree. It also makes people think that we are hedging our bet around structured parameters, and I don't think we are. If there are truly shortcomings with it, and I accept that there are, and we will fix those forward.

[pohly]

I've created #4904 to mark the KEP as "withdrawn" and notified folks on #wg-device-management.

[kannon92]

@pohly can you update the PR description to include the latest changes for 1.32?

[jenshu]

1.32 Enhancements team here. I see the updates have been made to withdraw this KEP, and I've updated the status to tracked for enhancements freeze

[rytswd]

Hi @pohly 👋 -- this is Ryota (@rytswd) from the v1.32 Communications Team!

For the v1.32 release, we are currently in the process of collecting and curating a list of potential feature blogs, and we are keen to hear if you would consider writing one for this withdrawal!

As you may be aware, feature blogs are a great way to communicate to users about features which fall into (but not limited to) the following categories:

• This introduces some breaking change(s)
• This has significant impacts and/or implications to users
• ...Or this is a long-awaited feature, which would go a long way to cover the journey more in detail 🎉

To opt in to write a feature blog, could you please let us know and open a "Feature Blog placeholder PR" (which can be only a skeleton at first) against the website repository by Wednesday, 30th Oct 2024? For more information about writing a blog, please find the blog contribution guidelines 📚

Tip

Some timeline to keep in mind:

• 02:00 UTC Wednesday, 30th Oct: Feature blog PR freeze
• Monday, 25th Nov: Feature blogs ready for review
• You can find more in the release document

Note

In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

[jenshu]

Hello @pohly 👋, v1.32 Enhancements team here.

With all the implementation (code related) PRs merged per the issue description:

• DRA: remove "classic DRA" kubernetes#128003

This enhancement is now marked as tracked for code freeze for the v1.32 Code Freeze!

Additionally, please let me know if there are any other PRs in k/k that we should track for this KEP, so that we can maintain accurate status.

[rytswd]

Hi @pohly 👋, v1.32 Communications Team here again!

This is a gentle reminder for the feature blog deadline mentioned above, which is 02:00 UTC Wednesday, 30th Oct. To opt in, please let us know and open a Feature Blog placeholder PR against k/website by the deadline. If you have any questions, please feel free to reach out to us!