DRA: control plane controller ("classic DRA")

[pohly]

Enhancement Description

• One-line enhancement description: dynamic resource allocation

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3063-dynamic-resource-allocation

• Discussion Link: CNCF TAG Runtime Container Device Interface (COD) Working Group meeting(s)

• Primary contact (assignee): @pohly

• Responsible SIGs: SIG Node

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.27
  • Beta release target (x.y): 1.29
  • Stable release target (x.y): 1.31

• Alpha (1.26)

  • KEP (k/enhancements) update PR(s):
    • KEP-3063: dynamic resource allocation #3064
    • KEP-3063: dynamic resource allocation updates for 1.26 #3502
    • KEP-3063 Update kep.yaml #3868
  • Code (k/k) update PR(s): dynamic resource allocation kubernetes#111023
  • Docs (k/website) update PR(s):
    • blog: add dynamic resource allocation feature blog post website#37434
    • dynamic resource allocation concepts website#37766

• Alpha (1.27)

  • KEP (k/enhancements) update PR(s):
    • KEP-3063 Update kep.yaml #3868
    • KEP 3063: Update to reflect changes merged into v1.27 #3802
  • Code (k/k) update PR(s):
    • api: resource.k8s.io v1alpha1 -> v1alpha2 kubernetes#116299
    • Update resource.AllocationResult with a slice of ResourceHandlers kubernetes#116332
    • dra: PodScheduling -> PodSchedulingContext kubernetes#116556
    • Update kubeletplugin API for DRA to v1alpha2 kubernetes#116558
    • optional: api: extend validation of dynamic resource allocation fields in PodSpec kubernetes#116576
  • Docs (k/website) update PR(s):
    • dynamic resource allocation: update for 1.27 website#40086

• Alpha (1.28)

  • KEP (k/enhancements) update PR(s):
    • KEP-3063: dra: pre-scheduled pods #4063
    • KEP-3063: dra: continue as alpha in 1.28 #4096
  • Code (k/k) update PR(s):
    • node authorizer: limit kubelet access to ResourceClaim objects kubernetes#116254
    • kubelet: support batched prepare/unprepare in v1alpha3 DRA plugin API kubernetes#119012
    • dra: pre-scheduled pods kubernetes#118209
    • DRA: generated resource claim names kubernetes#117351
    • dra: scheduler queueing hints kubernetes#119078
    • DRA controller: batch resource claims for Allocate kubernetes#118862
    • DRA: improve handling of completed pods kubernetes#118817
    • DRA: for delayed allocation, deallocate when no longer used kubernetes#118936
    • dynamic resource allocation: Improve code coverage of state checkpoint kubernetes#119058
    • DRA: don't query claims from API server kubernetes#119307
    • DRA: report NodePrepareResource errors kubernetes#118578
    • add unit test for dra/manager.go kubernetes#118711
    • Increase timeout for DRA kubelet plugin client kubernetes#119414
    • feature(scheduler): implement ClusterEventWithHint to filter out useless events kubernetes#118551
    • feature(scheduling_queue): track events per Pods kubernetes#118438
    • dynamic resource allocation: reuse gRPC connection kubernetes#118619 (v1.29)
  • Docs (k/website) update PR(s): dra: update for Kubernetes 1.28 website#41856

• Alpha (1.29)

  • KEP (k/enhancements) update PR(s):
    • DRA: handle non graceful node shutdowns #4260
    • KEP-3063: dynamic resource allocation: update kubelet plugin RPC versions supported. #4164
  • Code (k/k) update PR(s):
    • DRA: notify ResourceClaim or ResourceClass of params error kubernetes#121065
    • dra scheduler: fall back to SSA for PodSchedulingContext updates kubernetes#120534
    • dynamic resource allocation: reuse gRPC connection kubernetes#118619
    • dra e2e: fix "reallocation works" flake kubernetes#119195
    • DRA: Pass CDI devices with a new CRI field kubernetes#115891
    • DRA: call plugins for claims even if exist in cache kubernetes#120550
  • Docs (k/website) update PR(s):
    • Document impact of DRA on scheduling website#43907

• Alpha (1.30)

  • KEP (k/enhancements) update PR(s):
    • KEP-3063: DRA: 1.30 update #4181
  • Code (k/k) update PR(s):
    • dra: reserve + publish during pod binding kubernetes#121876
    • DRA: resource quotas kubernetes#120611
  • Docs (k/website) update PR(s):
    • Add Resource Drivers section to extending Kubernetes documentation website#43742
    • DRA in Kubernetes 1.30: adds structured parameters website#45287

• Alpha (1.31)

  • KEP (k/enhancements) update PR(s): KEP-4381: DRA update for 1.31 #4709
  • Code (k/k) update PR(s): DRA for 1.31 kubernetes#125488
  • Docs (k/website) update PR(s): DRA documentation for 1.31 website#46816

[pohly]

/assign @pohly
/sig node

[ahg-g]

do we have a discussion issue on this enhancement?

[pohly]

@ahg-g: with discussion issue you mean a separate issue in some repo (where?) in which arbitrary comments are welcome?

No, not at the moment. I've also not seen that done elsewhere before. IMHO at this point the open KEP PR is a good place to collect feedback and questions. I also intend to come to the next SIG-Scheduling meeting.

[ahg-g]

@ahg-g: with discussion issue you mean a separate issue in some repo (where?) in which arbitrary comments are welcome?

Yeah, this is what I was looking for, the issue would be under k/k repo.

No, not at the moment. I've also not seen that done elsewhere before.

That is actually the common practice, one starts a feature request issue where the community discusses initial ideas and the merits of the request (look for issues with label kind/feature). That is what I would expect in the discussion link.

IMHO at this point the open KEP PR is a good place to collect feedback and questions. I also intend to come to the next SIG-Scheduling meeting.

But the community have no idea what this is about yet, so better to have an issue discusses "What would you like to be added?" and "Why is this needed" beforehand. Also, meetings are attended by fairly small groups of contributors, having an issue tracking the discussion is important IMO.

[pohly]

In my work in SIG-Storage I've not seen much use of such a discussion issue. Instead I had the impression that the usage of "kind/feature" is discouraged nowadays.

https://github.com/kubernetes/kubernetes/issues/new?assignees=&labels=kind%2Ffeature&template=enhancement.yaml explicitly says

Feature requests are unlikely to make progress as issues. Please consider engaging with SIGs on slack and mailing lists, instead. A proposal that works through the design along with the implications of the change can be opened as a KEP.

This proposal was discussed with various people beforehand, now we are in the formal KEP phase. But I agree, it is hard to provide a good link to those prior discussions.

[ahg-g]

We use that in sig-scheduling, and it does serve as a very good place for initial rounds of discussions, discussions on slack and meetings are hard to reference as you pointed out.

I still have no idea what this is proposing, and I may not attend the next sig meeting for example...

[gracenng]

Hi @ ! 1.24 Enhancements team here.
Checking in as we approach enhancements freeze in less than a week on 18:00pm PT on Thursday Feb 3rd
Here’s where this enhancement currently stands:

• Updated KEP file using the latest template has been merged into the k/enhancements repo. KEP-3063: dynamic resource allocation #3064
• KEP status is marked as implementable for this release with latest-milestone: 1.24
• KEP has a test plan section filled out.
• KEP has up to date graduation criteria.
• KEP has a production readiness review that has been completed and merged into k/enhancements.

The status of this enhancement is track as at risk.
Thanks!

[gracenng]

The Enhancements Freeze is now in effect and this enhancement is removed from the release.
Please feel free to file an exception.

/milestone clear

[gracenng]

Hi Patrick, `tracked/yes` will be applied when the KEP is merged and all requirements for Enhancement Freeze are met. We are definitely keeping an eye on this! Feel free to ping me once it's merged

…

On Tue, Mar 1, 2022 at 11:17 AM Patrick Ohly ***@***.***> wrote: @gracenng <https://github.com/gracenng> : an exception was requested and granted for this enhancement to move to GA in 1.24: https://groups.google.com/g/kubernetes-sig-release/c/sUpd2H1wxnk/m/lL_I6GT-BwAJ Can you label it again as "tracked/yes"? — Reply to this email directly, view it on GitHub <#3063 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCRLO46IYCBRTXXZE6EHGTU5ZUM3ANCNFSM5JCIMTQA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kerthcet]

/remove-lifecycle stale

[marosset]

Hello @pohly 👋, 1.25 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PST on Thursday June 16, 2022.

For note, This enhancement is targeting for stage alpha for 1.25 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP file using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable
• KEP has a updated detailed test plan section filled out
• KEP has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

It looks like #3064 will address everything in this list.

For note, the status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[marosset]

Hello @pohly 👋, just a quick check-in again, as we approach the 1.25 enhancements freeze.

Please plan to get #3064 reviewed and merged before enhancements freeze on Thursday, June 23, 2022 at 18:00 PM PT.

For note, the current status of the enhancement is atat-risk. Thank you!

[sreeram-venkitesh]

Thanks! Marking this KEP as tracked for code freeze 🎉

[kannon92]

@pohly What is your intention for this feature going forward? DRA has split into sub issues but I guess you are keeping this open as the main feature to update?

[pohly]

I don't have any particular plan for this aspect of DRA. Given that others want it removed, some potential user would need to speak up soon or it will get removed, probably in 1.32.

[kannon92]

So then technically we may want to do some work with this even if its deprecation?

[kannon92]

I'm trying to figure out for sig-node what to consider this feature.

"Proposed For Release", "Not for Release"?

[johnbelamaric]

I suggest we raise it in SIG Node and WG Dev Mgmt with the following plan:

• In 1.32, unless we hear a compelling case before Enhancements Freeze, the plan will be to remove this feature.
• We prepare a KEP updated that would retire this in 1.32 and target that for 1.32. If no one pipes up, we move forward with removal in 1.32.
• If there are folks that need to keep this around for another release, we need to understand their use case. We then need to see if we will be able to meet it with any of the currently planned enhancements to the structured parameters code, or we need to determine the right path forward to otherwise meet it. That may entail new, different enhancements to structured parameters, or some version of this feature. In that event, we would withdraw the "removal" from 1.32.

[tjons]

Hi, enhancements lead here - I inadvertently added this to the 1.32 tracking board 😀. Please readd it if you wish to progress this enhancement in 1.32.

/remove-label lead-opted-in

[haircommander]

/milestone v1.32
/label lead-opted-in

[catblade]

Request for leaving this here a little longer, @klueska . We would like some time to go evaluate what is best, from the scheduling side. If we can have some time to try to resolve the complexity of the structured parameters and maybe simplify the classic, having this to play with would be really helpful. 1.33 should be okay to remove because by then we'll have a plan. Spoke with @johnbelamaric already and he suggested I leave this comment and request. We are also looking at handling CPU, as was referenced in the original DRA doc here https://docs.google.com/document/d/1XNkTobkyz-MyXhidhTp5RfbMsM-uRCWDoflUMqNcYTk/ but I'm aware that that may make this scope too complex.

[johnbelamaric]

@pohly can you lay out here the implications of #4381 going beta without first removing #3063? It's important that we make beta for #4381 in 1.32.

[pohly]

The two are independent since Kubernetes 1.31, with separate feature gates. Keeping #3063 as alpha does not block #4381 as beta. It also does not cause extra work (that was all already done for 1.31).

[johnbelamaric]

The two are independent since Kubernetes 1.31, with separate feature gates. Keeping #3063 as alpha does not block #4381 as beta. It also does not cause extra work (that was all already done for 1.31).

@klueska (or was it @SergeyKanzhelev?) mentioned that there are round tripping implications, is that accurate?

[pohly]

Probably Jordan.

If we don't remove it now, the following fields remain reserved forever:

• ResourceClaimSpec.Controller
• ResourceClaimStatus.DeallocationRequested
• AllocationResult.Controller

They don't get set, but the names are "burned" and cannot be used for something else in the future. I think that's okay and won't block future extensions.

[cyclinder]

We're still using Classic DRA at the moment, and if it's removed, it's a big breaking change for us, so before we move to the Structure Parameter, we want the Classic DRA to be here for a while, thanks

[sftim]

• We can rename the existing fields now and then keep them forever. The earlier such a rename happens, the fewer people who need to update their code / integrations.
• We can have certain fields as alpha (behind their own gate) in APIs that are otherwise beta.

[alculquicondor]

We're still using Classic DRA at the moment, and if it's removed, it's a big breaking change for us, so before we move to the Structure Parameter, we want the Classic DRA to be here for a while, thanks

Alpha features don't have backwards compatibility guarantees. I suggest you start the migration process to structured DRA or elaborate on why it's not possible for you to migrate.

[catblade]

So we can wait another cycle, but perhaps expect a rename?

[aojea]

We can rename the existing fields now and then keep them forever. The earlier such a rename happens, the fewer people who need to update their code / integrations.

I don't think we need to find a technical solution to perpetuate an alpha feature specially when we are developing the alternative that solves the problem with the original one, also if there is a bug with classic DRA it will not be backported and most probably not fixed

We're still using Classic DRA at the moment, and if it's removed, it's a big breaking change for us, so before we move to the Structure Parameter, we want the Classic DRA to be here for a while, thanks

Request for leaving this here a little longer, @klueska . We would like some time to go evaluate what is best, from the scheduling side. I

@catblade @cyclinder it seems you are working with old versions of Kubernetes, can you explain which versions are you using now and how far are you from current development? we need more information than "please don't remove it" to objectively evaluate the cost of maintaining code that should not be used ... also is important to describe the exact problems and why you can not use the new one, is a custom DRA driver you already have? or one you are using from a third party?

[pohly]

I had a call with @catblade. She cannot share in public yet what she is working on, but I found it interesting and worth supporting by keeping classic DRA as alpha for another release. It's important to note that it's not about supporting some existing solution. Instead, she is currently exploring both classic DRA and structured parameters and wants to have all options available until she reaches a conclusion of that exploration.

@cyclinder already said on Slack that they will use classic DRA only with older Kubernetes and want to migrate to structured parameters for Kubernetes >= 1.31.

[cyclinder]

We haven't delved into whether the structured parameter will be able to meet our needs, and it will take some time. I can give you feedback on the results. 1.31 has removed the ResourceClass resource, so we had to make some changes to get ClassicDRA to run at 1.31, so we're considering moving directly to structured parameters.

[aojea]

It's important to note that it's not about supporting some existing solution. Instead, she is currently exploring both classic DRA and structured parameters and wants to have all options available until she reaches a conclusion of that exploration.

@pohly but this is still confusing, classic DRA has some limitations and we invested and decided to move with structured DRA, what is the point of exploring classic DRA?
what happens if the result of the exploration is to use classic DRA? are we going to open the debate again?

[thockin]

I agree. I do not see any possible future where classic DRA is revived. "Exploring" can be done on 1.31 or 1.30 or ... - why do we need to keep it in 1.32?