KEP-5517: Introduce DRA for Native Resources KEP

[pravk03]

• One-line PR description: Support native resource management through DRA.

• Issue link: DRA: Native Resource Requests #5517

• Other comments:

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[ffromani]

/cc

[kad]

/cc

[pravk03]

/cc @johnbelamaric

[mortent]

/wg device-management

[dom4ha]

Have you considered defining some sort of framework level consumable counter that different plugins could consume capacity from? DRA devices could use this counter using Consumable Capacity feature and NodeResources plugin would be extended somehow to use it as well.

It would be a bit cleaner conceptually, but we'd introduce quite significant framework change, which may be harder to get through the process.

[dom4ha]

What about in case of "AddPerReference" accounting policy?

[johnbelamaric]

Have you considered defining some sort of framework level consumable counter that different plugins could consume capacity from? DRA devices could use this counter using Consumable Capacity feature and NodeResources plugin would be extended somehow to use it as well.

So, one other option we thought about was simply to treat the NodeAllocatable as CounterSet that is implicitly always available with a well-known name. If we additionally layer in a concept of translating consumed capacity into counter set consumption (ie, a mapping between our current consumable capacity concept and our counter set concept), then we can solve many of the same problems.

[pravk03]

What about in case of "AddPerReference" accounting policy?

Updated this section. In the case of unreferenced claims, we should account for it once when we update NodeInfo irrespective of the accounting policy.

[dom4ha]

So, one other option we thought about was simply to treat the NodeAllocatable as CounterSet that is implicitly always available with a well-known name. If we additionally layer in a concept of translating consumed capacity into counter set consumption (ie, a mapping between our current consumable capacity concept and our counter set concept), then we can solve many of the same problems.

@johnbelamaric I believe this is something that we can still consider post-alpha. Note that it may also address the problem of scoring (node fit would see a counter aggregating native resource request coming from different plugins.

Maybe we could mention this option for the future consideration.

[pravk03]

@dom4ha I have covered scoring in "Future Enhancements" section. In the current proposal (and draft implementation), scoring in NodeResourcesFit does not consider claim requests. But it should be possible to introduce unified scoring.

[pravk03]

Summary of modifications made KEP based on discussions. This should help reduce the scope and implementation complexity on the scheduler side.

1. Removed AccountingPolicy from Device and DeviceClass APIs. For the Alpha scope, we will support a single accounting policy where claim requests is allocated once and is additive to the pod's standard resource requests. The AccountingPolicy field along with supported policies has been moved to the "Future Enhancements" section for reconsideration in Alpha 2/Beta.
2. Sharing native resource claims between pods is excluded from alpha. The DynamicResources plugin will be updated to reject any pod that references a claim already in use by an existing pod. This reduces implementation complexity and allows for more discussion on node-level enforcement and cgroup management.
3. Pod Level Resources, if specified will take precedence and determine the overall resource footprint. Added a validation step in the DynamicResources plugin to ensure that container requests (including native resource claims) do not exceed pod-level requests.

cc @johnbelamaric @dchen1107 @SergeyKanzhelev @dom4ha

[dom4ha]

/approve

Thanks for reducing the scope, which should be helpful.

[johnbelamaric]

PRR is fine for alpha

[pravk03]

/hold

(for sig-node lgtm)

[SergeyKanzhelev]

how do we know that we are moving in the right direction by making kubelet read native resources added by DRA from Pod Status? Is it part of "Gather feedback from alpha"?

[pravk03]

Do you mean populating the claim information in pod status rather than ResourceClaim status ?. Or the general idea of having kubelet having to look outside of pod.spec ?

[SergeyKanzhelev]

The new scope seems more reasonable. I am still very un-easy on a few topics:

1. Can we GA the Native Resources DRA plugin so people can start adopting it (before we fully GA-ed this KEP)? If so - we may want to define a migration plan or just say that it will be a different DRA plugin once this KEP is GA.
2. Are there real world frameworks or users who lined up to try the fungibility between GPU and CPU? If we have nobody confirming that this is "likely" the way to go - it feels like we are implementing an expensive solution to the problem nobody is solving right now. Will we have enough feedback after alpha to decide on the path forward?

[johnbelamaric]

/approve for PRR

[pravk03]

1. Can we GA the Native Resources DRA plugin so people can start adopting it (before we fully GA-ed this KEP)? If so - we may want to define a migration plan or just say that it will be a different DRA plugin once this KEP is GA.

I think the answer depends heavily on the specific plugin implementation. Specifically thinking of dra-driver-cpu, we can support adoption now using existing workarounds, such as mirroring claim requests in the Pod Spec. I view having Pod Level Resources define the overall shape (including claims) as one clean path for migrating to this KEP, but this definitely needs more thought. I am not sure if the same approach works for Memory or other drivers. @ffromani, since you are working on both CPU and Memory drivers, you may have any thoughts here?

On the contrary, without the understanding on native claims in scheduler and kubelet we would have to rely on workarounds (potentially different) in all the native resource driver implementations ?.

2. Are there real world frameworks or users who lined up to try the fungibility between GPU and CPU? If we have nobody confirming that this is "likely" the way to go - it feels like we are implementing an expensive solution to the problem nobody is solving right now. Will we have enough feedback after alpha to decide on the path forward?

Yes, we have seen interest in GPU/CPU fungibility.

[ffromani]

1. Can we GA the Native Resources DRA plugin so people can start adopting it (before we fully GA-ed this KEP)? If so - we may want to define a migration plan or just say that it will be a different DRA plugin once this KEP is GA.

I think the answer depends heavily on the specific plugin implementation. Specifically thinking of dra-driver-cpu, we can support adoption now using existing workarounds, such as mirroring claim requests in the Pod Spec. I view having Pod Level Resources define the overall shape (including claims) as one clean path for migrating to this KEP, but this definitely needs more thought. I am not sure if the same approach works for Memory or other drivers. @ffromani, since you are working on both CPU and Memory drivers, you may have any thoughts here?

Like you, I tend to believe that combining with Pod Level Resources should work, but we need more data here.

I see the point about experimenting with drivers before adding important changes into core kubernetes. I'm not against it, even if I see clearly the value in this KEP. The issues with this approach (try external drivers first) boils down to chicken/egg problems:

• the amount of workarounds needed is likely to limit significantly the UX, then the adoption, which provides us too little data
• what drivers can actually do is limited, which in turn constraints negatively the adoption, which circles back to too little data.

What I'm trying to say is that I see the value in experimenting out of tree and I personally would encourage that but without some core kubernetes changes, be them in the scheduler, in the kubelet or likely both, what drivers for core resources can do is very limited, and that hampers early adoption.

[dchen1107]

/lgtm
/approve

Thank you all for the extensive discussions, both online and offline. We’ve successfully narrowed the scope to the most essential components. While some open questions remain, I believe it's ready to move forward into the implementation phase to continue exploring those details.

Agreed with Sergey above to move cpu dra driver to GA without being blocked by this KEP, so that we can have early and more feedbacks from the users.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dchen1107, dom4ha, johnbelamaric, pravk03

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS [johnbelamaric]
• keps/sig-scheduling/OWNERS [dom4ha,johnbelamaric]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[johnbelamaric]

/hold cancel