KEP-5677: DRA Resource Availability Visibility

[nmn3m]

• One-line PR description: Adding KEP-5677 for DRA Resource Availability Visibility

• Issue link: DRA: Resource Availability Visibility #5677

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

@nmn3m: GitHub didn't allow me to request PR reviews from the following users: kubernetes/sig-scheduling, kubernetes/sig-node, kubernetes/sig-cli, kubernetes/wg-device-management.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

/cc @kubernetes/sig-scheduling
/cc @kubernetes/sig-node
/cc @kubernetes/sig-cli
/cc @kubernetes/wg-device-management

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[nmn3m]

/cc @johnbelamaric
/cc @pohly

[nmn3m]

/cc @kubernetes/sig-cli-kubectl-maintainers

[mortent]

/wg device-management

[johnbelamaric]

First pass, this is looking really really good to me so far

[nmn3m]

Is there an existing pattern for kubectl to reach KCM endpoints?

I don't think so.

Should we reconsider the out-of-tree aggregated API server approach to avoid this connectivity challenge?

That would be the more standard approach, if someone was to built this out-of-tree. We look for in-tree, out-of-the-box support. It might still be the best solution, though.

Any other approaches I should explore?

We could add something directly to the apiserver. I am not sure how I feel about that. It adds some overhead there even when not in use and it's a fairly central component where we don't want to risk causing crashes.

1. If we go with apiserver, what would be the preferred pattern?

   • A new endpoint under /apis/resource.k8s.io/?
   • Something else?

2. For out-of-tree approach, would it make sense to start there and potentially graduate
   to in-tree later if there's demand?

[johnbelamaric]

@liggitt WDYT?

An always available, in-tree approach is preferable, but I am not sure there's a great option for that. Here's what I could think of:

1. A "request" is made by creating a ResourcePool object, or maybe it's even called something like "ResourcePoolStatusRequest" or something. A controller runs in KCM that sees that request, makes the calculation, and writes the result to the object's status, where it can be observed by the user. It is a one-time operation with a timestamp. To recalculate, the user has to delete and recreate the object. The object probably should be cluster scoped.
2. A specialized API endpoint built into API server. I suspect this is a no-go. Jordan, is there any precedent for that?
3. A specialized API endpoint in KCM that is then exposed via an aggregated API configuration. Jordan, any precedent?

The first one seems promising if we want to do this in-tree. I actually think it's fine, and there is precedent for similar "imperative operations through declarative APIs" with things like CSR and even the way device taints with "None" effect works. It gives us the ability to controller permissions on the object, too.

For out-of-tree (could be in k-sigs), we could implement JUST a kubectl plugin and rely on user permissions, to start. And add in some aggregated API server later, if we see the need.

The advantage of in-tree: always available and in-sync with K8s releases, all users can depend on it. Disadvantage: locked to K8s release cycle.

Advantage of out-of-tree: we can implement it independently of the release cycle.

My preference: the first in-tree option.

[nmn3m]

@johnbelamaric , Thanks for the detailed design options! I've updated the KEP to implement your first
suggestion - the CSR-like pattern with ResourcePoolStatusRequest.

[nmn3m]

/cc @kannon92

[kannon92]

Please check over the questionaire. There are missing questions.

[kannon92]

Please make sure to mark this off as you go.

[kannon92]

If we don't get TTL should we consider ways to limit the number of ResourcePoolStatusRequests?

I worry that this will be an informational API that could be useful for scheduling decisions or even information for quota management solutions like Kueue. TTL would be useful but we should also consider

[kannon92]

Would the default admin permissions allow access to this resource? Or would cluster admins need to also add these ClusterRoles?

[kannon92]

Should there also be e2e tests for kubectl?

[kannon92]

Can you please check this questionaire and make sure to follow the template?

Missing Questions

1. "Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?"
   This entire question is absent from the Scalability section (after line 809).

Incomplete Answers

2. "How can someone using this feature know that it is working for their instance?" (line 754-756)
   The template expects structured responses using checkboxes:
   - [ ] Events — with Event Reason
   - [ ] API .status — with Condition name / Other field
   - [ ] Other

The KEP just has a plain text answer. It should at minimum check off API .status with Condition name: Complete.
3. "What are the SLIs (Service Level Indicators)?" (line 764-767)
The template expects structured responses:
- [ ] Metrics — with Metric name, Aggregation method, Components exposing the metric
- [ ] Other

The KEP lists metric names but doesn't specify which component exposes them (should be kube-controller-manager), and doesn't use the
template checkbox format.
4. "Does this feature depend on any specific services running in the cluster?" (line 776-778)
The template expects a structured table/list per dependency with fields like: name, usage, impact of unavailability, impact of degraded
performance, whether it can operate with a degraded/unavailable dependency. The KEP just gives a brief bullet list.
5. "Will enabling / using this feature result in any new API calls?" (line 783-786)
The template expects specifics: API call type, estimated throughput, originating component. The KEP gives a general description without
these details.
6. "Will enabling / using this feature result in introducing new API types?" (line 788-789)
The template expects: API type, supported operations, estimated count. Only the type name is given.
7. "What are other known failure modes?" (line 819-822)
The template expects structured entries with: failure mode description, detection method, mitigations, diagnostics, testing (is it covered
by e2e tests?). The KEP uses a brief bullet list.

[nmn3m]

I followed the rules and fixed them. Thanks for detailed feedback.

[kannon92]

thanks looks great.

[kannon92]

@haircommander do you know who on sig-node signed up to approve this?

[kannon92]

https://github.com/kubernetes/community/blob/master/sig-scalability/slos/slos.md#kubernetes-slisslos

I think you are supposed to answer this with respect to kube-controller-manager SLI/SLO.

This isn't required for beta so I won't block the approval on this.

[kannon92]

/approve

For PRR.
This is a very well thoughout proposal for alpha.

Thank you!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kannon92, nmn3m
Once this PR has been reviewed and has the lgtm label, please ask for approval from mrunalp. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS [kannon92]
• keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment