Add support for a drop-in kubelet configuration directory

[haircommander]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add support for a kubelet drop-in configuration directory
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3983-drop-in-configuration
• Discussion Link: SIG-Node April 25th meeting SIG-Arch May 4 meeting
• Primary contact (assignee): @haircommander
• Responsible SIGs: SIG-Node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.28
  • Beta release target (x.y): 1.30
  • Stable release target (x.y): ？
• Alpha
  • KEP (k/enhancements) update PR(s): KEP-3983: Add support for a drop-in kubelet configuration directory #4031, KEP 3983: target to beta for 1.29 #4242
  • Code (k/k) update PR(s): cmd/kubelet: implement drop-in configuration directory for kubelet kubernetes#119390, Retarget drop-in kubelet configuration dir feature to Alpha kubernetes#121193
  • Docs (k/website) update PR(s): Update documentation to reflect kubelet.conf.d changes for k8s 1.28 website#42013, docs: prepare KubeletConfigDropinDir to Beta website#43542
• Beta
  • KEP (k/enhancements) update PR(s):
    • KEP-3983: target to beta for 1.30 #4419
  • Code (k/k) update PR(s):
    • [KEP-4419]: promote KubeletConfigDropInDir feature to beta kubernetes#122907
  • Docs (k/website) update(s):
    • [dev-1.30] docs: Update KubeletConfigDropinDir doc information website#45665

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[haircommander]

/sig node

[SergeyKanzhelev]

/milestone v1.28

Adding to the milestone for tracking based on discussions at sig node meeting.

[ffromani]

+1!!! (non binding)
@haircommander I'd love to help here to get this in kubelet

[SergeyKanzhelev]

/stage alpha

[SergeyKanzhelev]

/label lead-opted-in

[ruheenaansari34]

Hello @haircommander 👋, 1.28 Enhancements team here.

Just checking in as we approach enhancements freeze on 1:00 UTC on Friday 16th June 2023.

This enhancement is targeting stage alpha for 1.28 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.28
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would need to take care of:

• Merging KEP update PR
• The KEP number seems to be incorrect ie., should this be 3983 instead of 3893?

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[yuqi-zhang]

Hi @ruheenaansari34 , I've updated the ID, thanks for pointing that out!

Is there anything else needed at the moment other than getting consensus to merge the enhancement?

[ruheenaansari34]

@yuqi-zhang Please take care of the following before the Enhancement freeze to meet the requirements:

• The status in the kep.yaml should be implementable here -

  enhancements/keps/sig-node/3983-drop-in-configuration/kep.yaml

  Line 7 in 8e75faa

  status: provisional

• Merging KEP update PR

The status of this enhancement is still marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[yuqi-zhang]

Thanks! I've marked it as implementable after discussion with @haircommander . We will work to get it merged

[Atharva-Shinde]

Thanks @yuqi-zhang :)
With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze 🚀.
The status of this enhancement is marked as tracked. Please keep the issue description up-to-date with appropriate stages as well.

[Rishit-dagli]

Hello @haircommander 👋, 1.28 Docs Lead here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[Rishit-dagli]

Hey @haircommander , could you please create a docs PR even if it is a draft PR with no content yet against dev-1.28 branch in the k/website repo. The deadline to create this draft PR is Thursday 20th July 2023.

[haircommander]

yes I will take care of it, thanks @Rishit-dagli !

[haircommander]

done! kubernetes/website#42013

[ruheenaansari34]

Hey again @haircommander 👋
Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023 .

I don't see any code (k/k) update PR(s) in the issue description so if there are any k/k related PR(s) that we should be tracking for this KEP please link them in the issue description above.

As always, we are here to help if any questions come up. Thanks!

[Atharva-Shinde]

Hey @haircommander 👋 Enhancements Lead here,
With kubernetes/kubernetes#119390 merged as per the issue description I updated above, this enhancement is now tracked for v1.28 Code Freeze. Thanks!

[Rishit-dagli]

Hello @haircommander wave: please take a look at Documenting for a release - PR Ready for Review to get your docs PR ready for review before Tuesday 25th July 2023. Thank you!

Ref: kubernetes/website#42013

[haircommander]

thank you! I have moved it out of draft

[katcosgrove]

Hey y'all! I'm swinging by for the v1.28 Docs team. Today is the docs deadline. Is there anything we can help you with to get this merged?

[haircommander]

sorry I am updating it now, I will get this done soon.

[npolshakova]

/remove-label lead-opted-in

[SergeyKanzhelev]

/milestone v1.29
/stage beta
/label lead-opted-in

[sohankunkerkar]

wangzhen127 Thanks for your interest. Today, we can query the Kubelet config by setting up the kubectl proxy and using a curl command to query the /configz endpoint (curl -X GET http://127.0.0.1:8001/api/v1/nodes/<node-name>/proxy/configz | jq .). This is more of a gap in the Kubernetes documentation. Additionally, we need to refine the language used in the KEP to make it sound more like a documentation request than an implementation.

[wangzhen127]

Ah, I see. Thanks for the clarification! So basically kubectl get --raw "/api/v1/nodes/<nodename>/proxy/configz" | jq

[sohankunkerkar]

Ah, I see. Thanks for the clarification! So basically kubectl get --raw "/api/v1/nodes/<nodename>/proxy/configz" | jq

Yeah,that's another way to query the configz endpoint.

[sohankunkerkar]

@AnaMMedina21 Thanks for the reminder. Could you exclude the docs PR from the code freeze tracker? The code PR has already merged.

[salehsedghpour]

Hello @haircommander 👋 , Enhancements team here.

With all the implementation(code related) PRs merged as per the issue description:

• [KEP-4419]: promote KubeletConfigDropInDir feature to beta kubernetes#122907

This enhancement is now marked as tracked for code freeze for the v1.30 Code Freeze!

[sreeram-venkitesh]

Hi @haircommander 👋, 1.31 Enhancements Lead here.

If you wish to progress this enhancement in v1.31, please have the SIG lead opt-in your enhancement by adding the lead-opted-in label and set the milestone to v1.31 before the Production Readiness Review Freeze.

/remove-label lead-opted-in

[sreeram-venkitesh]

CC: @sohankunkerkar

[kannon92]

Do we have plans to work on this for 1.32?

[haircommander]

I put it in proposed for consideration. It's the earliest it could GA, and I'm not sure there are any end-users yet so we may want to soak it a bit more

[haircommander]

I think we should have more uses before we push for GA. I'm bumping this from the SIG node KEP planning for this release

[SergeyKanzhelev]

I think we should have more uses before we push for GA. I'm bumping this from the SIG node KEP planning for this release

One think I feel we need to change here is the GA criteria: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3983-drop-in-configuration#ga

Changes I suggest:

• Change the default of the argument while in beta with the ability to turn off. We MUST NOT go in GA AND change the logic at that time as the exit criteria suggests. GA must be a no-op for the already enabled beta feature.
• At least two reports of feature being used with the feedback given.

I would really like to get feedback (ideally real life feedback) from SIG Windows. I vaguely remember they wanted this feature to drop in some windows-specific changes

[ffromani]

* Change the default of the argument while in beta with the ability to turn off. We MUST NOT go in GA AND change the logic at that time as the exit criteria suggests. GA must be a no-op for the already enabled beta feature.

+1 (non-binding)

[haircommander]

yeah we could maybe make progress in 1.32 while in beta (defaulting to on) and then in 1.33 push for GA

[SergeyKanzhelev]

This seems to be a piece of feedback: kubernetes/kubernetes#126630

[brandond]

We're looking at using this for K3s and RKE2's managed configuration somewhat soonish. Would be nice if we could get the resolv-conf issue fixed first though.

[liggitt]

The merging process for files in the drop-in dir is incorrect and must be fixed before graduating. That is the root cause of the resolvConf issue (kubernetes/kubernetes#126630 (comment), kubernetes/kubernetes#79421 (comment)).

Opened kubernetes/kubernetes#127421 with a fix

[haircommander]

/milestone v1.32
/label lead-opted-in

[SergeyKanzhelev]

the PR kubernetes/kubernetes#127421 was merged with two TODO. We need to address todos as a criteria to move forward with this KEP.

[liggitt]

We need to address todos as a criteria to move forward with this KEP.

That's up to you. Those TODOs have to be resolved before kubelet revs its configuration format to v1beta2 or v1, certainly.

[SergeyKanzhelev]

We need to address todos as a criteria to move forward with this KEP.

That's up to you. Those TODOs have to be resolved before kubelet revs its configuration format to v1beta2 or v1, certainly.

Yep. There is little that will help us catch this when we have v1. So I think it's best fixed as part of this KEP.

[shecodesmagic]

Hello @haircommander 👋, v1.32 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024.

This enhancement is targeting for stage stable for v1.32 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has submitted a production readiness review request for approval and has a reviewer assigned.
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 3rd October 2024 so that the PRR team has enough time to review your KEP.

For this KEP, we would need to update the following:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has submitted a production readiness review request for approval and has a reviewer assigned.
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on
  Thursday 3rd October 2024 so that the PRR team has enough time to review your KEP.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!