.featureGates.RotateKubeletServerCertificate dropped from kubelet config in k8s 1.28

[errm]

Image I'm using:

bottlerocket-aws-k8s-1.28-x86_64-v1.15.0-c9af43ad

What I expected to happen:

I am benchmarking against CIS guidelines using kube-bench

Test 3.2.11 starts to fail when I upgrade to k8s 1.28.

This control is checking for the presence of .featureGates.RotateKubeletServerCertificate in the kubelet config.

What actually happened:

Images for kubernetes 1.27 and below included the following in packages/kubernetes-1.x/kubelet-config

 featureGates:
   RotateKubeletServerCertificate: true

This seems to have been removed from the kubernetes-1.28 package without any comment or information in the relevant commit bf9e579

I want to understand if this was an intentional change?

There seems to be some test code here https://github.com/bottlerocket-os/bottlerocket/blob/develop/sources/bloodhound/README.md that should run this check... but I can't tell if it is run as part of the build process ...

How to reproduce the problem:

Run kube-bench against a bottlerocket - k8s 1.28 image.

[stmcginnis]

Hi @errm - sorry, that should maybe have been called out more clearly than just passing comments.

This feature gate is actually enabled by default since Kubernetes 1.12. So there wasn't really a reason to ever have it present in the config file with any of the versions Bottlerocket supports.

The reference you found in Bloodhound may actually be very interesting to you if you are trying to run the CIS Benchmarks. I'm sure you've seen a few other things that are just a little bit different in Bottlerocket that causes problems trying to use some of the tools designed for general purpose OSs. Bloodhound is an internal tool that was added and exposed through apiclient for running the Bottlerocket or Kubernetes CIS benchmarks. They provide either a nicely formatted human-readable format or a JSON formatted report that can be convenient for piping in to other programmatic reporting tools.

You can get a little more background context with the Bottlerocket CIS report and Kubernetes CIS report PRs.

Just let me know if you have any questions.

[errm]

This feature gate is actually enabled by default since Kubernetes 1.12. So there wasn't really a reason to ever have it present in the config file with any of the versions Bottlerocket supports.

Yes thanks - that was my suspicion

I was a bit confused though because this config seems to have appeared in bottlerocket for k8s 1.23 - then went away again now for 1.28 - and that didn't seem like it lined up with this feature gate becoming a default.

I think the correct thing in my case is probably to make a PR to kube-bench so it can understand this default correctly, just like bloodhound does here https://github.com/bottlerocket-os/bottlerocket/blob/develop/sources/bloodhound/src/bin/kubernetes-checks/checks.rs#L596-L599