k8s: increase the kube-api-server QPS from 5/10 to 10/20

[tzneal]

Issue number:

N/A

Description of changes:

This changes the default setting for EKS v1.22+ where API Priority & Fairness is available and there is a specific queue for kubelet health.

Testing done:

cargo make unit-tests and

• Built a custom K8s v1.23 Bottlerocket AMI
• Launched a node using that AMI
• Verified that the config changes were applied by logging in via SSM

bash-5.1# cat /etc/kubernetes/kubelet/config  | grep kubeAPI
kubeAPIQPS: 10
kubeAPIBurst: 20

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[tzneal]

In testing increasing from 5 QPS to 10 QPS effectively halves the overall time for 100 pods to become ready on a single node from 67.5 to 32.8 seconds while also halving the time for the first pod to become ready from to 22 seconds to 10 second.

For larger scale-up events (in this case 3k pods and 30 nodes) the benefits above 10/20 QPS settings decrease, though a large benefit still remains if the values are increased from the default 5/10 QPS.

Any change in the kubelet QPS settings is scaled by the number of nodes in each cluster. This poses a risk that we can overwhelm the kube-api-server by increasing the amount of traffic received from kubelet running on each node. The primary mitigation to this risk is that for K8s v1.20+ the API Priority & Fairness feature is enabled in EKS. This added to Kubernetes the concept of priority levels where requests in different priority levels cannot starve each other. In v1.22+ specifically there is a separate queue for K8s health checks which will prevent increased load from pod churn from interfering with Kubelet health reporting. The increase here is small while realizing most of the benefit, which further decreases negative impact risks.

[etungsten]

This changes the default setting for EKS v1.22+

These kubelet configs are also used in non-EKS K8s variants as well, e.g. metal-k8s-*, vmware-k8s-*.

I think increasing the defaults still makes sense. I also agree with the assessment that the benefits here outweighs any potential downsides.

[jpmcb]

Very nice! 🏃🏼

[zmrow]

Thanks!

[bcressey]

I really appreciate the focus on metrics and data here. Looks like a nice win.

[tzneal]

I really appreciate the focus on metrics and data here. Looks like a nice win.

Thanks! I don't have permissions on this repo to merge, so feel free to merge whenever.

[arnaldo2792]

I'm merging since all tests passed (thanks @tzneal!)