kubelet: add setting for configuring kubeAPIQPS and kubeAPIBurst

[gthao313]

Issue number:
#1495

Description of changes:
Adds two new settings kubernetes.kube-api-qps and kubernetes.kube-api-burst for configuring

Testing done:

kube-api-qps

Creating large amount of pods (which needs to talk with kube-apiserver) at same time, and time how long it would take. Compare two argument values (smaller value and larger value ), the time that spends to create pods by smaller value is longer than by larger value, because small value limits the QPS of talking to kube-apiserver.

Step1: kube-api-qps = 100 (larger value)

kubectl apply -f amazonlinux.yaml -f debian.yaml -f ubuntu.yaml -f nginx.yaml -f python.yaml -f golang.yaml -f httpd.yaml -f node.yaml -f redis.yaml -f mysql.yaml -f postgres.yaml
time kubectl wait --for=condition=ready --all pod

Result:

real	1m5.026s

Step2: kube-api-qps = 1 lowest value (smaller value)

kubectl apply -f amazonlinux.yaml -f debian.yaml -f ubuntu.yaml -f nginx.yaml -f python.yaml -f golang.yaml -f httpd.yaml -f node.yaml -f redis.yaml -f mysql.yaml -f postgres.yaml
time kubectl wait --for=condition=ready --all pod

Result:

real	0m36.846s

Test result: The time that spends to create pods by smaller value kube-api-qps = 1 will longer than by larger value kube-api-qps = 100, which means setting kubernetes.kube-api-qps works.

kube-api-burst

Creating large amount of pods (which needs to talk with kube-apiserver) at same time, and time how long it would take. Compare two argument values (smaller value and larger value ), the time that spends to create pods by smaller value is longer than by larger value, because small value limits the burst of talking to kube-apiserver.

Step1: kube-api-burst= 100 (larger value)

kubectl apply -f amazonlinux.yaml -f debian.yaml -f ubuntu.yaml -f nginx.yaml -f python.yaml -f golang.yaml -f httpd.yaml -f node.yaml -f redis.yaml -f mysql.yaml -f postgres.yaml
time kubectl wait --for=condition=ready --all pod

Result:

real	0m37.224s

Step2: event-burst = 1 lowest value (smaller value)

kubectl apply -f amazonlinux.yaml -f debian.yaml -f ubuntu.yaml -f nginx.yaml -f python.yaml -f golang.yaml -f httpd.yaml -f node.yaml -f redis.yaml -f mysql.yaml -f postgres.yaml
time kubectl wait --for=condition=ready --all pod

Result:

real	0m46.027s

Test result: The time that spends to create pods by smaller value kube-api-burst = 1 will longer than by larger value kube-api-burst = 100, which means setting kubernetes.kube-api-burst works.

Migration test:

upgrade

Step1: Upgrade to v1.1.0

bash-5.0# updog check-update -a --json
[
  {
    "variant": "aws-k8s-1.19",
    "arch": "x86_64",
    "version": "1.1.0",
    "max_version": "1.1.0",
.....
bash-5.0# updog update -i 1.1.0 -r -n
Starting update to 1.1.0

Step2: Specify new setting kube-api-qps and kube-api-burst through control container

apiclient set -j '{"kubernetes": {"kube-api-qps": 50, "kube-api-burst": 100}}

bash-5.0# cat /var/lib/bottlerocket/datastore/current/live/settings/kubernetes/kube-api-qps
50
bash-5.0# cat /var/lib/bottlerocket/datastore/current/live/settings/kubernetes/kube-api-burst
100

downgrade

Step1: Check migration binary

ls -al /var/lib/bottlerocket-migrations
-rw-r--r--.  1 root root 500592 Apr 30 22:44 06c4eb9923cc3af6717b3caa8b628ceb87060522c1a75c1c1e5a9dcb23914922.migrate_v1.1.0_kubelet-kube-api-qps-kube-api-burst.lz4
-rw-r--r--.  1 root root 500376 Apr 30 22:44 20bf88b0fc969d079323303636d59f7dee19334d5330ca4d98f03e83dd0768bc.migrate_v1.1.0_kubelet-server-tls-bootstrap.lz4
-rw-r--r--.  1 root root   2970 Apr 30 22:44 2dd9fcbe78d91d931bb85a75d0124cdb25c71e99e5ad36ac600b0f258e2197e2.manifest.json
-rw-r--r--.  1 root root 500441 Apr 30 22:44 4a5d86d8b8342b562e43ad8aae876bf71249510ec6e1a83b52df998e7180299e.migrate_v1.1.0_kubelet-registry-qps-registry-burst.lz4
-rw-r--r--.  1 root root 500460 Apr 30 22:44 81d7535ebd76e9307766c658f1aaffdba91f227d31acb2421058a9d7ba7a53cb.migrate_v1.1.0_kubelet-cloud-provider.lz4

Step2: Downgrade to previous verison

signpost rollback-to-inactive
reboot

Step3: Check if kube-api-qps and kube-api-burst have been removed

bash-5.0# ls /var/lib/bottlerocket/datastore/current/live/settings/kubernetes
api-server	     cloud-provider	  cluster-dns-ip		    cluster-domain  max-pods			node-ip			   pod-infra-container-image			pod-infra-container-image.setting-generator  standalone-mode
authentication-mode  cluster-certificate  cluster-dns-ip.setting-generator  cluster-name    max-pods.setting-generator	node-ip.setting-generator  pod-infra-container-image.affected-services	server-tls-bootstrap			     static-pods.affected-service

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[gthao313]

Push above fix conflicts.

[gthao313]

Push above fix conflicts

[gthao313]

Push above fix conflicts again.