You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
How can pod start-up time be accelerated on nodes in large clusters? This is a common issue that cluster administrators may face.
11
11
12
-
This blog post focuses on methods to speed up pod start-up from the kubelet side. It does not involve the creation time of pods by controller-manager through kube-apiserver, nor does it include scheduling time for pods.
12
+
This blog post focuses on methods to speed up pod start-up from the kubelet side. It does not involve the creation time of pods by controller-manager through kube-apiserver, nor does it include scheduling time for pods or webhooks executed on it.
13
13
14
14
We have mentioned some important factors here to consider from the kubelet's perspective, but this is not an exhaustive list. As Kubernetes v1.27 is set to release soon, this blog will highlight significant changes in v1.27 that aid in speeding up pod start-up.
15
15
@@ -25,25 +25,26 @@ We are introducing a new feature in kubelet that sets a limit on the number of p
25
25
26
26
To limit the number of simultaneous image pulls, you can configure the `maxParallelImagePulls` field in kubelet. By setting `maxParallelImagePulls` to a value of _n_, only _n_ images will be pulled concurrently. Any additional image pulls beyond this limit will wait until at least one ongoing pull is complete.
27
27
28
-
More detials can be found in the KEP <https://kep.k8s.io/3673>.
28
+
You can find more details in the associated KEP: [Kubelet limit of Parallel Image Pulls](https://kep.k8s.io/3673)
29
+
(KEP-3673). The improvement will arrive as part of Kubernetes v1.27 (scheduled for release in April 2023).
29
30
30
-
## bump default API QPS limits for Kubelet
31
+
## Raised default API query-per-second limits for kubelet
31
32
32
-
To improve pod startup in scenarios with multiple pods on a node, particularly sudden scaling situations, it is necessary for Kubelet to synchronize the pod status and prepare configmaps, secrets or volumes. This requires a large bandwidth to access kube-apiserver.
33
+
To improve pod startup in scenarios with multiple pods on a node, particularly sudden scaling situations, it is necessary for Kubelet to synchronize the pod status and prepare configmaps, secrets, or volumes. This requires a large bandwidth to access kube-apiserver.
33
34
34
35
In versions prior to v1.27, the default `kubeAPIQPS` was 5 and `kubeAPIBurst` was 10. However, Kubelet has increased these defaults to 50 and 100 respectively for better performance during pod startup. It's worth noting that this isn't the only reason why we've bumped up the API QPS limits for Kubelet.
35
36
36
37
1. It has a potential to be hugely throttled now (default QPS = 5)
37
38
2. In large clusters they can generate significant load anyway as there are a lot of them
38
39
3. They have a dedicated PriorityLevel and FlowSchema that we can easily control
39
40
40
-
In old days, we always encounter `volume mount timeout` on kubelet in node with more than 50 pods during pod start up. We suggest our customers to bump 3 or 4 times of kube api QPS and especially with customers that is using bare-metal(physical machines).
41
+
In old days, we always encounter `volume mount timeout` on kubelet in node with more than 50 pods during pod start up. We suggest our customers to bump `kubeAPIQPS` to 20 and `kubeAPIBurst` to 40 and especially with customers that is using bare-metal(physical machines).
41
42
42
43
More detials can be found in the KEP <https://kep.k8s.io/1040> and <https://github.com/kubernetes/kubernetes/pull/116121>.
43
44
44
-
## Evented PLEG will enhance the scalability of kubelet
45
+
## Event triggered updates to container status
45
46
46
-
`Evented PLEG` is set to be in beta for v1.27, enabled by default. It aids pod startup in large clusters by reducing unnecessary work during inactivity through replacing periodic polling. The current `Generic PLEG` incurs non-negligible overhead due to frequent container status polling, which worsens with Kubelet's parallelism, leading to poor performance and reliability issues. For these reasons, it is recommended that we use Evented PLGE instead. Further details can be found in the KEP <https://kep.k8s.io/3386>.
47
+
`Evented PLEG`(PLEG is short for "Pod Lifecycle Event Generator") is set to be in beta for v1.27, disabled by default (needs to be enabled explicitly). It aids pod startup by reducing unnecessary work during inactivity through replacing periodic polling. The current `Generic PLEG` incurs non-negligible overhead due to frequent container status polling, which worsens with Kubelet's parallelism, leading to poor performance and reliability issues. For these reasons, it is recommended that we use Evented PLEG instead. Further details can be found in the KEP <https://kep.k8s.io/3386>.
0 commit comments