diff --git a/content/en/blog/_posts/2026-08-20-migrate-cgroup-v2.md b/content/en/blog/_posts/2026-08-20-migrate-cgroup-v2.md new file mode 100644 index 0000000000000..72b7f9ce74062 --- /dev/null +++ b/content/en/blog/_posts/2026-08-20-migrate-cgroup-v2.md @@ -0,0 +1,262 @@ +--- +layout: blog +title: 'The Shift to cgroup v2 in Kubernetes: What You Need to Know' +draft: true +slug: kubernetes-cgroups-v2-shift +author: > + Paco Xu (DaoCloud) +--- + +In Linux, _cgroups_ (control groups) are a kernel feature used for managing system resources. +Kubernetes uses cgroups to allocate resources like CPU and memory to containers, +ensuring that applications run smoothly without interfering with each other. +With the release of Kubernetes v1.31, support for v1 cgroup management moved into +[maintenance mode](/blog/2024/08/14/kubernetes-1-31-moving-cgroup-v1-support-maintenance-mode/). +Support for v2 cgroup management has been stable since Kubernetes v1.25. + +Compared with cgroup v1, cgroup v2 provides a single unified hierarchy, +a more consistent interface, and a stronger foundation for resource isolation +and modern resource-management features. + +## Deprecation of cgroup v1 + +FEATURE STATE: Kubernetes v1.35 [deprecated] + +Kubernetes has [deprecated cgroup v1](/docs/concepts/architecture/cgroups/#deprecation-of-cgroup-v1). +Starting with Kubernetes v1.35, `failCgroupV1` defaults to `true`, so the kubelet does not start +on a cgroup v1 node by default. Administrators can temporarily set `failCgroupV1: false` in the +[kubelet configuration file](/docs/tasks/administer-cluster/kubelet-config-file/), but removal will +follow the [Kubernetes deprecation policy](/docs/reference/using-api/deprecation-policy/). +Track that work in [KEP-5573: Remove cgroup v1 support](https://kep.k8s.io/5573). + +Before upgrading to v1.35 or later, migrate every Linux node to cgroup v2 or explicitly use the +temporary override. A node that still uses cgroup v1 otherwise fails during kubelet startup. + +For kubeadm-managed clusters, Kubernetes v1.35 also makes this an earlier, stricter check. The +`SystemVerification` preflight check, provided by `k8s.io/system-validators`, returns an error during +`kubeadm init`, `kubeadm join`, and `kubeadm upgrade` when it detects cgroup v1 with kubelet v1.35 or +later; with an older kubelet, the check remains a warning. See +[kubernetes/kubernetes#134744](https://github.com/kubernetes/kubernetes/pull/134744) for details. + +The top FAQs cover three main areas: why to migrate, the benefits and drawbacks, +and key points to keep in mind when using cgroup v2. + +## Limitations of cgroup v1 and Improvements with cgroup v2 + +The Linux kernel documentation describes both interfaces: + +- [cgroup v1 documentation](https://www.kernel.org/doc/Documentation/cgroup-v1/) +- [cgroup v2 documentation](https://docs.kernel.org/admin-guide/cgroup-v2.html) + +Let's enumerate some known issues. + +### `active_file` memory is not considered available memory + +The kubelet treats `active_file` memory as not reclaimable. For I/O-intensive workloads, a large +page cache can therefore make the kubelet report memory pressure and evict Pods. This is a +[known kubelet issue](/docs/concepts/scheduling-eviction/node-pressure-eviction/#active-file-memory-is-not-considered-as-available-memory) +([kubernetes/kubernetes#43916](https://github.com/kubernetes/kubernetes/issues/43916)); migrating to +cgroup v2 does not by itself change that calculation. The documented workaround is to set equal +memory requests and limits for containers that perform intensive I/O, after measuring an appropriate +value. + +### Memory QoS updates in Kubernetes v1.36 + +[Memory QoS](https://kep.k8s.io/2570) was introduced as an alpha feature in Kubernetes v1.22 and +updated in v1.27. It remains alpha in v1.36, but now separates memory throttling from memory +reservation and adds [tiered memory protection](/blog/2026/04/29/kubernetes-v1-36-memory-qos-tiered-protection/): + +**Note:** +Memory QoS is available only on Linux nodes that use cgroup v2. It relies on the cgroup v2 memory +controller: `memory.high` provides throttling, while `memory.min` and `memory.low` provide hard and +soft protection when tiered reservation is enabled. cgroup v1 cannot provide this protection model. + +- Enabling the `MemoryQoS` feature gate applies `memory.high` throttling to Burstable containers. + The threshold is derived from the request, limit, and `memoryThrottlingFactor` (default `0.9`). +- `memoryReservationPolicy: None` is the default. It does not write `memory.min` or `memory.low`. +- `memoryReservationPolicy: TieredReservation` maps Guaranteed Pod memory requests to `memory.min` + (hard protection) and Burstable Pod requests to `memory.low` (soft protection). BestEffort Pods + receive neither protection. +- The kubelet exposes alpha metrics for the total `memory.min` and `memory.low` reservations on a node. +- Kernel 5.9 or later is recommended. On older kernels, `memory.high` reclaim can trigger a known + livelock; from v1.36 the kubelet logs a warning when Memory QoS is enabled on an affected kernel. + +For example, to opt in to tiered protection: + +```yaml +apiVersion: kubelet.config.k8s.io/v1beta1 +kind: KubeletConfiguration +featureGates: + MemoryQoS: true +memoryReservationPolicy: TieredReservation +memoryThrottlingFactor: 0.9 +``` + +Because Memory QoS is still alpha and disabled by default, test the configuration and account for +hard-reserved memory before enabling `TieredReservation` in production. See the +[Pod QoS documentation](/docs/concepts/workloads/pods/pod-qos/#memory-qos-with-cgroup-v2) for the +current mapping of Kubernetes QoS classes to cgroup v2 controls. + +### Container-aware OOM handling + +On cgroup v2 nodes, the kubelet defaults `singleProcessOOMKill` to `false`. It therefore sets +`memory.oom.group` for each container cgroup so that an OOM event kills all processes in that +container together, rather than leaving a partially functioning multi-process container. Set +`singleProcessOOMKill: true` only if you need the cgroup v1-compatible behavior where the kernel +may kill one process at a time. See the +[`KubeletConfiguration` reference](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) +for this setting. + +This behavior is scoped to a container cgroup, not the whole Pod. Also, `cgroup.kill` is a separate +administrative interface: writing `1` to it sends `SIGKILL` to every process in that cgroup and its +descendants; it does not configure OOM behavior. The cgroup v2 memory controller additionally +provides `memory.events` counters that monitoring systems and userspace OOM managers can observe. + +### Rootless support + +In cgroup v1, delegating controllers to less privileged containers may be dangerous. + +Unlike cgroup v1, cgroup v2 officially supports delegation. +Most Rootless Containers implementations rely on systemd for +delegating v2 controllers to non-root users. + +This delegation mechanism is separate from Kubernetes Pod user namespaces, which map container +users to unprivileged host users. [Pod user namespace support](/docs/concepts/workloads/pods/user-namespaces/) +graduated to stable in Kubernetes v1.36; check its filesystem, kernel, CRI runtime, and OCI runtime +prerequisites before enabling it. + +### What else? + +1. eBPF stories: + - In cgroup v1, device access controls are exposed through interface files. + - The cgroup v2 device controller has no interface files, and is implemented on top of cgroup BPF. + - Cilium attaches BPF cgroup programs for socket-based load balancing. Its + [default cgroup root](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#validate-bpf-cgroup-programs-attachment) + is `/run/cilium/cgroupv2`. +2. [Pressure Stall Information (PSI)](/docs/reference/instrumentation/understand-psi-metrics/) + reports CPU, memory, and I/O contention at node, Pod, and container level. The `KubeletPSI` + feature graduated to stable in Kubernetes v1.36 and is always enabled. PSI requires cgroup v2, + Linux 4.20 or later, `CONFIG_PSI=y`, and a kernel not booted with `psi=0`. The kubelet exposes + the data through the Summary API and `/metrics/cadvisor`. +3. When migrating, update software that reads the cgroup filesystem directly. The + [migration guidance](/docs/concepts/architecture/cgroups/#migrating-cgroupv2) recommends + cAdvisor v0.43.0 or later and lists compatible Java, Node.js, and `automaxprocs` versions. + +### CPU weight conversion in newer OCI runtimes + +cgroup v1 uses `cpu.shares`, whereas cgroup v2 uses `cpu.weight`. Newer OCI runtimes use an improved +non-linear conversion that preserves the default priority and gives small CPU requests more usable +granularity. The change is implemented in the OCI runtime rather than Kubernetes: it is available +in runc v1.3.2 and crun v1.23. After upgrading a runtime, monitoring or policy tools that predict +exact `cpu.weight` values may need updates. Read +[New Conversion from cgroup v1 CPU Shares to v2 CPU Weight](/blog/2026/01/30/new-cgroup-v1-to-v2-cpu-conversion-formula/) +for the formula, examples, and compatibility considerations. + +### In-place resource updates + +[In-place Pod vertical scaling](/blog/2025/12/17/kubernetes-v1-35-release/#stable-in-place-update-of-pod-resources) +graduated to stable in Kubernetes v1.35. Kubernetes v1.36 then enabled +[in-place vertical scaling for Pod-level resources](/blog/2026/04/30/kubernetes-v1-36-inplace-pod-level-resources-beta/) +by default as a beta feature. The kubelet coordinates changes between the Pod-level and container +cgroups so that increases create headroom before container limits grow, while decreases constrain +containers before shrinking the Pod-level boundary. Accurate aggregate enforcement for this +v1.36 feature requires cgroup v2. + +## Adopting cgroup version 2 + +### Requirements + +Here's what you need to use cgroup v2 with Kubernetes. +First up, you need to be using a version of Kubernetes with support for v2 cgroup management; +that's been stable since Kubernetes v1.25 and all supported Kubernetes releases include this support. + +- OS distribution enabled cgroup v2 +- Linux kernel version is 5.8 or later (5.9 or later is recommended when using Memory QoS) +- Container runtime supports cgroup v2. For example: + - containerd v1.4 or later supports cgroup v2; use containerd v2.0 or later for automatic + cgroup-driver discovery + - CRI-O v1.20 or later +- The kubelet and the container runtime are configured to use the systemd cgroup driver + +The Kubernetes v1.35 release announced the end of the containerd 1.x support window. However, the +final Kubernetes v1.36 runtime documentation still describes a fallback for containerd 1.x, which +does not implement the `RuntimeConfig` CRI RPC. On those nodes the kubelet uses its configured +`cgroupDriver` value instead of automatic discovery. That fallback is scheduled for removal in +Kubernetes v1.38. For new or upgraded v1.36 nodes, prefer a currently supported containerd 2.x +release and review both the [Kubernetes runtime documentation](/docs/setup/production-environment/container-runtimes/#systemd-cgroup-driver) +and the [containerd compatibility matrix](https://github.com/containerd/containerd/blob/main/RELEASES.md#kubernetes-support). + +#### kernel updates around cgroup v2 + +When Kubernetes was first announced, in 2014, only v1 cgroup existed. +Version 2 cgroup management first appeared in Linux kernel 4.5, released in 2016. + +- In Linux 4.5, the cgroup v2 `io`, `memory`, and `pids` controllers were supported. +- Linux 4.15 added support for the cgroup v2 `cpu` controller. +- [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html) (PSI) support began with Linux 4.20. +- The Kubernetes project does not recommend using cgroup v2 with a Linux kernel older than 5.2 + due to lack of cgroup-level task freezer support. +- Kubernetes documents 5.8 as the minimum kernel version for cgroup v2; the root cgroup's + system-level `cpu.stat` file was added in Linux 5.8. +- The `memory.high` livelock fix used by Memory QoS is present in Linux 5.9 and later. +- `memory.peak` was added in Linux 5.19. + +### Use systemd as cgroup driver + +[Configure the kubelet's cgroup driver to match the container runtime cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/). + +The [Container runtimes](/docs/setup/production-environment/container-runtimes) +page explains that the `systemd` driver is recommended for kubeadm +based setups instead of the kubelet's default `cgroupfs` driver, +because kubeadm manages the kubelet as a systemd service. + +A minimal example of configuring the field explicitly: + +```yaml +apiVersion: kubelet.config.k8s.io/v1beta1 +kind: KubeletConfiguration +cgroupDriver: systemd +``` + +Automatic discovery of the runtime's cgroup driver through the CRI, tracked by +[KEP-4033](https://kep.k8s.io/4033), graduated to stable in Kubernetes v1.34. It requires a runtime +that implements the `RuntimeConfig` CRI RPC (containerd v2.0+ or CRI-O v1.28+). When available, +the kubelet uses the value reported by the runtime instead of its configured `cgroupDriver` value. + +### Tools and commands for troubleshooting + +Tools and commands that you should know about cgroups: + +- `stat -fc %T /sys/fs/cgroup/`: Check whether cgroup v2 is enabled; it returns `cgroup2fs`. +- `systemctl list-units 'kube*' --type=slice` or `--type=scope`: List Kubernetes-related units + that systemd currently has in memory. +- `bpftool cgroup list /sys/fs/cgroup/*`: List all programs attached to the cgroup CGROUP. +- `systemd-cgls /sys/fs/cgroup/*`: Recursively show control group contents. +- `systemd-cgtop`: Show top control groups by their resource usage. +- `tree -L 2 -d /sys/fs/cgroup/kubepods.slice`: Show Pods' related cgroups directories. + +How to check if a Pod CPU or memory limit is successfully applied to the cgroup file? + +- Kubernetes Pod spec: check limits `spec.containers[*].resources.limits.{cpu,memory}` and requests + `spec.containers[*].resources.requests.{cpu,memory}` +- CRI: `cpu_period`, `cpu_quota`, `cpu_shares` for CPU and `memory_limit_in_bytes` for memory limit +- OCI Spec: `memory.limit`, `cpu.shares`, `cpu.quota`, `cpu.period` +- Systemd Scope Unit: `CPUWeight`, `CPUQuotaPerSecUSec`, `CPUQuotaPeriodUSec`, `MemoryMax` +- Cgroupfs value: `/sys/fs/cgroup/../cpu.weight`, `/sys/fs/cgroup/../cpu.max`, `/sys/fs/cgroup/../memory.max` + +## Further reading + +- [Kubernetes 1.31: Moving cgroup v1 Support into Maintenance Mode](/blog/2024/08/14/kubernetes-1-31-moving-cgroup-v1-support-maintenance-mode/) +- [Kubernetes 1.35: Timbernetes](/blog/2025/12/17/kubernetes-v1-35-release/) +- [Kubernetes v1.36: Tiered Memory Protection with Memory QoS](/blog/2026/04/29/kubernetes-v1-36-memory-qos-tiered-protection/) +- [Kubernetes v1.36: PSI Metrics for Kubernetes Graduates to GA](/blog/2026/05/12/kubernetes-v1-36-psi-metrics-ga/) +- [Kubernetes v1.36: In-Place Vertical Scaling for Pod-Level Resources Graduates to Beta](/blog/2026/04/30/kubernetes-v1-36-inplace-pod-level-resources-beta/) +- [cgroup v2 in Kubernetes](/docs/concepts/architecture/cgroups/) +- [Kubernetes 1.27: Quality-of-Service for Memory Resources (alpha)](/blog/2023/05/05/qos-memory-resources/) +- [Kubernetes 1.25: cgroup v2 graduates to GA](/blog/2022/08/31/cgroupv2-ga-1-25/) +- KubeCon NA 2022 [cgroup v2: Before You Jump In](https://www.youtube.com/watch?v=WxZK-UXKvXk) by Tony Gosselin & Mike Tougeron, Adobe Systems +- KubeCon NA 2022 [Cgroupv2 Is Coming Soon To a Cluster Near You](https://www.youtube.com/watch?v=sgyFCp1CRhA) - David Porter, Google & Mrunal Patel, RedHat +- KubeCon EU 2020 [Kubernetes On cgroup v2](https://www.youtube.com/watch?v=u8h0e84HxcE&t=783s) by Giuseppe Scrivano, Red Hat. +- This blog only covers the basic requirements and configuration of Kubernetes components. + It will not include how to enable cgroup fs in OS distributions. + For migration, you can refer to [migrating cgroup v2](/docs/concepts/architecture/cgroups/#migrating-cgroupv2) diff --git a/content/en/blog/_posts/2026/migrate-cgroup-v2.md b/content/en/blog/_posts/2026/migrate-cgroup-v2.md new file mode 100644 index 0000000000000..faec8b798c86a --- /dev/null +++ b/content/en/blog/_posts/2026/migrate-cgroup-v2.md @@ -0,0 +1,263 @@ +--- +layout: blog +title: 'The Shift to cgroup v2 in Kubernetes: What You Need to Know' +draft: true +slug: kubernetes-cgroups-v2-shift +author: > + Paco Xu (DaoCloud) +--- + +In Linux, _cgroups_ (control groups) are a kernel feature used for managing system resources. +Kubernetes uses cgroups to allocate resources like CPU and memory to containers, +ensuring that applications run smoothly without interfering with each other. +With the release of Kubernetes v1.31, support for v1 cgroup management moved into +[maintenance mode](/blog/2024/08/14/kubernetes-1-31-moving-cgroup-v1-support-maintenance-mode/). +Support for v2 cgroup management has been stable since Kubernetes v1.25. + +Compared with cgroup v1, cgroup v2 provides a single unified hierarchy, +a more consistent interface, and a stronger foundation for resource isolation +and modern resource-management features. + +## Deprecation of cgroup v1 + +FEATURE STATE: Kubernetes v1.35 [deprecated] + +Kubernetes has [deprecated cgroup v1](/docs/concepts/architecture/cgroups/#deprecation-of-cgroup-v1). +Starting with Kubernetes v1.35, `failCgroupV1` defaults to `true`, so the kubelet does not start +on a cgroup v1 node by default. Administrators can temporarily set `failCgroupV1: false` in the +[kubelet configuration file](/docs/tasks/administer-cluster/kubelet-config-file/), but removal will +follow the [Kubernetes deprecation policy](/docs/reference/using-api/deprecation-policy/). +Track that work in [KEP-5573: Remove cgroup v1 support](https://kep.k8s.io/5573). + +Before upgrading to v1.35 or later, migrate every Linux node to cgroup v2 or explicitly use the +temporary override. A node that still uses cgroup v1 otherwise fails during kubelet startup. + +For kubeadm-managed clusters, Kubernetes v1.35 also makes this an earlier, stricter check. The +`SystemVerification` preflight check, provided by `k8s.io/system-validators`, returns an error during +`kubeadm init`, `kubeadm join`, and `kubeadm upgrade` when it detects cgroup v1 with kubelet v1.35 or +later; with an older kubelet, the check remains a warning. See +[kubernetes/kubernetes#134744](https://github.com/kubernetes/kubernetes/pull/134744) for details. + +The top FAQs cover three main areas: why to migrate, the benefits and drawbacks, +and key points to keep in mind when using cgroup v2. + +## Limitations of cgroup v1 and Improvements with cgroup v2 + +The Linux kernel documentation describes both interfaces: + +- [cgroup v1 documentation](https://www.kernel.org/doc/Documentation/cgroup-v1/) +- [cgroup v2 documentation](https://docs.kernel.org/admin-guide/cgroup-v2.html) + +Let's enumerate some known issues. + +### `active_file` memory is not considered available memory + +The kubelet treats `active_file` memory as not reclaimable. For I/O-intensive workloads, a large +page cache can therefore make the kubelet report memory pressure and evict Pods. This is a +[known kubelet issue](/docs/concepts/scheduling-eviction/node-pressure-eviction/#active-file-memory-is-not-considered-as-available-memory) +([kubernetes/kubernetes#43916](https://github.com/kubernetes/kubernetes/issues/43916)); migrating to +cgroup v2 does not by itself change that calculation. The documented workaround is to set equal +memory requests and limits for containers that perform intensive I/O, after measuring an appropriate +value. + +### Memory QoS updates in Kubernetes v1.36 + +[Memory QoS](https://kep.k8s.io/2570) was introduced as an alpha feature in Kubernetes v1.22 and +updated in v1.27. It remains alpha in v1.36, but now separates memory throttling from memory +reservation and adds [tiered memory protection](/blog/2026/04/29/kubernetes-v1-36-memory-qos-tiered-protection/): + +{{< note >}} +Memory QoS is available only on Linux nodes that use cgroup v2. It relies on the cgroup v2 memory +controller: `memory.high` provides throttling, while `memory.min` and `memory.low` provide hard and +soft protection when tiered reservation is enabled. cgroup v1 cannot provide this protection model. +{{< /note >}} + +- Enabling the `MemoryQoS` feature gate applies `memory.high` throttling to Burstable containers. + The threshold is derived from the request, limit, and `memoryThrottlingFactor` (default `0.9`). +- `memoryReservationPolicy: None` is the default. It does not write `memory.min` or `memory.low`. +- `memoryReservationPolicy: TieredReservation` maps Guaranteed Pod memory requests to `memory.min` + (hard protection) and Burstable Pod requests to `memory.low` (soft protection). BestEffort Pods + receive neither protection. +- The kubelet exposes alpha metrics for the total `memory.min` and `memory.low` reservations on a node. +- Kernel 5.9 or later is recommended. On older kernels, `memory.high` reclaim can trigger a known + livelock; from v1.36 the kubelet logs a warning when Memory QoS is enabled on an affected kernel. + +For example, to opt in to tiered protection: + +```yaml +apiVersion: kubelet.config.k8s.io/v1beta1 +kind: KubeletConfiguration +featureGates: + MemoryQoS: true +memoryReservationPolicy: TieredReservation +memoryThrottlingFactor: 0.9 +``` + +Because Memory QoS is still alpha and disabled by default, test the configuration and account for +hard-reserved memory before enabling `TieredReservation` in production. See the +[Pod QoS documentation](/docs/concepts/workloads/pods/pod-qos/#memory-qos-with-cgroup-v2) for the +current mapping of Kubernetes QoS classes to cgroup v2 controls. + +### Container-aware OOM handling + +On cgroup v2 nodes, the kubelet defaults `singleProcessOOMKill` to `false`. It therefore sets +`memory.oom.group` for each container cgroup so that an OOM event kills all processes in that +container together, rather than leaving a partially functioning multi-process container. Set +`singleProcessOOMKill: true` only if you need the cgroup v1-compatible behavior where the kernel +may kill one process at a time. See the +[`KubeletConfiguration` reference](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) +for this setting. + +This behavior is scoped to a container cgroup, not the whole Pod. Also, `cgroup.kill` is a separate +administrative interface: writing `1` to it sends `SIGKILL` to every process in that cgroup and its +descendants; it does not configure OOM behavior. The cgroup v2 memory controller additionally +provides `memory.events` counters that monitoring systems and userspace OOM managers can observe. + +### Rootless support + +In cgroup v1, delegating controllers to less privileged containers may be dangerous. + +Unlike cgroup v1, cgroup v2 officially supports delegation. +Most Rootless Containers implementations rely on systemd for +delegating v2 controllers to non-root users. + +This delegation mechanism is separate from Kubernetes Pod user namespaces, which map container +users to unprivileged host users. [Pod user namespace support](/docs/concepts/workloads/pods/user-namespaces/) +graduated to stable in Kubernetes v1.36; check its filesystem, kernel, CRI runtime, and OCI runtime +prerequisites before enabling it. + +### What else? + +1. eBPF stories: + - In cgroup v1, device access controls are exposed through interface files. + - The cgroup v2 device controller has no interface files, and is implemented on top of cgroup BPF. + - Cilium attaches BPF cgroup programs for socket-based load balancing. Its + [default cgroup root](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#validate-bpf-cgroup-programs-attachment) + is `/run/cilium/cgroupv2`. +2. [Pressure Stall Information (PSI)](/docs/reference/instrumentation/understand-psi-metrics/) + reports CPU, memory, and I/O contention at node, Pod, and container level. The `KubeletPSI` + feature graduated to stable in Kubernetes v1.36 and is always enabled. PSI requires cgroup v2, + Linux 4.20 or later, `CONFIG_PSI=y`, and a kernel not booted with `psi=0`. The kubelet exposes + the data through the Summary API and `/metrics/cadvisor`. +3. When migrating, update software that reads the cgroup filesystem directly. The + [migration guidance](/docs/concepts/architecture/cgroups/#migrating-cgroupv2) recommends + cAdvisor v0.43.0 or later and lists compatible Java, Node.js, and `automaxprocs` versions. + +### CPU weight conversion in newer OCI runtimes + +cgroup v1 uses `cpu.shares`, whereas cgroup v2 uses `cpu.weight`. Newer OCI runtimes use an improved +non-linear conversion that preserves the default priority and gives small CPU requests more usable +granularity. The change is implemented in the OCI runtime rather than Kubernetes: it is available +in runc v1.3.2 and crun v1.23. After upgrading a runtime, monitoring or policy tools that predict +exact `cpu.weight` values may need updates. Read +[New Conversion from cgroup v1 CPU Shares to v2 CPU Weight](/blog/2026/01/30/new-cgroup-v1-to-v2-cpu-conversion-formula/) +for the formula, examples, and compatibility considerations. + +### In-place resource updates + +[In-place Pod vertical scaling](/blog/2025/12/17/kubernetes-v1-35-release/#stable-in-place-update-of-pod-resources) +graduated to stable in Kubernetes v1.35. Kubernetes v1.36 then enabled +[in-place vertical scaling for Pod-level resources](/blog/2026/04/30/kubernetes-v1-36-inplace-pod-level-resources-beta/) +by default as a beta feature. The kubelet coordinates changes between the Pod-level and container +cgroups so that increases create headroom before container limits grow, while decreases constrain +containers before shrinking the Pod-level boundary. Accurate aggregate enforcement for this +v1.36 feature requires cgroup v2. + +## Adopting cgroup version 2 + +### Requirements + +Here's what you need to use cgroup v2 with Kubernetes. +First up, you need to be using a version of Kubernetes with support for v2 cgroup management; +that's been stable since Kubernetes v1.25 and all supported Kubernetes releases include this support. + +- OS distribution enabled cgroup v2 +- Linux kernel version is 5.8 or later (5.9 or later is recommended when using Memory QoS) +- Container runtime supports cgroup v2. For example: + - containerd v1.4 or later supports cgroup v2; use containerd v2.0 or later for automatic + cgroup-driver discovery + - CRI-O v1.20 or later +- The kubelet and the container runtime are configured to use the systemd cgroup driver + +The Kubernetes v1.35 release announced the end of the containerd 1.x support window. However, the +final Kubernetes v1.36 runtime documentation still describes a fallback for containerd 1.x, which +does not implement the `RuntimeConfig` CRI RPC. On those nodes the kubelet uses its configured +`cgroupDriver` value instead of automatic discovery. That fallback is scheduled for removal in +Kubernetes v1.38. For new or upgraded v1.36 nodes, prefer a currently supported containerd 2.x +release and review both the [Kubernetes runtime documentation](/docs/setup/production-environment/container-runtimes/#systemd-cgroup-driver) +and the [containerd compatibility matrix](https://github.com/containerd/containerd/blob/main/RELEASES.md#kubernetes-support). + +#### kernel updates around cgroup v2 + +When Kubernetes was first announced, in 2014, only v1 cgroup existed. +Version 2 cgroup management first appeared in Linux kernel 4.5, released in 2016. + +- In Linux 4.5, the cgroup v2 `io`, `memory`, and `pids` controllers were supported. +- Linux 4.15 added support for the cgroup v2 `cpu` controller. +- [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html) (PSI) support began with Linux 4.20. +- The Kubernetes project does not recommend using cgroup v2 with a Linux kernel older than 5.2 + due to lack of cgroup-level task freezer support. +- Kubernetes documents 5.8 as the minimum kernel version for cgroup v2; the root cgroup's + system-level `cpu.stat` file was added in Linux 5.8. +- The `memory.high` livelock fix used by Memory QoS is present in Linux 5.9 and later. +- `memory.peak` was added in Linux 5.19. + +### Use systemd as cgroup driver + +[Configure the kubelet's cgroup driver to match the container runtime cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/). + +The [Container runtimes](/docs/setup/production-environment/container-runtimes) +page explains that the `systemd` driver is recommended for kubeadm +based setups instead of the kubelet's default `cgroupfs` driver, +because kubeadm manages the kubelet as a systemd service. + +A minimal example of configuring the field explicitly: + +```yaml +apiVersion: kubelet.config.k8s.io/v1beta1 +kind: KubeletConfiguration +cgroupDriver: systemd +``` + +Automatic discovery of the runtime's cgroup driver through the CRI, tracked by +[KEP-4033](https://kep.k8s.io/4033), graduated to stable in Kubernetes v1.34. It requires a runtime +that implements the `RuntimeConfig` CRI RPC (containerd v2.0+ or CRI-O v1.28+). When available, +the kubelet uses the value reported by the runtime instead of its configured `cgroupDriver` value. + +### Tools and commands for troubleshooting + +Tools and commands that you should know about cgroups: + +- `stat -fc %T /sys/fs/cgroup/`: Check whether cgroup v2 is enabled; it returns `cgroup2fs`. +- `systemctl list-units 'kube*' --type=slice` or `--type=scope`: List Kubernetes-related units + that systemd currently has in memory. +- `bpftool cgroup list /sys/fs/cgroup/*`: List all programs attached to the cgroup CGROUP. +- `systemd-cgls /sys/fs/cgroup/*`: Recursively show control group contents. +- `systemd-cgtop`: Show top control groups by their resource usage. +- `tree -L 2 -d /sys/fs/cgroup/kubepods.slice`: Show Pods' related cgroups directories. + +How to check if a Pod CPU or memory limit is successfully applied to the cgroup file? + +- Kubernetes Pod spec: check limits `spec.containers[*].resources.limits.{cpu,memory}` and requests + `spec.containers[*].resources.requests.{cpu,memory}` +- CRI: `cpu_period`, `cpu_quota`, `cpu_shares` for CPU and `memory_limit_in_bytes` for memory limit +- OCI Spec: `memory.limit`, `cpu.shares`, `cpu.quota`, `cpu.period` +- Systemd Scope Unit: `CPUWeight`, `CPUQuotaPerSecUSec`, `CPUQuotaPeriodUSec`, `MemoryMax` +- Cgroupfs value: `/sys/fs/cgroup/../cpu.weight`, `/sys/fs/cgroup/../cpu.max`, `/sys/fs/cgroup/../memory.max` + +## Further reading + +- [Kubernetes 1.31: Moving cgroup v1 Support into Maintenance Mode](/blog/2024/08/14/kubernetes-1-31-moving-cgroup-v1-support-maintenance-mode/) +- [Kubernetes 1.35: Timbernetes](/blog/2025/12/17/kubernetes-v1-35-release/) +- [Kubernetes v1.36: Tiered Memory Protection with Memory QoS](/blog/2026/04/29/kubernetes-v1-36-memory-qos-tiered-protection/) +- [Kubernetes v1.36: PSI Metrics for Kubernetes Graduates to GA](/blog/2026/05/12/kubernetes-v1-36-psi-metrics-ga/) +- [Kubernetes v1.36: In-Place Vertical Scaling for Pod-Level Resources Graduates to Beta](/blog/2026/04/30/kubernetes-v1-36-inplace-pod-level-resources-beta/) +- [cgroup v2 in Kubernetes](/docs/concepts/architecture/cgroups/) +- [Kubernetes 1.27: Quality-of-Service for Memory Resources (alpha)](/blog/2023/05/05/qos-memory-resources/) +- [Kubernetes 1.25: cgroup v2 graduates to GA](/blog/2022/08/31/cgroupv2-ga-1-25/) +- KubeCon NA 2022 [cgroup v2: Before You Jump In](https://www.youtube.com/watch?v=WxZK-UXKvXk) by Tony Gosselin & Mike Tougeron, Adobe Systems +- KubeCon NA 2022 [Cgroupv2 Is Coming Soon To a Cluster Near You](https://www.youtube.com/watch?v=sgyFCp1CRhA) - David Porter, Google & Mrunal Patel, RedHat +- KubeCon EU 2020 [Kubernetes On cgroup v2](https://www.youtube.com/watch?v=u8h0e84HxcE&t=783s) by Giuseppe Scrivano, Red Hat. +- This blog only covers the basic requirements and configuration of Kubernetes components. + It will not include how to enable cgroup fs in OS distributions. + For migration, you can refer to [migrating cgroup v2](/docs/concepts/architecture/cgroups/#migrating-cgroupv2)