Do not set memory limits for Cgroup v2 "memory.swap.max: permission denied"

[costiser]

Steps to reproduce the issue:

1. clean install of docker and clean install minikube
2. docker group created and user added to it (as instructed here: https://docs.docker.com/engine/install/linux-postinstall/)
   3.minikube start --driver=docker fails

Full output of failed command:

Full output of minikube start command used, if not already included:

Command: minikube start --driver=docker:

😄  minikube v1.17.0 on Debian rodete
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🎉  minikube 1.17.1 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.17.1
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

🔥  Creating docker container (CPUs=2, Memory=16000MB) ...
🤦  StartHost failed, but will try again: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=16000mb --memory-swap=16000mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
abaa833909b92f1c668485dc022be8caa9c31d1eadd3da326fefea210ee311b0

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-abaa833909b92f1c668485dc022be8caa9c31d1eadd3da326fefea210ee311b0.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-abaa833909b92f1c668485dc022be8caa9c31d1eadd3da326fefea210ee311b0.scope/memory.swap.max: permission denied: unknown.

🤷  docker "minikube" container is missing, will recreate.
🔥  Creating docker container (CPUs=2, Memory=16000MB) ...
😿  Failed to start docker container. Running "minikube delete" may fix it: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=16000mb --memory-swap=16000mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
8755e8ad1f08a1700aa2a1cc25defe8bf3ccc56647403c86537a2cecfb5cd562

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-8755e8ad1f08a1700aa2a1cc25defe8bf3ccc56647403c86537a2cecfb5cd562.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-8755e8ad1f08a1700aa2a1cc25defe8bf3ccc56647403c86537a2cecfb5cd562.scope/memory.swap.max: permission denied: unknown.


❌  Exiting due to GUEST_PROVISION: Failed to start host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=16000mb --memory-swap=16000mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
8755e8ad1f08a1700aa2a1cc25defe8bf3ccc56647403c86537a2cecfb5cd562

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-8755e8ad1f08a1700aa2a1cc25defe8bf3ccc56647403c86537a2cecfb5cd562.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-8755e8ad1f08a1700aa2a1cc25defe8bf3ccc56647403c86537a2cecfb5cd562.scope/memory.swap.max: permission denied: unknown.


😿  If the above advice does not help, please let us know:
👉  https://github.com/kubernetes/minikube/issues/new/choose

Optional: Full output of minikube logs command:

🤷 The control plane node "" does not exist. 👉 To start a cluster, run: "minikube start"

[afbjorklund]

I think you need to configure those in the Kernel settings, when running on Debian ?

https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities

Currently we only check for "memsw.limit_in_bytes", but not for "memory.swap.max"

There should be a similar check to the one for cgroups v1, also for the new cgroups v2.

[costiser]

1. Has this changed recently? I'm trying to figure out what changed recently, because minikube was working fine on the same machine before upgrading minikube & docker...

2. do you plan to add checks also for memory.swap.max (similar to what you mentioned "memsw.limit_in_bytes")?

[awan1]

I ran into this problem today too, also after a restart, before which everything was working fine. Making the GRUB changes listed in the docs linked by @afbjorklund made it work. Thank you!

Edit: now that I'm thinking about it, I reinstalled Minikube because I thought maybe I needed to update to a new version, so it makes sense that I needed to follow the post-install instructions.

Edit 2: maybe part of the trouble here is that the error message did not contain the message indicated in the docs, "Your kernel does not support cgroup swap limit capabilities". Maybe the error message could be updated to be closer to that, or the docs could include a sample of the error message we encountered.

[afbjorklund]

1. Has this changed recently? I'm trying to figure out what changed recently, because minikube was working fine on the same machine before upgrading minikube & docker...

Older versions of Docker (before 20.10) did not support cgroups v2.

The kernel setting is the same, though. It's something specific to Debian, others have it on by default...

2. do you plan to add checks also for memory.swap.max (similar to what you mentioned "memsw.limit_in_bytes")?

Yes.

In this section: https://github.com/kubernetes/minikube/blob/v1.17.1/pkg/drivers/kic/oci/oci.go#L157_L161

[costiser]

I confirm it's working now with the changes suggested on https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities

[medyagh]

I confirm this is a problem that affects users with cgroup2

and one thing we could do on minikube side is NOT applying memory limits for cgroup v2 until it is fixed and also point them to the docs on Docker website to fix their cgroup

docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
 
Server:
 Containers: 16
  Running: 0
  Paused: 0
  Stopped: 16
 Images: 21
 Server Version: 20.10.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.7.17-1rodete5-amd64
 Operating System: Debian GNU/Linux rodete
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 125.5GiB
 Name: drewpca.sbo.corp.google.com
 ID: F4RJ:PGBV:QUDU:5P77:Y3AV:YEKE:RWX3:T6QM:MPCP:WREL:7FKF:WSY5
 Docker Root Dir: /usr/local/google/docker
 Debug Mode: true
  File Descriptors: 24
  Goroutines: 36
  System Time: 2021-02-12T13:40:50.695925257-08:00
  EventsListeners: 0
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://mirror.gcr.io/
 Live Restore Enabled: false
 
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support
WARNING: Support for cgroup v2 is experimental

[afbjorklund]

Basically they haven't configured their Docker installation "properly"

(alternatively you could say that their Linux vendor chose to disable some of the kernel capabilities for them, by default)

[afbjorklund]

Seems like it is now an error with Docker, like it was in Podman, so we should just avoid the flag. It was ignored, anyway...

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.

[medyagh]

this PR ://github.com//pull/10507 didnt fully fix

we need to check for cat /sys/fs/cgroup/memory/memory.limit_in_bytes
to not add memory limits ( I will add a PR for that)

[afbjorklund]

Pretty sure I misunderstood the settings, as well. Not specifiying swap means that you want swap...

By default, the container can swap the same amount of assigned memory, which means that the overall hard limit would be around 256m when you set --memory 128m.

https://thorsten-hans.com/limit-memory-for-docker-containers

As far as I know, memory.limit_in_bytes is available by default and it was memswap that was toggled ?

But I don't have a system to reproduce this, worked OK on Debian Sid.

[afbjorklund]

The current checks were explicitly only to check for the boot parameter of Debian/Ubuntu:

https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities

        // memory subsystem checks and adjustments
        if resources.Memory != 0 && resources.Memory < linuxMinMemory {
                return warnings, fmt.Errorf("Minimum memory limit allowed is 6MB")
        }
        if resources.Memory > 0 && !sysInfo.MemoryLimit {
                warnings = append(warnings, "Your kernel does not support memory limit capabilities or the cgroup is not mounted. Limitation discarded.")
                resources.Memory = 0
                resources.MemorySwap = -1
        }
        if resources.Memory > 0 && resources.MemorySwap != -1 && !sysInfo.SwapLimit {
                warnings = append(warnings, "Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.")
                resources.MemorySwap = -1
        }
        if resources.Memory > 0 && resources.MemorySwap > 0 && resources.MemorySwap < resources.Memory {
                return warnings, fmt.Errorf("Minimum memoryswap limit should be larger than memory limit, see usage")
        }
        if resources.Memory == 0 && resources.MemorySwap > 0 && !update {
                return warnings, fmt.Errorf("You should always set the Memory limit when using Memoryswap limit, see usage")
        }

cgroups v1

        mountPoint, ok := cgMounts["memory"]
        if !ok {
                warnings = append(warnings, "Your kernel does not support cgroup memory limit")
                return warnings
        }
        info.MemoryLimit = ok

        info.SwapLimit = cgroupEnabled(mountPoint, "memory.memsw.limit_in_bytes")
        if !info.SwapLimit {
                warnings = append(warnings, "Your kernel does not support swap memory limit")
        }

cgroups v2

       if _, ok := controllers["memory"]; !ok {
                warnings = append(warnings, "Unable to find memory controller")
                return warnings
        }

        info.MemoryLimit = true
        info.SwapLimit = true

Not check for some weird system with /sys/fs/cgroup/memory but without /sys/fs/cgroup/memory/memory.limit_in_bytes

[afbjorklund]

Noticed a funky detail on podman, as well...

$ sudo podman run --memory 256m --memory-swap -1 busybox true
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x15ee61e]

goroutine 1 [running]:
github.com/containers/podman/v2/cmd/podman/common.getMemoryLimits(0xc00035cc00, 0x27d8da0, 0x1b05550, 0x17, 0xc000594868)
	cmd/podman/common/specgen.go:153 +0x33e
github.com/containers/podman/v2/cmd/podman/common.FillOutSpecGen(0xc00035cc00, 0x27d8da0, 0xc0002b9f20, 0x2, 0x6, 0x0, 0x18ab660)
	cmd/podman/common/specgen.go:430 +0xda6
github.com/containers/podman/v2/cmd/podman/containers.run(0x2748b80, 0xc0002b9f20, 0x2, 0x6, 0x0, 0x0)
	cmd/podman/containers/run.go:171 +0x46f
github.com/spf13/cobra.(*Command).execute(0x2748b80, 0xc000138020, 0x6, 0x6, 0x2748b80, 0xc000138020)
	vendor/github.com/spf13/cobra/command.go:850 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0x275c140, 0xc000130010, 0x18d6c80, 0x280b340)
	vendor/github.com/spf13/cobra/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	vendor/github.com/spf13/cobra/command.go:895
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	vendor/github.com/spf13/cobra/command.go:888
main.Execute()
	cmd/podman/root.go:92 +0xec
main.main()
	cmd/podman/main.go:36 +0x92

💥

Reported as containers/podman#9429 (fixed on master, but not 3.0.x)

[afbjorklund]

this PR #10507 didnt fully fix

@medya : it seems like you didn't merge #10468 ? so there was no check in place, for cgroups v2 ?

[awan1]

I'm running into this issue again after my system OS updated, but I'm not sure what exactly was updated.

/etc/os-release says

NAME="Debian GNU/Linux"
ID=debian
VERSION_CODENAME=rodete
VERSION_ID=rodete

System Settings says

Cinnamon Version 4.8.6
Linux Kernel 5.7.17-1rodete5-amd64

Attempted: minikube delete followed by minikube start

Output:

ᐅ minikube start
😄  minikube v1.17.1 on Debian rodete
✨  Automatically selected the docker driver
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🤦  StartHost failed, but will try again: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=3900mb --memory-swap=3900mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 126
stdout:
d136aef3e96600c628f11fbe214ac5d579acb3b8601abe336a1c35b8ffbd2914

stderr:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "0" to "/sys/fs/cgroup/system.slice/docker-d136aef3e96600c628f11fbe214ac5d579acb3b8601abe336a1c35b8ffbd2914.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-d136aef3e96600c628f11fbe214ac5d579acb3b8601abe336a1c35b8ffbd2914.scope/memory.swap.max: permission denied: unknown.

Followed the post-install steps, which solved this problem for me before: https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities. Confirmed that /etc/default/grub contained GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1", ran sudo update-grub, rebooted machine. Did this twice, with same minikube start error message.

Happy to open new bug and/or provide new information, but this seemed like the same error message so I thought I'd reuse this bug.

[afbjorklund]

I think the latest Google Linux updates to cgroups v2, and there was a regression in how minikube handles that.

[sharifelgamal]

@awan1 I noticed you're using minikube 1.17.1, which is a full version old now. Can you try upgrading to v1.18.1 and see if that helps? It's working on my rodete machine.

[costiser]

Have you tried the fix indicated above?
https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities

[spowelljr]

Also working on my machine as well on v1.18.1

Linux version 5.7.17-1rodete5-amd64

[awan1]

@sharifelgamal great catch - updating to minikube v1.18.1 made minikube start work without doing anything else. Thank you! I'll add running minikube update-check to my debugging process.