Rootless K3s fails with "operation not permitted" error during mount operations

[nishantmunjal7]

When running K3s in rootless mode, I'm encountering an "operation not permitted" error during mount operations. This appears to be related to the handling of mount flags in user namespaces.

Environmental Info:
K3s Version:
k3s version v1.30.4+k3s1 (98262b5)
go version go1.22.5

Node(s) CPU architecture, OS, and Version:
Linux ip-** 5.14.0-427.31.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 9 14:06:03 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug:
failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting ".rancher/k3s/agent/kubelet/pods//volumes/kubernetes.io ~ configmap/config-volume" to rootfs at "/etc/coredns": mount .rancher/k3s/agent/kubelet/pods//volumes/kubernetes.io~configmap/config-volume:/etc/coredns (via /proc/self/fd/6), flags: 0x5021: operation not permitted: unknown" containerID=""

Steps To Reproduce:

• 1. Set up K3s in rootless mode
• 2. Create a test directory and mount it with specific options

mkdir -p ~/test_dir
sudo mount -o bind,rw,nosuid,nodev ~/somedir /home/user/.rancher

• 3. Attempt to start K3s Rootless service from user folder.

Expected behavior:
All K3s pods should start and operate normally in rootless mode.

Actual behavior:
All pods are in CrashLoopBackOff State

Additional context / logs:
I noticed a similar issue for this in Moby repo, do we need to implement something similar with K3s containerd version

[brandond]

sudo mount -o bind,rw,nosuid,nodev ~/somedir /home/user/.rancher

Why are you bind-mounting the rootless dir with weird options? Why not just set --data-dir=$HOME/somedir instead?

[brandond]

We're currently shipping v1.1.14 of runc, not v.1.12 - but that looks promising otherwise.

I've reached out to @cyphar to ask about getting that fix backported to v1.1.

[nishantmunjal7]

Thanks @brandond.
I'll hold off for now and will wait for Cyphar's response

[cyphar]

That patch is the wrong patch to backport to properly fix the issue, it was later reverted. The correct patch is opencontainers/runc#3967 but it is a bit too large to backport and it had to change some behaviour of mount options in a way that wouldn't be reasonable to backport to a patch release (see the 1.2.0-rc.1 release notes).

We plan to do a 1.2.0 release fairly soon -- if you can verify that 1.2.0-rc.3 works for you that would be very useful.

If it's really necessary, I can try to do a minimal backport of that PR that just adds the fallback code for locked mount options, without fixing the other issues with mount options (which cause the behaviour change). But that code is a little hairy and I'm a bit worried about regressions because the patch would be more of a rewrite than a backport. Also, one of the problems that opencontainers/runc#3967 fixed is that runc would ignore flags that users specified in some circumstances (this is the part of the patch that can't be backported as it could introduce a compatiblity issue) -- if you backport the patch without that part it would make the problem worse (which is why opencontainers/runc#3967 contains a fix for both issues).

[nishantmunjal7]

Thanks, @cyphar, for the detailed explanation.

I have verified that 1.2.0-rc.3 works with k3s, and it resolves the issue as expected. In the meantime, I can fork k3s and make it compatible with 1.2.0-rc.3 until the final 1.2.0 release is available.

Appreciate your offer for a minimal backport, but I understand that it could introduce potential regressions. For now, I'll work with a fork for k3s.

[geraldwuhoo]

Thanks all for the work on this one. I believe this is blocking my attempt at rootless podman k3d as well, since I am seeing the same error signature:

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/91f67bdd-aa11-46a4-a00f-ebe6d4744f72/volumes/kubernetes.io~configmap/config-volume" to rootfs at "/etc/coredns": mount /var/lib/kubelet/pods/91f67bdd-aa11-46a4-a00f-ebe6d4744f72/volumes/kubernetes.io~configmap/config-volume:/etc/coredns (via /proc/self/fd/6), flags: 0x5021: operation not permitted: unknown

I notice that runc 1.2.1 is coming with the next release, very exciting. I eagerly await the release.