Expose rootless containerd socket directories for external access

[hinshun]

Proposed Changes

When installing k3s as a rootless systemd user service, it'll have better usability if regular tools like nerdctl, ctr, and crictl can interact with the container runtime embedded in k3s. Due to the rootlesskit setup being embedded in the k3s binary, it's impossible to use k3s server --rootless with an external --container-runtime-endpoint even with nsenter because the embedded kubelet does a healthcheck:

1. kubelet (in k3s) fail to startup due to health check of containerd-runtime-endpoint
2. containerd (outside of k3s) fail to startup due to rootlesskit not yet self-exec'd (i.e. child_pid not yet available).

Thus, in rootless mode it's better to just leverage the embedded containerd.

Types of Changes

• Mount k3s rootless containerd & cri-dockerd socket directories to $XDG_RUNTIME_DIR/k3s/containerd and $XDG_RUNTIME_DIR/k3s/cri-dockerd respectively.

Verification

• Run k3s server --rootless
• Run ctr -a "$XDG_RUNTIME_DIR/k3s/containerd/containerd.sock" -n k8s.io image ls

Linked Issues

• Cannot use --container-runtime-endpoint with rootless k3s #9448

User-Facing Change

Mount k3s rootless containerd & cri-dockerd socket directories to `$XDG_RUNTIME_DIR/k3s/containerd` and `$XDG_RUNTIME_DIR/k3s/cri-dockerd` respectively.

[brandond]

Is this something that makes sense to do? Using rootless k3s with a rootful container runtime? The entire point of rootless k3s is that the container runtime is also rootless; if you're pointing k3s at a container runtime that is running as root outside the isolated namespace, you're missing most of the advantages of running rootless. I also think that the things will be very confused about the kubelet and container runtime being in different namespaces. Are you sure this works?

[codecov]

Codecov Report

Attention: 14 lines in your changes are missing coverage. Please review.

Comparison is base (6d77b7a) 40.56% compared to head (5e01c60) 40.49%.
Report is 26 commits behind head on master.

Files	Patch %	Lines
pkg/rootless/mounts.go	0.00%	14 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9309      +/-   ##
==========================================
- Coverage   40.56%   40.49%   -0.08%     
==========================================
  Files         154      154              
  Lines       16555    16569      +14     
==========================================
- Hits         6716     6710       -6     
- Misses       8692     8711      +19     
- Partials     1147     1148       +1     

Flag	Coverage Δ
inttests	37.66% <0.00%> (-0.07%)	⬇️
unittests	14.51% <0.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hinshun]

I’m still talking about a rootless container runtime. So I have systemd user service for rootless containerd, and I just wanted to re-use that and point k3s to use that. From a UX point of view, it’ll be nice to have parity with the rootful workflow, where you can do things like nerdctl image ls, and also use the same runtime for kubectl apply.

As you point out, it can’t work in separate namespaces, and a rootless containerd cannot be managed in a separate systemd unit even with nsenter because both are waiting for each other to be healthy.

So the conclusion I’ve arrived at is that with rootless k3s, you must use the embedded containerd, and we can still use off-the-shelf nerdctl if we simply expose the containerd.sock.

[brandond]

with rootless k3s, you must use the embedded containerd

Right, which is why I'm confused why you still want to expose the cri-dockerd socket?

[hinshun]

@brandond I'm fine with removing the cri-dockerd socket. I've updated the commit with only exposing the containerd.sock.

[hinshun]

@brandond I notice we are stalled on #9448 but can we still backport this PR #9309?