Rootless on Bottlerocker failed with failed to mount /run/user/1000/containerd-mount3852074643: operation not permitted

[AhmadMS1988]

Hi fellows in buildkit.
I know this might have open multiple times like here and here, but will try to bring it again with more details so you may be able to help more.
I am trying to run buildkit in rootless in EKS using bottlerocket, the infra information are below:

1. Arch: arm64
2. buildkit image: moby/buildkit:rootless
3. Bottlerocket OS 1.19.1 (aws-k8s-1.28)
4. k8s version: v1.28.5-eks-5e0fdde

The below pod definition is used:

apiVersion: v1
kind: Pod
metadata:
  name: buildkitd
  annotations:
    container.apparmor.security.beta.kubernetes.io/buildkitd: unconfined
spec:
  nodeSelector:
    workload: runners
  containers:
    - name: buildkitd
      image: moby/buildkit:rootless
      args:
        - --addr
        - tcp://0.0.0.0:1234
        - --oci-worker-no-process-sandbox
        - --debug
      securityContext:
        seccompProfile:
          type: Unconfined
        runAsUser: 1000
        runAsGroup: 1000
      volumeMounts:
        - mountPath: /home/user/.local/share/buildkit
          name: buildkitd
    - name: runner
      image: moby/buildkit:rootless
      command: [ "/bin/sh", "-c", "--" ]
      args: [ "while true; do sleep 30; done;" ]
      env:
        - name: BUILDKIT_HOST
          value: tcp://localhost:1234
  volumes:
    - name: buildkitd
      emptyDir: {}

Note the the runner is actually a custom image that we use in our CI, but replaced with the same buildkit container as it has buildctl to use, but buildkit container is the same.

When we run buildctl on the runner, we get the following error:

time="2024-02-19T12:34:34Z" level=warning msg="failed to compute blob by overlay differ (ok=false): failed to write compressed diff: mount callback failed on /run/user/1000/containerd-mount387897202: mount callback failed on /run/user/1000/containerd-mount1737412574: failed to record upperdir changes (close error: failed to close tar writer: context canceled): context canceled"
time="2024-02-19T12:34:34Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to mount /run/user/1000/containerd-mount3852074643: operation not permitted"

Bottlerocket is configured with:

    [settings.kernel.sysctl]
    "user.max_user_namespaces" = "63359"

Really appreciate your help in identifying where the missing peace to let this to work.
Thank you

[AkihiroSuda]

Does it work if you specify securityContext.privileged ?

[AkihiroSuda]

Does https://raw.githubusercontent.com/moby/buildkit/master/examples/kubernetes/job.rootless.yaml work?

[AhmadMS1988]

We do not want to run it in privileged mode.

[AkihiroSuda]

We do not want to run it in privileged mode.

Asking for a diagnosis purpose

[AhmadMS1988]

It worked actually, but still the purpose to run it without privileged.

[AhmadMS1988]

One question comes to my mind, as we by default use the oci worker, what is this containerd mount?

[AkihiroSuda]

One question comes to my mind, as we by default use the oci worker, what is this containerd mount?

OCI mode still consumes containerd as a library

[AhmadMS1988]

Is there any logs or commands that I can execute to help investigating more?

[AkihiroSuda]

Is there any logs or commands that I can execute to help investigating more?

cat /proc/mounts in the buildkitd container, and compare the result with Ubuntu nodes, etc.

[AhmadMS1988]

It worked as expected on both Amazon linux 2 and Ubuntu EKS optimized images based on 20.04.
The output of /proc/mounts is:

overlay / overlay rw,context="system_u:object_r:data_t:s0:c208,c287",relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/71/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/59/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/55/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/50/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/45/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/40/fs,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/425/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/425/work 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev tmpfs rw,context="system_u:object_r:data_t:s0:c208,c287",nosuid,size=65536k,mode=755 0 0
devpts /dev/pts devpts rw,context="system_u:object_r:data_t:s0:c208,c287",nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs ro,seclabel,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup cgroup2 ro,seclabel,nosuid,nodev,noexec,relatime 0 0
/dev/nvme1n1p1 /etc/hosts xfs rw,seclabel,nosuid,nodev,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
/dev/nvme1n1p1 /dev/termination-log xfs rw,seclabel,nosuid,nodev,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
/dev/nvme1n1p1 /etc/hostname xfs rw,seclabel,nosuid,nodev,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
/dev/nvme1n1p1 /etc/resolv.conf xfs rw,seclabel,nosuid,nodev,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
shm /dev/shm tmpfs rw,seclabel,nosuid,nodev,noexec,relatime,size=65536k 0 0
/dev/nvme1n1p1 /home/user/.local/share/buildkit xfs rw,seclabel,nosuid,nodev,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
tmpfs /run/secrets/kubernetes.io/serviceaccount tmpfs ro,seclabel,relatime,size=6931992k 0 0
proc /proc/bus proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/fs proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/irq proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sysrq-trigger proc ro,nosuid,nodev,noexec,relatime 0 0
tmpfs /proc/acpi tmpfs ro,context="system_u:object_r:data_t:s0:c208,c287",relatime 0 0
tmpfs /proc/kcore tmpfs rw,context="system_u:object_r:data_t:s0:c208,c287",nosuid,size=65536k,mode=755 0 0
tmpfs /proc/keys tmpfs rw,context="system_u:object_r:data_t:s0:c208,c287",nosuid,size=65536k,mode=755 0 0
tmpfs /proc/latency_stats tmpfs rw,context="system_u:object_r:data_t:s0:c208,c287",nosuid,size=65536k,mode=755 0 0
tmpfs /proc/timer_list tmpfs rw,context="system_u:object_r:data_t:s0:c208,c287",nosuid,size=65536k,mode=755 0 0
tmpfs /proc/scsi tmpfs ro,context="system_u:object_r:data_t:s0:c208,c287",relatime 0 0
tmpfs /sys/firmware tmpfs ro,context="system_u:object_r:data_t:s0:c208,c287",relatime 0 0

Can you please take a look and provide feedback so I can open a ticket to Bottlerocket team with the details?
Thanks

[AkihiroSuda]

Seems relevant to SELinux? Does this work?

securityContext:
  seLinuxOptions:
    level: s0
    type: spc_t

[AhmadMS1988]

Unfortunately, it did not work.
I got the same error.