rawhide (crun): rootless(?): podman exec broken: OCI runtime error

[edsantiago]

On current rawhide with crun:

$ podman run -d --name foo alpine sleep 60
8c0855798e92d718be496e428c66a4e702ec4921575f006a5848fe53e41d50b8
$ podman exec foo true
Error: non zero exit code: 1: OCI runtime error
$ echo $?
1

I can't check root because podman-root is 100% broken (rhbz1748214).

With --log-level=debug:

...
DEBU[0000] Successfully started exec session 3985743e00dad6f2710b08790fb2ea75f96572b520aa2c16dc296cfb927e2856 in container c5875947634e0bf1f5804b19f361e7123b253f523cf43e0494ad04450d508419
[conmon:d]: exec with attach is waiting for start message from parent
[conmon:d]: exec with attach got start message from parent
ERRO[0000] non zero exit code: 1: OCI runtime error

podman-1.5.2-0.46.dev.git1d8a940.fc32.x86_64
crun-0.7-2.fc31.x86_64
package runc is not installed

[giuseppe]

it should work well with crun 0.8. The new version has different fixes for interacting with Podman and reporting the correct exit code.

[rhatdan]

@edsantiago can you test it out, reopen if this does not fix the issue.

[edsantiago]

Problem persists with crun-0.8-1.fc32.x86_64 although the error is different:

$ podman exec foo true
Error: writing file '/sys/fs/cgroup//user.slice/user-1000.slice/[email protected]/user.slice/libpod-61c0251e87cd6312ce7e6176f2ad66f3260b675a079efa5d71ad79ed22e06671.scope/cgroup.procs': Permission denied: OCI runtime permission denied error

[rhatdan]

Progress...

[giuseppe]

it seems to fail moving a process from cgroup A to a cgroup B, where A is the current cgroup (owned by root) and B is the destination cgroup in the container.

Can you confirm the current cgroup is owned by root with `ls -l /sys/fs/cgroup/$(sed -e 's|0::||' /proc/self/cgroup)?

Does it work if you use something like systemd-run --user --scope podman exec foo echo hello ?

Not sure what would be the best way to handle it, we'd need to enforce somehow to run in a cgroup that is owned by the rootless user.

[giuseppe]

opened a PR to automatically create the scope: #3959

[edsantiago]

Sorry for the late response. In case it's still helpful:

Can you confirm the current cgroup is owned by root with `ls -l /sys/fs/cgroup/$(sed -e 's|0::||' /proc/self/cgroup)?

$ (I took the liberty of adding `-d` to the command above)
drwxr-xr-x. 2 root root 0 Sep  9 15:15 /sys/fs/cgroup//user.slice/user-0.slice/session-18.scope
(ls -l, no -d, shows everything owned by root)

Does it work if you use something like systemd-run --user --scope podman exec foo echo hello ?

$ systemd-run --user --scope podman exec foo echo hello
Failed to create bus connection: No such file or directory
$ XDG_RUNTIME_DIR=/run/user/1000 !!
Running scope as unit: run-r51595d4cfd9a41db8c2d73c407893090.scope
hello

[juhp]

I hit this too now while trying to test toolbox in Rawhide.

[popsUlfr]

I think I encountered this problem on podman 1.9.0 in rootless mode :

When I try to exec :

$ podman --log-level debug exec -ti d3bfcb240aed bash

DEBU[0000] Attaching to container d3bfcb240aed2e55863dbff58af373b80a284777dbc93eadab336a58a120f7a6 exec session 53430ac13aad60bfe302e5b1b6d583ed5c1276538cdbd605ba9f4c2d3dbac03d 
DEBU[0000] connecting to socket /run/user/974/libpod/tmp/socket/53430ac13aad60bfe302e5b1b6d583ed5c1276538cdbd605ba9f4c2d3dbac03d/attach 
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

                                                                          DEBU[0000] Received: 0                                  
DEBU[0000] Received: -1                                 
ERRO[0000] [conmon:d]: exec with attach is waiting for start message from parent
[conmon:d]: exec with attach got start message from parent
writing file `/sys/fs/cgroup//user.slice/user-974.slice/[email protected]/user.slice/libpod-d3bfcb240aed2e55863dbff58af373b80a284777dbc93eadab336a58a120f7a6.scope/cgroup.procs`: Permission denied: OCI runtime permission denied error

But if I prefix with systemd-run :

$ systemd-run --user --scope podman exec -ti d3bfcb240aed bash

It works.

Not sure if it is related but I have a dedicated podmanuser user on the system, so I SSH to my ssh user and then I sudo machinectl shell podmanuser@ to become the other user and get a systemd user session.

Here's my podman info :

host:
  arch: arm64
  buildahVersion: 1.14.8
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.15, commit: 1bddbf7051a973f4a4fecf06faa0c48e82f1e9e1'
  cpus: 4
  distribution:
    distribution: archarm
    version: unknown
  eventLogger: journald
  hostname: rock64
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 974
      size: 1
    - container_id: 1
      host_id: 200000
      size: 1001
    uidmap:
    - container_id: 0
      host_id: 974
      size: 1
    - container_id: 1
      host_id: 200000
      size: 1001
  kernel: 5.6.7-1-ARCH
  memFree: 3494100992
  memTotal: 4097687552
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.3.0
  swapFree: 0
  swapTotal: 0
  uptime: 58m 41.45s
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /var/lib/podmanuser/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /var/lib/podmanuser/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/974/containers
  volumePath: /var/lib/podmanuser/.local/share/containers/storage/volumes