Running multiple rootless containers in parallel with toolbox-like UID mapping results in "conmon.pid: permission denied"

[jistr]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running Fedora Silverblue 30 with Podman 1.3.1, i can't run multiple containers with toolbox-like UID mapping in parallel. First i tought this is #2935 or #2673 but i no longer think so. I ran podman system migrate, and eventually i ran also rm -rf .local/share/containers, so the containers (and images too) were created on the most recent Podman, but none of that helped. This worked fine before -- i'm not sure which version of Podman i was using previously but i think it was 1.2.

FYI @debarshiray, i guess this must be affecting toolbox on Silverblue 30.

Steps to reproduce the issue:

Here's a minimal reproducer script test.sh:

#!/bin/bash

UID_MAX=65536
UID_HIGHER_START=$(($UID+1))
UID_HIGHER_COUNT=$((UID_MAX-$UID))

function spawn_container() {
    podman run \
           --rm \
           --uidmap $UID:0:1 \
           --uidmap 0:1:$UID \
           --uidmap $UID_HIGHER_START:$UID_HIGHER_START:$UID_HIGHER_COUNT \
           --gidmap $UID:0:1 \
           --gidmap 0:1:$UID \
           --gidmap $UID_HIGHER_START:$UID_HIGHER_START:$UID_HIGHER_COUNT \
           registry.fedoraproject.org/fedora:30 \
           bash -c 'echo "Hi from container."; sleep 10' &
}

# Should print "Hi from container." twice.
spawn_container
sleep 5
spawn_container

Describe the results you received:

Run ./test.sh for the first time, it prints "Hi from container." just once. On a subsequent run it prints "Hi from container." just once, but it's then also followed by error conmon.pid: permission denied. Full output and full error message below:

$ ./test.sh 
Trying to pull registry.fedoraproject.org/fedora:30...Getting image source signatures
Copying blob 149da9c683f1 [==>-----------------------------------] 7.8MiB / 93.2MiB

Copying blob 149da9c683f1 [==========>---------------------------] 27.4MiB / 93.2MiB
Copying blob 149da9c683f1 [=============>------------------------] 34.9MiB / 93.2MiB
Copying blob 149da9c683f1 [======================>---------------] 55.5MiB / 93.2MiB
Copying blob 149da9c683f1 [======================>---------------] 57.0MiB / 93.2MiB
Writing manifest to image destination
Copying blob 149da9c683f1 [==============================>-------] 77.2MiB / 93.2MiB
Copying blob 149da9c683f1 done
Copying config edc5d31c98 done
Writing manifest to image destination
Storing signatures
Hi from container.


$ ./test.sh 
Hi from container.
$ ERRO[0000] open /var/home/jistr/.local/share/containers/storage/overlay-containers/8747fdf01fe9686eadc4e03d57f5946c5904bc67bf0c63906d9df373550e61a7/userdata/conmon.pid: permission denied 

Describe the results you expected:

The test.sh script should print "Hi from container." twice.

Output of podman version:

Version:            1.3.1
RemoteAPI Version:  1
Go Version:         go1.12.2
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.2
  podman version: 1.3.1
host:
  BuildahVersion: 1.8.2
  Conmon:
    package: podman-1.3.1-1.git7210727.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: c9a4c48d1bff85033b7fc9b62d25961dd5048689'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 21526147072
  MemTotal: 25196261376
  OCIRuntime:
    package: runc-1.0.0-92.dev.gitc1b8c57.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc7+dev
      commit: d38f8a2d00ff444e52d16ab1ea5bbe5c1cb471e8
      spec: 1.0.1-dev
  SwapFree: 8589930496
  SwapTotal: 8589930496
  arch: amd64
  cpus: 4
  hostname: split
  kernel: 5.0.16-300.fc30.x86_64
  os: linux
  rootless: true
  uptime: 42m 10.06s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/jistr/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/jistr/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /tmp/1000
  VolumePath: /var/home/jistr/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical host, Fedora Silverblue 30.

[mheon]

So you ran system migrate and you're still seeing a permission denied on the Conmon pidfile?

@giuseppe PTAL

[mheon]

(That error is actually exactly what system migrate was supposed to solve, so I'm very curious here)

[jistr]

@mheon One more thing to highlight which might be easy to miss in the logs. The error message in #2935 was with the old conmon.pid location in /run/user/1000, but the error message here is with the new location in ~/.local/share/containers. As i wrote, i eventually completely dropped the data directory, so this issue is not about migration of content from older Podman. In my later attempts, all the content was created from scratch on Podman 1.3.1, still producing the same error.

[giuseppe]

we managed to break it again when we dropped the intermediate namespace.

I am going to work on a fix for toolbox. This should not happen anymore in future as we have a pause process, but let's have a workaround for now to not have another breaking change for existing running containers

[giuseppe]

@jistr could try out the PR I've just opened?

[jistr]

@giuseppe Thanks for such a quick fix. I will do my best to try when i get home (i'm not hitting this bug at work). I'm not a Go dev (go Rust! :D ) and my dev workflow on my home machine is a bit troublesome right now because of this very bug, so i can't promise results with certainty, but i'll try :).

[floating-cat]

I reported a similar issue in Toolbox1, someone pointed this issue there (this issue may related to my original issue).
For that issue, I ran system migrate many times, I still can see permission denied if I try to enter Toolbox.
And in the new Toolbox version, if I enter Toolbox, I would see this outpout:

~> podman system migrate
container_linux.go:388: signaling init process caused "operation not permitted"
ERRO[0000] Error removing store for partially-created runtime: A layer is mounted: layer is in use by a container
Error: error migrating containers: cannot stop container a33655212b94c4b5eb9b59feec317f89dbc9771d2e0a3c51b573cccce4b1f68d: operation not permitted

[debarshiray]

Thanks for sorting this out, @jistr and @giuseppe !

[giuseppe]

@debarshiray from the next 1.4 release rootless containers are out of tech preview and there won't be more breaking changes.
@debarshiray could you help us to figure out the toolbox use cases so that we have proper tests for it?

I am going to work on something like --userns=keep-uid ASAP, so that we won't need the magic shell code for creating the nested user namespace

[jistr]

I tested latest-at-the-time commit 6bca80bdfebf9d390ce1a9786ae6101c87f1322d and it does fix the issue for me. Thanks @giuseppe! Let me know if you want me to retest again with later/final version.

Also +1 on the idea of --userns=keep-uid :).