[SOLVED] Inside a rootless container, using machinectl to run another (PINP) rootless container fails with 'newuidmap: write to uid_map failed'

[mohaa7]

It's failed to execute this command insde another container:

first: podman -d --name primary-container \
	--cap-add=SYS_ADMIN \
        ... \
	custom-rocky8
Then: machinectl -q shell --uid=<username> .host /usr/bin/env podman run hello

ERRO[0000] running `/usr/bin/newuidmap 5229 0 201 1 1 100000 65536`: newuidmap: write to uid_map failed: Operation not permitted 
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

ls -l /usr/bin/newuidmap /usr/bin/newgidmap:

-rwsr-xr-x. 1 root root 48944 Nov 21  2023 /usr/bin/newgidmap
-rwsr-xr-x. 1 root root 48904 Nov 21  2023 /usr/bin/newuidmap

sysctl kernel.unprivileged_userns_clone:

sysctl: cannot stat /proc/sys/kernel/unprivileged_userns_clone: No such file or directory

[rhatdan]

First you should not need --cap-add=SYS_ADMIN

sysctl: cannot stat /proc/sys/kernel/unprivileged_userns_clone: No such file or directory

This looks like something in the kernel is not compiled correctly.

[mohaa7]

I also tried with FROM registry.fedoraproject.org/fedora:latest and:

sh-5.2# cat /etc/redhat-release 
Fedora release 41 (Forty One)
sh-5.2# sysctl kernel.unprivileged_userns_clone
sysctl: cannot stat /proc/sys/kernel/unprivileged_userns_clone: No such file or directory

Then you mean an issue in the kernel of my host, right?

By removing --cap-add=SYS_ADMIN switch, the second container can not mount its required volume:

time="2025-01-13T15:07:37Z" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"

[rhatdan]

Does podman work on the rocky8 system at all, when you log in as a non-root user?

time="2025-01-13T15:07:37Z" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"

Indicates that newuidmap is not seeing entries for the podman user.

Does the following commands work, and get you closer?

# machinectl -q shell --uid=gpostgres .host /usr/bin/env /bin/sh
$ podman info
$ podman run alpine echo hi

[mohaa7]

In case the range "100000:65536" is defined in both /etc/subuid and /etc/subgid, I’m still encountering issues with newuidmap. The problem isn’t limited to the machinectl command.

# /usr/bin/machinectl -q shell --uid=user .host /usr/bin/env bash

$ podman info
ERRO[0000] running `/usr/bin/newuidmap 85 0 201 1 1 100000 65536`: newuidmap: write to uid_map failed: Operation not permitted 
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

$ podman run hello
ERRO[0000] running `/usr/bin/newuidmap 95 0 201 1 1 100000 65536`: newuidmap: write to uid_map failed: Operation not permitted 
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

$ podman run alpine echo hi
ERRO[0000] running `/usr/bin/newuidmap 615 0 201 1 1 100000 65536`: newuidmap: write to uid_map failed: Operation not permitted 
ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot set up namespace using "/usr/bin/newuidmap": exit status 1 

$ exit

But, If I echo "" into /etc/subuif and /etc/subgid, the podman info retures information, but I can't run conatainers as below:

# > /etc/subuid; > /etc/subgid

# cat /etc/subuid /etc/subgid

# /usr/bin/machinectl -q shell --uid=user .host /usr/bin/env bash

$ podman info
ERRO[0000] cannot find UID/GID for user: no subuid ranges found for user "user" in /etc/subuid - check rootless mode in man pages. 
WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user 
host:
  arch: amd64
  buildahVersion: 1.33.11
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
...

By removing the user from /etc/sub{u,g}id, Padman seems to be functional, but when writing images, it encounters the problem of insufficient UIDs or GIDs available in user namespace.

[rhatdan]

newuidmap can fail if the process does not have CAP_SETUID, CAP_SETGID in the bounding set. You can check this out in using capsh --print

Also make sure that the file caps are set.
getcap /usr/bin/newuidmap /usr/bin/newgidmap
/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep

Finally what OS are you doing this on?

[rhatdan]

@giuseppe PTAL

[rhatdan]

On my Fedora 41 system, this works fine.

# /usr/bin/machinectl -q shell --uid=cdoern .host /usr/bin/env  /usr/bin/podman run alpine echo hi
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 1f3e46996e29 done   | 
Copying config b0c9d60fc5 done   | 
Writing manifest to image destination
hi

[mohaa7]

On rocky8;
If I change the user range in /etc/subgid and /etc/subuid from 100000:65536 to 10000:5000, it works for me too! Why does the [gu]id ranges matter?

[rhatdan]

Your inside of a container and the container does not have UID 100000 mapped, would be my guess.

[mohaa7]

The issue solved by chaing UIDs range for the user who runs the outer container to remove the starting_id conflict with the user who runs the container inside the outer container.