Running singularity from within a singularity container

[sntgluca]

Dear all,
I am walking the first steps with singularity, and I have a use case I am not able to handle.

I am supporting a complex Jupyter environment with a Jupyter server and several backend kernels to connect to. Each component is a separate (conda) environment.
The Jupyter server starts one or more backend kernels on demand.

I am evaluating rebuilding the individual parts as containers.
For this to work I wish to run singularity from the Jupyter container:

shell --singularity--> jupyter container --singularity--> kernel container

Unfortunately I am always having issues with suid when trying to run singularity exec from the first container

Singularity> singularity exec <container> ls
ERROR : Failed to set effective UID to 0

or

Singularity> sudo mount -o remount,suid /
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?

May anyone suggest a configuration to mount / or another directory (/opt, /usr/local) with suid, or any other approach to overcome this problem?

Many thanks

Version of Singularity:

What version of Singularity are you using? Run:

$ singularity version
v3.5.2-1.el7.centos

Expected behavior

What did you expect to see when you do...?

I would expect the following command to work from within a container

Singularity> singularity exec <container> ls

Actual behavior

What actually happend? Why was it incorrect?

I experience issues with root permissions and SUID

ERROR : Failed to set effective UID to 0

Steps to reproduce this behavior

How can others reproduce this issue/problem?

AFAIK this applies to any image by default. For instance,

Bootstrap: docker

    From: continuumio/miniconda3

%runscript
    exec "$*"

What OS/distro are you running

$ cat /etc/os-release

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[DrDaveD]

Privileged singularity is never going to work inside of a singularity container, because singularity prevents it from happening for security reasons. Is enabling unprivileged user namespaces an option for you, and using a sandbox container (that is, unpacked in a directory tree) for the inner singularity invocation? If so you can run singularity unprivileged inside of another singularity container.

[sntgluca]

Gosh, I could not follow up for more than 20 days, sorry.
Thank you for the explanation, and suggesting two different ways to get around the issue.
I will try using a sandbox container. It seems the right strategy.

May I ask where I can find detailed documentation about enabling unprivileged user namespaces and/or running singularity unprivileged?

[dtrudg]

May I ask where I can find detailed documentation about enabling unprivileged user namespaces and/or running singularity unprivileged?

See: https://sylabs.io/guides/3.5/admin-guide/user_namespace.html

[DrDaveD]

To be clear, I wasn't suggesting 2 different ways to get around the issue. You need both both unprivileged user namespaces and a sandbox container for the inner invocation.