ERROR: Could not virtualize PID namespace: Invalid argument

[fmc2478]

Hello,

I've compiled singularity 2.1.2 from source on two different hosts, both running redhat clones, although of different versions. The compilation on both hosts was uneventful, and in both I used --prefix=/opt/singularity with the configure script. On the host with a more recent kernel version (2.6.32-573.22.1.el6.x86_64), things seem to work...

$ singularity shell container.img
Singularity.container.img> $ cat /etc/debian_version
8.5
Singularity.container.img> $ exit
/bin/sh: 2: Cannot set tty process group (No such process)
$

...while on the host with the older kernel version (2.6.32-220.23.1.bl6.Bull.28.8.x86_64), I run into this error when invoking singularity...

$ singularity -v shell container.img
increasing verbosity level (2)
Exec'ing: /opt/singularity/libexec/singularity/cli/shell.exec container.img
VERBOSE: Set messagelevel to: 2
LOG : Command=shell, Container=container.img, CWD=/panfs/vol/f/fachaud74, Arg1=(null)
VERBOSE: Creating/Verifying session directory: /tmp/.singularity-session-7001.19.4214115159
VERBOSE: Calculating image offset
VERBOSE: Found valid loop device: /dev/loop0
VERBOSE: Using loop device: /dev/loop0
VERBOSE: Creating namespace process
ERROR : Could not virtualize PID namespace: Invalid argument
VERBOSE: Cleaning sessiondir: /tmp/.singularity-session-7001.19.4214115159
$

The older host is running the equivalent of redhat 6.2 while the newer one is running stock 6.4. Short of upgrading the OS on older host, is there anything I can do to get singularity to work?

Thank you.
Regards,
Faisal

[gmkurtzer]

Hi Faisal,

Ohh, I would bet that there is a misalignment between userspace libraries that say CLONE_NEWPID is supported and the kernel that isn't supporting it. Can you send me the configure output so I can see what userspace supports (the configure script just looks at usespace, not kernel).

As a work around, you can set 'allow pid ns = no' in singularity.conf and it should work properly. Also in the new code I've been working on for 2.2 (due to release mid-end September) the PID namespace must be requested as it is not used by default.

Thanks and hope that helps!

[fmc2478]

Thanks for the quick response Greg. Turning off PID namespace in singularity.conf did the trick. I'm attaching the output of the configure script as requested.

Thanks for working on this wonderful project. We're looking forward to exploring both singularity and shifter on our soon-to-be-installed Cray XC system!

Regards,
Faisal
autogen-output.txt
configure-output.txt
make-output.txt

[gmkurtzer]

Yes, this confirmed my suspicion... The user space is claiming to support CLONE_NEWPID, but the kernel is not. I am pretty sure that a kernel update will fix it, but I also understand if you can't update it so just keeping the PID namespace disabled for now will work fine.

My pleasure on working on the project and thank you for the compliment! Once you are done with your investigation of Singularity and if you end up using it can you please send me a note to let me know what you are running it on? I keep a document that I share with management when the need arises. lol

Lastly, if you are investigating Singularity, I would also encourage you to look at the master branch which will soon be released as 2.2. Lots of really cool work going on there and I'd love the help testing it!

Thanks!

[fmc2478]

Thanks again. I will update you when we get to testing on our new system in a month or two.

Regards,
Faisal

Sent from my Samsung device

-------- Original message --------
From: "Gregory M. Kurtzer" [email protected]
Date: 9/1/16 21:34 (GMT+03:00)
To: gmkurtzer/singularity [email protected]
Cc: "Chaudhry, Faisal" [email protected], Author [email protected]
Subject: Re: [gmkurtzer/singularity] ERROR: Could not virtualize PID namespace: Invalid argument (#204)

Yes, this confirmed my suspicion... The user space is claiming to support CLONE_NEWPID, but the kernel is not. I am pretty sure that a kernel update will fix it, but I also understand if you can't update it so just keeping the PID namespace disabled for now will work fine.

My pleasure on working on the project and thank you for the compliment! Once you are done with your investigation of Singularity and if you end up using it can you please send me a note to let me know what you are running it on? I keep a document that I share with management when the need arises. lol

Lastly, if you are investigating Singularity, I would also encourage you to look at the master branch which will soon be released as 2.2. Lots of really cool work going on there and I'd love the help testing it!

Thanks!

�
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_gmkurtzer_singularity_issues_204-23issuecomment-2D244135005&d=CwMFaQ&c=bnyvWLdNsiLM7TAqr9FAIzD3xPtiaC-DpEuDzu-03E4&r=UMFCvhe2x8jHEAdiQFdjnNEY393CQd-U6XBz7dwJQxM&m=ZxOHz7GxI8i4Dcx3c0IsoRVmIe887tJOs3coBpgOBZc&s=aSa1WU6vw4pb3768U_ecLocKNU6oGQm6Bw1QvbFrgks&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AUff0ktsaSY4GIFYcMqL6ThJTwkw9et6ks5qlv1hgaJpZM4JyaZY&d=CwMFaQ&c=bnyvWLdNsiLM7TAqr9FAIzD3xPtiaC-DpEuDzu-03E4&r=UMFCvhe2x8jHEAdiQFdjnNEY393CQd-U6XBz7dwJQxM&m=ZxOHz7GxI8i4Dcx3c0IsoRVmIe887tJOs3coBpgOBZc&s=4tLFq5oHzg5EXYYJoGtfSCmVkJ_VOqhCcFYAX2dRlQc&e=.

[gmkurtzer]

My pleasure and please let me know if you have any other questions or problems.

Greg

[soichih]

Hello. I am testing singularity 2.2 on our Clay based HPC system (IU Bigred 2), and I am seeing this error message.

$ singularity shell -C soichi.img
increasing verbosity level (2)
Exec'ing: /N/soft/cle5/singularity/2.2/libexec/singularity/cli/shell.exec -CVERBOSE: Set messagelevel to: 2
VERBOSE: Running NON-SUID program workflow
VERBOSE: Opening configuration file: /N/soft/cle5/singularity/2.2/etc/singularity/singularity.conf
VERBOSE: Invoking SUID sexec: /N/soft/cle5/singularity/2.2/libexec/singularity/sexec-suid
VERBOSE: Set messagelevel to: 2
VERBOSE: Opening configuration file: /N/soft/cle5/singularity/2.2/etc/singularity/singularity.conf
VERBOSE: Not virtualizing user namespace: running SUID root
ERROR  : Could not virtualize PID namespace: Invalid argument
ABORT  : Retval = 255
VERBOSE: Cleaning sessiondir: /tmp/.singularity-session-740536.21.323635004

Is this issued fixed on 2.2? Thank you for working on this great project!

[gmkurtzer]

Two things are making me curious....

• I am seeing verbose output from Singularity. Is MESSAGELEVEL set in your environment?
• Luckily it is running in verbose mode because I can see that it is not running the SUID pathway. Was this installed as root, and if so is the directory it is installed to mounted with "nosuid "?

[soichih]

@gmkurtzer

For MESSAGELEVEL, I updated my earlier comment after I posted it to include output from -v but I forgot to update the actual command line.. sorry!

For SUID, yes the singularity was installed as root and /N/soft (alias for /gpfs/hps/soft) is mounted with following options.

$ mount | grep gpfs
/dev/gpfs on /gpfs type gpfs (rw,dev=GSB.gsb0:GSD,ldev=gpfs)

$ cat /etc/fstab | grep gpfs
/dev/gpfs            /gpfs                gpfs       rw,dev=GSB.gsb0:GSD,ldev=gpfs,noauto 0 0

By the way, this is on SLE11 (IU BigRed2). Will mounting singularity binaries on GPFS be a problem?

[bbockelm]

For distributing setuid binaries via a shared filesystem: probably technically works, but most admins will be reluctant to put that much trust in GPFS. In most setups, the filesystem is mounted with the nosuid option.

Sent from my iPhone

On Oct 18, 2016, at 8:18 AM, Soichi Hayashi [email protected] wrote:

@gmkurtzer

For MESSAGELEVEL, I updated my earlier comment after I posted it to include output from -v but I forgot to update the actual command line.. sorry!

For SUID, yes the singularity was installed as root and /N/soft (alias for /gpfs/hps/soft) is mounted with following options.

$ mount | grep gpfs
/dev/gpfs on /gpfs type gpfs (rw,dev=GSB.gsb0:GSD,ldev=gpfs)

$ cat /etc/fstab | grep gpfs
/dev/gpfs /gpfs gpfs rw,dev=GSB.gsb0:GSD,ldev=gpfs,noauto 0 0
By the way, this is on SLE11 (IU BigRed2). Will mounting singularity binaries on GPFS be a problem?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[soichih]

@bbockelm Does this mean admin have to install singularity locally on all nodes? GPFS is our current shared file system of choice and we use it to distribute applications (modules) as well as user's home directories across our cluster.

By the way, I can still launch singularity container on our cluster (binaries installed on GPFS). If suid is disabled, I would think that I won't be able to launch singularity at all.. let alone PID namespace. Does it use different mechanism to do chroot / launch container?

[bbockelm]

On Oct 18, 2016, at 3:40 PM, Soichi Hayashi [email protected] wrote:

@bbockelm https://github.com/bbockelm Does this mean admin have to install singularity locally on all nodes? GPFS is our current shared file system of choice and we use it to distribute applications (modules) as well as user's home directories across our cluster.

Like I said, it's a bit up to the admin whether they trust the shared filesystem with setuid executables. At a technical level, it should work just fine.

By the way, I can still launch singularity container on our cluster (binaries installed on GPFS). If suid is disabled, I would think that I won't be able to launch singularity at all.. let alone PID namespace. Does it use different mechanism to do chroot / launch container?

Nope - it's all the same underlying mechanism. If the simple tests work fine, I would expect everything else to work.

Brian

[soichih]

I've confirmed that SUID is working on our GPFS. Yet, I am seeing following error message on BigRed 2 (SLE11)

VERBOSE: Not virtualizing user namespace: running SUID root
ERROR  : Could not virtualize PID namespace: Invalid argument

@gmkurtzer

Earlier you said this

I would bet that there is a misalignment between userspace libraries that say CLONE_NEWPID is supported and the kernel that isn't supporting it.

So, I am understaing we won't be able to do PID name spacing - because SLE11's kernel (3.0.101) doesn't support it (and there is nothing we can do about it)?

[gmkurtzer]

Hi @soichih,

The PID namespace has technically been available since 2.6.24, so I am not sure why CLONE_NEWPID is failing aside from the fact that it is a Cray. I don't have direct access to a Cray to test, but I can ask around.

Can you do this and let me know what the result is please:

$ grep CONFIG_PID_NS /boot/config-`uname -r`

Thanks!

[bbockelm]

@gmkurtzer - while CLONE_NEWPID was added in 2.6.24 for the clone() syscall, support was not added to unshare until 3.8. Singularity uses the latter.

[gmkurtzer]

Yes, you are correct. I need to stop trusting RedHat as they have spoiled me by back-porting so much into the RHEL6 kernel (and removing the upstream kernel version it was actually supported in their man pages).

[soichih]

OK. Thanks. I think it will be nice if there is a table (matrix) of various major OSes that singularity runs on, and which features works and which doesn't. It looks like there are a lot of red flags for SLE11..

[antunderwood]

I am also having this problem on a Redhat 6.4 system.

Singularity is installed as root

When I run

singularity -v shell singularity/phenix-2017-05-15-862ad3f0cdad.img

I get the following error

Increasing verbosity level (2)
Singularity version: 2.3.1-dist
Exec'ing: /phengs/hpc_software/singularity/2.3.1/libexec/singularity/cli/shell.exec
Evaluating args: 'singularity/phenix-2017-05-15-862ad3f0cdad.img'
VERBOSE: Set messagelevel to: 2
VERBOSE: Initialize configuration file: /etc/singularity/singularity.conf
VERBOSE: Initializing Singularity Registry
VERBOSE: Invoking the user namespace
VERBOSE: Not virtualizing USER namespace: running as SUID
ERROR  : Could not virtualize PID namespace: Invalid argument
ABORT  : Retval = 255

Is there an incompatibility with singularity and Redhat 6.4? Kernel is 2.6.32-358.el6.x86_64

[vsoch]

I anticipate @gmkurtzer will ask for this too - could you put --debug with the command to get the full debug output? Thanks!

[antunderwood]

The output using --debug is

Increasing verbosity level (2)
Enabling debugging
Ending argument loop
Singularity version: 2.3.1-dist
Exec'ing: /phengs/hpc_software/singularity/2.3.1/libexec/singularity/cli/shell.exec
Evaluating args: 'singularity/phenix-2017-05-15-862ad3f0cdad.img'
VERBOSE [U=0,P=23694]      message_init()                            Set messagelevel to: 5
VERBOSE [U=0,P=23694]      singularity_config_parse()                Initialize configuration file: /etc/singularity/singularity.conf
DEBUG   [U=0,P=23694]      singularity_config_parse()                Starting parse of configuration file /etc/singularity/singularity.conf
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key allow setuid = 'yes'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key allow pid ns = 'yes'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key enable overlay = 'no'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key config passwd = 'yes'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key config group = 'yes'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key config resolv_conf = 'yes'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key mount proc = 'yes'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key mount sys = 'yes'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key mount dev = 'yes'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key mount home = 'yes'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key mount tmp = 'yes'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key mount hostfs = 'no'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key bind path = '/etc/hosts'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key user bind control = 'yes'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key mount slave = 'yes'
VERBOSE [U=0,P=23694]      singularity_config_parse()                Got config key container dir = '/var/singularity/mnt'
DEBUG   [U=0,P=23694]      singularity_config_parse()                Finished parsing configuration file '/etc/singularity/singularity.conf'
VERBOSE [U=0,P=23694]      singularity_registry_init()               Initializing Singularity Registry
VERBOSE [U=0,P=23694]      singularity_registry_set()                Adding value to registry: 'LIBEXECDIR' = '/phengs/hpc_software/singularity/2.3.1/libexec'
DEBUG   [U=0,P=23694]      singularity_registry_set()                Returning singularity_registry_set(libexecdir, /phengs/hpc_software/singularity/2.3.1/libexec) = 0
VERBOSE [U=0,P=23694]      singularity_registry_set()                Adding value to registry: 'COMMAND' = 'shell'
DEBUG   [U=0,P=23694]      singularity_registry_set()                Returning singularity_registry_set(COMMAND, shell) = 0
VERBOSE [U=0,P=23694]      singularity_registry_set()                Adding value to registry: 'MESSAGELEVEL' = '5'
DEBUG   [U=0,P=23694]      singularity_registry_set()                Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
VERBOSE [U=0,P=23694]      singularity_registry_set()                Adding value to registry: 'VERSION' = '2.3.1-dist'
DEBUG   [U=0,P=23694]      singularity_registry_set()                Returning singularity_registry_set(version, 2.3.1-dist) = 0
VERBOSE [U=0,P=23694]      singularity_registry_set()                Adding value to registry: 'LOCALSTATEDIR' = '/phengs/hpc_software/singularity/2.3.1/var'
DEBUG   [U=0,P=23694]      singularity_registry_set()                Returning singularity_registry_set(localstatedir, /phengs/hpc_software/singularity/2.3.1/var) = 0
VERBOSE [U=0,P=23694]      singularity_registry_set()                Adding value to registry: 'SYSCONFDIR' = '/etc'
DEBUG   [U=0,P=23694]      singularity_registry_set()                Returning singularity_registry_set(sysconfdir, /etc) = 0
VERBOSE [U=0,P=23694]      singularity_registry_set()                Adding value to registry: 'BINDIR' = '/phengs/hpc_software/singularity/2.3.1/bin'
DEBUG   [U=0,P=23694]      singularity_registry_set()                Returning singularity_registry_set(bindir, /phengs/hpc_software/singularity/2.3.1/bin) = 0
VERBOSE [U=0,P=23694]      singularity_registry_set()                Adding value to registry: 'IMAGE' = 'singularity/phenix-2017-05-15-862ad3f0cdad.img'
DEBUG   [U=0,P=23694]      singularity_registry_set()                Returning singularity_registry_set(IMAGE, singularity/phenix-2017-05-15-862ad3f0cdad.img) = 0
DEBUG   [U=0,P=23694]      singularity_registry_get()                Returning NULL on 'HOME'
DEBUG   [U=0,P=23694]      singularity_registry_get()                Returning NULL on 'TARGET_UID'
DEBUG   [U=0,P=23694]      singularity_registry_get()                Returning NULL on 'TARGET_GID'
DEBUG   [U=0,P=23694]      singularity_priv_init()                   Initializing user info
DEBUG   [U=0,P=23694]      singularity_priv_init()                   Set the calling user's username to: anthony
DEBUG   [U=0,P=23694]      singularity_priv_init()                   Marking uinfo structure as ready
DEBUG   [U=0,P=23694]      singularity_priv_init()                   Obtaining home directory
VERBOSE [U=0,P=23694]      singularity_priv_init()                   Set home (via getpwuid()) to: /home/anthony
VERBOSE [U=0,P=23694]      singularity_suid_init()                   Running SUID program workflow
VERBOSE [U=0,P=23694]      singularity_suid_init()                   Checking program has appropriate permissions
VERBOSE [U=0,P=23694]      singularity_suid_init()                   Checking configuration file is properly owned by root
VERBOSE [U=0,P=23694]      singularity_suid_init()                   Checking if singularity.conf allows us to run as suid
DEBUG   [U=0,P=23694]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow setuid, yes)
DEBUG   [U=0,P=23694]      singularity_config_get_value_impl()       Returning configuration value allow setuid='yes'
DEBUG   [U=0,P=23694]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow setuid, yes) = 1
DEBUG   [U=0,P=23694]      singularity_registry_get()                Returning NULL on 'NOSUID'
VERBOSE [U=0,P=23694]      singularity_priv_userns()                 Invoking the user namespace
DEBUG   [U=0,P=23694]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow user ns, yes)
DEBUG   [U=0,P=23694]      singularity_config_get_value_impl()       No configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG   [U=0,P=23694]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow user ns, yes) = 1
VERBOSE [U=0,P=23694]      singularity_priv_userns()                 Not virtualizing USER namespace: running as SUID
DEBUG   [U=0,P=23694]      singularity_priv_userns()                 Returning singularity_priv_init(void)
DEBUG   [U=0,P=23694]      singularity_priv_drop()                   Dropping privileges to UID=501, GID=1001 (8 supplementary GIDs)
DEBUG   [U=0,P=23694]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=501,P=23694]    singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=501,P=23694]    singularity_registry_get()                Returning NULL on 'CLEANUPDIR'
DEBUG   [U=501,P=23694]    singularity_registry_get()                Returning NULL on 'NOSESSIONCLEANUP'
DEBUG   [U=501,P=23694]    singularity_registry_get()                Returning NULL on 'NOCLEANUP'
DEBUG   [U=501,P=23694]    singularity_cleanupd()                    Not running a cleanup thread, no 'SINGULARITY_CLEANUPDIR' defined
DEBUG   [U=501,P=23694]    singularity_runtime_ns()                  Calling: _singularity_runtime_ns_pid()
DEBUG   [U=501,P=23694]    singularity_runtime_ns_pid()              Using PID namespace: CLONE_NEWPID
DEBUG   [U=501,P=23694]    singularity_priv_escalate()               Temporarily escalating privileges (U=501)
DEBUG   [U=0,P=23694]      singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=23694]      singularity_runtime_ns_pid()              Virtualizing PID namespace
ERROR   [U=0,P=23694]      singularity_runtime_ns_pid()              Could not virtualize PID namespace: Invalid argument
ABORT   [U=0,P=23694]      singularity_runtime_ns_pid()              Retval = 255