singularity shell fails with various errors

[vladfi1]

After creating an ubuntu image with the following definition

DistType "debian"
MirrorURL "http://us.archive.ubuntu.com/ubuntu/"
OSVersion "xenial"

Setup 
Bootstrap

Cleanup

I try to enter into a shell, but randomly get one of two errors:

ERROR  : Failed to execv() /bin/sh: Operation not permitted
ABORT  : Retval = 255

or

ERROR  : Could not read /var/singularity/mnt/final/etc/passwd: Operation not permitted
ERROR  : Failed copying template passwd file to sessiondir: Operation not permitted
ABORT  : Retval = 255

Rarely, but especially the first time, I do actually get into a shell.

[gmkurtzer]

Can you run with the --debug option and send the output?

Thanks!

[vladfi1]

Thanks for the swift reply! I'll do that right away. I'm also in the irc channel - that might be better for quicker debugging?

[vladfi1]

@gmkurtzer Output here: http://lpaste.net/6861355957830549504

[gmkurtzer]

Quick but not productive. On iPhone now and only for a few lol. Will be able to help more tomorrow but wanted to get the info needed.

[gmkurtzer]

Hrmmmm. That is interesting. Singularly output looks fine. Can you mount the image and do a ls -l and file on /bin/sh inside the container image?

Also if you want to compare, might be interesting to test against github master branch.

[gmkurtzer]

Oh wait. You are using master branch. Hrmmmmmmm. Yes. Please mout the image and let's verify that /bin/sh is as it should be. Also can you show me the output of the containers /.shell wrapper?

[vladfi1]

I'll give that a try. I am using the master branch, installed as of a few hours ago. I managed to figure out the ldconfig issue that satra was having on my own :)

[gmkurtzer]

It just occurred to me. What is your umask? Maybe those files got created with restrictive permissions?

[vladfi1]

After mounting:

$ ls -la /var/singularity/mnt/final/bin/sh
lrwxrwxrwx 1 root root 4 Feb 17  2016 /var/singularity/mnt/final/bin/sh -> dash

I'm not sure what you mean by the "containers /.shell" wrapper.

My umask inside the image is 0022, outside is 0002.

[gmkurtzer]

Ls -l /var/singularity/mint/final/.shell

Also can you do an ls -l on dash and also do a file on it?

Lastly a dmesg.

[vladfi1]

# ls -l .shell 
-rwxr-xr-x 1 root root 366 Sep 13 00:02 .shell
# ls -l bin/dash 
-rwxr-xr-x 1 root root 154072 Feb 17  2016 bin/dash
# file bin/dash 
bin/dash: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=504637666875a5d526ef51acfe601c99efc99114, stripped

Dmesg: http://lpaste.net/94411657568059392

[gmkurtzer]

At quick glance it looks like a kernel bug in overlayfs. Can you please try to disable the overlayFS support in Singularity Config file and try again?

Thanks!

[vladfi1]

Ok, I set enable overlay = no and now I can shell 100% of the time!

[gmkurtzer]

Yep. Kernel bug! Are you running the distributions latest? Can you try a dist-upgrade and see if a newer kernel solves the issue?

Thanks!

[vladfi1]

I'm running the 4.5.0-040500-generic kernel on ubuntu 16.04. Interestingly, I had no problems with the singularity shell on a remote Centos 7 machine running linux 3.10.0-229.el7.x86_64. I'll try a newer mainline kernel and see how it goes.

[vladfi1]

Upgrading to linux 4.8.0 seems to have fixed it (after setting enable overlay = yes).

[gmkurtzer]

Excellent, thank you!