User running podman segfaults #1189

afbjorklund · 2018-07-30T16:08:15Z

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

Description

Running podman as user causes crashes.

Steps to reproduce the issue:

podman version

Describe the results you received:

$ podman version
ERRO[0000] open /etc/subuid: no such file or directory  
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x3926000 pc=0xf61a89]

Describe the results you expected:

$ sudo podman version
Version:       0.7.4
Go Version:    go1.10.2
OS/Arch:       linux/amd64

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

(paste your output here)

Output of podman info:

(paste your output here)

Additional environment details (AWS, VirtualBox, physical, etc.):

VirtualBox

The text was updated successfully, but these errors were encountered:

mheon · 2018-07-30T16:17:09Z

@giuseppe PTAL

giuseppe · 2018-07-30T16:37:38Z

@afbjorklund is the podman distributed by Fedora? I cannot reproduce with the last version available there.

Could you try https://koji.fedoraproject.org/koji/buildinfo?buildID=1131468 ?

afbjorklund · 2018-07-30T16:48:55Z

Nope, I built it from source (using buildroot for minikube)

deploy/iso/minikube-iso/package/podman/podman.mk

Had to do some vendor hack for "varlink" - whatever that is...

Seems like it wasn't getting installed with the install.tools ?

afbjorklund · 2018-07-30T16:51:18Z

@giuseppe : maybe it requires that /etc/subuid to be present ? (rootless mode)

But maybe not for the "version" command, if that happens to be the case...

afbjorklund · 2018-07-30T16:56:08Z

Yes, that seems to have been it:

$ echo $USER:10000:65536 | sudo tee /etc/subuid
docker:10000:65536
$ echo $USER:10000:65536 | sudo tee /etc/subgid
docker:10000:65536
$ podman version
Version:       0.7.4
Go Version:    go1.10.2
OS/Arch:       linux/amd64

Same as documented in #1185

mheon · 2018-07-30T16:56:19Z

@afbjorklund re: varlink - If you don't want if in your build, edit the Makefile and remove varlink from BUILDTAGS (or manually set the environment variable)

afbjorklund · 2018-07-30T16:59:39Z

@mheon : I have no idea, I was just updating podman (trying to get away from kpod eventually)

kubernetes/minikube#3026

Happy to build with the varlink feature, as long as it works... Maybe I could go get it instead ?

mheon · 2018-07-30T17:04:06Z

@afbjorklund There's a dependency package (libvarlink, I think?) as well. Generally speaking, if you're not thinking you need a remote API, it's safe to build without varlink.

giuseppe · 2018-07-30T17:06:30Z

@afbjorklund it is required for rootless containers, but if it is not present it should definitely not crash.

I've just built the same version you are using but still it works here, I just get the error message and podman exits.

On what system are you trying to run it? I've tried to use it only on Fedora and Ubuntu

afbjorklund · 2018-07-30T17:11:52Z

Users seem to be happy about hi-jacking the old docker daemon (of minikube), not sure if that should be allowed to continue or if they should use their own container infrastructure instead of the (mini) cloud's ?

https://kubernetes.io/docs/setup/minikube/#reusing-the-docker-daemon

Currently I'm just ssh'ing into the machine, happy that it can show the images now (cache got turned off...) even if seems like listing containers (i.e. ps) got broken out-of-the-box again. Probably just a socket move.

Hacking on: kubernetes/minikube#2757 (running Kubernetes with CRI-O)

afbjorklund · 2018-07-30T17:14:52Z

@giuseppe : it is running the "minikube.iso", which is a buildroot 2018.05 environment

giuseppe · 2018-07-31T09:04:24Z

I'm still not able to reproduce the segfault when the /etc/subuid file is missing. What changes have you done to varlink? If you have the binary you have built somewhere, that could help as well.

afbjorklund · 2018-08-02T10:33:15Z

@giuseppe : I used the vendored version of varlink, to avoid the out-of-the-box build error when just running "make install.tools podman" that the buildroot make naively tried to do. Did not modify anything.

Will see if I can upload the minikube.iso and the podman binary somewhere for easier access... There was a stacktrace as well, after the error printout. But I'm not sure if this segfault is "important" or not ?

afbjorklund · 2018-08-02T11:00:37Z

Here is the ISO: https://github.com/afbjorklund/minikube/releases/download/15e56aa/minikube.iso
Built from: https://github.com/afbjorklund/minikube/tree/15e56aa8e13d97a1bb6a0813281f9bc886816f4d

As requested, direct link to the binary itself: podman

Example output

                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
$ podman version

ERRO[0000] open /etc/subuid: no such file or directory

$ fatal error: unexpected signal during runtime execution

[signal SIGSEGV: segmentation violation code=0x1 addr=0x3b56000 pc=0xf61a89]
runtime stack:

runtime.throw(0x122f0e5, 0x2a)

/home/anders/KUBE/minikube/out/buildroot/output/host/lib/go/src/runtime/panic.go:616 +0x81

runtime.sigpanic()

/home/anders/KUBE/minikube/out/buildroot/output/host/lib/go/src/runtime/signal_unix.go:372 +0x28e
goroutine 1 [syscall, locked to thread]:

runtime.cgocall(0xf618e0, 0xc420281470, 0x4971f1)

/home/anders/KUBE/minikube/out/buildroot/output/host/lib/go/src/runtime/cgocall.go:128 +0x64 fp=0xc420281430 sp=0xc4202813f8 pc=0x4090c4

github.com/projectatomic/libpod/pkg/rootless._Cfunc_reexec_in_user_namespace(0xc400000005, 0x0)

_cgo_gotypes.go:43 +0x4d fp=0xc420281470 sp=0xc420281430 pc=0xd36d5d

github.com/projectatomic/libpod/pkg/rootless.BecomeRootInUserNS(0x101c300, 0x0, 0x0, 0x0)

/home/anders/KUBE/minikube/out/buildroot/output/build/podman-v0.7.4/_output/src/github.com/projectatomic/libpod/pkg/rootless/rootless_linux.go:92 +0x1ff fp=0xc420281680 sp=0xc420281470 pc=0xd375bf

main.main()

/home/anders/KUBE/minikube/out/buildroot/output/build/podman-v0.7.4/_output/src/github.com/projectatomic/libpod/cmd/podman/main.go:32 +0x71 fp=0xc420281f88 sp=0xc420281680 pc=0xf33e51

runtime.main()

/home/anders/KUBE/minikube/out/buildroot/output/host/lib/go/src/runtime/proc.go:198 +0x212 fp=0xc420281fe0 sp=0xc420281f88 pc=0x4331b2

runtime.goexit()

/home/anders/KUBE/minikube/out/buildroot/output/host/lib/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420281fe8 sp=0xc420281fe0 pc=0x45ee81
goroutine 5 [chan receive]:

github.com/projectatomic/libpod/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1d102a0)

/home/anders/KUBE/minikube/out/buildroot/output/build/podman-v0.7.4/_output/src/github.com/projectatomic/libpod/vendor/github.com/golang/glog/glog.go:882 +0x8b

created by github.com/projectatomic/libpod/vendor/github.com/golang/glog.init.0

/home/anders/KUBE/minikube/out/buildroot/output/build/podman-v0.7.4/_output/src/github.com/projectatomic/libpod/vendor/github.com/golang/glog/glog.go:410 +0x203
goroutine 6 [syscall]:

os/signal.signal_recv(0x0)

/home/anders/KUBE/minikube/out/buildroot/output/host/lib/go/src/runtime/sigqueue.go:139 +0xa6

os/signal.loop()

/home/anders/KUBE/minikube/out/buildroot/output/host/lib/go/src/os/signal/signal_unix.go:22 +0x22

created by os/signal.init.0

/home/anders/KUBE/minikube/out/buildroot/output/host/lib/go/src/os/signal/signal_unix.go:28 +0x41

afbjorklund · 2018-08-02T12:09:38Z

Using a previous version works fine. So it's probably something with "varlink" ?

$ podman version
Version:       0.4.1
Go Version:    go1.10.2
Git Commit:    "f3d114a1effd8a6ef773bee14fe49ea6d8d7c350"
Built:         Thu Aug  2 11:58:17 2018
OS/Arch:       linux/amd64

^ The git commit above refers to buildroot, not to libpod (which is from tarball)

https://github.com/buildroot/buildroot/releases/tag/2018.05

mheon · 2018-08-02T12:29:25Z

Call stack looks like the rootless code, and not Varlink. Something seems to be going wrong in the CGo code in pkg/rootless.

…

On Thu, Aug 2, 2018, 08:09 Anders Björklund ***@***.***> wrote: Using a previous version works fine. So it's probably something with "varlink" ? $ podman version Version: 0.4.1 Go Version: go1.10.2 Git Commit: "f3d114a1effd8a6ef773bee14fe49ea6d8d7c350" Built: Thu Aug 2 11:58:17 2018 OS/Arch: linux/amd64 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1189 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHYHCFBvqye_EeLxdVzTwcRYwIMBX8WHks5uMuwDgaJpZM4VmtF4> .

giuseppe · 2018-08-02T12:30:48Z

I am still not able to reproduce locally but inspecting the code and your call stack, I think the error can be fixed by: #1201.

Could you please verify if it solves the problem for you?

afbjorklund · 2018-08-02T13:08:30Z

Didn't seem to help, unfortunately.

>>> podman v0.7.4 Patching

Applying 1201.patch using patch: 
patching file pkg/rootless/rootless_linux.c

Think I need to read up on my CGo :-)
Or opt out of this "rootless" thing perhaps.

runtime.cgocall(0xf618e0, 0xc42027d470, 0x4971f1)
	/home/anders/KUBE/minikube/out/buildroot/output/host/lib/go/src/runtime/cgocall.go:128 +0x64 fp=0xc42027d430 sp=0xc42027d3f8 pc=0x4090c4
github.com/projectatomic/libpod/pkg/rootless._Cfunc_reexec_in_user_namespace(0xc400000005, 0x0)
	_cgo_gotypes.go:43 +0x4d fp=0xc42027d470 sp=0xc42027d430 pc=0xd36d5d
github.com/projectatomic/libpod/pkg/rootless.BecomeRootInUserNS(0x101c300, 0x0, 0x0, 0x0)
	/home/anders/KUBE/minikube/out/buildroot/output/build/podman-v0.7.4/_output/src/github.com/projectatomic/libpod/pkg/rootless/rootless_linux.go:92 +0x1ff fp=0xc42027d680 sp=0xc42027d470 pc=0xd375bf
main.main()
	/home/anders/KUBE/minikube/out/buildroot/output/build/podman-v0.7.4/_output/src/github.com/projectatomic/libpod/cmd/podman/main.go:32 +0x71 fp=0xc42027df88 sp=0xc42027d680 pc=0xf33e51

afbjorklund · 2018-08-02T17:09:01Z

No luck with gdb, so added some oldschool printf debugging instead...

It seems like when the program crashes, that argc is 0 (cmdline gone?)

$ ./podman version
ERRO[0000] open /etc/subuid: no such file or directory  
argc: 2
used: 17
$ ./podman version
ERRO[0000] open /etc/subuid: no such file or directory  
argc: 0
used: 0
fatal error: unexpected signal during runtime execution

So that termination code needs to check for that possibility too...

  argv = malloc (sizeof (char *) * (argc + 1));
  if (argv == NULL)
    return NULL;
  argc = 0;

  argv[argc++] = buffer;
  for (i = 0; i < used - 1; i++)
    if (buffer[i] == '\0')
      argv[argc++] = buffer + i + 1;

  argv[argc] = NULL;

afbjorklund · 2018-08-02T17:15:19Z

Like so: 054d9bc

afbjorklund · 2018-08-03T11:01:28Z

Thanks, with those additions the segfault doesn't happen anymore.

$ podman version
ERRO[0000] open /etc/subuid: no such file or directory  
$ sudo podman version
Version:       0.7.4
Go Version:    go1.10.2
OS/Arch:       linux/amd64

hsmiranda · 2019-07-08T20:20:48Z

Yes, that seems to have been it:

$ echo $USER:10000:65536 | sudo tee /etc/subuid
docker:10000:65536
$ echo $USER:10000:65536 | sudo tee /etc/subgid
docker:10000:65536
$ podman version
Version:       0.7.4
Go Version:    go1.10.2
OS/Arch:       linux/amd64

Same as documented in #1185

it's worked for me, thanks.

mheon added the bug label Jul 30, 2018

rhatdan closed this as completed in c7b6403 Aug 2, 2018

This was referenced Aug 2, 2018

Check for missing arguments in /proc/self/cmdline #1206

Closed

rootless: do not segfault if the parent already died #1201

Merged

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 24, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

User running podman segfaults #1189

User running podman segfaults #1189

afbjorklund commented Jul 30, 2018 •

edited

Loading

mheon commented Jul 30, 2018

giuseppe commented Jul 30, 2018

afbjorklund commented Jul 30, 2018

afbjorklund commented Jul 30, 2018

afbjorklund commented Jul 30, 2018

mheon commented Jul 30, 2018

afbjorklund commented Jul 30, 2018

mheon commented Jul 30, 2018

giuseppe commented Jul 30, 2018

afbjorklund commented Jul 30, 2018

afbjorklund commented Jul 30, 2018

giuseppe commented Jul 31, 2018

afbjorklund commented Aug 2, 2018

afbjorklund commented Aug 2, 2018

afbjorklund commented Aug 2, 2018 •

edited

Loading

mheon commented Aug 2, 2018 via email

giuseppe commented Aug 2, 2018

afbjorklund commented Aug 2, 2018

afbjorklund commented Aug 2, 2018

afbjorklund commented Aug 2, 2018

afbjorklund commented Aug 3, 2018

hsmiranda commented Jul 8, 2019 •

edited

Loading

User running podman segfaults #1189

User running podman segfaults #1189

Comments

afbjorklund commented Jul 30, 2018 • edited Loading

mheon commented Jul 30, 2018

giuseppe commented Jul 30, 2018

afbjorklund commented Jul 30, 2018

afbjorklund commented Jul 30, 2018

afbjorklund commented Jul 30, 2018

mheon commented Jul 30, 2018

afbjorklund commented Jul 30, 2018

mheon commented Jul 30, 2018

giuseppe commented Jul 30, 2018

afbjorklund commented Jul 30, 2018

afbjorklund commented Jul 30, 2018

giuseppe commented Jul 31, 2018

afbjorklund commented Aug 2, 2018

afbjorklund commented Aug 2, 2018

afbjorklund commented Aug 2, 2018 • edited Loading

mheon commented Aug 2, 2018 via email

giuseppe commented Aug 2, 2018

afbjorklund commented Aug 2, 2018

afbjorklund commented Aug 2, 2018

afbjorklund commented Aug 2, 2018

afbjorklund commented Aug 3, 2018

hsmiranda commented Jul 8, 2019 • edited Loading

afbjorklund commented Jul 30, 2018 •

edited

Loading

afbjorklund commented Aug 2, 2018 •

edited

Loading

hsmiranda commented Jul 8, 2019 •

edited

Loading