-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime: fatal error: unexpected signal during runtime execution #13046
Comments
/cc @aclements |
Is this reproducible? If so how? On Sunday, October 25, 2015, Burcu Dogan [email protected] wrote:
|
@RLH Sorry. This is a panic that a user reported to me. We tried to reproduce it but failed. And it seems to happen very infrequently. I suspect this is a go runtime bug. If you think this is a user program bug, I can try to check the related code. Also I am happy to look into the related code inside go if you are sure this is a go runtime issue. Let me know if you want more information. |
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x80 addr=0x0 pc=0x40f024]
The code is interesting. Normally, it should 0x1 (SEGV_MAPERR).
(The other possibilities are 2 for SEGV_ACCERR, and 3 for
SEGV_BNDERR)
code 0x80 just signifies that it's coming from kernel, but doesn't
give a reason for the signal.
And looking at the code:
609 if v.ptr() == nil {
610 systemstack(func() {
611 mCache_Refill(c, int32(sizeclass))
612 })
613 shouldhelpgc = true
614 s = c.alloc[sizeclass]
615 v = s.freelist
616 }
617 s.freelist = v.ptr().next
v.ptr() shouldn't be nil at line 617 unless someone else is also
modifying it. (but it's from gomcache, nobody else should be
modifying it concurrently.
Could you tell us more about the program? Does it use cgo or
unsafe? Have you tried to run it with race detector enabled?
|
If this is coming from the kernel, there will be a report in dmesg (there is on arm64 systems)
|
The program is etcd (https://github.com/coreos/etcd). The core part of etcd does not involve cgo/unsafe, but we are using external pkgs. So I am not sure.
A normal go user program can corrupt runtime data? Or the race detector can detect data race in cgo? I cannot reproduce the issue myself right now. I do not know if it helps to collect data by enabling race detector. |
A program with a data race is undefined, it can do anything including corrupting runtime data structures. |
@xiang90 i had a quick play with coreos/etcd and found several packages fail the race detector. Are these know failures ? I think you should address those in coreos/etcd before we can investigate. |
@davecheney I encountered the issue and checked dmesg, nothing was printed there. |
@lorenz, can you post your panic message and traceback? |
Is in coreos/etcd#3745. |
Ah, sorry, I hadn't realized you were the poster of the original bug. I'm inclined to agree with @davecheney that there's not much we can do until the races are fixed. |
I agree with Austin and Dave that this should be closed until the program On Mon, Oct 26, 2015 at 10:52 AM, Austin Clements [email protected]
|
After reading runtime malloc code, I agree with @davecheney that the user code can potentially corrupt runtime heap. But to be honest, I still feel the possibility is still too low that a newly allocated freelist is corrupted. I would like to close this issue on go upstream for now. I will reopen it if I find something. |
@davecheney Just want to make sure we are fixing the current thing (at least fixing what you have seen)
How did you play with coreos/etcd? How did you find out the races? |
|
Same issue here when running fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x0 pc=0x0]
runtime stack:
runtime.throw(0x196e440, 0x2a)
/usr/lib/go/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
/usr/lib/go/src/runtime/sigpanic_unix.go:12 +0x5a
runtime.systemstack(0xc820022000)
/usr/lib/go/src/runtime/asm_amd64.s:295 +0x79
runtime.mstart()
/usr/lib/go/src/runtime/proc.go:1051
goroutine 8 [running]:
runtime.systemstack_switch()
/usr/lib/go/src/runtime/asm_amd64.s:245 fp=0xc82002ceb0 sp=0xc82002cea8
runtime.writebarrierptr_nostore1(0xc8204ba300, 0xc82002cfa0)
/usr/lib/go/src/runtime/mbarrier.go:121 +0x71 fp=0xc82002cee8 sp=0xc82002ceb0
runtime.writebarrierptr(0xc8204ba300, 0xc82002cfa0)
/usr/lib/go/src/runtime/mbarrier.go:143 +0x72 fp=0xc82002cf18 sp=0xc82002cee8
runtime.gopark(0xc8202cc3b8, 0xc82002cfa0, 0x18616a0, 0x10, 0x14, 0x0)
/usr/lib/go/src/runtime/proc.go:255 +0x1b8 fp=0xc82002cf40 sp=0xc82002cf18
runtime.gcBgMarkWorker(0xc82001a000)
/usr/lib/go/src/runtime/mgc.go:1403 +0xda fp=0xc82002cfb8 sp=0xc82002cf40
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc82002cfc0 sp=0xc82002cfb8
created by runtime.gcBgMarkStartWorkers
/usr/lib/go/src/runtime/mgc.go:1330 +0x92
goroutine 1 [runnable]:
strings.Join(0xc820251ce8, 0x2, 0x2, 0x175b6f8, 0x1, 0x0, 0x0)
/usr/lib/go/src/strings/strings.go:369 +0x3c3
path/filepath.join(0xc820251ce8, 0x2, 0x2, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path_unix.go:42 +0xfa
path/filepath.Join(0xc820251ce8, 0x2, 0x2, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:206 +0x41
path/filepath.walk(0xc820523d40, 0xcc, 0x7fd82ab46468, 0xc820523e10, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:367 +0x37a
path/filepath.walk(0xc8205225b0, 0xc4, 0x7fd82ab46468, 0xc820522680, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc82017e000, 0xc2, 0x7fd82ab46468, 0xc82017e0d0, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc8203dcc00, 0xba, 0x7fd82ab46468, 0xc820454d00, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc8203dc840, 0xb2, 0x7fd82ab46468, 0xc820454c30, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc8203efe40, 0xa6, 0x7fd82ab46468, 0xc8203ed110, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc820156640, 0x9c, 0x7fd82ab46468, 0xc820555ad0, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc82031a1e0, 0x96, 0x7fd82ab46468, 0xc820133110, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc8203de3f0, 0x8e, 0x7fd82ab46468, 0xc820133040, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc8203de900, 0x89, 0x7fd82ab46468, 0xc82017c1a0, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc82042de00, 0x7f, 0x7fd82ab46468, 0xc820425380, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc82026fc00, 0x72, 0x7fd82ab46468, 0xc8202201a0, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc82039bf20, 0x5d, 0x7fd82ab46468, 0xc820163ee0, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc8203ac5a0, 0x50, 0x7fd82ab46468, 0xc820163c70, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc8203ac460, 0x4c, 0x7fd82ab46468, 0xc820163040, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc8203ac320, 0x41, 0x7fd82ab46468, 0xc820162f70, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc8203bcd40, 0x3d, 0x7fd82ab46468, 0xc8203f6340, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.walk(0xc82038dc40, 0x31, 0x7fd82ab46468, 0xc8203b1930, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:374 +0x4fc
path/filepath.Walk(0xc82038dc40, 0x31, 0xc8203b2060, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:396 +0xe1
github.com/docker/docker/builder.ValidateContextDirectory(0xc82038d800, 0x31, 0x0, 0x0, 0x0, 0x0, 0x0)
/build/docker/src/docker/.gopath/src/github.com/docker/docker/builder/context.go:68 +0x10b
github.com/docker/docker/api/client.(*DockerCli).CmdBuild(0xc82026fa80, 0xc82000a440, 0x1, 0x1, 0x0, 0x0)
/build/docker/src/docker/.gopath/src/github.com/docker/docker/api/client/build.go:144 +0x1e72
reflect.callMethod(0xc820389e30, 0xc820253c58)
/usr/lib/go/src/reflect/value.go:631 +0x1fc
reflect.methodValueCall(0xc82000a440, 0x1, 0x1, 0x1, 0xc820389e30, 0x0, 0x0, 0xc820389e30, 0x0, 0x47a3c4, ...)
/usr/lib/go/src/reflect/asm_amd64.s:29 +0x36
github.com/docker/docker/cli.(*Cli).Run(0xc820389620, 0xc82000a430, 0x2, 0x2, 0x0, 0x0)
/build/docker/src/docker/.gopath/src/github.com/docker/docker/cli/cli.go:89 +0x384
main.main()
/build/docker/src/docker/docker/docker.go:63 +0x435
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:1998 +0x1
goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
/usr/lib/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/usr/lib/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/usr/lib/go/src/os/signal/signal_unix.go:28 +0x37 |
@fusion809, please open a new issue. Thanks! |
I'm getting this error when I run the cf push command. unexpected fault address 0x2a20eeb8ee7a goroutine 405 [running]: goroutine 1 [select]: goroutine 17 [syscall, locked to thread]: goroutine 19 [syscall]: goroutine 320 [select]: goroutine 319 [IO wait]: |
@vkg70, this bug is old and closed. Please file a new bug if you have a problem, and include all the details we might need to debug. |
Go version: 1.5
Arch: linux/amd64. kernel 4.1.7
stack trace
The text was updated successfully, but these errors were encountered: