segmentation violation

[MorphBonehunter]

• Gitea version (or commit ref): Gitea v1.1.1 built with: bindata, sqlite
• Git version: 2.12.2
• Operating system: Arch Linux
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

Description

Today i upgrade to gitea 1.1.1.
I've used the binary from
https://github.com/go-gitea/gitea/releases/download/v1.1.1/gitea-1.1.1-linux-amd64
First all runs as expected but after pushing to an repo which have an webhook i got the following:

Mai 05 14:27:04 pandora gitea[813]: [Macaron] 2017-05-05 14:27:04: Started HEAD /underverse/arch-packages/tasks/trigger?branch=master&secret=XXX&pusher=1 for 127.0.0.1
Mai 05 14:27:04 pandora gitea[813]: [Macaron] 2017-05-05 14:27:04: Completed /underverse/arch-packages/tasks/trigger?branch=master&secret=XXX&pusher=1 202 Accepted in 4.985499ms
Mai 05 14:27:04 pandora gitea[813]: fatal error: unexpected signal during runtime execution
Mai 05 14:27:04 pandora gitea[813]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x7fa9468d04e3]
Mai 05 14:27:04 pandora gitea[813]: runtime stack:
Mai 05 14:27:04 pandora gitea[813]: runtime.throw(0x13666ae, 0x2a)
Mai 05 14:27:04 pandora gitea[813]:         /usr/local/go/src/runtime/panic.go:596 +0x95
Mai 05 14:27:04 pandora gitea[813]: runtime.sigpanic()
Mai 05 14:27:04 pandora gitea[813]:         /usr/local/go/src/runtime/signal_unix.go:274 +0x2db
Mai 05 14:27:04 pandora gitea[813]: goroutine 2168 [syscall, locked to thread]:
Mai 05 14:27:04 pandora gitea[813]: runtime.cgocall(0xfc3cd0, 0xc421e0ddd8, 0x13649ff)
Mai 05 14:27:04 pandora gitea[813]:         /usr/local/go/src/runtime/cgocall.go:131 +0xe2 fp=0xc421e0dd98 sp=0xc421e0dd58

Since this message gitea is "dead" and could not restarted as it dies on startup:

Mai 05 14:44:34 pandora gitea[5157]: 2017/05/05 14:44:34 [T] Custom path: /usr/bin/custom
Mai 05 14:44:34 pandora gitea[5157]: 2017/05/05 14:44:34 [T] Log path: /var/log/gitea
Mai 05 14:44:35 pandora gitea[5157]: fatal error: unexpected signal during runtime execution
Mai 05 14:44:35 pandora gitea[5157]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x7fa2000224e3]
Mai 05 14:44:35 pandora gitea[5157]: runtime stack:
Mai 05 14:44:35 pandora gitea[5157]: runtime.throw(0x13666ae, 0x2a)
Mai 05 14:44:35 pandora gitea[5157]:         /usr/local/go/src/runtime/panic.go:596 +0x95
Mai 05 14:44:35 pandora gitea[5157]: runtime.sigpanic()
Mai 05 14:44:35 pandora gitea[5157]:         /usr/local/go/src/runtime/signal_unix.go:274 +0x2db
Mai 05 14:44:35 pandora gitea[5157]: goroutine 163 [syscall, locked to thread]:
Mai 05 14:44:35 pandora gitea[5157]: runtime.cgocall(0xfc3cd0, 0xc420947dd8, 0x13649ff)
Mai 05 14:44:35 pandora gitea[5157]:         /usr/local/go/src/runtime/cgocall.go:131 +0xe2 fp=0xc420947d98 sp=0xc420947d58
Mai 05 14:44:35 pandora gitea[5157]: net._C2func_getaddrinfo(0x7fa1f0000c20, 0x0, 0xc420794ea0, 0xc420167fe8, 0x0, 0x0, 0x0)
Mai 05 14:44:35 pandora gitea[5157]:         net/_obj/_cgo_gotypes.go:86 +0x68 fp=0xc420947dd8 sp=0xc420947d98
Mai 05 14:44:35 pandora gitea[5157]: net.cgoLookupIPCNAME.func2(0x7fa1f0000c20, 0x0, 0xc420794ea0, 0xc420167fe8, 0x42d256, 0x0, 0x0)
Mai 05 14:44:35 pandora gitea[5157]:         /usr/local/go/src/net/cgo_unix.go:151 +0x171 fp=0xc420947e38 sp=0xc420947dd8
Mai 05 14:44:35 pandora gitea[5157]: net.cgoLookupIPCNAME(0xc42077d540, 0x11, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Mai 05 14:44:35 pandora gitea[5157]:         /usr/local/go/src/net/cgo_unix.go:151 +0x1bd fp=0xc420947f38 sp=0xc420947e38
Mai 05 14:44:35 pandora gitea[5157]: net.cgoIPLookup(0xc4201624e0, 0xc42077d540, 0x11)
Mai 05 14:44:35 pandora gitea[5157]:         /usr/local/go/src/net/cgo_unix.go:203 +0x4d fp=0xc420947fc8 sp=0xc420947f38
Mai 05 14:44:35 pandora gitea[5157]: runtime.goexit()
Mai 05 14:44:35 pandora gitea[5157]:         /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420947fd0 sp=0xc420947fc8
Mai 05 14:44:35 pandora gitea[5157]: created by net.cgoLookupIP
Mai 05 14:44:35 pandora gitea[5157]:         /usr/local/go/src/net/cgo_unix.go:213 +0xb4
Mai 05 14:44:35 pandora gitea[5157]: goroutine 1 [runnable]:
Mai 05 14:44:35 pandora gitea[5157]: compress/flate.(*decompressor).huffmanBlock(0xc4207c6c00)
Mai 05 14:44:35 pandora gitea[5157]:         /usr/local/go/src/compress/flate/inflate.go:477 +0xc86
Mai 05 14:44:35 pandora gitea[5157]: compress/flate.(*decompressor).nextBlock(0xc4207c6c00)
Mai 05 14:44:35 pandora gitea[5157]:         /usr/local/go/src/compress/flate/inflate.go:326 +0x1ea

[lunny]

@tboerger have no idea about this, maybe a problem of xgo?

[tboerger]

I think it's a duplicate of #1408. Please try to launch it with the environment variable GODEBUG=netdns=go mentioned at #1408 (comment)

Maybe somebody wants to contribute an Arch package to avoid this problem?

[MorphBonehunter]

@lunny @tboerger i can reproduce this behavior.
I've done an new installation, setup users, orgs and repositorys and push my repos to new gitea installation.
This works as expected.
After that i activate drone on one of my repos and push to it...it breaks instant:

Mai 05 16:29:40 pandora gitea[2243]: [Macaron] 2017-05-05 16:29:40: Started HEAD /underverse/arch-packages/tasks/trigger?branch=master&secret=aae4e1615d1242bf95051b9db4766d24&pusher=1 for 127.0.0.1
Mai 05 16:29:40 pandora gitea[2243]: [Macaron] 2017-05-05 16:29:40: Completed /underverse/arch-packages/tasks/trigger?branch=master&secret=aae4e1615d1242bf95051b9db4766d24&pusher=1 202 Accepted in 11.177545ms
Mai 05 16:29:40 pandora gitea[2243]: fatal error: unexpected signal during runtime execution
Mai 05 16:29:40 pandora gitea[2243]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x7fdf430d14e3]
Mai 05 16:29:40 pandora gitea[2243]: runtime stack:
Mai 05 16:29:40 pandora gitea[2243]: runtime.throw(0x13666ae, 0x2a)
Mai 05 16:29:40 pandora gitea[2243]:         /usr/local/go/src/runtime/panic.go:596 +0x95
Mai 05 16:29:40 pandora gitea[2243]: runtime.sigpanic()
Mai 05 16:29:40 pandora gitea[2243]:         /usr/local/go/src/runtime/signal_unix.go:274 +0x2db
Mai 05 16:29:40 pandora gitea[2243]: goroutine 512 [syscall, locked to thread]:
Mai 05 16:29:40 pandora gitea[2243]: runtime.cgocall(0xfc3cd0, 0xc420687dd8, 0x13649ff)
Mai 05 16:29:40 pandora gitea[2243]:         /usr/local/go/src/runtime/cgocall.go:131 +0xe2 fp=0xc420687d98 sp=0xc420687d58
Mai 05 16:29:40 pandora gitea[2243]: net._C2func_getaddrinfo(0x7fdf44024480, 0x0, 0xc421b24c00, 0xc421926f60, 0x0, 0x0, 0x0)
Mai 05 16:29:40 pandora gitea[2243]:         net/_obj/_cgo_gotypes.go:86 +0x68 fp=0xc420687dd8 sp=0xc420687d98
Mai 05 16:29:40 pandora gitea[2243]: net.cgoLookupIPCNAME.func2(0x7fdf44024480, 0x0, 0xc421b24c00, 0xc421926f60, 0x42d256, 0xc42046eea0, 0xc420687ea8)
Mai 05 16:29:40 pandora gitea[2243]:         /usr/local/go/src/net/cgo_unix.go:151 +0x171 fp=0xc420687e38 sp=0xc420687dd8
Mai 05 16:29:40 pandora gitea[2243]: net.cgoLookupIPCNAME(0xc421b22760, 0x11, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Mai 05 16:29:40 pandora gitea[2243]:         /usr/local/go/src/net/cgo_unix.go:151 +0x1bd fp=0xc420687f38 sp=0xc420687e38
Mai 05 16:29:40 pandora gitea[2243]: net.cgoIPLookup(0xc421970f00, 0xc421b22760, 0x11)
Mai 05 16:29:40 pandora gitea[2243]:         /usr/local/go/src/net/cgo_unix.go:203 +0x4d fp=0xc420687fc8 sp=0xc420687f38
Mai 05 16:29:40 pandora gitea[2243]: runtime.goexit()
Mai 05 16:29:40 pandora gitea[2243]:         /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420687fd0 sp=0xc420687fc8
Mai 05 16:29:40 pandora gitea[2243]: created by net.cgoLookupIP
Mai 05 16:29:40 pandora gitea[2243]:         /usr/local/go/src/net/cgo_unix.go:213 +0xb4
Mai 05 16:29:40 pandora gitea[2243]: goroutine 1 [select, 4 minutes]:
Mai 05 16:29:40 pandora gitea[2243]: code.gitea.io/gitea/vendor/github.com/facebookgo/grace/gracehttp.Serve(0xc42062faf8, 0x1, 0x1, 0xa85531, 0x123ba60)
Mai 05 16:29:40 pandora gitea[2243]:         /srv/app/src/code.gitea.io/gitea/vendor/github.com/facebookgo/grace/gracehttp/http.go:162 +0x5e6
Mai 05 16:29:40 pandora gitea[2243]: code.gitea.io/gitea/cmd.runHTTP(0xc4214c9600, 0xc, 0x1c6eec0, 0xc4214de6c0, 0x0, 0xc4214c9600)
Mai 05 16:29:40 pandora gitea[2243]:         /srv/app/src/code.gitea.io/gitea/cmd/web_graceful.go:21 +0xc7
Mai 05 16:29:40 pandora gitea[2243]: code.gitea.io/gitea/cmd.runWeb(0xc42042c140, 0x0, 0xc42042c100)
Mai 05 16:29:40 pandora gitea[2243]:         /srv/app/src/code.gitea.io/gitea/cmd/web.go:676 +0x1cf1
Mai 05 16:29:40 pandora gitea[2243]: code.gitea.io/gitea/vendor/github.com/urfave/cli.HandleAction(0x11d6cc0, 0x137e850, 0xc42042c140, 0xc4203b4c00, 0x0)

[MorphBonehunter]

@tboerger after reading you post i have to think about this one: harness/harness#1943
Indeed, after adding the line InaccessiblePaths=/etc/nsswitch.conf in my service file, gittea is starting again and the drone integration works.
So with this harness/harness#1765, i'm not sure if building gitea on my arch from source make it better (BTW is build an PKGBUILD based on the bin files released here).
I also try your GODEBUG=netdns=go and this does also work.

[tboerger]

Looks like we have to build the binary with the build tag netgo, because the default netcgo seems to fail on some systems. It have been introduced with golang/go@b615ad8

[lunny]

It seems many of this-like issues are reported from Arch Linux users?

[MorphBonehunter]

@lunny maybe 😄 seem "freshness" isn't always the best.
But in this particular case i think this problem could hit also users of other distros.
I will rewrite my PKGBUILD and compile it on my own, this should fix the problem but i think considering @tboerger netgo suggestion could also be valid for fixing the downloadable binaries.

[tboerger]

I have created the pull request #1690 which should solve this issue as it enforces the Go name resolution instead of CGO for cross-compiled binaries.

[MorphBonehunter]

@tboerger sorry if i stress this topic further but after suggesting this possible fix in the drone project, @bradrydzewski told me, that the project use the netgo build tag already.
As i could not find something about this in the drone Makefile/build commands, i google a little bit to understand the stuff with the netgo and found the go 1.5 release notes:

The DNS resolver in the net package has almost always used cgo to access the system interface. A change in Go 1.5 means that on most Unix systems DNS resolution will no longer require cgo, which simplifies execution on those platforms. Now, if the system's networking configuration permits, the native Go resolver will suffice. The important effect of this change is that each DNS resolution occupies a goroutine rather than a thread, so a program with multiple outstanding DNS requests will consume fewer operating system resources.

The decision of how to run the resolver applies at run time, not build time. The netgo build tag that has been used to enforce the use of the Go resolver is no longer necessary, although it still works. A new netcgo build tag forces the use of the cgo resolver at build time. To force cgo resolution at run time set GODEBUG=netdns=cgo in the environment. More debug options are documented here.

This change applies to Unix systems only. Windows, Mac OS X, and Plan 9 systems behave as before.

So maybe i understand this wrong as i'm not an developer, but it reads as is netdns=go the default and can changed at runtime to cgo.
Also in Code https://github.com/golang/go/blob/b615ad8fd57f9394db14e403d12061c369379c52/src/net/conf.go#L22 it reads that this is the default.

After reading those, i'm a litte bit confused about this as your hint to use GODEBUG=netdns=go works and suggests netdns=go isn't the default. If this is true, how doe's this then compare to the statement that drone use the already the netgo build tag (but i have to use my systemd workaround with InaccessiblePaths=/etc/nsswitch.conf to get it work).

I know, this is not relevant for this project, but as gitea "co-operate" with drone i would be pleased if my question could answered so that i could understand this topic 🤔

[lunny]

@MorphBonehunter could you confirm #1690 could resolve your problem?

[MorphBonehunter]

@lunny for beeing close to your release mechanism, i do the following steps (based on project drone.yml):

build

docker run --rm -it -e 'TAGS=bindata sqlite' -e 'GOPATH=/srv/app' -v /home/daniel/repos/verschiedenes/:/srv/app/src/code.gitea.io/:rw webhippie/golang:edge

apk -U add openssh-client \
&& git config --global user.email "[email protected]" \
&& git config --global user.name "Your Name" \
&& cd /srv/app/src/code.gitea.io/ \
&& git clone https://github.com/go-gitea/gitea.git \
&& cd gitea \
&& git checkout v1.1.1 \
&& git cherry-pick c864ccf9b1414dfdae1fd271511853e058b9e7c9 \
&& make clean \
&& make generate \
&& make vet \
&& make lint \
&& make build

release

docker run --rm -it -e 'TAGS=bindata sqlite' -e 'GOPATH=/srv/app' -e 'CI=drone' --entrypoint=/bin/bash -v /home/daniel/repos/verschiedenes/:/srv/app/src/code.gitea.io/:rw karalabe/xgo-latest:latest

cd /srv/app/src/code.gitea.io/gitea/ \
&& make release

The produced dist/release/gitea-master-linux-amd64 works on my Arch without any error until now.
So yes, i can confirm the problem is fixed for me (although i'm not fully understand this...see comment above 😃 ).
Thanks @ALL!

[tboerger]

As far as I can say it's related to xgo (the tool we are using for cross-compiling). Since this is a tool for cross-compiling programs with CGO dependencies I can just imagine that it enables the CGO name resolver which crashs in your case. So really enforcing the Go name resolver seems to fix this problem on the supported platforms.