runtime: frequent crashes with "exit status 16962" on netbsd-arm64-bsiegert builder #42422
Comments
|
/cc @bsiegert |
bcmills
added
arch-arm64
OS-NetBSD
NeedsInvestigation
|
@bsiegert: which version of netbsd/arm64 is this running? |
|
It's a -current from May: I can try a newer kernel if you would like. |
The code in question specifically calls |
bcmills
changed the title
|
Change https://golang.org/cl/375834 mentions this issue: |
Instead of exiting with status 16962 when we fail to call SYS_setcontext in sigreturn, reference a null pointer and crash. Hopefully, this will enable grabbing a core dump to debug. Updates #42422 Change-Id: If02c14a0a37084351f3f00db3dc9766cb68ae4b8 Reviewed-on: https://go-review.googlesource.com/c/go/+/375834 Reviewed-by: Tobias Klauser <[email protected]> Trust: Tobias Klauser <[email protected]> Trust: Benny Siegert <[email protected]>
|
The thing that's surprising to me is: I actually use Go on netbsd-arm64 productively. My main development laptop is a Pinebook Pro. And I have built Go a number of times, but I have not seen the "exit 16962" crash a single time. Maybe it does have something to do with the builder machine. |
|
Can you turn off ASLR and see if that fixes the problem? |
|
I turned off ASLR by setting I'll keep watching the build results. FWIW, the Pinebook Pro has been continuously building Go HEAD for netbsd-arm for the last 24 hours, without a single crash. |
|
So has it failed since? |
|
I'll keep looking :-) |
|
The failures are typically early in the bootstrap phase, which means they are in binaries built with Go 1.14.2. I did the following two things:
|
|
I think your builder updates have resolved this:
|
|
@aclements, are you also fetching subrepos? This failure mode is definitely lower, but I'm still seeing it in 2 out of 3 failures after the configuration change:
2022-02-11T22:01:05-badba35/netbsd-arm64-bsiegert ( 2022-02-09T21:30:12-5430f30-0a6cf87/netbsd-arm64-bsiegert |
|
@golang/netbsd, any forward progress on this? |
|
Sorry, I am too busy to try my theory (from above). Perhaps next month. |
|
A large crop of these yesterday. @bsiegert, did something odd happen with the builder yesterday? Maybe that's a clue.
|
|
Ah, no. That large crop is on Should we just turn this builder off for that branch? |
|
Yes, that seems like a good idea. |
Instead of exiting with status 16962 when we fail to call SYS_setcontext in sigreturn, reference a null pointer and crash. Hopefully, this will enable grabbing a core dump to debug. Updates golang#42422 Change-Id: If02c14a0a37084351f3f00db3dc9766cb68ae4b8 Reviewed-on: https://go-review.googlesource.com/c/go/+/375834 Reviewed-by: Tobias Klauser <[email protected]> Trust: Tobias Klauser <[email protected]> Trust: Benny Siegert <[email protected]>
gopherbot
added
the
compiler/runtime
|
Found new dashboard test flakes for: 2022-10-12 19:16 netbsd-arm64-bsiegert go@4a4de141 (log)2022-10-12 23:00 netbsd-arm64-bsiegert go@19095e10 (log) |
|
@bsiegert, should we configure this builder with GoBootstrap: "none" in order to avoid this failure mode? (Does the builder have a bootstrap toolchain installed that is newer than 1.17.13?)Message ID: ***@***.***>
Does the builder do anything special to ensure a certain bootstrap version? What does GoBootstrap=none imply?There are two arm64 builder machines, called fluebrig and rpi4. On rpi4, bootstrap is with Go 1.19, I haven't had time to make the same change on the other box. As far as I can tell, none of these errors occur with Go 1.19 on bootstrap.
|
|
If the problem is Go 1.17.13, then we can have the coordinator install something newer. |
|
Installing the latest 1.18 would fix this.
|
|
Change https://go.dev/cl/442855 mentions this issue: |
|
Change https://go.dev/cl/442787 mentions this issue: |
The NetBSD builders are failing in a way that only manifests with Go 1.17. The OpenBSD builders are also very flaky. Update both operating systems to Go 1.19.2 in hopes of eliminating those flakes. For golang/go#42422. Change-Id: I10a736c12b38924f6fed5269c2cde21c16de4887 Reviewed-on: https://go-review.googlesource.com/c/build/+/442787 Run-TryBot: Russ Cox <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> TryBot-Result: Gopher Robot <[email protected]>
|
Found new dashboard test flakes for: 2022-10-12 20:23 netbsd-arm64-bsiegert go@987f94fa (log)2022-10-13 15:31 netbsd-arm64-bsiegert go@4a0ce469 (log)2022-10-13 20:22 netbsd-arm64-bsiegert go@005c78d8 (log)2022-10-13 21:14 netbsd-arm64-bsiegert go@c79b2009 (log)2022-10-14 12:48 netbsd-arm64-bsiegert go@a4b4717f (log) |
|
No |
bcmills
added
the
WaitingForInfo
|
Yes, ever since the bootstrap was bumped to 1.18, this has not been a problem any more. |
github-project-automation
bot
moved this from Triage Backlog
to Done
in Go Compiler / Runtime
The netbsd/arm64 builder regularly fails with a process (usually
compile) exiting withexit status 16962- some examples:https://build.golang.org/log/21955ac5c870203c1e6ff0c9c4f3437c10778dbf
https://build.golang.org/log/f58d31621ecf6eb5317790d7683479bb44b9561d
https://build.golang.org/log/96764e29f501a683cccf7006af49b865d0e2f133
16962 is 0x4242 hexadecimal, which seems to suggest that
sigreturn_trampis failing to callsetcontext:https://github.com/golang/go/blob/master/src/runtime/sys_netbsd_arm64.s#L275
The text was updated successfully, but these errors were encountered: