runtime: mayMoreStackPreempt tests flaky after stack frame CLs

[prattmic]

greplogs --dashboard -md -l -e 'SignalInVDSO'

2022-09-05T21:39:28-4c1ca42/linux-386-longtest
2022-09-05T08:28:34-bd5595d/linux-386-longtest
2022-09-05T08:08:24-af7f417/linux-386-longtest
2022-09-05T08:08:18-4ad55cd/linux-386-longtest
2022-09-05T08:07:47-357b922/linux-386-longtest
2022-09-05T07:17:56-4e7e7ae/linux-386-longtest
2022-09-05T07:14:08-3fbcf05/linux-386-longtest
2022-09-04T04:17:04-535fe2b/linux-386-longtest
2022-09-03T18:21:45-a0f0582/linux-386-longtest
2022-09-03T15:45:36-f798dc6/linux-386-longtest
2022-09-02T19:22:26-0fda8b1/linux-386-longtest
2022-09-02T19:09:03-55ca6a2/linux-386-longtest
2022-09-02T19:08:56-b91e373/linux-386-longtest
2022-09-02T19:08:53-dbf442b/linux-386-longtest

##### maymorestack=mayMoreStackPreempt
--- FAIL: TestVDSO (0.43s)
    testenv.go:468: [/workdir/tmp/go-build2185756284/testprog.exe SignalInVDSO] exit status: signal: segmentation fault (core dumped)
    crash_test.go:144: output:
        
        
        wanted:
        success
FAIL
FAIL	runtime	95.198s
FAIL

The first failure occurred on https://go.dev/cl/424514. It is flaky, so it may not be precisely that CL at fault, but it seems likely that one of the stack frame CLs is to blame.

[aclements]

I think this also affects linux-amd64-longtest and fails in other mayMoreStackPreempt tests as well:

greplogs -dashboard -md -l -E '(?s)mayMoreStackPreempt.*segmentation fault'
2022-09-06T11:14:09-1c50484/linux-amd64-longtest
2022-09-05T21:39:28-4c1ca42/linux-386-longtest
2022-09-05T21:39:28-4c1ca42/linux-amd64-longtest
2022-09-05T08:28:34-bd5595d/linux-386-longtest
2022-09-05T08:12:37-67e6542/linux-amd64-longtest
2022-09-05T08:08:24-af7f417/linux-386-longtest
2022-09-05T08:08:18-4ad55cd/linux-386-longtest
2022-09-05T08:07:47-357b922/linux-386-longtest
2022-09-05T08:07:47-357b922/linux-amd64-longtest
2022-09-05T07:17:56-4e7e7ae/linux-386-longtest
2022-09-05T07:14:08-3fbcf05/linux-386-longtest
2022-09-04T04:17:04-535fe2b/linux-386-longtest
2022-09-04T04:17:04-535fe2b/linux-amd64-longtest
2022-09-03T18:21:45-a0f0582/linux-386-longtest
2022-09-03T18:21:45-a0f0582/linux-amd64-longtest
2022-09-03T15:45:36-f798dc6/linux-386-longtest
2022-09-03T15:45:36-f798dc6/linux-amd64-longtest
2022-09-02T20:14:40-3e11e61/linux-amd64-longtest
2022-09-02T20:14:16-646c3ee/linux-amd64-longtest
2022-09-02T19:40:34-9154d4a/linux-amd64-longtest
2022-09-02T19:38:45-e406112/linux-amd64-longtest
2022-09-02T19:22:26-0fda8b1/linux-386-longtest
2022-09-02T19:09:03-55ca6a2/linux-386-longtest
2022-09-02T19:08:56-b91e373/linux-386-longtest
2022-09-02T19:08:56-b91e373/linux-amd64-longtest
2022-09-02T19:08:53-dbf442b/linux-386-longtest
2022-09-02T19:08:53-dbf442b/linux-amd64-longtest

[aclements]

Quick reproducer:

cd runtime/testdata/testprogcgo
go build -gcflags=runtime=-d=maymorestack=runtime.mayMoreStackPreempt
./testprogcgo CgoPprofCallback

This fails ~25% on linux/amd64 at tip.

(Note that, relative to what dist passes, -asmflags isn't necessary, and we only need to cover the runtime package itself, not the several other packages that dist does.)

[aclements]

Bisecting points at 511cd9b (CL 424257), which is a bit surprising because I was pretty sure that couldn't affect any existing logic. I'll dig into it.

[aclements]

I found the cause:

1. We force a preemption on g1. (This could happen in any test, but mayMoreStackPreempt makes it way more likely, which is great, because it means that test mode is doing what it's supposed to!)
2. The preemption goes into the scheduler, which picks another G, g2, to run and enters execute.
3. execute sets mp.curg = g2, but hasn't yet set g2.m = mp.
4. A profiling signal comes in and starts a stack trace with _TraceJumpStack.
5. Traceback sees the morestack from the preemption, and sees that curg != nil, so it follows curg into g2. This is a little weird because g2 hasn't really started running yet, but should be harmless.
6. On the next frame, gentraceback tries to check gp.m.g0, but g2.m is nil because g2 is not quite running yet, so it segfaults. Since we're in a signal, the process goes down hard.

[gopherbot]

Change https://go.dev/cl/428656 mentions this issue: runtime: in traceback, only jump stack if M doesn't change

[mknyszek]

There are a number of related failures from this week's triage, but AFAICT they were all from more than 6 hours ago.