runtime: macOS Sierra builders spinning

[bradfitz]

I finally (sorry) set up macOS Sierra VMs last week.

They keep hanging in infinite loops. I just attached to one of them to see what's going on:

The go_bootstrap install -v std cmd command is spinning.

Using dtrace -p <pid>, I see an endless stream of:

select(0x0, 0x0, 0x0, 0x0, 0xC82003DF00) = 0 0
select(0x0, 0x0, 0x0, 0x0, 0xC82003DF00) = 0 0
select(0x0, 0x0, 0x0, 0x0, 0xC82003DF00) = 0 0
select(0x0, 0x0, 0x0, 0x0, 0xC82003DF00) = 0 0
select(0x0, 0x0, 0x0, 0x0, 0xC82003DF00) = 0 0

The GOROOT_BOOTSTRAP is the Go 1.4 branch with the Sierra fixes. (it barely makes it a second into the build otherwise)

The VMs are running under VMWare ESXi, just like all the OS X 10.8, 10.10, and 10.11 builders, which are all fine.

[bradfitz]

(Other comment deleted. The SIGQUIT shown in the deleted post was the buildlet process, not the spinning one.)

[josharian]

Probably unrelated, but just in case...yesterday, the net/http tests starting hanging on my laptop, and required a reboot to get passing again. The test that was hanging was TestTransportPersistConnReadLoopEOF. I'm afraid I don't have the stack traces anymore. Again, probably unrelated, many of the net/http benchmarks fail on Sierra with 'too many open files'.

[bradfitz]

@josharian, this is just go install, so we're not even running tests yet. But I could imagine the highly-parallel net/http tests using lots of FDs. Please open a separate bug for that.

[aclements]

@bradfitz, can you kill -QUIT the go_boostrap process directly to get a traceback? Or if that doesn't work maybe send SIGQUIT to the whole make.bash process group (like ^[ at the terminal would).

[bradfitz]

I did kill -QUIT the go_bootstrap process directly to no avail, several times. It didn't appear to quit. Maybe that's telling in itself.

I then also tried a Ctrl-\ at parent (the buildlet) and got a backtrace of only the buildlet (backtrace paths were like /home/bradfitz instead of the expected /Users/gopher)

Trying again now.

[JayNakrani]

Possibly related: #17161 . Infinite retry was added to solve that issue. From the second stacktrace, it looks like it may be going into that infinite retry loop.

[aclements]

It didn't appear to quit. Maybe that's telling in itself.

That's too bad. A few other things come to mind: Does it freeze even if you run a trivial subcommand like, say, go_bootstrap version? Can you dtrace go_bootstrap from the start? Can this be bisected?

Possibly related: #17161 . Infinite retry was added to solve that issue. From the second stacktrace, it looks like it may be going into that infinite retry loop.

I would expect to see Mach semaphore calls in the dtrace if this were the case, though maybe those are at the wrong level in the OS X kernel for dtrace? (I've always found this confusing.) The second traceback was just from the buildlet, so I don't think it's related. That's also exactly what I would expect a normal parked P's stack to look like.

[ianlancetaylor]

Can you add a -x to the go_bootstrap command near the end of make.bash so that we can see where it is hanging?

In case it's not obvious the select calls are from calls to runtime.usleep, but unfortunately we don't know the duration of the sleep so we don't know precisely which call is looping. It could just be the sysmon thread.

[quentinmit]

@aclements dtrace only shows traditional BSD-style syscalls, not Mach calls.