CockroachDB hangs under hermit

[srosenberg]

Describe the bug

Running cockroach appears to hang,

./hermit --log=trace --log-file=hermit.log run ./cockroach version
WARNING: --preemption-timout requires hardware perf counters which is not supported on this host, resetting preemption-timeout to 0
^C
thread 'main' panicked at 'Deadlock detected: thread(s) waiting on futex, but no runnable threads left.
 queue: RunQueue { queue: {}, last_back_turn: 1310, last_front_turn: -2, tentative_selection: None, sched_strategy: None, prng: Mcg128Xsl64 {}, sticky_random_param: 0.0, sticky_random_selection: None }
  next_turns: {DetPid(3): ThreadNextTurn { dettid: DetPid(3), child_tid_addr: 0, req: Ivar { inner: Mutex { data: Shared { contents: Some(Ok(Resources { tid: DetPid(3), resources: {FutexWait: R}, poll_attempt: 0, fyi: "" })), waiter: None }, poisoned: false, .. } }, resp: Ivar { inner: Mutex { data: Shared { contents: None, waiter: Some(Waker { data: 0x55df4302e2f0, vtable: 0x55df427d2900 }) }, poisoned: false, .. } } }}
  blocked: BlockedPool { futex_waiters: {(DetPid(3), 162650784): [], (DetPid(3), 163037944): [], (DetPid(3), 824638579016): [], (DetPid(3), 824634362184): [], (DetPid(3), 162655624): [(DetPid(3), Ivar { inner: Mutex { data: Shared { contents: None, waiter: Some(Waker { data: 0x55df4302e2f0, vtable: 0x55df427d2900 }) }, poisoned: false, .. } })], (DetPid(3), 163038336): []}, timed_waiters: TimedEvents { map: {}, alarm_times: {} }, external_io_blockers: {} }
', detcore/src/scheduler.rs:1498:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }', detcore/src/tool_global.rs:822:14
fatal runtime error: failed to initiate panic, error 5

Indicate any of these common scenarios that apply:

• a program hangs under hermit

To Reproduce
Minimal input to reproduce the behavior.

Install Nightly

wget -O cockroach https://storage.googleapis.com/cockroach-edge-artifacts-prod/cockroach/cockroach.linux-gnu-amd64.225c06dffe37b2db3913fd73d195cdf5c507427a && chmod u+x cockroach

Verify

./cockroach version
Build Tag:        v22.2.0-beta.2-972-g225c06dffe
Build Time:       2022/12/02 19:14:37
Distribution:     CCL
Platform:         linux amd64 (x86_64-pc-linux-gnu)
Go Version:       go1.19.1
C Compiler:       gcc 6.5.0
Build Commit ID:  225c06dffe37b2db3913fd73d195cdf5c507427a
Build Type:       development

Run Hermit

./hermit --log=trace --log-file=hermit.log run ./cockroach version

Expected behavior
A clear and concise description of what you expected to happen.

Environment

• Linux kernel version (uname -a): Linux gceworker-srosenberg 5.13.0-1021-gcp #25~20.04.1-Ubuntu SMP Thu Mar 17 04:09:01 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• CPU version (/proce/cpuinfo): GCE's custom-24-32768
• Linux distro flavor (/etc/issue, /etc/redhat-release): Ubuntu 20.04.3 LTS \n \l

Additional context
Attach the logs to this issue as a text file generated by hermit --log=trace --log-file=FOO run.

hermit.log.gz