Remove the hang guard that killed a live worker and named no test

[leaf-agent]

Problem

Run 32954830645 reached 99% of the suite, tripped the 180s faulthandler guard, killed gw1, and then sat silent for twenty-five minutes until the step's 45-minute bound ended the job. The guard bought a stack dump that named no test, and gave the rest of the budget back.

It cannot name one. Under the sync Playwright API a test blocked on the browser has its own frames in a suspended greenlet, and faulthandler dumps OS threads — so the trip printed the page server's thread, execnet's reader, and Playwright's event loop, and nothing else. Exiting the worker is then worse than doing nothing: xdist clones a replacement, and under --dist loadgroup the clone never picks the run back up.

What it fired on was probably not a hang. The suite's slowest browser sweeps run 60–105s on four cores at -n 2 — the shape of machine CI runs on, at the same wall time — so a three-minute clock sits 1.7× above work every run does, and one noisy runner spends that margin.

Solution

Remove faulthandler_timeout and faulthandler_exit_on_timeout. Both things the guard was meant to buy are already held elsewhere, by machinery that was not added for it:

• Naming a stalled browser wait. Playwright's own 30-second timeouts fail it with the test's nodeid. That is the shape of the three other recent ci failures on main — each one FAILED tests/test_render_projection.py::test_a_message_reference_travels_or_says_it_cant - TimeoutError, named, at ordinary cost.
• Making a true wedge visible. The workflow's 45-minute step bound from Bound the suite so a wedged run fails instead of vanishing #30 is what turns it into the failure tend-ci-fix answers, and that is the fault Bound the suite so a wedged run fails instead of vanishing #30 was actually about.

Net two lines fewer than main, and nothing added.

What the dump actually printed

Timeout (0:03:00)!
Thread 0x00007f9be7fff6c0 (most recent call first):
  File "/usr/lib/python3.12/socketserver.py", line 235 in serve_forever
  ...
Thread 0x00007f9c057ff6c0 (most recent call first):
  File ".../execnet/gateway_base.py", line 1160 in _thread_receiver
  ...
Thread 0x00007f9c0694f080 (most recent call first):
  File ".../playwright/sync_api/_context_manager.py", line 56 in greenlet_main
....[gw1] node down: Not properly terminated
##[error]The action '🧪 Test suite' has timed out after 45 minutes.

Three threads, no test frame among them: the page's dev server, execnet's reader, and the driver greenlet the main thread was sitting in.

What this gives up

One class the guard could have named, had it ever fired there: a wait with no timeout of its own outside the browser. The cold-cache launcher runs in test_interact_layer.py shell out to bin/leaf and let uv resolve through the network with no timeout=. Those have never hung, and the direct fix if they ever do is a timeout= on the call — which names the test and raises in place, rather than exiting a worker the run cannot continue without. Not doing that here; no evidence asks for it yet.

The pytest>=9.1 floor was raised in aef67af solely for faulthandler_exit_on_timeout support. Left alone — uv.lock pins 9.x either way, and lowering it would be a guess at a version the suite has not been run on.

Testing

uv run pytest tests — 574 passed, 6 skipped in 141.73s. pre-commit passes on the changed file. This PR's own ci run exercises the config over the full --run-nightly suite.

[max-sixty]

can we instead try removing the faulthandler configs? might that help? I'm loath to add more configs

[leaf-agent]

Yes — it helps more than the tuning did, and I've replaced the PR with it. Pushed as 3337041: both faulthandler lines gone, --max-worker-restart=0 dropped too, so it's now two lines fewer than main and nothing added.

Checking the premise turned up a better reason than the config count. The guard cannot name a stalled browser test. Under the sync Playwright API the test's own frames sit in a suspended greenlet, and faulthandler dumps OS threads — so the trip on main printed the page server's thread, execnet's reader, and Playwright's greenlet_main, and nothing else. Exiting the worker after that is worse than doing nothing: xdist clones a replacement and under --dist loadgroup the clone never picks the run back up, which is the twenty-five silent minutes.

Both things it was meant to buy are already held, by machinery that wasn't added for it. Playwright's own 30-second timeouts fail a stalled browser wait with the nodeid — the three other recent ci failures on main are all exactly that shape, named and cheap. And #30's 45-minute step bound is what makes a true wedge a failure tend-ci-fix answers, which was the fault #30 was about.

One thing it gives up, in the PR body rather than here: the untimed cold-cache launcher runs in test_interact_layer.py shell out to bin/leaf over the network with no timeout=, and faulthandler would have named those. They've never hung, and the fix if they do is a timeout= on the call rather than a clock over the whole suite, so I've left them.

Also noting rather than changing: pytest>=9.1 was raised in aef67af only for faulthandler_exit_on_timeout. uv.lock pins 9.x regardless, so lowering it would be a guess at a version the suite hasn't run on.

[leaf-agent]

The red ci on 3337041 says nothing about this change — no step ran. Both jobs concluded cancelled with an empty runner_name, zero recorded steps, and one annotation: The job was not acquired by Runner of type hosted even after multiple attempts. They sat 15 minutes and were reaped without ever being assigned a machine.

It's a GitHub-side runner-allocation problem, not one of ours, and it's hitting the repo more widely: main's ci on e0cd607 concluded startup_failure in the same window, and publish-site has been queued 40+ minutes. I've re-queued this run's jobs; it's still waiting on a runner, so the verdict here will arrive whenever allocation recovers. Nothing to do but re-run if it reaps again.

Evidence

• Run 32984346543 — conclusion: failure, both jobs cancelled, started_at 15:11:54Z → completed_at 15:26:56Z, runner_name: "", steps: [].
• Check-run annotation on lint (98228236219): The job was not acquired by Runner of type hosted even after multiple attempts.
• Run 32984483627 — ci on main, startup_failure, 15:14:37Z.
• Jobs are ubuntu-24.04 across all 14 job definitions in .github/workflows/, and the tend-* runs on the same label acquired normally through this window — so it's intermittent allocation, not the label.