Read a response body only once the browser has finished delivering it

[leaf-agent]

Problem

Three ci runs on main have now spent the whole 45-minute step bound and left no failure to read: 33130006054 (45de22f), 33130455313 (d4472ad), 33131937176 (caa8460). Under the -v #81 added, each names a different test that a worker picked up and never reported — test_a_resolved_thread_can_be_reopened, test_a_sent_comment_is_revealed_in_the_panel, test_an_answer_carrying_an_older_pick_cannot_undo_a_newer_one — so the wedge is not a test's, and the other worker stops minutes later in a module of its own.

It is Traffic. _responded queues a response when its headers arrive, and settle then reads the body with response.json(). That call waits on the request's finished fact and takes no timeout in any binding. A page that abandons an answer it no longer wants leaves a request Chromium never reports finished, and the read then blocks for as long as the job lives. It is the one wait in this harness that cannot run out: _until's own 30-second deadline guards wait_for_event and never gets a turn, so the worker stops inside a helper every render test calls, with no nodeid, no traceback and no Playwright timeout.

This is measured, not inferred — the full suite was run locally under a call tracer and both workers were caught inside Response.json, unmoved across two snapshots a minute apart (below).

Solution

Settle a response only once requestfinished says the browser holds the whole body, and wake _until on that same fact rather than on arriving headers — a body is what the counters are read from, and that event is the browser saying it has one. settle keeps an unfinished response queued instead of reading it, so the shape that used to stop the run now lets _until reach its deadline and print its counters, which is the naming a wedge could never do.

The counters themselves are untouched: acked and heard still move on the response event, so what a trip means to round_trip is what it meant before.

This is at a different site from #84, which bounds page.evaluate's frame promises on the same three runs' evidence and says of itself that it does not prove which wait wedged them. The two do not overlap in render_harness.py, and #84 still closes a real unbounded-wait class — it is just not the one that stopped these runs.

Testing

uv run pytest tests --run-nightly — 1357 passed, 6 skipped in 19:36, on the machine that had just wedged twice in a row at 65% and 91% without the change. Rebased onto dd42ddb: ruff check, ruff format, tests/test_render_conversations.py and tests/test_render_reactions.py (48 passed), and the three traffic-wait gates.

Bug-back: with settle restored to its unconditional drain, test_traffic_leaves_a_body_the_browser_has_not_finished_handing_over fails with a body was read before the browser had all of it.

How the blocking call was identified

The suite was run as CI runs it (uv run pytest tests --run-nightly -v) with a plugin wrapping the Playwright sync surface — Page.close, Page.evaluate, Page.wait_for_function, Response.body/json/text, Locator.*, Mouse.*, Keyboard.* — each writing the call it is inside to a per-worker file. A watchdog snapshotted those files after the log had been silent for 150 seconds, and again a minute later:

--- snapshot A
30159 1787884275.442 ENTER Response.json stack=['Response.json']
27463 1787884129.338 ENTER Response.json stack=['Response.json']
--- snapshot B
30159 1787884275.442 ENTER Response.json stack=['Response.json']
27463 1787884129.338 ENTER Response.json stack=['Response.json']

Both workers, unmoved, with the report count frozen at 1315 of 1395. py-spy cannot reach these (the sandbox denies ptrace), and faulthandler names nothing here for the reason pyproject.toml records — the test's frames sit in a suspended greenlet.

The wedge reproduces without instrumentation too. An earlier plain run stopped at 91% with gw1 inside test_a_resolved_thread_can_be_reopened after test_a_thread_the_agent_closed_names_who_closed_it — the same pair, in the same order, as run 33130006054. Both wedges left their page server answering: the served page's viewed.json kept ticking every 30 seconds, and ss showed one established connection per server with empty queues, so nothing was stalled on the wire and no flock was held. Per-test pacing in the failed CI runs matches a green run's decile for decile, which rules out the machine slowing down before it stopped.

---

Automated fix for failed run

[leaf-agent]

Independently reproduced the bug-back in this checkout: with settle restored to the unconditional drain, test_traffic_leaves_a_body_the_browser_has_not_finished_handing_over fails on a body was read before the browser had all of it. Two points, neither about the mechanism.

#84 lands a comment that attributes the same two runs to a different wait. #84's body hedges — "This bounds the class; it does not prove which wait wedged those two runs" — but the comment it commits above FRAME_DEADLINE_MS does not: "Runs 33130006054 and 33130455313 ended exactly there, each inside a render test that had reached no assertion." This PR's tracer caught both workers inside Response.json, which is a measurement that says otherwise. Whichever of the two lands second, render_harness.py ends up carrying a causal claim about those runs that the other's evidence refuted, in permanent source prose where the next reader takes it as measured. Worth settling which claim survives before either lands rather than after — the two fixes don't conflict, only their accounts of the same three runs do.

One stale sentence in _until's docstring — inline. The paragraph above it now says the loop wakes on finished trips; the closing paragraph still describes it waking on responses.

ci / test was still running when the poll cap expired, so I have not verified this head green.

[leaf-agent]

Review actioned. The docstring sentence is fixed in 9026293; the reply is inline.

On the two accounts of the same runs — this PR's measurement survives, and #84 no longer claims otherwise. The tracer caught both workers of a locally reproduced wedge inside Response.json, unmoved across two snapshots a minute apart, with Page.evaluate absent from either; #84's evidence for the frame promise is that the CI runs ended with no traceback, which is a shape both waits produce, and those runs were never traced themselves. A measurement outranks a shape, so the causal claim was #84's to drop rather than this one's to soften. Pushed to that branch as e6db018: the sentence above FRAME_DEADLINE_MS now says no run has been measured ending there, names where the wedge that was traced stopped, and states that it bounds a class rather than the wedge those runs took — a follow-up (4fca229) tightened the same sentence further, since the CI runs were never under the tracer and there are five of them now, not three. The frame-wait fix itself is untouched — it still closes a real unbounded-wait class, and neither PR's source now carries a claim the other's evidence refuted, whichever lands first.

ci / test is green on this head (9026293) and on #84's e6db018.

[leaf-agent]

A fourth run on main has now ended this way: 33138394016 (7892125), 45 minutes spent, no failure reported, 241 of 1367 tests never run. It is the same wedge as 33131937176 down to the test — gw0 reported test_render_options.py::test_a_pick_states_the_whole_set, picked up test_render_options.py::test_an_answer_carrying_an_older_pick_cannot_undo_a_newer_one, and never spoke again; gw1 drained for another 100 seconds and stopped in test_render_outbox.py. Two commits apart, same worker, same preceding test, same test wedged.

Its log reads as naming nothing, which is worth recording since the next run to land here may look the same. -v writes a worker's nodeid without a newline, so the line only reaches the log when the other worker's report closes it. gw0's wedged nodeid got flushed by gw1's next report; gw1's own never did, because nothing reported after it. So the wedged nodeid is there, but as an unpaired start line in the middle of the file rather than at the end — tail shows a clean PASSED.

Extracting the unpaired nodeid

gh run view <run-id> --log-failed \
  | sed -E 's/\r$//; s/^test\t[^\t]*\t//' \
  | awk '{ if ($2 ~ /^tests\//) print "START", $2
           else if ($0 ~ /\] (PASSED|FAILED|SKIPPED|ERROR|XFAIL|XPASS) tests\//) {
             line=$0; sub(/.*\] [A-Z]+ /,"",line); sub(/[[:space:]]+$/,"",line)
             print "END", line } }' \
  | awk '{ if ($1=="START") s[$2]++; else e[$2]++ }
         END { for (k in s) if (s[k] > e[k]) print "UNPAIRED:", k }'

On 33138394016 and 33131937176 that prints test_an_answer_carrying_an_older_pick_cannot_undo_a_newer_one and nothing else. The surrounding lines on this run:

03:34:01.7179649 [gw1] [ 79%] PASSED tests/test_render_options.py::test_a_pick_states_the_whole_set
03:34:05.4389495 tests/test_render_options.py::test_an_answer_carrying_an_older_pick_cannot_undo_a_newer_one
03:34:05.4472678 [gw1] [ 79%] PASSED tests/test_render_options.py::test_a_question_says_what_the_agent_is_doing_about_it
...
03:35:41.0169943 [gw1] [ 82%] PASSED tests/test_render_outbox.py::test_a_refused_draft_keeps_newer_authoritative_words_under_its_editor
04:02:08.7897928 ##[error]The action '🧪 Test suite' has timed out after 45 minutes.

([gw1] labels the report, not the start line above it — that start is gw0's, flushed by gw1's write.)

The three ci runs after this one on main are green, so the shape stays intermittent rather than having become constant.

No separate fix opened for this run — it is the failure this PR already measures.

[leaf-agent]

Two more runs on main ended this way, both starting while the note above was being written and both spending the full 45 minutes: 33139867444 (1d1a92c) and 33139894911 (f339a40). The unpaired-nodeid recipe above prints exactly one test on each, and nothing else: test_render_navigation.py::test_the_resolve_key_changes_the_focused_threads_resolution and test_render_conversations.py::test_a_resolved_thread_can_be_reopened.

The second is the test 33130006054 wedged on, a repeat three hours and 22 commits apart rather than two commits apart. The first is new, and test_render_navigation.py is a third module. Both call round_trip, so both were inside the wait this PR bounds, and neither reached an assertion. That is six runs on four distinct tests in three modules.

The three green runs that note cites (9ed2b73, cab8723, 7164e68) are followed immediately by these two, so the shape is still intermittent but has not receded. No fix was opened for either — they are the failure this PR measures.

Both timelines

33139867444 — 1345 items collected, 1217 reported, 128 never run. gw0 picked the test up and never spoke again; gw1 drained for another seven minutes, then 26 minutes of silence to the step bound.

03:58:23.2  [gw0] [ 75%] PASSED tests/test_render_navigation.py::test_the_key_line_names_what_this_press_will_comment_on
03:58:24.4         tests/test_render_navigation.py::test_the_resolve_key_changes_the_focused_threads_resolution
04:05:18.9  [gw1] [ 90%] PASSED tests/test_render_projection.py::test_agent_places_its_live_line_before_command_evidence
04:32:01.1  ##[error]The action '🧪 Test suite' has timed out after 45 minutes.

33139894911 — 1345 items collected, 858 reported, 487 never run. Same shape with the workers swapped: gw1 took the test at 03:55:52 and stopped there, gw0 drained for 44 more seconds, then 36 minutes of silence.

03:55:52.1         tests/test_render_conversations.py::test_a_resolved_thread_can_be_reopened
03:56:36.4  [gw0] [ 63%] PASSED tests/test_render_conversations.py::test_the_line_offers_the_list_its_own_keys_rather_than_the_way_deeper_in
04:32:38.5  ##[error]The action '🧪 Test suite' has timed out after 45 minutes.

Neither is contention: per-test pacing on both runs is ordinary right up to the wedge — the two reports bracketing the start line on 33139867444 are 1.2 seconds apart — and the other worker keeps reporting at its usual rate after its partner has gone quiet.

[leaf-agent]

Two more, and the second one repeats a wedge this PR already names — same worker, same preceding test, same test. 33139867444 (1d1a92c) stopped on test_render_navigation.py::test_the_resolve_key_changes_the_focused_threads_resolution at 75%; 33139894911 (f339a40), the very next push, stopped on test_render_conversations.py::test_a_resolved_thread_can_be_reopened at 61%, picked up by gw1 immediately after gw1 reported test_a_thread_the_agent_closed_names_who_closed_it.

That pair is the one 33130006054 landed on and the one this PR's local run reproduced, down to the worker and the 62% mark. So across six runs the wedged test keeps moving while the pairs that recur, recur exactly — which is what a shared harness wait looks like and not what a test-specific hang looks like. No separate fix opened; this is the failure the PR measures.

The two logs

Extracted with the unpaired-nodeid pass from the comment above.

33139867444  1d1a92c
  03:58:23.2 [gw0] [ 75%] PASSED test_render_navigation.py::test_the_key_line_names_what_this_press_will_comment_on
  03:58:24.3            START test_render_navigation.py::test_the_resolve_key_changes_the_focused_threads_resolution   <- never reported
  04:05:18.8 [gw1] [ 90%] PASSED test_render_projection.py::test_agent_places_its_live_line_before_command_evidence    <- gw1 drains 7 min, stops
  04:32:01.0 ##[error] timed out after 45 minutes

33139894911  f339a40
  03:54:34.0 [gw1] [ 61%] PASSED test_render_conversations.py::test_a_thread_the_agent_closed_names_who_closed_it
  03:55:52.1            START test_render_conversations.py::test_a_resolved_thread_can_be_reopened                     <- never reported
  03:56:36.3 [gw0] [ 63%] PASSED test_render_conversations.py::test_the_line_offers_the_list_its_own_keys_rather_than_the_way_deeper_in
  04:32:38.5 ##[error] timed out after 45 minutes

33130006054  45de22f   (for comparison)
  00:41:47.8 [gw1] [ 62%] PASSED test_render_conversations.py::test_a_thread_the_agent_closed_names_who_closed_it
  00:44:27.8            START test_render_conversations.py::test_a_resolved_thread_can_be_reopened                     <- never reported

Each run has exactly one unpaired start line. The three ci runs after f339a40 on main (7868411, 68fad80, 7acb80c) are green, so it stays intermittent.

[leaf-agent]

A ninth run ended this way: 33146543802 (e0fe67f, "Name stamp transaction phases"), 45 minutes spent, no failure reported, stopped at 65%. The unpaired-nodeid recipe prints one test and nothing else — tests/test_render_conversations.py::test_a_thread_reopened_mid_fold_folds_again_when_it_settles, started at 06:10:20 and never reported, while the other worker drained to test_render_drafts.py::test_a_remove_failure_cannot_resurrect_an_accepted_draft at 06:12:03 and stopped there too. It reaches Traffic three times before its first assertion (panel_settled, round_trip, undo), which is this PR's site.

Two runs from before this PR was opened also belong in the tally and were never named here, so the full run of the shape on main is nine, unbroken since 45de22f — the three failing ci runs immediately before it (728707f, 9c5a5d4, 080a331) fail with real assertions, not the timeout. Nine wedges on eight distinct tests across six modules, which is the same "the wedge is not a test's" reading the PR body makes, now with four times the sample.

The nine, with the test each stopped on

run	commit	wedged on
33130006054	45de22f	test_render_conversations.py::test_a_resolved_thread_can_be_reopened
33130455313	d4472ad	test_render_conversations.py::test_a_sent_comment_is_revealed_in_the_panel
33131937176	caa8460	test_render_options.py::test_an_answer_carrying_an_older_pick_cannot_undo_a_newer_one
33133078347	888006e	test_render_outbox.py::test_z_waits_for_an_unanswered_thread_resolution
33133680899	36ba137	test_render_controls.py::test_a_stale_package_widget_uses_recursive_parent_eligibility
33138394016	7892125	test_render_options.py::test_an_answer_carrying_an_older_pick_cannot_undo_a_newer_one
33139867444	1d1a92c	test_render_navigation.py::test_the_resolve_key_changes_the_focused_threads_resolution
33139894911	f339a40	test_render_conversations.py::test_a_resolved_thread_can_be_reopened
33146543802	e0fe67f	test_render_conversations.py::test_a_thread_reopened_mid_fold_folds_again_when_it_settles

Rows 4 and 5 are the two this thread had not named. Every one of the eight distinct tests reaches Traffic through round_trip, told, undo or a bare _until before it stops.

No separate fix opened for this run — this PR is the fix, and it is still unmerged.

[leaf-agent]

A tenth run ended this way: 33146978933 (abf03ba, "Name GET route phases"), 45 minutes spent, no failure reported, 1021 of 1346 tests run before it stopped at 75%.

The unpaired nodeid is tests/test_render_drafts.py::test_page_round_trip, picked up at 06:18:40 and never reported, while the other worker drained to test_render_navigation.py::test_a_key_on_screen_is_a_key_that_works at 06:25:30 and stopped there too. It is a module the table above does not carry, and it reaches Traffic through round_trip partway in and told after that, so it sits in the same class as the rest rather than adding a new one. settle on main still drains unconditionally into response.json().

Nothing here moves the diagnosis, so this is the last per-run row I'll add — the shape is established at ten and another identical row is noise, not evidence. A later ci-fix run reading this thread should stay silent on a wedge that matches this shape and comment only if one diverges from it: a wedge that reports a traceback, one on a main that already carries this PR, or a run whose other worker keeps going to the end. This PR is still the fix, and still unmerged.

[leaf-agent]

An eleventh run ended this way — 33151490961 (b13dfd4, "Split render command browser tests"), unpaired nodeid tests/test_render_options.py::test_a_pick_offered_can_be_pointed_at_too, which reaches Traffic through round_trip before it stops. By the rule the note above set, that is a matching row and I would have stayed silent on it. What follows is not the row: it is a structural fact about the other worker, which holds in this run and in the tenth, and which bears on what merging this PR should be expected to clear.

The second worker does not stop where the first one does. It stops in teardown. xdist reschedules a worker only on runtest_protocol_complete, which the worker sends after setup, call and teardown — so a worker whose last line is its own PASSED has sent the call report and then stopped before the protocol ended, while a worker that wedges in the test body leaves the unpaired logstart this thread's recipe already reads. Both runs I parsed show exactly one of each: one unpaired nodeid, and one worker whose last line is a PASSED followed by silence to the cap.

Nothing on a render test's teardown path calls settle. Traffic registers listeners and has no teardown; what runs after the call phase is serve's httpd.shutdown() and the autouse _no_page_outlives_its_test. So this PR's site and #84's are both places the second worker has already left. That is why it is worth naming before a merged fix is read as the whole answer: it is both workers being stopped, not one, that spends the full 45 minutes instead of the remaining quarter of the suite.

How the teardown placement was established, and the two runs it was read from

The scheduling fact, from the installed pytest-xdist 3.8.0: the worker sends runtest_protocol_complete after pytest_runtest_protocol returns (xdist/remote.py, run_one_test), the controller turns that event alone into sched.mark_test_complete (xdist/dsession.py, worker_runtest_protocol_complete), and mark_test_complete is the only caller of _reschedule (xdist/scheduler/loadscope.py). pytest_runtest_logstart is forwarded live and separately. So the two silences read differently: no PASSED after a logstart means the call phase never returned; no further logstart after a PASSED means the protocol never completed, and the call phase already had.

run	unpaired nodeid (worker A, mid-test)	worker B's last line, then silence
33151490961	test_render_options.py::test_a_pick_offered_can_be_pointed_at_too (gw0)	gw1 PASSED test_render_options.py::test_what_a_widget_paints_it_says_to_a_reader_listening 07:38:49 → cap at 08:12:48
33146978933	test_render_drafts.py::test_page_round_trip (gw0)	gw1 PASSED test_render_navigation.py::test_a_key_on_screen_is_a_key_that_works 06:25:30 → cap

Neither run reports a crashed or replaced worker, and the job's orphan list holds both worker processes and both chrome-headless-shell at kill time, so worker B is alive and blocked rather than gone.

What blocks it is inference; the placement above is not. The one unbounded wait I can find on that path is shutdown() itself. LeafHTTPServer.shutdown delegates to socketserver.BaseServer.shutdown, which waits on an Event with no timeout until serve_forever leaves its loop; that loop can be held inside get_request(), where a listening socket reported readable by select still blocks in accept() if the peer aborted the connection in between. A page abandoning a request it no longer wants — the behaviour this PR's body describes at the response end — is one way to produce that abort. Worth a measurement rather than a patch.

This does not contradict the tracer snapshot in the PR body, and may be reconcilable with it: a worker already in teardown makes no further Playwright calls, so if the recorder's per-worker file shows the last call entered rather than one provably still live, worker B would read as sitting in the Response.json of the test it had just finished. Whether the recorder pops on exit is the check that separates those two readings.

[leaf-agent]

Merged main in — the branch had conflicted on ac8fe9e.

The textual conflict was one docstring paragraph in _until: this branch rewrote it around finished trips, and main rewrote it around the final reading being taken after the timeout. Both statements are still true of the merged code, so the paragraph carries both, in this branch's vocabulary.

The conflict git couldn't see mattered more. Main added test_a_traffic_wait_accepts_completion_delivered_with_its_timeout, which asserts event == "response" inside its fake page — the event name this branch changes to requestfinished. It auto-merged clean and would have failed on the merged tree. Its subject (the ordinary listener settling the fact as the timeout is delivered) is unchanged by this branch, since Traffic still registers its own requestfinished handler alongside the waiter, so it is rewritten in the new vocabulary rather than dropped.

lint passes. test is red on test_a_label_press_keeps_the_controls_keyboard_standing, which is the deterministic main-wide failure #88 exists to fix and is the only test named in this run's failures — it is not reachable from anything on this branch. The four traffic tests pass here.

[leaf-agent]

Correcting the last line of the comment above: #88 was closed as superseded a few minutes before I posted it, so "the failure #88 exists to fix" no longer names anything pending. The fix itself did land — e4be680 ("Make Python the canonical browser projector", #87) added the same panel_settled(page) call at that site on main.

Merged main again to pick it up, pushed as 211eb13. The branch was sitting on a merge of ac8fe9e, which predates that commit, so its red test was a failure main had already fixed. This merge was clean and the four traffic tests still pass on it.

[leaf-agent]

Re-reviewed the merge and 9026293. Both concerns from the review on 17c83ca are cleared: the stale _until sentence reads correctly now, and #84's comment above FRAME_DEADLINE_MS no longer attributes those runs to the frame wait. Reproduced the bug-back independently in this checkout — with settle restored to the unconditional drain, test_traffic_leaves_a_body_the_browser_has_not_finished_handing_over fails on a body was read before the browser had all of it — and the four traffic-wait tests pass on the merged tree.

One finding, on the comment this PR commits rather than on the mechanism.

settle's docstring attributes the whole 45-minute bound to this call, and the thread has since established it can't be. The eleventh-run note works out from xdist's scheduling that each wedged run has two stopped workers — one mid-test with an unpaired logstart, one whose last line is its own PASSED, which places it past the call phase in teardown — and says plainly that "this PR's site and #84's are both places the second worker has already left." A run reaches the cap only because both are stopped. But the docstring says "three runs on main spent their 45-minute bound inside this call", in permanent source prose, with no measurement behind the second worker being here; the tracer snapshot in the PR body is a local reproduction, and that same note flags that a recorder which doesn't pop on exit would show a teardown-blocked worker as still inside the Response.json it had just left.

That matters after this lands rather than before. The next reader to hit a wedge on a main that carries this fix reads that sentence, sees the run spend its full bound anyway, and concludes the fix didn't take — when what it says is that the other site (httpd.shutdown(), by the note's inference) is still open. It is the same shape as the claim the earlier review asked #84 to drop, at this PR's own site. Inline suggestion states what was traced and leaves the second worker where the thread put it; the count is dropped rather than moved to eleven, since it ages either way.

Not blocking — the mechanism, the gate, and the bug-back all hold. ci / test was still in progress on this head when I finished, so I have not seen it green here.

[leaf-agent]

A twelfth run ended this way, on the pair this thread has already named twice: 33187521630 (d79a08b, "Extract authored anchor capture"). The unpaired-nodeid pass finds exactly one — gw1 printed tests/test_render_conversations.py::test_a_resolved_thread_can_be_reopened at 16:05:15 and never reported it, picked up immediately after gw1 reported test_a_thread_the_agent_closed_names_who_closed_it. Same worker, same preceding test, same test as 33130006054, 33139894911, and this PR's local run.

16:04:33.7 [gw1] [ 61%] PASSED test_render_conversations.py::test_a_thread_the_agent_closed_names_who_closed_it
16:05:15.5            START test_render_conversations.py::test_a_resolved_thread_can_be_reopened   <- never reported
16:06:46.6 [gw0] [ 63%] PASSED test_render_drafts.py::test_failed_settlement_keeps_the_base_for_a_chained_nondurable_edit
16:41:36.9 ##[error] timed out after 45 minutes

gw0 drained its share and stopped thirty-five minutes before the bound. 884 of 1382 reported (878 passed, 6 skipped); 498 tests never ran.

What this adds over the eleventh note is only that the wedge survives this morning's five module-refactor commits (ae66e73..47f1c35) — nothing that landed since 08:23 has touched it. The branch is green and mergeable: true against current main; blocked is the admin-only merge ruleset, not a check.

The lint half of the same run is #96's, and is not duplicated here.

[leaf-agent]

A thirteenth run ended this way, and it is the first to name a test outside test_render_conversations.py and test_render_options.py: 33189277312 (47f1c35, "Split served browser projection by scope"), 45 minutes spent, 1079 of 1382 tests reported. gw0 printed tests/test_render_drafts.py::test_failed_settlement_keeps_the_base_for_a_chained_nondurable_edit at 16:27:08 and never reported it. gw1 reported tests/test_render_options.py::test_what_a_widget_paints_it_says_to_a_reader_listening at 16:34:35, picked up nothing after it, and left 303 items pending — one worker stopped inside a test and the other past the call phase in teardown, which is the pairing this PR's body says it parsed on the earlier runs.

Every wait the named test makes is bounded: expect, wait_for_function, and _until's own 30-second deadline. The one unbounded site it reaches is the one this PR closes — round_trip → _traffic → settle → response.json(). A drafts module reaching it too is what the "the wedge is not a test's" claim predicts.

The same run's lint job failed for the unrelated import-order drift #96 fixes, so no fix PR was opened for either half: both are already in open, mergeable PRs.

How the nodeid was derived, and the lint reading

The wedged nodeid is the set difference between the bare nodeids -v writes at pickup and the nodeids in [gwN] [ n%] PASSED … lines, over the whole job log — one entry, gw0's. Reading the log in order finds nothing, because the two workers interleave and the last line of the log is a report, not a pickup.

Counts: 2 workers [1382 items]; 408 reported by gw0, 671 by gw1; gw0's last report was tests/test_render_drafts.py::test_a_held_comment_send_leaves_the_passage_picked_out_behind_it at 16:27:07, one second before the pickup that never returned.

Lint, on main at 5c5547a rather than on the failing commit, with the versions the hooks pin:

$ uvx ruff@0.16.1 check --no-fix .
Found 8 errors.
$ uvx ruff@0.16.1 format --check .
2 files would be reformatted, 138 files already formatted

That is the same 8 I001 errors and the same 2 files #96 rewrites, so #96 still covers main as it stands.

[leaf-agent]

Rather than name a fourteenth run, here is the whole day measured at once — and then I'll stop posting recurrences individually, since eleven anecdotes on this thread have said less than one count does.

18 of the 93 ci runs on main today ended with the test job spending the full 45-minute step bound and reporting no failure — about 13.5 hours of runner time, and eighteen commits on main left red with nothing to read. Every one of them has exactly one unpaired nodeid, and across the eighteen those nodeids are 11 distinct tests in 6 modules. That spread is the part the individual reports could not show: no test appears often enough to be the cause, which is what a wait inside shared harness code rather than inside a test looks like.

None of the three branches is on main. At 1c5edad (this run's commit) tests/render_harness.py still reads the response body unconditionally in settle — grep for requestfinished, FRAME_DEADLINE_MS, and #95's never ran out returns nothing across tests/.

The latest is 33209352445 (1c5edad, "Build the Command Hub coordination loop"): gw1 printed tests/test_render_conversations.py::test_a_resolved_thread_can_be_reopened at 20:54:00 after passing test_a_thread_the_agent_closed_names_who_closed_it at 20:53:11, and never reported it; gw0 drained on alone to 99% and was still inside test_smoke.py::test_a_shipped_page_passes_the_real_browser_gate when the cap cut the job off 34 minutes later. Same worker, same preceding test, same test as 33130006054, 33139894911, 33187521630 and 33192065454. It is intermittent rather than constant — the five runs immediately before it were green.

I'm not re-adjudicating which of #84, #85 and #95 owns which wedge; that argument is already on these threads. The measurement is offered as evidence about the class.

All 18 wedged runs, with the unpaired nodeid each left

Wedge = the test job's own start-to-finish duration is the 45-minute step bound, and the run reports no failed test. Unpaired nodeid = a nodeid the -v log dispatches but no [gwN] [nn%] PASSED/FAILED line ever reports. Five of these (888006e, 36ba137, d63ae90, ac8fe9e, 5c5547a) had not been named on this thread before.

time (UTC)	commit	run	unpaired nodeid
00:31	45de22f	33130006054	test_render_conversations.py::test_a_resolved_thread_can_be_reopened
00:40	d4472ad	33130455313	test_render_conversations.py::test_a_sent_comment_is_revealed_in_the_panel
01:08	caa8460	33131937176	test_render_options.py::test_an_answer_carrying_an_older_pick_cannot_undo_a_newer_one
01:30	888006e	33133078347	test_render_outbox.py::test_z_waits_for_an_unanswered_thread_resolution
01:42	36ba137	33133680899	test_render_controls.py::test_a_stale_package_widget_uses_recursive_parent_eligibility
03:16	7892125	33138394016	test_render_options.py::test_an_answer_carrying_an_older_pick_cannot_undo_a_newer_one
03:46	1d1a92c	33139867444	test_render_navigation.py::test_the_resolve_key_changes_the_focused_threads_resolution
03:46	f339a40	33139894911	test_render_conversations.py::test_a_resolved_thread_can_be_reopened
06:01	e0fe67f	33146543802	test_render_conversations.py::test_a_thread_reopened_mid_fold_folds_again_when_it_settles
06:09	abf03ba	33146978933	test_render_drafts.py::test_page_round_trip
07:26	b13dfd4	33151490961	test_render_options.py::test_a_pick_offered_can_be_pointed_at_too
07:56	d63ae90	33153356042	test_render_navigation.py::test_the_resolve_key_changes_the_focused_threads_resolution
09:21	ac8fe9e	33159049900	test_render_drafts.py::test_a_draft_explains_its_change_and_restores_history_as_an_edit
11:03	4591cfd	33165671503	test_render_conversations.py::test_a_thread_reopened_mid_fold_folds_again_when_it_settles
15:55	d79a08b	33187521630	test_render_conversations.py::test_a_resolved_thread_can_be_reopened
16:17	47f1c35	33189277312	test_render_drafts.py::test_failed_settlement_keeps_the_base_for_a_chained_nondurable_edit
16:53	5c5547a	33192065454	test_render_conversations.py::test_a_resolved_thread_can_be_reopened
20:42	1c5edad	33209352445	test_render_conversations.py::test_a_resolved_thread_can_be_reopened

By module: test_render_conversations 8, test_render_options 3, test_render_drafts 3, test_render_navigation 2, test_render_outbox 1, test_render_controls 1.

The other 27 failed ci runs on main today are not this shape — their test job ran 16–29 minutes, and they are lint failures (mostly ruff format) or named test failures such as 33185431703, which #97 fixes.

How the counts were taken

Runs enumerated from repos/max-sixty/leaf/actions/runs?branch=main (pages 1–6, filtered to name == "ci" and created_at >= 2026-08-28T00:00:00Z): 93 runs, 45 with conclusion == "failure". For each failure, the test job's started_at/completed_at from .../runs/<id>/jobs; the 18 above are the ones where that span is 45 minutes.

The unpaired nodeid, per run, from gh run view <id> --log:

grep -oE 'tests/[^ ]+\.py::[^ ]+' log | sed 's/[[:space:]]*$//' | sort -u > all
grep -E '\[gw[0-9]+\] \[' log | grep -oE 'tests/[^ ]+\.py::[^ ]+' | sed 's/[[:space:]]*$//' | sort -u > done
comm -23 all done

On the eighteen this returns one nodeid each, except 1c5edad, where it returns two: the wedged one plus test_smoke.py::test_a_shipped_page_passes_the_real_browser_gate, which gw0 was simply still inside when the cap arrived.