cmux-tui: no request waits for the disk while it holds the state lock or the registry connection by lawrencecchen · Pull Request #28159 · manaflow-ai/cmux

lawrencecchen · 2026-10-30T08:23:52Z

Summary

On a loaded Mac (#27698: daemon lock waits p90 5 s, p99 17 s), two request paths waited for the disk while they held a lock. Other requests queued behind them:

rename-workspace held the mux state lock across its journal writer receipt wait (up to the 30 s durable wait). Every state reader (list-workspaces, identify, terminal event placement) waited for the disk. The rename now releases state before the wait and takes it again to apply the committed name, finding the workspace by its key. The registry stays held, so no other topology writer interleaves.
A request that found an earlier indeterminate commit waited for its answer (2 s) with the registry connection pinned. The journal writer needs that connection to commit and answer, so the wait could only run out and fail the request. lock_state_pinned now waits for those answers before it pins, and drain_unsettled never waits while the thread holds the connection: it takes only the answers that already arrived. A commit still unanswered after the unpinned wait is refused at once, as before.

With nothing left to unpin, PinnedState::unpin and the release_pin hook are removed. A debug assert now refuses any journal writer receipt wait while the thread holds the mux state lock. It did not fire in any full test run of this branch.

What callers see after a rename. The rename replies only after the new name is in state, so a caller that renames and then lists sees the new name. Until the rename takes state again after its commit is durable, other readers see the old name. A journal subscriber that the writer wakes can read the old name for that short time, as after other writer-batch commits (resource_effects.rs:217). The test checks both: no reply before durability, and only the old name visible to readers during the wait.

This PR also carries one follow-up from the review of #28147: server-stats locks now formats its site labels after it releases the site table, and two schema doc lines are corrected.

This PR contains #28147 (the deadlock fix and the lock statistics). Review only the commits after 100b8e3; once #28147 lands, the diff shrinks to them. #27780 (Austin) touches the same registry lock, in a different function.

Owner: PL-daemon (session cmux-b6).

Testing

Regression tests, both in cmux-tui-core/src/mux/tests/writer_lock_order.rs:

a_rename_waiting_for_its_writer_receipt_does_not_hold_state pauses the journal writer inside its batch and probes the state lock while the rename waits.
waiting_for_an_earlier_indeterminate_commit_leaves_the_connection_free holds an unanswered receipt and probes the connection while a rename waits for it.
Red, fleet step a7af786698ead5c101d1d25c: scripts/ci/cmux-tui-rust-check.sh test writer_lock_order at b702c98 (tests only). Both new tests FAILED on their "lock is free" asserts (state_free, line 238; connection_free, line 292). The other four tests passed.
Green: scripts/ci/cmux-tui-rust-check.sh all at 352210f, fleet step 4aa52296785c6974c78b8d10: fmt and clippy pass. 12705 tests ran and 12703 passed. The 2 failures are base reds on feat-cmux-next: every_safe_transport_operation_has_a_noun_first_path (fix cmux-tui: git diff and checkpoint diff take --after-path, --patch-offset and --patch-revision (base red) #28119) and the acpmux hermetic_daemon_guard check for web_image_attachments (fix acpmux tests: web_image_attachments runs hermetic, with node as its one host tool (base red) #28150). The new debug assert did not fire.

Observed / Not run

Observed: the red failures above. In the earlier full run 87ecb023 (at 04bb5f0), the only failures were the two base reds named in cmux-tui: the daemon no longer deadlocks settling a late registry receipt; server-stats reports both locks by site #28147 (CLI noun-first path, fixed by cmux-tui: git diff and checkpoint diff take --after-path, --patch-offset and --patch-revision (base red) #28119; acpmux hermetic guard), plus the unexplained reds below.
Reds this PR does not cause. They are listed in the handoff followups for a fixer, not cleared as flaky; each failed once and passed when run again alone:
- surface::tests::local_resize_order::local_resize_bounds_the_drain_to_a_queued_snapshot ("the resize consumed output without publishing its title", local_resize_order.rs:111; step dd8687a1). This PR cannot reach it: its title event takes only mux.state, through none of the changed functions.
- activity::v2_terminal_input_counts_only_from_a_persons_attached_client (a 12 s read timeout, activity.rs:25; step 87ecb023). This PR cannot reach it: it runs on an in-memory mux with no journal writer, so there are no unsettled receipts, and it does no rename.
- acpmux daemon_restart_sends_close_1012_before_wireguard_shutdown and subsessions_safety::a_message_never_reaches_a_session_that_asks_less_than_its_sender. This PR cannot reach them: acpmux does not depend on cmux-tui-core.
Not run: Austin's-day before and after numbers, which the harness owner runs and posts on this PR. My own diagnostic bench (bench-daemon-latency.py --scenarios agents, before: step 73e93323) only ranks lock holders and is not the merge evidence. Perf claim: none until those numbers are posted.

Known follow-ups (from review, not regressions)

A pinned flow can still miss a late answer that arrives between its pin and its settle. It then builds on state without that answer. The window existed before this PR and is now smaller (Low 1 in the review). The fix, refusing or settling an answer that resolves after the pin's settle, goes in the next PR of this series, with a correction of the drain_unsettled doc text that this PR overstates.
The error path for a workspace that left state after a durable rename should raise the revisions first. It cannot be reached today (Low 3).

Performance

No perf numbers: this PR claims no measured speedup. The Austin's-day before and after numbers come from the harness owner and will be posted on this PR when they exist.

Changelog

Fixed: Renaming a workspace on a busy Mac no longer makes the rest of cmux NEXT wait for the disk.

Proof

No capture: daemon-only change, no UI.

80 fake terminal agents run the real hook helper (one durable journal append per turn event) and a CLI call in a loop, a churn client creates and closes terminals, and an app client sends what the cmux app sends (put-frontend-projection window saves, list-workspaces, session.snapshot, identify). It reports the app requests' latency, the requests over the app's 2 s timeout and both daemon locks by site, the shape of the 2026-10-29 report (#27698). Run: cmux-ci run --class exclusive --measurement --script scripts/measure/perf-10x-daemon.sh --ref SHA --arg=scenarios=agents Measurement only; no CI step, gate or budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…olders by site server-stats include=["locks"] reports the workspace registry lock and, new, the mux state lock: wait and hold histograms, the current holder, the last stall, and the top twenty acquiring sites by total hold and by p99 hold, each with its hold and wait p99 and maximum. Sites come from #[track_caller] (lock_state_pinned now carries it too, so its callers are named instead of the wrapper). Per site, two power-of-two histograms (256 bytes each) are updated under the site table lock that the release already took, while the lock is still held: no allocation after a site's first acquisition and no new contention. The state lock's existing rank and holds bookkeeping is unchanged. The section is optional (absent unless named), so older SDK decoders keep working; schema, bindings, api-compat surface and commands.md updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…pers in lock stats #[track_caller] on the Mux wrappers that take the workspace registry or the mux state lock (with_state, surface, resolve_resource_path, commit_state, commit_resource_mutation_plan, with_resource_projection, the resource effect helpers, tree_decorations and others), so server-stats locks names the request path that holds the lock instead of the wrapper's own line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…again (red) lock_state_pinned settles the late answers of indeterminate registry commits (cx-g1fa.3) while it holds the registry and state. The settle then ran publish_resource_event, whose pending-directory step locks state again: a lock-rank panic in debug builds and a self-deadlock of the whole daemon in release builds (std Mutex), on a path a loaded Mac reaches when a journal commit outlives its 30 s receipt wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…urnal event The settle runs under the registry and state; publish_resource_event also publishes pending terminal directories, which locks state again (and then the registry). Publish the journal event only: subscribers still read the new head, and a directory still pending is published by the terminal's next output or the next committed graph change, as before the settle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e connection across an unsettled wait (red) Two holders that wait while they hold a lock (#27698 audit, lock waits p90 5 s / p99 17 s on a loaded Mac): - rename-workspace held the mux state lock across its journal writer receipt wait (up to the 30 s durable wait); - a request that found an earlier indeterminate commit waited for its answer (2 s) with the registry connection pinned, though the writer needs that connection to commit and answer: the wait could only run out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ate or the connection - rename-workspace releases the mux state lock before its journal writer receipt wait and takes it again to apply the committed name (found by workspace key). The registry stays held, so no topology writer interleaves; state readers see the old name until the commit is durable, instead of queueing behind the disk for up to the 30 s durable wait. - lock_state_pinned waits for the answers of earlier indeterminate commits before it pins the registry connection, and drain_unsettled never waits while this thread holds the connection: the writer needs the connection to commit and answer, so that wait could only run out (2 s) and fail the request. A commit still unanswered after the unpinned wait is refused at once, as it was after the pinned wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…tate With rename no longer holding state across its receipt, nothing unpins a held state any more: PinnedState always holds its connection pin, and commit_registry_intent_with loses its release_pin hook. A debug assert refuses a journal writer receipt wait while the thread holds the mux state lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…aders see the old name only until then Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…lippy redundant_clone) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…leases the site table Every release of the registry and state locks takes the site table mutex; detail() now copies the raw records out and formats the file:line labels after the unlock (review of #28147). Schema text: the holder site names either lock, and sites counts the locations that released the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cursor · 2026-10-30T08:23:59Z

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

github-actions · 2026-10-30T08:24:06Z

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 1722dc5: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28159 -f ref=fix/pa-b3-waits -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 2a822f8 to compare; do not merge until green; see lane-rules

Shipped in:

cmux NEXT nightly 1.0.0-nightly.4130052256501 (28aa638015)

blacksmith-sh · 2026-10-30T08:30:28Z

Found 9 test failures on Blacksmith runners:

Failures

Test	View Logs
`/tmp/tmp0bq0f54y/summary-fail.json (live):/summary-fail.json (live):`	View Logs
`/tmp/tmpu4bgm7z0/summary-missing.json (live):/summary-missing.json (live):`	View Logs
`[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/` `[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows`	View Logs
`[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames`	View Logs
`[tabs] debug.tab_switch timed 0 of 40 switches/` `[tabs] debug.tab_switch timed 0 of 40 switches`	View Logs
`[tabs] the window draws no frames/[tabs] the window draws no frames`	View Logs
`convention violations found (ERROR lines above)./` `convention violations found (ERROR lines above).`	View Logs
`test_every_part_of_bun_run_check_runs_in_the_warn_branch (main.WebviewsWarnSplitCov` `ersCheck.test_every_part_of_bun_run_check_runs_in_the_warn_branch)/` `test_every_part_of_bun_run_check_runs_in_the_warn_branch (main.WebviewsWarnSplitCov` `ersCheck.test_every_part_of_bun_run_check_runs_in_the_warn_branch)`	View Logs
`test_no_runner_picker_returns_a_hosted_mac (main.NoHostedMacInNightlies.test_no_run` `ner_picker_returns_a_hosted_mac)/` `test_no_runner_picker_returns_a_hosted_mac (main.NoHostedMacInNightlies.test_no_run` `ner_picker_returns_a_hosted_mac)`	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

lawrencecchen · 2026-10-30T08:48:15Z

Review (claude-subagent): SHIP at 352210f
R9: pass (no new cap, queue or deadline; UNSETTLED_DRAIN_WAIT keeps its 2 s value and now waits before the pin. Older R-L3/R-L5 debt stays: the 'still indeterminate; retry' refusal has no retry_after)
perf claim: none

Independent subagent, given the diff after 100b8e3 and the PR body. No correctness defect found.

Rename: while the registry is held, no state-only path removes, moves or renames a workspace or changes workspace_revision. The delta is built from the state taken again. The reply comes after the apply.
The wait before the pin cannot deadlock. The debug assert is unreachable with state held on every commit_* call site.
Low 1 (a pinned flow can miss a late answer between its pin and its settle; existed before, window now smaller) and Low 3 (the 'left state' path after a durable commit cannot be reached today): follow-ups, listed in the PR body.

Owner PL-daemon (cmux-b6): this changes the daemon's lock path.

lawrencecchen · 2026-10-30T11:29:12Z

Review (claude-subagent-fresh): SHIP at 352210f
R9: pass (no new cap, queue bound or deadline. The existing UNSETTLED_DRAIN_WAIT (2 s) is now also used in lock_state_pinned. That wait ends on the receipt channel signal, and 2 s is only its upper limit.)
perf claim: none

Scope: git diff 100b8e317c35..352210fa9976 -- . ':!cmux-tui/bindings', read at the exact sha. I wrote this verdict before I read any other review comment.

Correctness checks (no finding)

Races while the rename does not hold state (mux/workspace_rename.rs:234-253). The rename writes only name, name_source, name_revision, workspace_revision and resource_revision. In the source, every writer of a workspace name or of workspace_revision, and every push_workspace, remove_workspace and move_workspace, holds the workspace registry. The rename holds the registry for the full wait, so no writer can run in that gap. Paths that use only state (focus, terminal I/O) do not touch these fields. The rename finds the workspace again by key, and it uses max on both revisions, so a revision cannot go back. This is the same release-and-retake pattern that commit_resource_effect_projection already uses (resource_effects.rs:213-220).
Reply order. The name and the revisions go into state first. Then tree_entity_json reads them, the state guard drops, emit_committed_workspace_delta runs under the registry, then publish_resource_event, then Ok(result). A caller that renames and then lists sees the new name.
Second lock_state_pinned after the commit. Its drain cannot wait. The prepare drain ran pinned with no wait and refuses an unresolved entry, and the registry is held, so no new unsettled entry can appear. Its settle first applies any answer that came after the first settle, then the rename applies. If that answer is an earlier rename of the same workspace, the final name and revision are still correct (r1 > r0, and max is used).
Pre-pin wait (signaled_mutex.rs:372, registry_intent.rs:227-249). The waiter holds the registry and the unsettled mutex, and not the connection. The writer takes neither of them. Only registry holders lock unsettled: hold_unsettled and take_settled_receipts are both on the registry path. Thus there is no cycle. The wait is bounded at 2 s in total, not per entry. A writer that dropped the receipt sender returns at once (Disconnected). A caller that already holds the connection gets try_take, which does not wait.
Debug assert (registry_intent.rs:309-314). A PinnedState holder returns to commit_registry_intent_locally before the assert. If a holder of unpinned state takes the connection, the existing lock-order assert in RegistryConnection::get fires first. I found no caller that holds unpinned state and reaches the sink. commit_registry_intent_with(.., Some(settle)) now has only one caller (the rename), and it releases state before the call. The test profile keeps debug-assertions on (no override in [profile.dev]), so the green run exercised the assert. In release builds it does nothing.
NotQueued path. The rename no longer holds state there, so ensure!(!current_thread_holds_state()) passes and the commit falls back to a local commit. Before this change, the request failed.
diagnostics.rs detail(): it copies the records under the mutex and formats the labels after the unlock. That is correct.

Lows (none blocks)

Low: new bounded wait on pinned flows that do not commit. lock_state_pinned now waits up to 2 s under the registry (not under state) whenever an earlier commit is still unanswered. This includes flows that never prepare a commit, for example terminal_host_link.rs:121,188 and layout_undo.rs:92. Before, flows that commit already waited the same 2 s in prepare, and they held the registry, the connection pin and state during that wait. Thus the change is a net improvement. The new cost is limited to flows that do not commit, and only while the writer is more than about 30 s behind.
Low: the red evidence is for an earlier version of the test. Red step a7af7866 ran the b702c98 version of a_rename_waiting_for_its_writer_receipt_does_not_hold_state. Commit f3dd51c added the old_name read through with_state. On the base, that read would block until the rename's durable wait ends, and the test would then fail at !renamer.is_finished(), not at line 238. Thus it still fails without the fix, but no red run of the head version exists.
Low: test timing. (a) The hook is one-shot (before_commit.take()). If an unrelated writer batch takes the hook first, the rename blocks at pin() while it does not hold state, and the test passes without testing the receipt wait. The red run shows that the hook took the rename's own batch there. (b) waiting_for_an_earlier_indeterminate_commit_leaves_the_connection_free sends the answer about 300 ms into the 2 s drain bound. It can fail only if the test thread stops for more than about 1.7 s, or if replay() (which takes the connection for a short time) is still running after the 100 ms sleep. The risk is small.
Known Low 1 (late answer between the pin and the settle): I agree. It existed before this PR, and the PR makes the window smaller. Known Low 3 (the "left state" path after a durable rename): I confirm that it cannot be reached. A settle cannot remove a workspace, because only the rename passes a SettleState. Every removal of a workspace holds the registry, and the rename holds the registry from start to end. Raising the revisions before that .context(..)? is still a good defensive follow-up.

Fleet logs read

Red a7af786698ead5c101d1d25c (checkout b702c984cf62e94ee16c2979f1fcbc0c89223a71, test writer_lock_order). Both new tests FAIL, including on retry: writer_lock_order.rs:238:5 (state_free) and :292:5 (connection_free). The other 4 tests in the module PASS.
Green 4aa52296785c6974c78b8d10 (checkout 352210fa997654cd98abe52878bb441770a3cf9f, mode all). fmt pass, clippy -D warnings pass. 12705 tests ran, 12703 passed, 2 failed, and both new tests PASS. The only 2 failures are acpmux::hermetic_daemon_guard (tests/web_image_attachments.rs lacks HERMETIC, fix in acpmux tests: web_image_attachments runs hermetic, with node as its one host tool (base red) #28150) and transport_path_coverage_tests::every_safe_transport_operation_has_a_noun_first_path (git.checkpoint.diff fields after_path/patch_offset/patch_revision, fix in cmux-tui: git diff and checkpoint diff take --after-path, --patch-offset and --patch-revision (base red) #28119). Neither one touches the code this PR changes. The assert message "a request waits for a journal writer receipt while it holds the mux state lock" does not occur in the log.

…ic path (panic ratchet) diagnostics.rs: the per-site octave histogram indexed its bucket array and added with +=, which the Rust panic ratchet counts (cmux-tui-core indexing_slicing 854 > baseline 853; arithmetic_side_effects rose too; review of #28147 at c88bc3a). The bucket is now taken with get_mut, and the counts, the running sum, the p99 bound and the acquisitions use saturating or checked arithmetic. The values are unchanged: bucket 0 holds 0, bucket i holds 2^(i-1)..2^i - 1, the last bucket everything above. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lawrencecchen · 2026-10-30T16:12:35Z

Second correction (harness owner pa-d): the "whose connections fill the 64" part of my correction above was wrong.

I inferred from the clients' own refusal counts that "~99 % are hook helpers". The process samples of the #28041 A/B, on the same scenario (#28041 (comment)), contradict that.

At any time, a median of 62-63 cmux CLI processes are alive, and 0 hook-helper processes.
So in this scenario the pool is held by cmux CLI calls, mostly the fake agents' cmux workspace list (one per 3 tool calls, a load choice of the harness, not from Austin's data).
The CLI retries a refused connect inside the call, so its refusals show as latency (CLI p99 about 4 s), not as errors.

The other numbers in the comments above are unchanged.

…ench) (#28077) * scripts(cmux-next): Austin's day, the P-AUSTIN load scenario on an exclusive fleet mini 80 terminal agents through the real claude shim, wrapper and hooks (a fake Claude that runs the hook settings it is given on every tool call and stop, and calls the cmux CLI), a partial clone with 19,110 promisor packs, 25-chat acpmux waves, a bounded memory hog up to swap, and workspace closes through the app's closeWorkspace action. Measures app daemon requests (probe and log time-outs), refused connections, lock waits, close time, acpmux restarts, memory, load and swap, and the debug.hangs stalls over 100 ms by top cmux symbol. One JSON plus a text summary; --base/--head alternate on one host. A fleet bench each agent runs itself; not a CI job or gate. selftest.py (run by hand): 4 tests OK on cmux-orchestrator-1. * austins-day: resolve a job id without an empty --ref argument First run on the feat-cmux-next tip (146a973, build ab32ee84) on cmux20: 80/80 agents hooked, 13,693 hook runs; daemon refused 378 connections (peak 64/64); registry lock wait p99 393 ms (max 5.2 s); person CLI p99 4.65 s; 7 of 10 workspace closes timed out at 2 s (p95 8.7 s, one never closed); 71 cmux main-thread stalls over 100 ms. Not seen: acpmux restarts, app no-reply log lines (the log query found none; the app probe is pending). * austins-day: agents draw to their terminals and use CPU; swap to 14 GB; stalls by own cmux symbol Run 1 had load 35-54 (Austin 150-252) and swap 9 GB (Austin 17 of 18): the fake agents were silent between tool calls. Each now draws as Claude Code's TUI does while it thinks (a spinner at 10 Hz, ~300 chars/s of streamed text) and keeps 15 % of a core busy, and the hog targets 14 GB of swap. A stall is named by its first frame in cmux's own code (a CmuxNext module), not by a stdlib generic specialized into the binary. The app log query also counts the daemon category's 'failed' lines. * austins-day: validity rules, owner, 5 runs per arm with bootstrap verdicts, long-arm metrics (cmux-ad) - An A/B gives no verdict when any base run shows none of Austin's failures ('invalid: base did not reproduce') or when the head changes this folder ('invalid: head edits the harness'). - The README names the owner (the P-AUSTIN harness owner, agent pa-d): only the owner edits the folder and runs each P-AUSTIN PR's A/B. - 5 alternating runs per arm by default. Per metric: a paired percentile bootstrap 95 % CI of mean(head - base) over the rounds (10,000 resamples, fixed seed); 'better'/'worse' only outside the interval, else 'no difference'. - Long-arm metrics: open fds per process (lsof) and a per-hour growth slope of RSS, footprint and fds. The 60 min arm itself is blocked by the 10 min capture-host lease (README). selftest.py: 7 tests OK (cmux-orchestrator-1). * austins-day: stalls split into CODE and WAIT (P-MAIN rule G, lead 2026-10-30) A stall over 100 ms is CODE when cpu_ms is at least half of duration_ms (or the app's own debug.hangs cause, from fix/stall-hang-wait-split on), else a WAIT. Only CODE stalls with a cmux frame are reds by top cmux symbol and count in austin_failure; WAITs are counted apart (count, total ms, largest CPU share, page-ins) and reported next to swap and memory level, owner B8. Run 1 re-read: 92 stalls over 100 ms, 0 CODE, all WAITs (max CPU share 27 %, 43.9 s wall, 0.89 s CPU), so its 3 named sites are not reds. selftest.py: 8 tests OK (cmux-orchestrator-1). * austins-day: hook CPU per hook run, per-metric validity, CLI and close p99 rows For the P-AUSTIN A/Bs (#28041 B2: hook CPU; #28159 B3: lock waits, app requests, workspace close): - fake_claude logs each hook's CPU (wait4 user + system of the hook command and the children it waited for). A hook past its timeout is killed only while it is not yet reaped (WNOWAIT, then a lock), never a reused pid. - Each metric needs its own failure on every base run (METRIC_FAILURE: the daemon's 100 ms lock-stall line, the B3 50 ms app-request target, the 2 s action deadline, refused connections for CLI and hook rows); else that metric says 'invalid: base did not reproduce'. No pairs: 'not measured'. - New rows: agent CLI p99, hook CPU total and p99, workspace close p99. selftest.py: 9 tests OK (cmux-orchestrator-1). * austins-day: each run records the harness commit; refuse uncommitted harness edits * austins-day: hook waits work on the minis' Python 3.9; harness faults void a run; a failed arm is retried once The #28159 A/B (10:15Z-12:05Z) logged no hook runs: the minis run Python 3.9.6, which has no os.waitid on macOS, so each hook thread died right after it started its hook. The hooks still ran but nobody waited for them (Claude Code's sync hooks became fire-and-forget) and their time and CPU were lost. - The hook wait is a kqueue process-exit event on macOS (a pidfd on Linux); the child stays unreaped until wait4, so a time-out kill never reaches a reused pid. - A crash in a hook thread is logged as the hook's result. - A run whose hooked agents logged no hooks, or whose hooks hit a harness error, makes the A/B 'invalid: harness faults'. - A run that got no lease or no app is retried once (head round 5 got no capture host before the acquire time-out). selftest.py: 9 tests OK. * austins-day: refused as a rate and a share; base must be the PR's parent; who fills the pool From the #28159 review (pa-b3, issuecomment-6523481295), both points real: - 'refused' was a cumulative count read at each run's last answered server-stats, which came 28-47 s earlier on some base runs, so the arms were read at different times. Arms now compare refused per second over each run's own server-stats window and refused share of connects. - The base was the merge base with feat (9fa133b) while #28159 sits on the open #28147 (100b8e3), so the A/B measured both PRs. The README now says the base is the PR's parent; the summary lists base..head and refuses a base that is not the head's ancestor. - New: pool_holders, sampled with server-stats: the daemon's list-clients by kind/name, live hook-helper and CLI processes, and the refusals each client kind saw (lead: whose connections fill the 64). selftest.py: 11 tests OK. * austins-day: A/A self-test (cmux-ad) austins-day.sh --aa JOB runs one build in both arms (5 alternating rounds). summarize.py --aa passes a metric whose 95 % interval contains 0 and fails the rest; the owner records failures in aa-failures.json, and every later summary marks those metrics 'invalid (fails A/A)'. An A/A with two builds is refused. selftest.py: 13 tests OK. * austins-day: paired t-interval instead of the bootstrap; ABBA run order; a position estimate per metric (lead decision) The A/A on build 84c705ad (3 rounds) failed 10 of 28 metrics on identical arms with the percentile bootstrap, which is far too narrow over 5 or fewer pairs; a paired t-interval (95 %, t with n - 1 df) failed 2. The runner now alternates which arm runs first (ABBA; arm.json records the position), and each metric gets a position estimate (second run - first run) so a warm-up or drift effect shows apart from the arm difference. selftest.py: 13 tests OK. * austins-day: A/A of 2026-10-30 recorded (passes with the t-interval in ABBA order); the runner runs a private copy A/A on build 84c705ad (3ecd802), cmux21, 5 rounds, paired t-interval: - as run (base first in every round): 27 of 28 metrics pass; acpmux RSS growth MB/h fails, a run-position effect (the second run grows less); - labels in ABBA order: 28 of 28 pass. aa-failures.json records no failed metric and the position effect; app request p99 is not tested (no probe). The A/A run died at its very end because austins-day.sh was edited while it ran (bash reads a script as it runs); the runner now execs a private copy. * austins-day: usage reads the harness file, not the unlinked private copy * austins-day: acpmux chat-open failures per run (pa-b10's measure) Every A/A run and every #28159 rerun run failed most acpmux chat opens: session/new answered 'agent host start did not finish within 10s' (and some 'agent_start did not finish within 90s') from the second 25-chat wave on. New rows, carried in every A/B table: acpmux chat opens, open failures, failure share, host-start 10 s time-outs, agent_start 90 s time-outs; open failures count as an Austin failure and need a failing base per metric. selftest.py: 13 tests OK. * austins-day: fixes from the review of aa7c09e (CHANGES) Should-fix: 1. Workspace close: the app and the daemon are polled in their own threads from the moment the action is sent, so a slow 'cmux workspace list' or the action's own reply no longer delays the app's reading. 2. The whole harness (scripts, summarize.py, aa-failures.json, build_names.py, the acpmux test agent) runs from a private copy made at the start; an edit to the checkout during an A/B changes nothing in it. 3. A run whose agents did not all start, or did not all get hooks, is a harness fault: the A/B is invalid (less load would look 'better'). 4. packs.py compares only its parameters with READY ('made' is a result), so the 19,110-pack fixture is built once per host, not in every lease. 5. Every step of run_arm that must work checks itself (set -e does not act inside && and ||): staging, copy, fixture build, launch. Notes: the app probe without a start read is 'not measured'; a harness-fault message names each broken run; a killed script releases its lease (EXIT trap) and never signals a reaped acquire pid; the fake Read tool reads the fixture's dir*/ files; README says which series cover only the load phase and what 5-round ABBA leaves of a position effect; hog docstring corrected. selftest.py: 14 tests OK. * austins-day: fixes from the re-review of 2b7fd10 (CHANGES) Should-fix: 1. A workspace close starts only when it can end before the load deadline, and its watchers stop at their own 30 s bound, not at the run's stop flag: the end of a run no longer cuts a close and counts it 'not done'. 2. Each Chief-wave chat is one attempt (the driver records chats per wave); a chat whose connect is refused, closed or timed out counts as a failed open, so an acpmux restart or hang is not hidden. New row: connection failures. 3. 'acpmux chat opens' is no longer a lower-is-better verdict row (a faster head runs more waves). Notes: missing agent terminals (fewer than --agents) are a harness fault; the uncommitted-edit check also covers build_names.py and the acpmux test agent; the private copy is removed on every exit, usage errors too; on INT/TERM/HUP the runner stops a waiting lease request of its own (still a live job, never a reaped pid), removes the remote run folder and releases the lease; the close watchers poll every 100 ms; the RETURN-trap comment is corrected. selftest.py: 14 tests OK. * austins-day: a refused acpmux connect is a failed chat open (re-review of a84ddb0) Should-fix: one_chat kept answers[1:], which dropped the failed connection (the only entry when the connect is refused or initialize never answers), so the chat was stored empty and counted nothing. It now drops the initialize answer by its method and never stores an empty chat; summarize counts an empty chat from older runs as a failed open. New test ChatRecord runs the driver's own one_chat against a socket nobody listens on (red on the old line: IndexError, the chat was empty; green now). Notes: the close bound is the real worst case, 125 s; the private copy is removed only under $TMPDIR/austins-day.* again; the fallback for runs without a chat record counts session/new only (an undercount, in the README); the raw open-failure count is no longer a verdict row (the share is). selftest.py: 15 tests OK. * austins-day: summarize reads runs from drivers before the chat record (review of b7574be) Drivers before 152ee4f (the A/A's 02ebc4c among them) wrote "chats": <wave size>; chat_firsts looped over that int and raised TypeError, so those runs never reached the session/new fallback. It now reads chats only when it is a list. New test OldRuns (a wave with "chats": 25). The A/A runs re-read with this summarize: 31 of 31 tested metrics pass in ABBA order. selftest.py: 16 tests OK. * austins-day: the answers fallback reads chats only when it is a list (delta review of 76f61c1) An old run's wave with "answers": [] and "chats": 25 (every chat failed to connect) raised TypeError in the answers line. Same isinstance check as chat_firsts; OldRuns now has such a wave. selftest.py: 16 tests OK. * austins-day: label the agent CLI rate as assumed (lead decision 2026-10-30) The fake agents' 'cmux workspace list' every 3 tool calls is assumed, not from Austin's data; the #28041 A/B's process samples show it fills the 64-slot pool with CLI calls (median 62-63 live cmux CLI processes, 0 hook helpers), so hook-path fixes cannot show in refused connections. The README and every summary (text and JSON) now say so. Kept for comparability until Austin's real client mix is measured. selftest.py: 16 tests OK. * austins-day: registry-lock held share; pool holders from 'cmux client list' - summarize: registry_lock_held_share, the change of hold count x mean hold between the run's first and last server-stats, over that window (lead 2026-10-30, for the B3 #28147 A/B; 81-82 % in both arms of the B1 A/B re-read). A new row; no existing computation changes. - driver: the pool-holder sample reads 'cmux client list' (client_kind, name, transport, connected_seconds); 'raw command list-clients' was refused (usage.invalid) in every run so far. Descriptive, outside every verdict; it runs in the sampler as before. selftest.py: 16 tests OK. * austins-day: the client list is read as the CLI prints it; held share is descriptive (review of f8a8afb) - 'cmux --json client list' prints the daemon's bare array; the driver's value() turned it into {}, so the pool-holder sample still read nothing. client_list_entries() takes a bare array or a {value|clients|items} list. New selftest ClientList runs it on a bare-array sample. - The registry lock held share is not A/A-tested and has no base-failure rule: it is no longer a verdict row; each run's text line shows it as descriptive, with its integer-mean bias (~1-2 points low at most). No existing metric's computation changes. selftest.py: 17 tests OK.

…aits Conflicts were feat's squash of #28147 (pa-b3) plus its later hook_queue section against this branch's copy of the pa-b3 commits. Kept this PR's changes: SiteStats copied out before the site labels are formatted, the new writer_lock_order tests, and the state-lock comment. Took feat's hook_queue (diagnostics, server_stats, tests), try_lock outside tests, and feat's generated bindings and spec (this PR changes no protocol).

…napshot Same resolution as #28159's merge 1722dc5: this PR's lock-site copy, writer_lock_order tests and state-lock comment; feat's hook_queue, try_lock outside tests, and feat's generated bindings and spec (no protocol change).

lawrencecchen · 2026-11-01T03:14:42Z

Review (claude-review-subagent-cw): SHIP at 1722dc5

Scope: the merge 1722dc5 (remerge diff) and the net change over feat 2a822f8. I read the code only. I did not compile or run tests.

Merge

The remerge diff keeps both sides. diagnostics.rs keeps the copy of SiteStats out of the sites mutex before site_label (Clone, Copy on OctaveHistogram and SiteStats) and adds feat's hook_queue field. signaled_mutex.rs keeps this PR's comment and feat's try_lock without #[cfg(test)]. server_stats.rs and server_stats_tests.rs are feat's. writer_lock_order.rs adds this PR's two tests only.
git diff 2a822f8157d 1722dc57f4c changes 5 files under cmux-tui-core/src and no file under cmux-tui/bindings or cmux-tui/spec. No protocol type changes, so the generated files agree with the Rust types.
Feat since 855fef6: only babe9dc (the squash of cmux-tui: the daemon no longer deadlocks settling a late registry receipt; server-stats reports both locks by site #28147) touches lock_state_pinned, commit_registry_intent*, drain_unsettled, PinnedState or the receipt settle. No feat caller uses PinnedState::unpin or commit_registry_intent_with; the only caller is workspace_rename.rs. I found no feat code that holds the connection pin or the state lock across a writer receipt wait.

Net change

signaled_mutex.rs:372-373: the drain runs before the pin, so the writer can answer during the wait. Correct. The caller holds the registry, so no new indeterminate commit can appear.
registry_intent.rs:230: a thread that holds the connection does not wait. It only takes answers that already arrived. Correct.
workspace_rename.rs:234-242: state is released before the receipt wait and taken again after it. The registry stays held, so no topology writer can interleave. The workspace is found again by key, which is correct if the index moved.
registry_intent.rs:311: the new debug_assert fires if a sender holds state without the pin. I read the commit callers (commit_effect_intent, commit_terminal, commit_workspace_registry, feed_local, terminal_move, surface_spawn, adoption, lifecycle commits). The callers that hold state are pinned and commit locally before the assert. The other callers hold no state.

Not blocking

signaled_mutex.rs:372: every pinned flow now waits up to UNSETTLED_DRAIN_WAIT (2 s) under the registry while an indeterminate answer is missing. This includes pinned reads (with_resource_projection, frontend_projection, full_projection). A failed drain leaves the entry unresolved, so each later pin waits again until the writer answers. Before, only the prepare paths waited. The writer answers within a bounded time, so the effect is temporary. But it is a new wait on read paths when the writer stalls.
writer_lock_order.rs:229, :288-292: the tests use fixed sleeps and sampling loops (REVIEW.md section 6). The checks are negative ("state is free", "connection is free"), so a sample loop is acceptable. But connection_free can fail if a background thread takes the connection for a moment during the 200 ms sample.

lawrencecchen · 2026-11-01T03:33:02Z

Conflict wave: merged feat-cmux-next 2a822f8 into this branch (merge 1722dc5). The conflicts were feat's squash of #28147 (pa-b3), plus its later hook_queue section, against this branch's copy of the pa-b3 commits. I kept this PR's changes: the SiteStats copy out before the labels are formatted, the new writer_lock_order tests and the state-lock comment. I took feat's hook_queue (diagnostics, server_stats, tests), try_lock outside tests, and feat's generated bindings and spec (this PR changes no protocol).

Fleet evidence at 1722dc5:

Compile (cmux-ci compile-check --package CmuxNext --app): CmuxNext passed (126 s), app passed.
cmux-tui-rust-check.sh test writer_lock_order (macOS): 6 tests run, 6 passed.
clippy: RED, a base finding: redundant closure in acpmux hub/processes.rs, fixed on feat by base red: acpmux clippy redundant_closure in process_values #30009 (55361f1) after this merge. This PR does not touch acpmux.
Review: SHIP at this head (comment 6552755355).

…-tui, tool argv, held-share test (#28505) * scripts(cmux-next): Austin's day, the P-AUSTIN load scenario on an exclusive fleet mini 80 terminal agents through the real claude shim, wrapper and hooks (a fake Claude that runs the hook settings it is given on every tool call and stop, and calls the cmux CLI), a partial clone with 19,110 promisor packs, 25-chat acpmux waves, a bounded memory hog up to swap, and workspace closes through the app's closeWorkspace action. Measures app daemon requests (probe and log time-outs), refused connections, lock waits, close time, acpmux restarts, memory, load and swap, and the debug.hangs stalls over 100 ms by top cmux symbol. One JSON plus a text summary; --base/--head alternate on one host. A fleet bench each agent runs itself; not a CI job or gate. selftest.py (run by hand): 4 tests OK on cmux-orchestrator-1. * austins-day: resolve a job id without an empty --ref argument First run on the feat-cmux-next tip (146a973, build ab32ee84) on cmux20: 80/80 agents hooked, 13,693 hook runs; daemon refused 378 connections (peak 64/64); registry lock wait p99 393 ms (max 5.2 s); person CLI p99 4.65 s; 7 of 10 workspace closes timed out at 2 s (p95 8.7 s, one never closed); 71 cmux main-thread stalls over 100 ms. Not seen: acpmux restarts, app no-reply log lines (the log query found none; the app probe is pending). * austins-day: agents draw to their terminals and use CPU; swap to 14 GB; stalls by own cmux symbol Run 1 had load 35-54 (Austin 150-252) and swap 9 GB (Austin 17 of 18): the fake agents were silent between tool calls. Each now draws as Claude Code's TUI does while it thinks (a spinner at 10 Hz, ~300 chars/s of streamed text) and keeps 15 % of a core busy, and the hog targets 14 GB of swap. A stall is named by its first frame in cmux's own code (a CmuxNext module), not by a stdlib generic specialized into the binary. The app log query also counts the daemon category's 'failed' lines. * austins-day: validity rules, owner, 5 runs per arm with bootstrap verdicts, long-arm metrics (cmux-ad) - An A/B gives no verdict when any base run shows none of Austin's failures ('invalid: base did not reproduce') or when the head changes this folder ('invalid: head edits the harness'). - The README names the owner (the P-AUSTIN harness owner, agent pa-d): only the owner edits the folder and runs each P-AUSTIN PR's A/B. - 5 alternating runs per arm by default. Per metric: a paired percentile bootstrap 95 % CI of mean(head - base) over the rounds (10,000 resamples, fixed seed); 'better'/'worse' only outside the interval, else 'no difference'. - Long-arm metrics: open fds per process (lsof) and a per-hour growth slope of RSS, footprint and fds. The 60 min arm itself is blocked by the 10 min capture-host lease (README). selftest.py: 7 tests OK (cmux-orchestrator-1). * austins-day: stalls split into CODE and WAIT (P-MAIN rule G, lead 2026-10-30) A stall over 100 ms is CODE when cpu_ms is at least half of duration_ms (or the app's own debug.hangs cause, from fix/stall-hang-wait-split on), else a WAIT. Only CODE stalls with a cmux frame are reds by top cmux symbol and count in austin_failure; WAITs are counted apart (count, total ms, largest CPU share, page-ins) and reported next to swap and memory level, owner B8. Run 1 re-read: 92 stalls over 100 ms, 0 CODE, all WAITs (max CPU share 27 %, 43.9 s wall, 0.89 s CPU), so its 3 named sites are not reds. selftest.py: 8 tests OK (cmux-orchestrator-1). * austins-day: hook CPU per hook run, per-metric validity, CLI and close p99 rows For the P-AUSTIN A/Bs (#28041 B2: hook CPU; #28159 B3: lock waits, app requests, workspace close): - fake_claude logs each hook's CPU (wait4 user + system of the hook command and the children it waited for). A hook past its timeout is killed only while it is not yet reaped (WNOWAIT, then a lock), never a reused pid. - Each metric needs its own failure on every base run (METRIC_FAILURE: the daemon's 100 ms lock-stall line, the B3 50 ms app-request target, the 2 s action deadline, refused connections for CLI and hook rows); else that metric says 'invalid: base did not reproduce'. No pairs: 'not measured'. - New rows: agent CLI p99, hook CPU total and p99, workspace close p99. selftest.py: 9 tests OK (cmux-orchestrator-1). * austins-day: each run records the harness commit; refuse uncommitted harness edits * austins-day: hook waits work on the minis' Python 3.9; harness faults void a run; a failed arm is retried once The #28159 A/B (10:15Z-12:05Z) logged no hook runs: the minis run Python 3.9.6, which has no os.waitid on macOS, so each hook thread died right after it started its hook. The hooks still ran but nobody waited for them (Claude Code's sync hooks became fire-and-forget) and their time and CPU were lost. - The hook wait is a kqueue process-exit event on macOS (a pidfd on Linux); the child stays unreaped until wait4, so a time-out kill never reaches a reused pid. - A crash in a hook thread is logged as the hook's result. - A run whose hooked agents logged no hooks, or whose hooks hit a harness error, makes the A/B 'invalid: harness faults'. - A run that got no lease or no app is retried once (head round 5 got no capture host before the acquire time-out). selftest.py: 9 tests OK. * austins-day: refused as a rate and a share; base must be the PR's parent; who fills the pool From the #28159 review (pa-b3, issuecomment-6523481295), both points real: - 'refused' was a cumulative count read at each run's last answered server-stats, which came 28-47 s earlier on some base runs, so the arms were read at different times. Arms now compare refused per second over each run's own server-stats window and refused share of connects. - The base was the merge base with feat (9fa133b) while #28159 sits on the open #28147 (100b8e3), so the A/B measured both PRs. The README now says the base is the PR's parent; the summary lists base..head and refuses a base that is not the head's ancestor. - New: pool_holders, sampled with server-stats: the daemon's list-clients by kind/name, live hook-helper and CLI processes, and the refusals each client kind saw (lead: whose connections fill the 64). selftest.py: 11 tests OK. * austins-day: A/A self-test (cmux-ad) austins-day.sh --aa JOB runs one build in both arms (5 alternating rounds). summarize.py --aa passes a metric whose 95 % interval contains 0 and fails the rest; the owner records failures in aa-failures.json, and every later summary marks those metrics 'invalid (fails A/A)'. An A/A with two builds is refused. selftest.py: 13 tests OK. * austins-day: paired t-interval instead of the bootstrap; ABBA run order; a position estimate per metric (lead decision) The A/A on build 84c705ad (3 rounds) failed 10 of 28 metrics on identical arms with the percentile bootstrap, which is far too narrow over 5 or fewer pairs; a paired t-interval (95 %, t with n - 1 df) failed 2. The runner now alternates which arm runs first (ABBA; arm.json records the position), and each metric gets a position estimate (second run - first run) so a warm-up or drift effect shows apart from the arm difference. selftest.py: 13 tests OK. * austins-day: A/A of 2026-10-30 recorded (passes with the t-interval in ABBA order); the runner runs a private copy A/A on build 84c705ad (3ecd802), cmux21, 5 rounds, paired t-interval: - as run (base first in every round): 27 of 28 metrics pass; acpmux RSS growth MB/h fails, a run-position effect (the second run grows less); - labels in ABBA order: 28 of 28 pass. aa-failures.json records no failed metric and the position effect; app request p99 is not tested (no probe). The A/A run died at its very end because austins-day.sh was edited while it ran (bash reads a script as it runs); the runner now execs a private copy. * austins-day: usage reads the harness file, not the unlinked private copy * austins-day: acpmux chat-open failures per run (pa-b10's measure) Every A/A run and every #28159 rerun run failed most acpmux chat opens: session/new answered 'agent host start did not finish within 10s' (and some 'agent_start did not finish within 90s') from the second 25-chat wave on. New rows, carried in every A/B table: acpmux chat opens, open failures, failure share, host-start 10 s time-outs, agent_start 90 s time-outs; open failures count as an Austin failure and need a failing base per metric. selftest.py: 13 tests OK. * austins-day: fixes from the review of aa7c09e (CHANGES) Should-fix: 1. Workspace close: the app and the daemon are polled in their own threads from the moment the action is sent, so a slow 'cmux workspace list' or the action's own reply no longer delays the app's reading. 2. The whole harness (scripts, summarize.py, aa-failures.json, build_names.py, the acpmux test agent) runs from a private copy made at the start; an edit to the checkout during an A/B changes nothing in it. 3. A run whose agents did not all start, or did not all get hooks, is a harness fault: the A/B is invalid (less load would look 'better'). 4. packs.py compares only its parameters with READY ('made' is a result), so the 19,110-pack fixture is built once per host, not in every lease. 5. Every step of run_arm that must work checks itself (set -e does not act inside && and ||): staging, copy, fixture build, launch. Notes: the app probe without a start read is 'not measured'; a harness-fault message names each broken run; a killed script releases its lease (EXIT trap) and never signals a reaped acquire pid; the fake Read tool reads the fixture's dir*/ files; README says which series cover only the load phase and what 5-round ABBA leaves of a position effect; hog docstring corrected. selftest.py: 14 tests OK. * austins-day: fixes from the re-review of 2b7fd10 (CHANGES) Should-fix: 1. A workspace close starts only when it can end before the load deadline, and its watchers stop at their own 30 s bound, not at the run's stop flag: the end of a run no longer cuts a close and counts it 'not done'. 2. Each Chief-wave chat is one attempt (the driver records chats per wave); a chat whose connect is refused, closed or timed out counts as a failed open, so an acpmux restart or hang is not hidden. New row: connection failures. 3. 'acpmux chat opens' is no longer a lower-is-better verdict row (a faster head runs more waves). Notes: missing agent terminals (fewer than --agents) are a harness fault; the uncommitted-edit check also covers build_names.py and the acpmux test agent; the private copy is removed on every exit, usage errors too; on INT/TERM/HUP the runner stops a waiting lease request of its own (still a live job, never a reaped pid), removes the remote run folder and releases the lease; the close watchers poll every 100 ms; the RETURN-trap comment is corrected. selftest.py: 14 tests OK. * austins-day: a refused acpmux connect is a failed chat open (re-review of a84ddb0) Should-fix: one_chat kept answers[1:], which dropped the failed connection (the only entry when the connect is refused or initialize never answers), so the chat was stored empty and counted nothing. It now drops the initialize answer by its method and never stores an empty chat; summarize counts an empty chat from older runs as a failed open. New test ChatRecord runs the driver's own one_chat against a socket nobody listens on (red on the old line: IndexError, the chat was empty; green now). Notes: the close bound is the real worst case, 125 s; the private copy is removed only under $TMPDIR/austins-day.* again; the fallback for runs without a chat record counts session/new only (an undercount, in the README); the raw open-failure count is no longer a verdict row (the share is). selftest.py: 15 tests OK. * austins-day: summarize reads runs from drivers before the chat record (review of b7574be) Drivers before 152ee4f (the A/A's 02ebc4c among them) wrote "chats": <wave size>; chat_firsts looped over that int and raised TypeError, so those runs never reached the session/new fallback. It now reads chats only when it is a list. New test OldRuns (a wave with "chats": 25). The A/A runs re-read with this summarize: 31 of 31 tested metrics pass in ABBA order. selftest.py: 16 tests OK. * austins-day: the answers fallback reads chats only when it is a list (delta review of 76f61c1) An old run's wave with "answers": [] and "chats": 25 (every chat failed to connect) raised TypeError in the answers line. Same isinstance check as chat_firsts; OldRuns now has such a wave. selftest.py: 16 tests OK. * austins-day: label the agent CLI rate as assumed (lead decision 2026-10-30) The fake agents' 'cmux workspace list' every 3 tool calls is assumed, not from Austin's data; the #28041 A/B's process samples show it fills the 64-slot pool with CLI calls (median 62-63 live cmux CLI processes, 0 hook helpers), so hook-path fixes cannot show in refused connections. The README and every summary (text and JSON) now say so. Kept for comparability until Austin's real client mix is measured. selftest.py: 16 tests OK. * austins-day: registry-lock held share; pool holders from 'cmux client list' - summarize: registry_lock_held_share, the change of hold count x mean hold between the run's first and last server-stats, over that window (lead 2026-10-30, for the B3 #28147 A/B; 81-82 % in both arms of the B1 A/B re-read). A new row; no existing computation changes. - driver: the pool-holder sample reads 'cmux client list' (client_kind, name, transport, connected_seconds); 'raw command list-clients' was refused (usage.invalid) in every run so far. Descriptive, outside every verdict; it runs in the sampler as before. selftest.py: 16 tests OK. * austins-day: the client list is read as the CLI prints it; held share is descriptive (review of f8a8afb) - 'cmux --json client list' prints the daemon's bare array; the driver's value() turned it into {}, so the pool-holder sample still read nothing. client_list_entries() takes a bare array or a {value|clients|items} list. New selftest ClientList runs it on a bare-array sample. - The registry lock held share is not A/A-tested and has no base-failure rule: it is no longer a verdict row; each run's text line shows it as descriptive, with its integer-mean bias (~1-2 points low at most). No existing metric's computation changes. selftest.py: 17 tests OK. * austins-day: held-share bias stated per hold; selftest for lock_held_share (review follow-ups of 856cf72) The daemon's mean hold is a whole number of microseconds, so the held share is low by up to about 1 µs per hold in the window (an exact value needs the daemon to report its hold sum). New test LockHeldShare: the change of count x mean over the window, no window, no stats, and a counter reset clamped at 0. selftest.py: 18 tests OK. * austins-day: pool holders through cmux-tui; each tool call records its command - 'cmux client list' is refused (usage.invalid) by design, not by a stale bundled CLI: 'client' and 'raw' are in CMUX_TUI_ONLY_SCOPES (cli.rs), and the app bundles cmux-tui beside cmux. The holder sample now runs the bundled cmux-tui. This is why both 'raw command list-clients' and 'client list' read nothing in every run so far. - fake_claude logs each tool call's argv: the #28147 A/B's first run on cmux17 had 157 git calls hit the 120 s time-out, and the log did not say which command. (A cold fixture was ruled out: every first run of the 63 earlier runs also built the fixture and had 0 tool time-outs.) Neither changes a verdict-row input: the holder sample is descriptive and the argv is one more field in a log line. selftest.py: 18 tests OK. * austins-day: main arm without agent CLI calls; lock detail and client list after the load (lead 2026-10-30) - Main arm: --cli-every 0 is the default. No shipped hook, plugin or status line calls 'cmux workspace list' per turn (hooks send only session.journal.append), so the old one-per-3-tool-calls had no evidence and drove 43 % of the registry lock busy share. --cli-every 3 stays as the labeled 'assumed' sensitivity arm; the summary names the arm and refuses runs with mixed rates. This changes a verdict-row input: the new main arm needs its own A/A. - server-stats include ["locks"] (cmux-tui raw command), read once right after the load: lock_top_sites, the top 20 sites of the registry lock and mux.state by total and by p99 hold. Descriptive. - The daemon's client list is read only once, after the load; during the load the holder sample counts processes only, as in the earlier A/As (review note of cdd562d: the working probe added a connection and a registry-lock acquisition every 15 s to the measured window). selftest.py: 20 tests OK. * austins-day: notes of the review of 3bf19c2; the A/A record is per scenario - The samplers stop before the post-load reads (locks, client list), so the end read stays the last server-stats and no sample falls in those calls. - The post-load client list is summarized (pool_holders_after_load). - aa-failures.json is keyed by scenario (agent CLI rate): '3' holds the earlier A/As; a summary whose scenario has no A/A marks every verdict 'not A/A-checked' (the main arm until its A/A is recorded). Runs that recorded no rate are counted as the old default, 3. - The scenario note names the real rate; the lock-detail line says its numbers count since the daemon started. selftest.py: 21 tests OK. * austins-day: a revert pair is a valid base; requested paired comparisons; named lock sites (lead, B3 1d) - B3 1d's 'before' is the fix's own tree plus one commit reverting the fix (b472663 = fee4db7 + revert). The base is then not an ancestor of the head, which the validity rule refused. A base whose parent is the head and whose subject starts with 'Revert ' is now accepted; measured_commits names the reverted commit. - 'requested paired comparisons': the same paired t-interval and n for values that are not verdict rows (registry lock held share, lock wait p99, app request p99, and the hold p99 and total hold of the registry lock at mux/frontend_projection.rs:9 and mux/terminal_progress.rs:18, from the post-load lock detail). No verdict, no A/A check. No verdict-row input changes. selftest.py: 22 tests OK. * austins-day: a strict revert pair; named lock sites by function (review of 2ea5ef9) Should-fix 1: is_revert_pair now needs exactly one parent (the head), a 'This reverts commit X' line with X an ancestor of the head, and the base-to-head diff equal to X's own diff; the summary prints every measured commit. The test adds an empty 'Revert' commit and a merge (both refused). Should-fix 2: B3 1d moves the terminal revision read's lock from frontend_projection.rs:9 to its fallback at :20, so a site named by line never paired. Sites are now named by function (NAMED_LOCK_SITES lists the lines per build); several lines add up (total) or take the max (p99); a site in neither top-20 list counts as 0 when the lock's sites count shows all were listed, else unknown; a requested comparison shows how many rounds it dropped for an unknown value. New test NamedSites. selftest.py: 23 tests OK. * austins-day: an upper bound for a named lock site absent from a full list; bytes-safe revert diff (review of 7b1d2cd) The registry lock has far more than 20 sites, so a named site that is rarely used (B3 1d's fallback at frontend_projection.rs:20) is in neither top-20 list: its value is unknown and its round is dropped. Each run now also records an upper bound for it, the smallest listed value (hold_total_us_at_most, hold_p99_us_at_most), shown on the run's line and never mixed into a paired difference. The revert-pair diffs are read as bytes. README notes NAMED_LOCK_SITES fits the builds it names. selftest.py: 23 tests OK. * austins-day: a lock-site bound is rounded up on the text line; no site lines without lock detail (notes of the review of 748dc58) A 40,000 us bound printed as 'at most 0.0 s', a smaller bound than the data gives; it is now rounded up (0.1 s). Runs from builds without the lock detail print no named-site lines. Text only: no recorded value changes. selftest.py: 24 tests OK. * austins-day: selftest: move an assertion back into its own test 61b4d29 was committed with selftest.py red (NameError): the new test was inserted above the last line of NamedSites' first test, so that line ran in the new test. Code unchanged. selftest.py: 24 tests OK. * austins-day: record the main-arm A/A; the holder sample names what its processes run - aa-failures.json scenarios['0']: the main-arm A/A of 2026-10-30 (build 84c705ad, cmux24, harness 8d4f908, 5 ABBA rounds): 30 of 30 tested metrics pass, none fails; agent CLI p99 and app request p99 not tested; workspace close not tested (base did not fail). The base round 1 note (one ssh date by the owner at the start of its load) is recorded. - With no agent CLI calls, the A/A still had a median of 48 live processes of the bundle's 'cmux' and the pool still full (refused 6.2/s); the role alone did not say what they run. The holder sample now records, from the same ps read, each role's command shapes (binary, then the command words without --socket, --json and ids), by count; summarize keeps the median per shape (process_kinds_median). Descriptive; no verdict input changes. selftest.py: 25 tests OK. * austins-day: hook-helper shapes keep their event; A/A record wording (notes of the review of 59ea3fd) - The bundle's hook helper path has spaces ('cmux DEV <tag>.app'), so the shape lost its event word; the app path is now replaced before the split. ProcessKinds tests a hook helper inside the bundle. - aa-failures.json scenarios['0']: the workspace close rows were tested in the A/A (their intervals contain 0); they need a failing base only in an A/B. The record also says 59ea3fd adds under 1 ms per sampler tick during the load, judged too small to need a new A/A. selftest.py: 25 tests OK. * austins-day: the app path is replaced in the cli role's arguments too (note of the review of 771cc4f) A cmux CLI argument that holds a bundle path ('cmux DEV <tag>.app', with spaces) split at its spaces; the replace now runs on the arguments of both roles. Descriptive shapes only. selftest.py: 25 tests OK. * austins-day: time to first answer per acpmux chat; #28442 rows in the requested comparisons For the #28442 A/B (pa-b10, lead 2026-10-30): the driver records, for each acpmux request, when the first agent_message_chunk for it arrived (first_chunk_ms, read from the session/update notifications it already skipped). summarize: acpmux_first_answer_ms per chat = session/new ms + its prompt's first-chunk ms, for chats that got an answer; acpmux_answered_chats. Requested paired comparisons gain the 90 s agent_start time-outs, the first answer median and p95, and the answered-chat count (no verdict, n shown). No verdict input changes. Tests FirstAnswer (the summary, and the driver's own acpmux_call against a tiny ACP server). selftest.py: 27 tests OK. * austins-day: first answer over all chats (unanswered at their time to failure); labels; one time-out row (notes of the review of d965fcf) - Survivor bias: the first-answer rows over answered chats only are not comparable when the arms answer different numbers of chats. New row over all chats: an answered chat at its first-answer time, an unanswered one at the time it failed (its recorded steps, or the 120 s client time-out), a lower bound; the rows say which they are. - The 90 s agent_start time-outs are a verdict row; the duplicate requested row is gone. selftest.py: 27 tests OK. * austins-day: every unanswered acpmux chat ranks last at 150 s; chats still running at the wave limit are recorded (review of bcbeee4, CHANGES) Should-fix: 'time to failure' for an unanswered chat differed by failure path (a lost prompt after session/new counted as ~2 s, a refused connect as 120 s), so the all-chats first-answer rows could flip the sign of an A/B. Every chat with no answer now counts at one value above any answer, NO_ANSWER_MS = 150 s (the wave's join limit), so p50 and p95 are exact order statistics with no-answer last; rows relabeled. Test chats for a lost prompt after session/new and a refused connect. Note: a chat still running when its wave stops waiting was left out of the record (survivor bias for the slowest); the driver now records it as 'still running at the 150 s wave limit' (from a snapshot of the finished chats, so a late append is never read). selftest.py: 27 tests OK.

…napshot #28159 landed on feat (a220bf2). The three conflicts are this PR's own later changes to that code (the pinned prepare refuses a late commit, the rename applies its name after the revisions); this PR's side kept.

…ly what they answer (#28289) * bench(cmux-tui): agents load scenario for the daemon latency bench 80 fake terminal agents run the real hook helper (one durable journal append per turn event) and a CLI call in a loop, a churn client creates and closes terminals, and an app client sends what the cmux app sends (put-frontend-projection window saves, list-workspaces, session.snapshot, identify). It reports the app requests' latency, the requests over the app's 2 s timeout and both daemon locks by site, the shape of the 2026-10-29 report (#27698). Run: cmux-ci run --class exclusive --measurement --script scripts/measure/perf-10x-daemon.sh --ref SHA --arg=scenarios=agents Measurement only; no CI step, gate or budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cmux-tui): server-stats locks section: registry and state lock holders by site server-stats include=["locks"] reports the workspace registry lock and, new, the mux state lock: wait and hold histograms, the current holder, the last stall, and the top twenty acquiring sites by total hold and by p99 hold, each with its hold and wait p99 and maximum. Sites come from #[track_caller] (lock_state_pinned now carries it too, so its callers are named instead of the wrapper). Per site, two power-of-two histograms (256 bytes each) are updated under the site table lock that the release already took, while the lock is still held: no allocation after a site's first acquisition and no new contention. The state lock's existing rank and holds bookkeeping is unchanged. The section is optional (absent unless named), so older SDK decoders keep working; schema, bindings, api-compat surface and commands.md updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(cmux-tui): name the callers of the registry and state lock wrappers in lock stats #[track_caller] on the Mux wrappers that take the workspace registry or the mux state lock (with_state, surface, resolve_resource_path, commit_state, commit_resource_mutation_plan, with_resource_projection, the resource effect helpers, tree_decorations and others), so server-stats locks names the request path that holds the lock instead of the wrapper's own line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(cmux-tui): settling a late registry receipt must not lock state again (red) lock_state_pinned settles the late answers of indeterminate registry commits (cx-g1fa.3) while it holds the registry and state. The settle then ran publish_resource_event, whose pending-directory step locks state again: a lock-rank panic in debug builds and a self-deadlock of the whole daemon in release builds (std Mutex), on a path a loaded Mac reaches when a journal commit outlives its 30 s receipt wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmux-tui): settling a late registry receipt publishes only the journal event The settle runs under the registry and state; publish_resource_event also publishes pending terminal directories, which locks state again (and then the registry). Publish the journal event only: subscribers still read the new head, and a directory still pending is published by the terminal's next output or the next committed graph change, as before the settle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(cmux-tui): drop unnecessary path qualifications in the lock tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(cmux-tui): no request holds state across a writer receipt, or the connection across an unsettled wait (red) Two holders that wait while they hold a lock (#27698 audit, lock waits p90 5 s / p99 17 s on a loaded Mac): - rename-workspace held the mux state lock across its journal writer receipt wait (up to the 30 s durable wait); - a request that found an earlier indeterminate commit waited for its answer (2 s) with the registry connection pinned, though the writer needs that connection to commit and answer: the wait could only run out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmux-tui): no request waits for a writer answer while it holds state or the connection - rename-workspace releases the mux state lock before its journal writer receipt wait and takes it again to apply the committed name (found by workspace key). The registry stays held, so no topology writer interleaves; state readers see the old name until the commit is durable, instead of queueing behind the disk for up to the 30 s durable wait. - lock_state_pinned waits for the answers of earlier indeterminate commits before it pins the registry connection, and drain_unsettled never waits while this thread holds the connection: the writer needs the connection to commit and answer, so that wait could only run out (2 s) and fail the request. A commit still unanswered after the unpinned wait is refused at once, as it was after the pinned wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(cmux-tui): PinnedState has no unpin; no receipt wait under state With rename no longer holding state across its receipt, nothing unpins a held state any more: PinnedState always holds its connection pin, and commit_registry_intent_with loses its release_pin hook. A debug assert refuses a journal writer receipt wait while the thread holds the mux state lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(cmux-tui): a rename's reply comes after its name is in state; readers see the old name only until then Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(cmux-tui): move the mux into the settle test's pinning thread (clippy redundant_clone) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmux-tui): server-stats locks formats its site labels after it releases the site table Every release of the registry and state locks takes the site table mutex; detail() now copies the raw records out and formats the file:line labels after the unlock (review of #28147). Schema text: the holder site names either lock, and sites counts the locations that released the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(cmux-tui): reads run no write, one-collection reads read one collection, a pinned prepare refuses a late commit (red) #27698 lock work, design rule 1 (no write or I/O under the registry lock) and the next steps after #28159: - session.snapshot deleted stale notification read marks under the registry lock (a read that writes); the open must drop them instead; - workspace.list and session.ping built the whole session snapshot (topology, terminals, projections, journal head) for their workspaces or cursor; - a pinned prepare took a late answer that landed after its settle and built on state without it (#28159 review, Low 1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmux-tui): reads run no write; workspace.list and session.ping read only what they answer; a pinned prepare refuses a late commit Design rule 1 of the #27698 lock work: no write under the registry lock in a read. - The session snapshot no longer deletes read marks of notifications outside the retained window; it skips them, and the registry open drops them (the runtime prune queue does not survive a restart). - The snapshot no longer publishes pending terminal directories (a full terminal.cwd commit) before it reads; terminal output, resize, reconnect and every committed graph change publish them. - workspace.list builds the workspaces alone (workspace rows, the topology frame, workspace decorations) under the registry lock only, not the whole session snapshot under the registry and state; session.ping reads the topology frame for its cursor. One builder makes the workspace values for both the snapshot and the list. Under an 80-agent load the session snapshot was the registry lock's longest holder (resource_api.rs:551, 25.7 s held in 60 s, fleet step 73e93323); most of its calls were agents' list calls. #28159 review, Low 1 and Low 3: lock_state_pinned settles the late answers that arrived when it pinned, but an answer can land after that settle (the writer drops the connection, then sends its receipts). drain_unsettled now refuses a pinned caller while such an answer is unsettled instead of letting it build on state without the late commit; the next pin settles it. A rename whose workspace left state after a durable commit raises the in-memory revisions before it reports the error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmux-tui): drop an unnecessary path qualification (clippy unused_qualifications) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(cmux-tui): the durable restart test waits for lifecycle readiness, not for the socket named_server_status_stop_alias_and_durable_restart_preserve_topology sent server status and reload-config right after the socket accepted. The server answers ordinary requests from that moment but its lifecycle commands only once it is lifecycle-ready (Mux::mark_server_lifecycle_ready, 'Ordinary control clients may connect before this point'); before that they refuse with 'still starting' (exit 3). The test passed only while the workspace list in between took longer than the restore. With workspace.list now building only the workspaces (this branch), it failed once on a loaded host (fleet step 23b44075: reload-config exit 3, 'the local server is still starting'). It now waits for lifecycle_ready on identify after each start, the product's own readiness signal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(cmux-tui): an adopted idle shell presents its reported directory after a restart (red) Review of #28289 (M1): the session snapshot no longer publishes pending terminal directories (a read writes nothing), and nothing else published the directory report a host replays at adoption: an idle shell after a daemon restart showed its launch directory until it printed or the graph changed (#12978's startup race). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmux-tui): adoption publishes the shell's replayed directory; workspace.list reads under one connection hold Review of #28289: - M1: the host replays the shell's last directory and progress reports at adoption; they stay pending until the terminal is live. The session snapshot used to publish them on its first read; a read writes nothing now, so finish_terminal_adoption publishes them once the terminal is live, as a reconnect does. An idle shell after a daemon restart presents its reported directory, not its launch directory. - L1: workspace.list holds the registry connection for its whole read, as the snapshot's pin did, so no late writer commit lands between the workspace rows, the topology frame and the decorations. - The terminal_directory doc names who retries a pending report now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmux-tui): keep a handle to the adopted surface to publish its replayed reports (E0382) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(cmux-tui): an app terminal's first view publishes the directory its shell reported before it (red) Review of #28289 (fresh review, finding 1): a byte-backend shell can report its directory before the first view writes its durable record; the report stays pending, terminal.project's own publish finds the record still Launching, and finish_app_terminal_first_view commits Running with no publish after it. The session snapshot used to publish it on its next read; a read writes nothing now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmux-tui): an app terminal's first view publishes its shell's pending directory and progress The first view commits the byte-backend terminal Running; a directory or progress report the shell made before the record existed stays pending until then. Publish it once the terminal runs, as adoption and a reconnect do (review of #28289, finding 1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmux-tui): the two unsettled-commit refusals say the request is retryable now Both refusals in drain_unsettled reach a client as an untyped operation_failed; until the typed busy error with retry_after_ms lands (B2, R9 R-L3), the message says the request is retryable and can be sent again at once, so an agent reading it is not misled into a hard failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmux-tui): StateGuard holds its MutexGuard directly; no expect (crash ratchet) The state guard kept its MutexGuard in an Option only to record the release before the unlock, and its Deref used two expect() calls: the crash ratchet counted them (cmux-tui-core expect 190 -> 192, #28147's red 'cmux-next crash ratchet'). Fields drop after Drop::drop, so the release is recorded in the body while the lock is still held, with no Option. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmux-tui): the lock site histograms index and count without a panic path (panic ratchet) diagnostics.rs: the per-site octave histogram indexed its bucket array and added with +=, which the Rust panic ratchet counts (cmux-tui-core indexing_slicing 854 > baseline 853; arithmetic_side_effects rose too; review of #28147 at c88bc3a). The bucket is now taken with get_mut, and the counts, the running sum, the p99 bound and the acquisitions use saturating or checked arithmetic. The values are unchanged: bucket 0 holds 0, bucket i holds 2^(i-1)..2^i - 1, the last bucket everything above. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…lane (on top of #27719's 48-of-64 cap) (#28343) * test(cmux-tui): the app keeps its daemon connections under terminal load (red, #27698 B1) Austin's nightly-next (2026-10-29): about 80 terminal agents held all 64 daemon connections, 55,162 were refused, the app's among them. Red tests, with behavior-neutral seams (ConnectionBudget at the old 64, ConnectionLane/ConnectionSlot, app_lane_socket_path) so they compile on the base: - an_app_connection_is_served_while_the_session_socket_is_full - one_terminal_opening_many_connections_cannot_starve_another - the_connection_budget_follows_the_open_file_limit - a_client_that_stops_reading_for_a_while_keeps_its_connection - AppLaneConnectionTests (Swift): the app connects on <socket>.app * fix(cmux-tui): the app has its own daemon socket; per-client fair shares; budget from RLIMIT_NOFILE; no 2 s write kill (#27698 B1) Austin's nightly-next (2026-10-29): about 80 terminal agents held all 64 daemon connections; 55,162 were refused in 2.5 h, the app's among them. The Austin's-day A/B on #28159 showed 64/64 with ~3,700-4,250 refusals per 6-min run. - The app's own socket, `<session socket>.app` (app_lane_socket_path), bound under the same start lock. Each socket has its own budget, so terminals, agents, hooks and the CLI (session socket) can never take the app's connections. The app (LineTransport) connects there first and falls back to the session socket (older daemon, stale file, long path). Chromium: the browser's own IPC channels never compete with renderer connections (render_process_host_impl.cc gives each renderer its own). - A fair share per client on the session socket: the client is the peer's process session (getsid), so a terminal's shell, agent, hooks and cmux calls count as one. A client may take a connection only while it then holds fewer than the socket has left free: one terminal stops at half the lane, and every new client still finds room. Remote entry and WebSocket clients keep the lane budget only (no identity at accept). - The budget comes from the soft RLIMIT_NOFILE the owner already raises at start (owner_start, raise_open_file_limit: hard limit, at most kern.maxfilesperproc / OPEN_MAX): half of the descriptors left after start, 8 per slot (4 descriptors per connection, 2 connections per slot with the hung-up pool), half per socket; on macOS also bounded by kern.num_taskthreads (2 threads per connection). 64 and MAX_SERVER_CONNECTIONS are gone; the parked-connection bound follows it. - No write deadline: STREAM_WRITE_TIMEOUT (2 s) closed a connection whose client read late (a stalled app main thread lost its connection and resynced). Now a client is ended only when its replies overflow the bounded control queue (QueuedSink evicts and shuts the socket); the WebSocket ping never waits on a peer that does not read, and the ping clock shuts a dead peer's socket. Replaces the stopgap split of #27719 (48/64 for terminals). * test(cmux-tui): the slow-reader test declares the stall as a signal, not a 3 s sleep (R5, #27698 B1) StalledSocket models the kernel in memory: a daemon write waits while the client's buffer is full, and one that waits when the test declares the stall fails as SO_SNDTIMEO does if a write deadline is set (red on the base: 2 s STREAM_WRITE_TIMEOUT), else keeps waiting (green). The app-lane test reads with no deadline: the daemon answers or refuses with one line. * test(cmux-tui): read the child's budget anywhere in its line (libtest prints its own prefix first) (#27698 B1) the_connection_budget_follows_the_open_file_limit failed on both the red and the green fleet runs (16894a02, 3830e38f) at connection_lanes.rs:123: the child printed "connection-budget=30" after libtest's "test <name> ... " on the same line, and the parser only took a line that starts with it. * fix(cmux-tui): the app socket gets its own outbound budgets; clean up .app first; clippy (review of 6aaecf6, #27698 B1) Review FIX (landing/reviews/6aaecf6a9207...md): - Blocker: clippy -D warnings on changed files: SynchronizedTcpStream:: set_write_timeout is dead with no write deadline; same-type casts, unnecessary qualifications and a redundant clone in the tests. - The two sockets shared one RenderService, whose outbound byte budgets a session-socket client that stops reading now holds until its queue overflows (no 2 s write kill). The app's replies could then fail on a full pool and the app's connection close: #27698 by another path. The app socket now has its own RenderService. - cleanup removes <socket>.app before the session socket, so a daemon that binds right after the first removal keeps its fresh .app. * test(cmux-tui): the app's replies never wait for the session socket's outbound bytes (review of 6aaecf6, #27698 B1) Regression test for the app socket's own RenderService (0a0796c): spend the session socket's outbound and control budgets, as clients that stopped reading hold them now that no write deadline frees them; a session-socket identify then gets no reply (the test checks it proves something), and the app socket's identify is still answered. ServedSession::new builds the session's two budget sets in one place. Red proof: cp-scratch/pa-b1-red-budget (6aaecf6 + this test, with ServedSession::new serving both sockets from one RenderService). * test(cmux-tui): a one-connection lane still admits a client; the daemon starts without its app socket (red, #27698 B1) Lead review of 5631798, two defects for Austin's load: - every_lane_admits_a_client_at_the_smallest_budget: at the smallest budget (one connection per lane) the fair share refuses every client. - the_session_socket_serves_when_the_app_socket_cannot_be_bound: a directory nobody may remove at <socket>.app makes the whole daemon fail to start. * fix(cmux-tui): a client's first connection always gets in while the lane has room; the daemon starts without its app socket (#27698 B1) - The fair share admitted a further connection only while the client then held fewer than the lane had left free, so a lane of one (the smallest budget an fd limit derives) refused every client. A client's first connection now gets in whenever the lane has room. - A failure to bind <socket>.app (a path that can't be removed, Windows) failed the whole start. The app socket is an addition: the daemon now logs why and serves the session socket alone; the app's LineTransport already falls back to it. * revert(cmux-tui): narrow this PR to build on #27719 (lead decision 2026-11-01 05:20Z) The app socket, the RLIMIT_NOFILE budget and the removal of the 2 s write stall leave this PR: #27719 (fbb9e39) already keeps 16 of the 64 connections for the app by capping terminal clients at 48, and the lead chose to build on that cap. The tree is feat-cmux-next again here; the next commits add only the per-terminal fair share inside the terminal lane. * test(cmux-tui): one terminal cannot take the terminal lane from the others (red) Builds on #27719's 48-of-64 terminal cap (lead decision 2026-11-01 05:20Z). The accept loop already knows which terminal a peer runs in (its process session); this commit carries that id to the terminal claim and the lease, with no behavior change, and adds the test. Red here: terminal 101 takes all 48 terminal connections and terminal 102 finds none. * fix(cmux-tui): a terminal's connections get a fair share of the terminal lane On top of #27719's 48-of-64 terminal cap (lead decision 2026-11-01 05:20Z): a terminal's first connection gets in whenever the lane has room; a further one only while that terminal then holds fewer than the lane has left free. One terminal opening connections in a loop stops at 24 of the 48, and every other terminal still finds room. The share and the count change together under one lock; a hung-up client's reclaim gives back its terminal's share too. A connection whose terminal is unknown keeps the lane bound alone. * test(cmux-tui): the fair-share test proves a released terminal's share is cleared; the share is "no more than" the free lane (review 6553892057) * fix(cmux-tui): contains_peer is test-only now (clippy dead_code; the accept loop asks terminal_of_peer) clippy step b7e7a3f2c43e at 8fc1e8a: method contains_peer is never used.

lawrencecchen and others added 15 commits October 30, 2026 06:32

test(cmux-tui): drop unnecessary path qualifications in the lock tests

2e50444

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(cmux-tui): a rename's reply comes after its name is in state; re…

f3dd51c

…aders see the old name only until then Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(cmux-tui): move the mux into the settle test's pinning thread (c…

50819b3

…lippy redundant_clone) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge branch 'fix/pa-b3' into fix/pa-b3-waits

04bb5f0

Merge remote-tracking branch 'origin/feat-cmux-next' into fix/pa-b3

100b8e3

Merge branch 'fix/pa-b3' into fix/pa-b3-waits

d58a066

lawrencecchen added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 30, 2026

lawrencecchen temporarily deployed to sentry-dsyms October 30, 2026 08:28 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts October 30, 2026 08:28 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts October 30, 2026 08:29 — with GitHub Actions Inactive

github-actions Bot temporarily deployed to ci-routing-agent October 30, 2026 08:31 Inactive

lawrencecchen temporarily deployed to artifacts October 30, 2026 08:39 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts October 30, 2026 08:41 — with GitHub Actions Inactive

lawrencecchen added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 30, 2026

lawrencecchen mentioned this pull request Oct 30, 2026

cmux-tui: reads run no write; workspace.list and session.ping read only what they answer #28289

Merged

lawrencecchen removed the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 30, 2026

lawrencecchen mentioned this pull request Oct 30, 2026

scripts(cmux-next): Austin's day, the P-AUSTIN load scenario (fleet bench) #28077

Merged

lawrencecchen and others added 2 commits October 30, 2026 15:46

Merge branch 'fix/pa-b3' into fix/pa-b3-waits

855fef6

lawrencecchen temporarily deployed to ci-routing-agent October 30, 2026 15:48 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to sentry-dsyms October 30, 2026 15:55 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts October 30, 2026 15:55 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts October 30, 2026 15:56 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts October 30, 2026 16:05 — with GitHub Actions Inactive

lawrencecchen mentioned this pull request Oct 30, 2026

cmux-tui hook: one queue and one connection per agent; a busy daemon never costs a hook event #28041

Merged

lawrencecchen temporarily deployed to artifacts October 30, 2026 16:15 — with GitHub Actions Inactive

lawrencecchen mentioned this pull request Oct 30, 2026

feat-cmux-next: cmux-next swift test red on unrelated PRs (status-read crash, routerResetDrop, acpmux restart, warm-up and cold-chat tests) #28492

Open

lawrencecchen temporarily deployed to ci-routing-agent November 1, 2026 03:03 — with GitHub Actions Inactive

lawrencecchen added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Nov 1, 2026

lawrencecchen merged commit a220bf2 into feat-cmux-next Nov 1, 2026
166 of 173 checks passed

lawrencecchen deleted the fix/pa-b3-waits branch November 1, 2026 04:03

lawrencecchen temporarily deployed to sentry-dsyms November 1, 2026 04:03 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts November 1, 2026 04:03 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts November 1, 2026 04:11 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts November 1, 2026 04:17 — with GitHub Actions Inactive

Navigation Menu

Repository navigation

cmux-tui: no request waits for the disk while it holds the state lock or the registry connection - #28159

cmux-tui: no request waits for the disk while it holds the state lock or the registry connection#28159
lawrencecchen merged 21 commits into
feat-cmux-nextfrom
fix/pa-b3-waits

lawrencecchen commented Oct 30, 2026 •

edited

Loading

Uh oh!

cursor Bot commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026 •

edited

Loading

Uh oh!

blacksmith-sh Bot commented Oct 30, 2026 •

edited

Loading

Uh oh!

lawrencecchen commented Oct 30, 2026

Uh oh!

lawrencecchen commented Oct 30, 2026

Uh oh!

lawrencecchen commented Oct 30, 2026

Uh oh!

lawrencecchen commented Nov 1, 2026

Uh oh!

lawrencecchen commented Nov 1, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lawrencecchen commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Observed / Not run

Known follow-ups (from review, not regressions)

Performance

Changelog

Proof

Uh oh!

cursor Bot commented Oct 30, 2026

Bugbot needs on-demand usage enabled

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blacksmith-sh Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Failures

Uh oh!

lawrencecchen commented Oct 30, 2026

Uh oh!

lawrencecchen commented Oct 30, 2026

Correctness checks (no finding)

Lows (none blocks)

Fleet logs read

Uh oh!

lawrencecchen commented Oct 30, 2026

Uh oh!

lawrencecchen commented Nov 1, 2026

Merge

Net change

Not blocking

Uh oh!

lawrencecchen commented Nov 1, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

lawrencecchen commented Oct 30, 2026 •

edited

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading

blacksmith-sh Bot commented Oct 30, 2026 •

edited

Loading