[lawrencecchen]

Summary

On Austin's day (9 runs of the pa-d harness, /work/tmp/pa-d/ab-28147), the workspace registry lock is held about 78 % of the time. On the hook-only load that the shipped agent hooks make, two of its largest holders take the lock for work that does not need it:

• terminal_registry_revision (mux/frontend_projection.rs:9): one committed value that identify and every list-workspaces reply read. 151,415 acquisitions in 9 runs, 7.0 % of the run time, mean 1.7 ms, max 1.9 s, spent mostly waiting for the connection behind writer commits.
• publish_terminal_progress (mux/terminal_progress.rs:18): a terminal's progress report (OSC 9;4, program status) read the whole session topology (every screen, pane, tab and browser, with their side tables) to find that one terminal's tabs. 7.1 % of the run time, mean 5.4 ms, max 2.1 s.

This PR:

• Gives the registry a read-only connection to its own file (RegistryConnection::read_committed). The database runs in WAL mode, so this connection reads the state committed when its read starts. It never returns a partial or uncommitted write, and an open write transaction never blocks it. Its own mutex is a leaf lock, held only for the read. An in-memory registry (no file) has no reader, and callers keep the locked read.
• Reads terminal_registry_revision on that connection, with no registry lock. If no reader exists, or the read fails, it falls back to the locked read.
• Makes publish_terminal_progress read only this terminal's own tab rows (resource_tabs_of_content). It still holds the registry for its commit.
• Adds an agents-hooks scenario to the diagnostic bench (bench-daemon-latency.py --scenarios agents-hooks). It is the agents load with no CLI calls: no shipped hook, plugin or status line calls the cmux CLI on each turn. The bench is not a CI step. scripts/measure/perf-10x-daemon.sh stays at its reviewed version.

This is the next PR of the #27698 lock series. It is stacked on #28147 (head d8f9ee9), so review only the commits after it. 1e (an agent hook that changes no projection takes no registry lock) builds on this PR's reader connection. #27780 (Austin) touches the same lock.

Owner: PL-daemon (session cmux-b6).

Testing

Tests, in cmux-tui-core/src/mux/tests/registry_reads.rs:

• the_terminal_revision_read_does_not_wait_for_the_registry_lock holds the registry lock and reads the revision from another thread, with a 5 s bound.
• a_progress_report_reads_only_its_own_terminals_tabs records each table the report reads, using a SQLite authorizer on the registry connection. It checks that the report reads none of the per-screen side tables that only a whole-topology load reads (resource_screen_rows, resource_column_docks, resource_lone_docks, resource_parked_splits), and that the report still publishes a revision.
• a_committed_read_during_an_open_write_transaction_sees_only_committed_data holds a write transaction open that changes the revision, and reads on the reader connection from another thread. The read must not wait, must return the committed value and not the uncommitted one, and must return the new value after the commit. It guards the reader; it is not a red test.

Red, fleet step 1e49666eaf503381f65e5869, cmux-tui-rust-check.sh test registry_reads at 044b745 (the tests, without the fix):

• the_terminal_revision_read_does_not_wait_for_the_registry_lock FAILED: "the terminal revision read waited for the held registry lock" (registry_reads.rs:40, after 5.07 s).
• a_progress_report_reads_only_its_own_terminals_tabs FAILED. At that commit the test still had an earlier assertion, which was too strict. Its failure printed every table the report read (log line at 20:10:20Z):
  a progress report read resource_screens: {"json_each", "meta", "resource_browsers", "resource_column_docks", "resource_lone_docks", "resource_mutations", "resource_panes", "resource_parked_splits", "resource_screen_rows", "resource_screens", "resource_tabs", "resource_terminals", "resource_workspaces", "sqlite_master", "stash_capture_workspaces", "state_pending_changes", "terminal_hosts"}
  All four side tables that the shipped assertion checks are in it. So the shipped assertion also fails without the fix.

Green: cmux-tui-rust-check.sh all at the head 5b321e6, fleet step 1323ed5be0f10f315cc33d30. It compiled, fmt and clippy pass, and 12970 of 12983 tests passed. All three registry_reads tests passed. None of the 13 failures is this PR's:

• Base reds, the same as at this PR's base (cmux-tui: the daemon no longer deadlocks settling a late registry receipt; server-stats reports both locks by site #28147's merge-head run aa49f608): the CLI noun-first path (cmux-tui: git diff and checkpoint diff take --after-path, --patch-offset and --patch-revision (base red) #28119); the acpmux hermetic guard (acpmux tests: web_image_attachments runs hermetic, with node as its one host tool (base red) #28150); 8 feat-cmux-next reds (agents_chat_tools x5, reply_in_place, pane-policy known params, integrity_check_scope), which feat-tip-reds is fixing.
• Unexplained reds, listed in the handoff followups for a fixer (not cleared as flaky). This PR changes no file in their crates or paths:
  • graphics_pointer::evicted_known_undelivered_motion_allows_same_cell_retry: 6 single runs passed, one attempt each, 3 at the head and 3 at the base (e364c319, 44e68d12, f3a4ab25, 58063d9e, 8be434c3, 0479c3ff).
  • local_resize_bounds_the_drain_to_a_queued_snapshot.
  • apps_store::the_customize_folder_installs_itself_and_reloads_when_it_changes.
• The gate skipped every alone re-run because 13 tests failed, more than its limit of 12.

Observed / Not run

• Observed: the red and green above. The holder numbers come from the 9 Austin's-day runs of the pa-d harness, from the registry's top 8 holders. Those runs did not read the locks section, so they have no p99 per holder.
• Not run: the before/after A/B of the two holders on the hook-only load. pa-d runs it on one host (before b472663, a scratch ref of this tree with the fix reverted; after fee4db7; the product code is unchanged at this head). It will report the LockStats p99 of both holders, each round's difference, and a paired t-interval with n.

Performance

No perf numbers: perf claim: pending A/B (pa-d). This PR claims no measured speedup until the A/B above is posted here.

Changelog

Fixed: On a busy Mac, agents' terminal progress and cmux NEXT's identify requests no longer wait behind other saves.

Proof

No capture: daemon-only change, no UI.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Reduces time waiting on the shared workspace registry lock by moving two of its largest read holders onto paths that don't take it, and fixes a self-deadlock on the settle of a late registry commit.

The terminal registry revision, read by identify and every list-workspaces reply, now reads on a new read-only WAL connection to the registry's own file, with no registry lock; an in-memory registry or a failed read keeps the locked read. Terminal progress reports now read only their own terminal's tab rows instead of the whole session topology.

Bug Fixes

• Settling a late indeterminate registry receipt no longer relocks state: the settle publishes only the journal event, avoiding a self-deadlock while it holds both the registry and the state lock.
• Replaced the Option/expect() StateGuard with a direct MutexGuard, and removed panic-ratchet hits in the new histogram's indexing and arithmetic.

New Features

• server-stats include=["locks"] reports the registry and state lock holders by site, with #[track_caller] naming the request path that holds the lock.
• Added an agents-hooks scenario to the daemon latency bench (hook-only load, no CLI calls); diagnostic use only, not a CI step.
• New tests cover the lock-free revision read, the narrowed progress-report read, and committed reads during open write transactions.

^{Written for commit c076a3d. Summary will update on new commits.}

[lawrencecchen]

Austin's-day A/B for B3 1d (harness owner pa-d), as the lead asked: before b472663 (scratch/pa-b3-1d-before: the 1d tree with fix 5cc5b7e reverted, never merge) against after fee4db7 (fix/pa-b3-holders). The terminal revision read no longer appears among the registry lock's top 20 holders (from about 35-47 s of hold per run to at most about 0.4 s), the registry lock held share falls by 0.7 points, and refused connections fall by 12 %. Lock wait p99 and app no-reply do not change.

• Pair: a revert pair. The base is the head plus one commit reverting 5cc5b7e, and the base-to-head diff is exactly that commit.
• Builds: before c4508a305e6709d5f6217112, after a878900da0504d8e4c648f51.
• Harness: runs made with 9cfb9f5 (scripts(cmux-next): Austin's day follow-up: pool holders through cmux-tui, tool argv, held-share test #28505, SHIP). Main arm: no agent CLI calls. The table was computed with 50d4a70 (SHIP), which reads the main-arm A/A record (scripts(cmux-next): Austin's day, the P-AUSTIN load scenario (fleet bench) #28077 (comment)); the run-time summary had marked every verdict "not A/A-checked" only because that record was added afterwards.
• Host and runs: cmux24 for all 10 runs, exclusive leases, 5 rounds in ABBA order. Base round 1 was retried once because it got no lease at first: the mini was busy with another job, and the lease waited for it.
• Validity: every base run reproduced Austin's day. No harness faults.
• Method: paired t-interval, 95 %, n = 5.

Metric	before	after	after - before [95 % CI], n	Verdict / kind
Registry lock held share	80.8 %	79.8 %	-0.66 pt [-1.31, -0.02], n=5	requested, no verdict
Registry lock wait p99	786 ms	786 ms	+26 ms [-47, +99], n=5	no difference
Refused per second	6.38	5.59	-0.79 [-1.50, -0.09], n=5	better
Refused share of connects	17.4 %	15.0 %	-2.4 pt [-4.3, -0.5], n=5	better
App request p99	-	-	n=0	not measured: no probe in these builds
App "no reply within" (log)	6	3	-1.6 [-5.0, +1.8], n=5	no difference
Agent hook p99	4,789 ms	4,600 ms	-260 ms [-652, +133], n=5	no difference
acpmux open failure share	85 %	86 %	-0.5 pt [-16, +15], n=5	no difference
terminal_registry_revision (frontend_projection.rs:9 / :20) total registry-lock hold per run	35.5-47.4 s (4 runs)	absent from the top 20: at most 0.29-0.38 s (3 runs)	no paired difference: the value is unknown when absent	requested
its hold p99	65-131 ms (4 runs)	at most 61-66 ms (3 runs)		requested
terminal_progress.rs:18 total hold per run	43.0-50.5 s	51.8-56.3 s	+5.9 s [-5.3, +17.1], n=3	requested, no verdict
its hold p99	131 ms	131 ms	0, n=3	requested

The lock detail is missing for 3 of 10 runs: base round 2 and head rounds 1 and 2. The read after the load got transport.closed from the daemon. So the named-site rows have n=3 or none.

• terminal_registry_revision: in all 4 before runs with detail, it held the registry lock 35-47 s per run. In all 3 after runs with detail, it is not among the top 20 holders, whose smallest total was 0.29-0.38 s. So the fix removes about 35-47 s of registry-lock hold per run. That is an upper-bound comparison, not a paired difference.
• terminal_progress.rs:18: no change.

Note: the held share falls by much less than that site's hold (0.7 points, about 2.7 s of a 390 s window). The removed hold was mostly not on the path that keeps the lock busy, or other holders took its place.

Leases: base r1 pos 1 slot-cmux24-5b938a5c9204; head r1 pos 2 slot-cmux24-a567cadae62e; head r2 pos 1 slot-cmux24-638500b88d78; base r2 pos 2 slot-cmux24-2fcbeb86041f; base r3 pos 1 slot-cmux24-917460d1729c; head r3 pos 2 slot-cmux24-a9e2171a4d5b; head r4 pos 1 slot-cmux24-cf70c191e467; base r4 pos 2 slot-cmux24-9f48a4e5277f; base r5 pos 1 slot-cmux24-6713c4d86e6d; head r5 pos 2 slot-cmux24-6ee5a59481c4;
The data is on cmux-orchestrator-1 at /work/tmp/pa-d/ab-1d (summary-50d4.txt).

[lawrencecchen]

Review (claude-review-subagent-cw): SHIP at c076a3d

I found no blocking defect. I did not compile or run tests. I read the code and the CI results only.

Merge (c076a3d = 5b321e6 + feat 1209ff7)

• The merge result is the same as feat in diagnostics.rs, signaled_mutex.rs, server_stats.rs and server_stats_tests.rs. Feat's hook_queue section and its non-test try_lock stay. Before the merge, the PR versions of these 4 files were the same as the squash babe9dc, except for feat's later hook_queue work. Thus the merge drops no PR content. The panic-ratchet fix d8f9ee9 is in the squash.
• git diff 1209ff76731 c076a3dbc19 -- cmux-tui/bindings cmux-tui/spec is empty. The Rust diff over feat changes no serde or protocol type. It changes only crate-internal code: a pub(crate) on current_terminal_revision (workspace_registry.rs:4967), a new RegistryConnection field and method, and the two call sites. Thus all generated files agree with the Rust protocol.
• bench-daemon-latency.py: the merge keeps the PR's --agent-cli option and the agents-hooks scenario on top of feat's agents(). The line args.agent_cli = 0 comes before b.agents("agents-hooks"). Thus, when you run both scenarios, agents runs with the CLI and agents-hooks runs without it.
• Feat changes since the PR head (5b321e6..1209ff7) did not touch registry_connection.rs. No feat change replaces the registry connection after open. RegistryConnection::new has one caller (workspace_registry.rs:2910). That caller runs after the schema setup and journal_mode=WAL, so the reader opens a file that exists and is in WAL mode. I found no semantic conflict.

Net change over feat

• read_committed (registry_connection.rs:199-210): the reader is a separate read-only connection behind a leaf mutex that the method holds only for the read. A failed open gives None, and the caller uses the lock path. A failed read (Some(Err)) also uses the lock path (frontend_projection.rs:14-20). The fallback is correct.
• Freshness: each read is an autocommit read transaction (query_row resets the statement), so the read sees all commits that completed before it started. Every reply that gives a client a terminal revision is sent after its commit. Thus identify and list-workspaces never return a value older than a revision that the client saw before. The value can be newer than the in-memory tree when a writer has committed but still holds the registry lock. This race was already possible: list_workspaces_reply (server.rs:2870) reads the tree in with_state before it reads the revision, and the old read took the registry lock after it released state. identify's workspace_revision and terminal_revision are different counters. A client that sends this value as expected_terminal_revision gets a CAS against the committed value under the lock, which is correct. I found no consumer that a newer value breaks.
• terminal_progress.rs:29-37: resource_tabs_of_content(id) uses load_tabs with the same filter (deleted_revision IS NULL) as the full topology. It uses the resource_tabs(content_id) index. Terminal and browser public IDs have different prefixes, so removing the ContentPublicId::Terminal match changes no result. terminal_tab_ids_in_canonical_order still sorts. This is the same pattern as terminal_directory.rs:47-55 on feat.

Non-blocking notes

1. frontend_projection.rs:6-7: the old doc lines ("the registry lock is held for one indexed read") stay above the new doc and contradict it. Remove them in a follow-up.
2. registry_connection.rs:79-85: the struct drops connection before reader. On a clean shutdown the read-only reader is the last connection to close. A read-only connection cannot checkpoint, so the WAL and SHM files stay. The next open replays the WAL, the same as after a crash. refuse_orphaned_journal acts only when the database file is missing, so this is safe. To keep the old clean-close behavior, close the reader first.
3. The reader has no explicit busy timeout, so it uses rusqlite's 5 s default. A WAL reader rarely gets SQLITE_BUSY (WAL recovery only). If it does, an identify can wait up to 5 s before the fallback. A short busy_timeout on the reader would limit this.
4. CI on this head: clippy (changed cmux-tui packages) is red because of crates/acpmux/src/hub/processes.rs:256 (redundant_closure). This PR does not touch that file, and the same error makes unrelated PRs red (for example, run 41266836439). This is a base finding. Because of it, clippy did not finish on this PR's cmux-tui-core code. pr-cloud-backend-check is red, but this PR changes no web or backend path. Many checks were still pending when I reviewed.

Observed / Not run

• Observed: git show --remerge-diff c076a3dbc19, git diff 1209ff76731 c076a3dbc19, the feat log for workspace_registry and mux, the consumers of terminal_registry_revision, load_tabs and load_resource_topology, and the CI annotations and clippy log on this head.
• Not run: no Rust build, clippy or test on this machine. I did not run the 3 new tests in mux/tests/registry_reads.rs.

[lawrencecchen]

Conflict wave: merged feat-cmux-next 1209ff7 into this branch (merge c076a3d). The conflicts were feat's squash of #28147 (pa-b3), plus its later hook_queue section, against this branch's copy of the pa-b3 commits. Rust: I took feat's side (hook_queue, try_lock outside tests). Generated bindings and spec: I took feat's files (this PR changes no protocol). bench-daemon-latency.py: I kept this PR's --agent-cli option and its agents-hooks scenario.

Fleet evidence at c076a3d:

• Compile (cmux-ci compile-check --package CmuxNext --app, job in the log): CmuxNext passed (241 s), app passed.
• cmux-tui-rust-check.sh test registry_reads (step 1038c5559019ce8d3d85d925, aws-m4pro-6, macOS): 3 tests run, 3 passed.
• clippy (step 88b1b421bbd6d2fc851a8350): RED, a base finding. It is redundant closure in acpmux hub/processes.rs, the base red that base red: acpmux clippy redundant_closure in process_values #30009 (55361f1) fixed on feat after this merge. This PR does not touch acpmux.
• Review: SHIP at this head (comment 6552682730).

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review c076a3d: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28534 -f ref=fix/pa-b3-holders -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base 1209ff7; does not block

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4130052256501 (28aa638015)

[blacksmith-sh]

Found 8 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmpfqsy2n0d/summary-fail.json (live):/summary-fail.json (live):	View Logs
/tmp/tmpvlvyc0ff/summary-missing.json (live):/summary-missing.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
convention violations found (ERROR lines above)./ convention violations found (ERROR lines above).	View Logs
test_every_part_of_bun_run_check_runs_in_the_warn_branch (main.WebviewsWarnSplitCov ersCheck.test_every_part_of_bun_run_check_runs_in_the_warn_branch)/ test_every_part_of_bun_run_check_runs_in_the_warn_branch (main.WebviewsWarnSplitCov ersCheck.test_every_part_of_bun_run_check_runs_in_the_warn_branch)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}