[lawrencecchen]

Summary

Under load, the cmux-tui session daemon could deadlock on its own mux state lock. When a registry commit outlives its 30 s journal receipt wait (a loaded Mac, #27698), its late answer is settled by the next request that calls Mux::lock_state_pinned, and that request holds the workspace registry and mux.state while it settles. The settle ran publish_resource_event. That function's publish_pending_terminal_directories step locks mux.state again. A debug build panics on the lock rank. A release build deadlocks the whole daemon (std Mutex), so every later request and every connection hangs.

The settle now publishes only the journal event. Subscribers still read the new head. A terminal directory that is still pending is published by that terminal's next output or the next committed graph change, the same as before the settle existed.

This PR also makes the daemon's two big locks measurable by call site:

• server-stats with include:["locks"] reports the workspace registry lock and, for the first time, mux.state. For each lock it gives wait and hold histograms, the current holder, the last stall, the number of sites, and the top 20 sites by total hold and by p99 hold. Each site row has its hold and wait p99 and maximum. Each site keeps two 256-byte power-of-two histograms, updated under the site table lock that the release already takes, while the lock is still held. After a site's first acquisition this costs no allocation. The section is optional, so older SDK decoders keep working. Schema, bindings, api-compat surface and spec/commands.md are updated.
• About 40 Mux lock wrappers get #[track_caller] (with_state, surface, resolve_resource_path, commit_state, commit_resource_mutation_plan, with_resource_projection, the resource effect helpers, lock_state_pinned and others). The statistics now name the request path that holds a lock, not the wrapper's own line.
• cmux-tui/scripts/bench-daemon-latency.py --scenarios agents is a diagnostic fleet bench, not a CI step. It runs 80 fake terminal agents (the real hook helper plus a CLI call), terminal churn, and an app client (window saves, list-workspaces, session.snapshot, identify). It prints the app requests' latency and the lock sites.

#27780 (Austin) touches the same registry lock. The only file both PRs change is mux/resource_effects.rs, at different functions; I noted the overlap there.

Owner: PL-daemon (session cmux-b6), please review the daemon lock change.

Testing

Regression test: settling_a_late_registry_receipt_does_not_relock_state in cmux-tui-core/src/mux/tests/writer_lock_order.rs. It holds an indeterminate receipt, answers it, and pins state from another thread; the pin must return within 5 s and run the owed state update.

• Red, fleet step a551489abaea4cfd8815ecf7: scripts/ci/cmux-tui-rust-check.sh test settling_a_late_registry_receipt_does_not_relock_state at 0f46ae7 (test only, no fix). It FAILED with lock order violation: acquiring mux.state (MuxState) while this thread holds mux.state at mux/journal.rs:11, the deadlock path.
• At the head d8f9ee9 (feat-cmux-next merged, plus the Rust panic ratchet fix), each fleet run is done (passed): settling_a_late_registry_receipt_does_not_relock_state (bd8f77630117abe7d6290e2d), lock_detail_reports_per_site_hold_and_wait_percentiles (e6b30baeb75b78399af62863), server_stats_include_reports_registry_and_state_lock_sites (197f5fab7581ddbebcc2fefb), clippy (a361a9dd9077dec93763fcd5). The PR's CI panic ratchet and crash ratchet pass.
• Full run at the merge head c88bc3a, fleet step aa49f608737c04b58e23478f: it compiled, fmt and clippy pass. Its 11 failures are base reds, not caused by this PR: the CLI noun-first path (cmux-tui: git diff and checkpoint diff take --after-path, --patch-offset and --patch-revision (base red) #28119), the acpmux hermetic guard (acpmux tests: web_image_attachments runs hermetic, with node as its one host tool (base red) #28150), and 9 new feat-cmux-next reds (agents_chat_tools x5, reply_in_place, a_daemon_whose_home_is_replaced_stops, integrity_check_scope, the_known_params_are_the_macos_hosts), whose fix feat-tip-reds has in progress. d8f9ee9 adds only the histogram change on top of it.
• Earlier heads: green 52b69830 at 3b1cb41 and 78f9e8ec at 100b8e3 each failed on the two base reds only.
• New tests for the statistics: lock_detail_reports_per_site_hold_and_wait_percentiles and server_stats_include_reports_registry_and_state_lock_sites.

Observed / Not run

• Observed: the red failure above, and the green run. Earlier full runs on this branch also showed reds this PR cannot reach. They passed when run again alone and are listed for a fixer, not cleared as flaky: history_agent_fold::agent_history_save_does_not_wait_for_the_workspace_registry_lock failed at its precondition (line 109, before it takes any lock): a read-after-write gap between the agent-hook reply and the journal ring. B3 design item 3 owns it. Also apps_store::the_customize_folder_installs_itself_and_reloads_when_it_changes (step ba4f8faf). I also ran the agents bench at 6340c54 (statistics only, no fix; fleet step 73e933233cd19e66b80674ea, aws-m4pro-9). It gave app requests p50 13 ms, p95 57 ms, p99 91 ms (1,868 requests, none over 2 s) and registry lock wait p99 82 ms. The longest registry holder is resource_api.rs:551, the snapshot readers: 25.7 s held in 60 s. These numbers rank the holders for the follow-up PRs; this PR claims no speedup.
• Not run: the deadlock was not reproduced in a release build on a live loaded Mac. The release behavior follows from std Mutex relocking on one thread, which deadlocks or panics. Austin's-day numbers: pending harness owner. Perf claim: none; the statistics are instrumentation.

Performance

No perf numbers: the lock statistics are instrumentation and the fix removes a deadlock; this PR claims no speedup.

Austin's-day A/B: pending, in the pa-d queue (harness owner). It runs with an A/A-valid harness sha; the first A/A pass is #28077 (comment). The A/B comment will be linked here.

Changelog

Fixed: The cmux NEXT session daemon no longer hangs when a busy Mac delays a save past its timeout.

Proof

No capture: daemon-only change, no UI.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Fixes the cmux-tui session daemon deadlocking when a late registry commit receipt is settled while the registry and state locks are held. The settle previously called publish_resource_event, which locks the state again — a self-deadlock in release builds and a lock-rank panic in debug. The settle now publishes only the journal event; a terminal directory still pending is published by the terminal's next output or the next committed graph change.

Bug Fixes

• Regression test settling_a_late_registry_receipt_does_not_relock_state answers an indeterminate receipt and pins state from another thread; the pin must return within 5 s.
• StateGuard now holds its MutexGuard directly, removing two expect() calls that tripped the crash ratchet.
• The lock-site histogram uses bounds-checked bucket access and saturating arithmetic so it stays under the panic ratchet baseline.

New Features

• server-stats with include:["locks"] now reports both the workspace registry lock and mux.state: wait/hold histograms, current holder, last stall, and the top 20 sites by total and p99 hold.
• ~40 Mux lock wrappers get #[track_caller], so the reported site names the request path holding the lock, not the wrapper's own line.
• cmux-tui/scripts/bench-daemon-latency.py --scenarios agents is a diagnostic fleet bench with 80 fake terminal agents, terminal churn, and an app client; it prints app-request latency and lock sites. Not a CI step.
• The locks section is optional, so older SDK decoders keep working; schema, bindings, and spec/commands.md are updated.

^{Written for commit c842b42. Summary will update on new commits.}

[lawrencecchen]

Correctness review (independent subagent, given the diff and the PR body): SHIP

• Deadlock real; fix complete. Before the fix, lock_state_pinned (registry and mux.state held) called settle_registry_receipts → publish_resource_event → publish_pending_terminal_directories, which locks mux.state again. After the fix, the settle takes only leaf mutexes (unsettled, journal_kernel.state, journal_event_epoch). No step takes state or the registry again. lock_state_pinned is the only caller of the settle.
• A pending directory is late, not lost. Terminal output, resize and reconnect, session.snapshot, and every committed topology change still publish it. Before the fix, the publish at settle never finished.
• The red test exercises the production path (hold_unsettled → lock_state_pinned → settle). Only the two hooks are #[cfg(test)].
• Statistics are correct:
  • The p99 bucket math is right: the value is never above the maximum and is less than twice the true sample.
  • The release is recorded while the inner mutex is still held.
  • The LockStats mutexes never nest, so there is no new deadlock or rank panic.
  • Schema, commands.md and the api-compat surface match the serialized structs.
• #[track_caller] (42 functions) changes only Location::caller().
• The bench is not in any CI workflow, and its quoting is safe.
• Non-blocking, carried into the follow-up PR:
  • detail() formats the site labels while it holds the sites mutex. This happens only on an include:["locks"] request.
  • Two schema doc nits: the holder site text names only the registry lock, and sites counts releases, not acquisitions.

Owner PL-daemon (cmux-b6): this changes the daemon's lock path, please take a look.

Review (claude-subagent): SHIP at 100b8e3
R9: pass (DETAIL_SITES = 20 and SITE_OCTAVES = 32 are report sizes. The PR adds no admission cap, queue bound or production deadline.)
perf claim: none

[blacksmith-sh]

Found 6 test failures on Blacksmith runners:

Failures

Test	View Logs
main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/ test_download_prints_the_active_profile_only	View Logs
stubs: /System/Library/Frameworks/Security.framework/Versions/A/Security has no stub/ Security has no stub	View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/ CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_CFRetain']/ CoreFoundation, missing 1: ['_CFRetain']	View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/ CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_log2']/ CoreFoundation, missing 1: ['_log2']	View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/ test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)	View Logs
test_the_ten_first_party_gpl_crates_are_named_exactly (main.AppCrateLicenseGate.tes t_the_ten_first_party_gpl_crates_are_named_exactly)/ test_the_ten_first_party_gpl_crates_are_named_exactly (main.AppCrateLicenseGate.tes t_the_ten_first_party_gpl_crates_are_named_exactly)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[lawrencecchen]

Review (claude-subagent): SHIP at 100b8e3
R9: pass (DETAIL_SITES = 20 and SITE_OCTAVES = 32 are report sizes; no admission cap, queue bound or production deadline is added)
perf claim: none

The same verdict as the review comment above, with the receipt line first.

[lawrencecchen]

Review (claude-subagent-fresh): SHIP at 100b8e3
R9: pass
perf claim: none

Fresh, independent correctness review. I read the code at the exact head through git show/git diff origin/feat-cmux-next...100b8e317c35, with the bindings spot-checked. I did not read earlier review comments before this verdict.

Deadlock: real, and fully fixed

• Real. Mux::lock_state_pinned (mux/signaled_mutex.rs:371-385) holds the registry, the connection pin and mux.state, then calls settle_registry_receipts. On the base, when take_settled_receipts() is not empty, that called publish_resource_event -> publish_pending_terminal_directories (terminal_directory.rs:95-97), which calls self.state.lock() again on the same thread. In debug builds this is the lock-rank panic. In release builds std Mutex relocks on one thread and the whole daemon hangs. The path runs every time a late receipt is settled, not only sometimes.
• Fixed. Settle now calls only publish_journal_event (effect_commit.rs:89). That function takes only leaf locks: journal_kernel.state (journal_kernel.rs:316-322) and journal_event_epoch. commit_effect_intent already called it while holding the registry. settle_registry_receipts has one caller (lock_state_pinned). The one production SettleState closure (workspace_rename.rs:214-229) only changes State and takes no lock. I checked other publish_resource_event callers next to state, e.g. resource_effects.rs:168-170 and 219-223, and state/commit.rs:47-65: each releases state (and the registry) first.
• Terminal directory update: not lost, possibly published later. publish_pending_terminal_directories only retries reports whose directory_pending flag is still set, and settle does not clear that flag. A report stays pending and is retried by the public session snapshot (resource_api.rs:541), every committed topology change (publish_resource_event), and terminal output, resize and reconnect (surface/hosted_reader/*, local_output.rs:250). The removed call could never run without deadlocking, so it never worked, and removing it loses no behavior. Informational only: a quiet shell whose launch row came in through a late receipt shows its cwd at the next snapshot read or topology commit, not immediately.

locks statistics (diagnostics.rs, StateMutex)

• No new deadlock. LockStats' holder, last_stall and sites mutexes are leaves: LockStats methods take them, and those methods take no other lock. debug_spans::mark_with uses a thread-local RefCell. server-stats locks reads stats().detail() and never takes the registry or mux.state.
• Release ordering is correct. StateGuard::drop records the release and clears holder while it still holds the inner guard, then unlocks, then note_state_lock(false), then _rank drops. The lock-rank release still comes after the unlock. A thread-local hold count that is briefly too high on the same thread is harmless.
• Poison path. A poisoned guard is still recorded and released. Deref uses expect, and the guard is None only inside drop.
• OctaveHistogram math is correct. Bucket i holds 2^(i-1)..2^i-1, and p99 is the bucket's upper bound clamped to the maximum, so it is less than 2x the true value and never above the maximum.
• Bounds. The sites map is keyed by &'static Location, so it can only hold as many entries as there are source locations. DETAIL_SITES = 20 limits the report size, not a resource, queue or deadline. Under R9, no new production cap, queue or deadline (the only new timeout is the test's 5 s recv_timeout).

Schema and spec

ServerStatsLocks, ServerStatsLockDetail and ServerStatsLockSiteDetail in spec/sdk-schema.json match the Rust structs field for field. holder and last_stall are required and nullable, and Rust serializes Option without skip_serializing_if. locks is optional and present only when include requests it (server_stats.rs:25). The thresholds in the constraints match LOCK_CONTENDED_THRESHOLD (1 ms) and LOCK_STALL_THRESHOLD (100 ms). Spot-checked: the Rust and Python bindings, and commands.md. The green run passed cmux-api-compat.

#[track_caller]

It does not change program semantics. It does change panic locations: an .unwrap()/.expect() or a lock-rank panic inside a wrapper now points at the wrapper's caller. The red log shows this (journal.rs:11:14 and not terminal_directory.rs). This is acceptable and arguably more useful. A few wrappers lock inside a closure (e.g. surface_workspace's or_else), so those sites name the wrapper line. That affects attribution only.

Bench script

cmux-tui/scripts/bench-daemon-latency.py was already on the base. The new agents scenario is not in the default --scenarios small,scale. No .github workflow references the script or scripts/measure/perf-10x-daemon.sh. It is not a CI step.

Non-blocking notes (no fix required)

• Low, unmeasured overhead: each mux.state acquisition now calls Instant::now() about 4 times, takes the holder mutex 2-3 times, and does one sites map update. That update runs while state is held, so it extends every state hold by a short amount. The PR claims no speedup, so R10 does not apply. The agents bench 73e933233cd19e66b80674ea ran with the instrumentation in place but has no before/after.
• Nit: diagnostics.rs:166 and the PR body say "256 bytes". OctaveHistogram is 32x8 + 16 = 272 bytes.
• Nit: sites keys include the column, but labels are file:line, so two lock calls on one line give two rows with the same label.

Fleet logs read (cmux-ci log --full)

• Red a551489abaea4cfd8815ecf7 at 0f46ae7 (test-only commit; checkout confirmed). settling_a_late_registry_receipt_does_not_relock_state FAILED with lock order violation: acquiring mux.state (MuxState) while this thread holds mux.state at mux/journal.rs:11:14, then the 5 s assertion "deadlocked or panicked". This is the deadlock path.

• Green 78f9e8ec8657b35e1ba9ee88 at 100b8e3 (checkout confirmed). fmt and clippy -D warnings passed (165 s). Nextest: 12703 run, 12701 passed, 2 failed, 130 skipped; doc tests passed. The new tests all PASS: settling_a_late_registry_receipt_does_not_relock_state, lock_detail_reports_per_site_hold_and_wait_percentiles and server_stats_include_reports_registry_and_state_lock_sites. No other test failed. The 2 failures are outside this diff:

  • acpmux::hermetic_daemon_guard: tests/web_image_attachments.rs lacks HERMETIC, and the PR touches no acpmux file.
  • transport_path_coverage_tests::every_safe_transport_operation_has_a_noun_first_path: git.checkpoint.diff fields after_path, patch_offset and patch_revision, and the PR touches no checkpoint or CLI file.

  Both are base findings.

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 51e2350: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28147 -f ref=fix/pa-b3 -f platform=macos -f backend_mode=local

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4116264351601 (f8c1ccbeda)

[lawrencecchen]

Review (Claude Opus 5.5 reviewer): SHIP at c842b42
reviewer-session: 4fb13519-fac9-4592-9954-a8497408f2ca
author-session: pa-b3-author
R9: pass (SITE_OCTAVES = 32 and DETAIL_SITES = 20 set the histogram and report sizes. The PR adds no cap on a resource, a queue or a production deadline. The only new timeout is the regression test's 5 s recv_timeout.)
perf claim: none

I read the code at the exact head in a detached worktree. I did not compile or run tests (fleet only). I ran only the pure-Python codegen check.

Deadlock fix (correct)

• mux/resource_effects/effect_commit.rs:89: the settle now calls publish_journal_event, not publish_resource_event. Before, lock_state_pinned (mux/signaled_mutex.rs:371-385) held the registry, the connection pin and mux.state, then publish_pending_terminal_directories (mux/terminal_directory.rs:95-97) locked mux.state again on the same thread. After the fix, the settle takes only leaf mutexes: journal_kernel.state (journal_kernel.rs:316), the maintenance control (mux/journal_maintenance.rs:94) and journal_event_epoch (mux/journal.rs:14-20). None of them takes the registry, the connection or state.
• settle_registry_receipts has one caller (lock_state_pinned).
• A pending terminal directory is published later, not lost: terminal output, resize and reconnect (surface/hosted_reader/*, surface/local_output.rs:250) and the next publish_resource_event retry it.
• Regression test mux/tests/writer_lock_order.rs:165: it uses the production path (hold_unsettled, then lock_state_pinned, then settle). Only the two seams are #[cfg(test)] (journal_ingress/effect_intents.rs:42, workspace_registry/registry_intent.rs:196). The red fleet step a551489abaea4cfd8815ecf7 at 0f46ae7 is in the PR body.

Lock statistics (correct)

• StateGuard (mux/signaled_mutex.rs:219-270): Drop::drop records the release while the inner guard is still held. Then the fields drop in declaration order: guard unlocks first, and _rank releases after the unlock. The poisoned path records and releases as well.
• LockStats mutexes (holder, last_stall, sites) are leaves and never nest. detail() (diagnostics.rs:315) takes neither the registry nor state, so server-stats include:["locks"] cannot deadlock.
• OctaveHistogram (diagnostics.rs:175-205): the bucket index is bounds-checked, the arithmetic saturates, and p99 is the bucket's upper bound, clamped to the maximum. There is no panic path, so the panic and crash ratchets pass.
• Wire output: locks is skipped unless include names it (server/server_stats.rs:25, diagnostics.rs:724-726), so older decoders keep working.

Merge c842b42 (clean)

• I compared the PR's net diff before the merge (d0d32d3a6e0..d8f9ee99c02) with its net diff after (HEAD^2..HEAD). Both diffs change the same 68 files. In every file outside bindings/, the +/- lines are identical, so no PR change was lost and no feat change was undone.
• In bindings/, the files differ only because feat changed the schema too (new hashes and new generated content). python3 cmux-tui/bindings/codegen/generate.py --check passes for cpp, go, java, python, rust, typescript, zig and pageRouter.

Evidence

• Fleet compile-check 37d91b3098752797cf3775ff at c842b42 shows done.
• gh pr checks at this head: 25 pass, 30 skipping, 0 red. This PR causes no red.

Non-blocking

• Every mux.state acquisition now calls Instant::now() about three times, takes the holder mutex twice, and makes one sites map update under state. The map update SipHashes the Location's file string. Nobody has measured this cost on the per-key paths. The PR claims no speedup, so nothing blocks on it. A follow-up bench of the agents scenario before and after would confirm the cost.
• diagnostics.rs:165 says 256 bytes. OctaveHistogram is 272 bytes.

[lawrencecchen]

Review (claude-subagent-merge-review): SHIP at 51e2350
R9: pass
perf claim: none

Independent review of the merge head 51e2350 (parents c842b42, the PR head, and 7008ce7, the feat-cmux-next tip). I did not write this code. I read the code only. I did not build or test on the review machine.

Merge resolution (the locks section from this PR and the hook_queue section from #28041)

• diagnostics.rs ServerStatsSnapshot: locks: Option<LocksSnapshot> comes after write_path, then hook_queue: Option<HookQueueSnapshot>. Both fields have skip_serializing_if = "Option::is_none". The diff of this file against feat has the same size as the PR's own change to the merge base (+166/-15). Nothing from either side is lost.
• server/server_stats.rs: both constants (LOCKS, HOOK_QUEUE) and both fields are present. locks is wants(LOCKS).then(|| mux.lock_detail_stats()). The hook_queue expression from feat did not change.
• server/server_stats_tests.rs: server_stats_include_reports_registry_and_state_lock_sites is complete, and its closing brace is restored. server_stats_include_reports_the_hook_queue follows it and has its #[cfg(unix)] from feat. Each test is identical to its parent's version.
• spec/commands.md: the include list is resource_projection, write_path, locks and hook_queue. The result shape has locks?:..., (with a comma) and then hook_queue?:.... The lock_detail/lock_site definitions and both prose paragraphs are present. The "reads atomics (and, for hook_queue, ...)" text from feat is kept.
• spec/sdk-schema.json: against feat, the only additions are the locks field (optional, not nullable, after write_path), its constraint, the three types ServerStatsLockSiteDetail, ServerStatsLockDetail and ServerStatsLocks, and the include constraint with the four names. Against the PR parent, the only additions are feat's hook_queue field, its constraint and ServerStatsHookQueue. In the Rust structs, holder and last_stall are Option without skip_serializing_if, so they serialize as null. The schema has required + nullable: true for them, and this agrees. All site fields are u64/String, required.
• spec/api-compat.surface.json: the automatic merge has both /data/locks/** (with holder, last_stall and last_stall/blocker nullable) and /data/hook_queue/**.
• Bindings: python3 -I bindings/codegen/generate.py --check passes for cpp, go, java, python, rust, typescript, zig and the webviews page router.
• I found no duplicate item and no missing item.

The PR as a whole

• Deadlock fix (mux/resource_effects/effect_commit.rs:75-90): settle_registry_receipts runs from lock_state_pinned (mux/signaled_mutex.rs:384) while the registry and mux.state are held. Before the fix, it called publish_resource_event (mux/journal.rs:6). That function calls publish_pending_terminal_directories (mux/terminal_directory.rs:95), which does self.state.lock(). On one thread this relocks a std Mutex. The fix calls only publish_journal_event, which takes only the leaf journal_event_epoch lock. take_settled_receipts has no other caller, so this is the only path. The terminal-directory publication that is now skipped has other triggers: resource_effects.rs:223, resource_workspace.rs:233, terminal_move.rs:124, resource_api.rs:541 and the terminal's own output. Thus no state is lost; the publication is only deferred. The fix is correct.
• Regression test settling_a_late_registry_receipt_does_not_relock_state: it uses hold_unsettled_for_test to keep an answered indeterminate receipt, then pins from a second thread. A panic or a deadlock fails recv_timeout (5 s). It also asserts that the owed settle ran. The PR body records the red result on the base (lock-rank panic at mux/journal.rs:11).
• StateMutex/StateGuard (mux/signaled_mutex.rs:214-327): wait_started runs before the block. acquired runs after the acquire. released runs in Drop::drop before the MutexGuard field drops, so the release is recorded while the lock is held, and an older unlock cannot erase the holder record of the next holder. This is the same order as SignaledMutexGuard. The poisoned path and try_lock record the same way. LockStats takes only its own leaf mutexes (holder, last_stall, sites) and never takes state or the registry. Thus it adds no lock-order edge.
• OctaveHistogram (diagnostics.rs:164-212): bucket i holds 2^(i-1)..2^i-1, and the p99 rank is ceil(0.99·n). The bound is clamped to max_us. All arithmetic is saturating or checked. sites grows only by &'static Location, so its bound is the number of call sites in the binary. DETAIL_SITES (20) limits only the report.
• #[track_caller] on about 40 wrappers changes only the Location that is recorded. It does not change behavior.

Findings

No correctness finding.

• Low (not blocking), diagnostics.rs:182-186: a sample of 2^31 µs or more (about 36 minutes) goes into the last bucket. That bucket's p99 bound (2^31-1) is then below the true value. The docs say "bucket upper bound". This applies only to holds or waits that last more than half an hour, where hold_max_us still shows the true value.
• Note (not blocking): every mux.state acquisition now takes about four more uncontended leaf mutex operations and one HashMap lookup on release. The release work occurs inside the state hold. The PR claims no speedup, and I expect this cost to be about 0.1 µs, far below the 10 ms action budget. No measurement of this overhead is attached.

Fleet evidence at 51e2350

• Compile: 3f6e8bcf32f8c46dd9346b33 is done. CmuxNext, app and cmux-tui have ok: true (I read the controller log).
• settling_a_late_registry_receipt_does_not_relock_state (the deadlock red test): 290d1a83dc69b37fee962a02 done, 1 test run: 1 passed (PASS mux::tests::writer_lock_order::...).
• lock_detail_reports_per_site_hold_and_wait_percentiles: 5a591e100d4ac31c43b420ca done, 1 test run: 1 passed.
• server_stats_include_reports_registry_and_state_lock_sites (exercises the merged server_stats.rs): c37071d49c4d1fa15e57ec5f done, 1 test run: 1 passed.
• Not run at this head: feat's server_stats_include_reports_the_hook_queue (no fleet job named it; the resolution leaves its body byte-identical to feat's), and a full cmux-tui test run.

[lawrencecchen]

Checks at the head 51e2350 (pa-b3), gh pr checks 28147, one row per red. None is this PR's own:

check	cause	class	evidence
cmux-tui spec inventory / clippy (changed cmux-tui packages)	Rust panic ratchet RED: cmux-tui-core indexing_slicing 856 > 853 and string_slice 57 > 52, at agent_hooks.rs:1013-1015, agent_message_prompt.rs, apps/, browser_, fs_ops, git_ops, launch_credential.rs, mux/presentation.rs. The warnings also include acpmux, cmux-tui, cmux-wg	base	the same check is red with the same counts on the feat-cmux-next tip (push run 41158092570 at a112726, job 124410703378: indexing_slicing 856 > baseline 853, string_slice 57 > baseline 52). None of the named lines is in a file this PR changes; this PR's own histogram hits were removed in d8f9ee9
rust workspaces / Native/OptChat/optchat-chief	memory_sync_state::tests::unasked_detect_never_lists_downloads_or_desktop panicked (memory_sync_state.rs:461)	base	the same test fails on an unrelated PR (lane/ao-takeover-22030, run 41159553538). This PR does not touch Native/OptChat; the job runs because the cmux-tui crates changed
rust workspaces	wrapper of optchat-chief	wrapper
cmux-next guards / tooling-tests	test_the_ten_first_party_gpl_crates_are_named_exactly: the list now has cmux-chats-mirror; plus no ACTIVE profile named 'cmux NEXT Developer ID'	base	red on the feat tip (guards push run 41157955814 at ef2c8ee, job tooling-tests)
cmux-next guards / release-notary	GitHub API rate limit exceeded for installation (POST 403, backoff to 300 s)	infrastructure, also red on the feat tip	the same job is red on the feat tip (run 41157955814)
nightly-guards / Guard status	wrapper of the guard jobs	wrapper
pr-red-checks / comment	cancelled	wrapper
PR checks / link	GitHub API rate limit exceeded for installation	infrastructure	reran once on the same sha (run 41157832409)

Still pending: cmux-next swift test, fast tier, risk evidence, merge freeze, pr-perf-section-check, pr-capture-check, changes, plan.

This PR's diff against its merge base changes only cmux-tui-core, cmux-tui/spec, cmux-tui/bindings and one bench script. Its focused tests (290d1a83, 5a591e10, c37071d4: 1 passed each) and its compile (3f6e8bcf) pass at this head.