Skip to content

cmux-tui: a daemon start adopts its terminal hosts in parallel (perf100, cx-p00l3.1) - #23180

Closed
lawrencecchen wants to merge 3 commits into
feat-cmux-nextfrom
perf100-tui-handoff
Closed

lawrencecchen wants to merge 3 commits into
feat-cmux-nextfrom
perf100-tui-handoff

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

perf100 tui lane, bead cx-p00l3.1 (P1): after an update, the new daemon adopts every terminal host one at a time before it answers. On Lawrence's laptop, 113 terminals stayed blank for 8 to 20 s. On mac26 with 112 terminals, the daemon took 8.4 s to start, and the first terminal frame came at 14.7 s.

Change (mux/terminal_adoption.rs)

  • The startup pass still makes every registry decision serially and in record order: exit sidecars, tombstones, incarnation checks, the template claim, the legacy import, orphan recovery, and the Adopting transition.
  • It then collects the live hosts and runs their handshakes at once on up to 16 scoped threads. A handshake connects and replays the host's snapshot into the mirror.
  • The results commit serially in record order, through the same success and failure paths as before. Lifecycle and topology decisions therefore do not change.
  • Output order does not change either. Each host replays its own snapshot, and later output follows on that host's stream.
  • The batch sets a Kitty budget owner floor while it runs, so each side-by-side reservation gets its final share at once. Without the floor, a reservation could wait 500 ms and then start with graphics disabled. A drop guard ends the floor, also when the commit loop returns early.
  • A debug-only seam, CMUX_TUI_TEST_ADOPT_HANDSHAKE_DELAY_MS, slows each handshake for the test.

Evidence

tests/terminal_host_recovery/parallel_handoff.rs runs on rbx. It uses 16 cat terminals, SIGKILLs the daemon, and restarts it with every handshake slowed to 400 ms.

  • Red at b7bf0b6: answered in 6.57 s (serial: 6.4 s).
  • Green at dbeaf2d: answered in 0.63 s.
  • After the handoff, every terminal is running on its old host pid, keeps its screen, and new input lands after the old screen.

Real handoff with no seam, on rbx (Linux, release build). The script is handoff_bench.py. Setup: 100 terminals with 20,000 lines of scrollback each. The old daemon is SIGKILLed, and the new binary starts on the same state.

Run Base 3c73ce6 After Ratio
spawn to first answer, run 1 3.80 s 1.09 s 3.5x
spawn to first answer, run 2 3.87 s 1.31 s 3.0x
spawn to all 100 running 3.81 / 3.88 s 1.09 / 1.32 s 3.0-3.5x

The GUI measurement with scripts/cmux-next/quit-launch-bound.py --handoff (#22919) runs on cmux-lawrence-2, before and after. The results will follow as a comment.

Not in this PR (named, decision for the coordinator)

  • The daemon still answers only after the whole adoption pass. The coordinator asked for the next step: answer at once, attach visible terminals first, and adopt the rest in the background. That needs the app to accept tabs in the adopting state at launch, which the async retry path already publishes (PendingTerminal::Adopting). It also needs an order of visible terminals first.
  • The handoff case is not yet added to the CI guard. quit-launch-bound.py lives in cmux NEXT: a keep quit ends within 4 s whatever any step does; Restart to Update never leaves cmux invisible #22919, which is still open. I will add the case there, or in a follow-up once it lands.

Observed / Not run

  • Ran on rbx:
    • clippy: 2 base errors (agent_reports.rs:458, event_shape_tests.rs:148), none from this change.
    • terminal_host_recovery: 204 pass, 4 fail. The same 4 fail on base 3c73ce6 (sized_attach x3, a_respawned_terminal_resumes_its_agent_session).
    • Kitty tests: 34/34 pass.
  • Review: the subagent findings and their resolution are in a PR comment.

Changelog

  • Fixed: after an update, terminals show again in about a third of the time.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

After an update, the new daemon adopted each terminal host one at a time before answering, leaving 113 terminals blank for 8-20s. It now runs the host handshakes (connect plus snapshot replay) in parallel on up to 16 threads while keeping every registry and lifecycle decision serial and in record order, so the daemon answers in about a third of the time with no change to output order.

  • The startup pass still makes all registry decisions serially and in record order; only the host handshakes run concurrently, and their results commit in the same order as before.
  • While the batch runs, a Kitty budget owner floor gives side-by-side reservations their final share at once; a drop guard ends the floor even on early error.
  • Adds a debug-only seam (CMUX_TUI_TEST_ADOPT_HANDSHAKE_DELAY_MS) and a test that SIGKILLs the daemon and restarts it with 16 slowed handshakes, down from 6.4s to 0.63s.

Changelog: Fixed: after an update, terminals show again in about a third of the time.

Written for commit dbeaf2d. Summary will update on new commits.

View guided diff Turn on auto-fix

lawrencecchen and others added 3 commits October 20, 2026 22:58
…erf100, cx-p00l3.1)

16 cat terminals, the daemon SIGKILLed and restarted with every host
handshake slowed to 400 ms by a debug seam (CMUX_TUI_TEST_ADOPT_HANDSHAKE_DELAY_MS).
The new daemon must answer within 3 s (serial: 6.4 s), every terminal must
be running on its old host with its screen, and input after the handoff
must land after the old screen. Red on the base: the startup pass adopts
one host after another.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…00, cx-p00l3.1)

After an update the new daemon adopted every terminal host one after
another before it answered: 113 terminals kept every terminal blank for 8
to 20 s on Lawrence's laptop (mac26: 8.4 s to start, first frame at 14.7 s).
The startup pass now makes every registry decision in record order as
before, collects the live hosts, runs their handshakes (connect, snapshot
replay into the mirror) at once on up to 16 scoped threads, then commits
the results in record order with the same success and failure paths, so
topology and lifecycle decisions are unchanged and no output is reordered
(each host replays its own snapshot; later output follows on its stream).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…h (review)

A reservation not yet committed cannot be shrunk, so side-by-side
reservations that changed the budget bucket each waited out the 500 ms
reservation wait and adopted with Kitty graphics disabled. The batch now
sets an owner floor (existing owners + handshakes) for the capacity while
it runs, so every reservation gets its final share at once; a drop guard
ends the floor after the commits, also on an early error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@lawrencecchen lawrencecchen added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 21, 2026
@lawrencecchen
lawrencecchen temporarily deployed to cmux-tui-merge-tree October 21, 2026 08:01 — with GitHub Actions Inactive
@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Correctness review (review subagent, read-only) and resolution

# Finding Severity Resolution
1 Side-by-side Kitty reservations cannot shrink an entry that is not yet committed, so each one that changed the bucket waited 500 ms and adopted with graphics disabled Medium Fixed in dbeaf2d. The batch sets an owner floor, so capacity is sized for every owner at once. A drop guard ends the floor. Kitty tests 34/34 pass. The test handoff went from 1.03 s to 0.63 s.
2 An early ? in the commit loop drops adopted surfaces that were never committed, without disconnect_for_daemon_shutdown Low Mux::new fails in that case, and the daemon exits. Recorded; not changed.
3 Test margin, and the test passes when some handshakes fall back to async retry Low The margin is now about 5x (0.63 s against 3 s). The test asserts that every terminal is running on its old host pid.

The reviewer traced these and found them clean: template claim, legacy import, orphan recovery (each creates its workspace inside the serial loop), handled_terminals, the sweeps after the loop, restored_terminal_binding, ids, locks (no lock is held during the scope), panics and spawn failure, FD pressure, and the wider window before commit (the is_dead check still applies).

@github-actions

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review dbeaf2d: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23180 -f ref=perf100-tui-handoff -f platform=macos -f backend_mode=local

@lawrencecchen lawrencecchen added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 21, 2026
@blacksmith-sh

blacksmith-sh Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Found 29 test failures on Blacksmith runners:

Failures

Test View Logs
.github/workflows/
nightly-next-tick.yml: triggers on the display names ['cmux-next', 'cmux-tui artifacts'
] and declares no SOURCE_WORKFLOW_PATHS, so renaming any of those workflows stops this
one from ever triggering and nothing reports it
View Logs
main.IOSWiring.test_test_ios_macos_jobs_take_the_runner_jobs_pool/
test_test_ios_macos_jobs_take_the_runner_jobs_pool
View Logs
main.SideLanes.test_a_completed_failed_blacksmith_retry_is_not_watched_again/
test_a_completed_failed_blacksmith_retry_is_not_watched_again
View Logs
main.SideLanes.test_a_real_failure_on_cmux_next_attempt_two_is_not_rescued/
test_a_real_failure_on_cmux_next_attempt_two_is_not_rescued
View Logs
cmux-next.yml job swift-canary step 'Report the build fleet compile verdict' uses the t
oolchain (swift) before scripts/select-ci-xcode.sh runs
View Logs
debug info {'CMUX_SWIFT_SANITIZE': 'address', 'CMUX_SWIFT_TEST_DEBUG_INFO': 'none'}: 's
wift --version' lacks '-debug-info-format none'/
debug info {'CMUX_SWIFT_SANITIZE': 'address', 'CMUX_SWIFT_TEST_DEBUG_INFO': 'none'}: 's
wift --version' lacks '-debug-info-format none'
View Logs
debug info {'CMUX_SWIFT_SANITIZE': 'address', 'CMUX_SWIFT_TEST_DEBUG_INFO': 'none'}: 's
wift package resolve --package-path Packages/macOS/Pkg --scratch-path Packages/macOS/
Pkg/.build-sanitize-address' lacks '-debug-info-format none'
View Logs
debug info {'CMUX_SWIFT_SANITIZE': 'address', 'CMUX_SWIFT_TEST_DEBUG_INFO': 'none'}: wa
nt one build and two suites: ['swift package resolve --package-path Packages/macOS/
Pkg --scratch-path Packages/macOS/Pkg/
.build-sanitize-address', 'swift --version', 'swift build --build-tests -c debug -debug
-info-format none --sanitize=address --scratch-path Packages/macOS/Pkg/
.build-sanitize-address -Xlinker -no_compact_unwind --package-path Packages/macOS/
Pkg', 'swift test -c debug -debug-info-format none --sanitize=address --scratch-path Pa
ckages/macOS/Pkg/
.build-sanitize-address -Xlinker -no_compact_unwind --package-path Packages/macOS/
Pkg --skip-build --filter ATests', 'swift test -c debug -debug-info-format none --sanit
ize=address --scratch-path Packages/macOS/Pkg/
.build-sanitize-address -Xlinker -no_compact_unwind --package-path Packages/macOS/
Pkg --skip-build --filter BTests']
View Logs
debug info {'CMUX_SWIFT_SANITIZE': 'address'}: want one build and two suites: ['swift p
ackage resolve --package-path Packages/macOS/Pkg --scratch-path Packages/macOS/Pkg/
.build-sanitize-address', 'swift --version', 'swift build --build-tests -c debug --sani
tize=address --scratch-path Packages/macOS/Pkg/
.build-sanitize-address -Xlinker -no_compact_unwind --package-path Packages/macOS/
Pkg', 'swift test -c debug --sanitize=address --scratch-path Packages/macOS/Pkg/
.build-sanitize-address -Xlinker -no_compact_unwind --package-path Packages/macOS/
Pkg --skip-build --filter ATests', 'swift test -c debug --sanitize=address --scratch-pa
th Packages/macOS/Pkg/
.build-sanitize-address -Xlinker -no_compact_unwind --package-path Packages/macOS/
Pkg --skip-build --filter BTests']
View Logs
debug info {'CMUX_SWIFT_TEST_DEBUG_INFO': 'dwarf'}: want one build and two suites: ['sw
ift package resolve --package-path Packages/macOS/
Pkg', 'swift --version', 'swift build --build-tests -c debug --package-path Packages/
macOS/Pkg', 'swift test -c debug --package-path Packages/macOS/
Pkg --skip-build --filter ATests', 'swift test -c debug --package-path Packages/macOS/
Pkg --skip-build --filter BTests']
View Logs

...and 19 more test failures. View all on Blacksmith: run 39566126416, run 39566126375, run 39566126252

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator

cmux-tui SDKs / protocol contract is red on 10 of 11 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@lawrencecchen

Copy link
Copy Markdown
Contributor Author

GUI handoff measurement (quit-launch-bound.py --handoff from #22919, cmux-lawrence-2)

The session has 38 workspaces, 152 terminals and 5 browser tabs, with no agent. Quit mode is interactive. The launch measured is the one after the daemon version handoff. There is one run per build.

Metric Base 3c73ce6 (job 1021-010217-e93b27) This PR (job 1021-010218-6f1429) Change
First live terminal frame (interactive_ms) 8292 ms (FAIL, bound 8000) 6948 ms (pass) -1344 ms, 1.19x
First window frame 904 ms 892 ms same
Longest main-thread stall 354 ms 321 ms same

The parallel adoption saves about 1.3 s here. About 6.9 s still passes before the first terminal frame. Most of that time is outside the adoption pass. The next step is to answer at once and attach visible terminals first, which needs the app side. It is a separate item.

@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Closed by the landing captain (coordinator decision, 2026-10-22): #23561 (merged 17:23Z) already makes a daemon start adopt its terminal hosts in parallel. This branch conflicts with it in mux/terminal_adoption.rs and tests/terminal_host_recovery.rs. The branch stays on the remote for reference.

This branch was previously deployed

4 inactive deployments
ci-routing-agent — dbeaf2d6 Deployed Oct 21, 2026 by github-actions[bot]
artifacts — dbeaf2d6 Deployed Oct 21, 2026 by lawrencecchen
sentry-dsyms — dbeaf2d6 Deployed Oct 21, 2026 by lawrencecchen
cmux-tui-merge-tree — dbeaf2d6 Deployed Oct 21, 2026 by lawrencecchen
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants