Skip to content

cmux NEXT: a restored terminal pane comes back without a key press - #23145

Merged
austinywang merged 4 commits into
feat-cmux-nextfrom
23029-restore-pane-waits-input
Oct 21, 2026
Merged

austinywang merged 4 commits into
feat-cmux-nextfrom
23029-restore-pane-waits-input

Conversation

@austinywang

@austinywang austinywang commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #23029. A restored terminal pane in cmux NEXT, usually an agent, stayed on "Restoring…" and loaded only once the user typed.

Cause. A restored view whose first attaches fail (the daemon is busy or slow to answer while restoring) re-attached by itself with the same backoff as any dropped link: 500 ms doubling to 30 s. A key press is the only thing that cuts that wait short; input typed while an attach is in flight is just queued. So the pane sat on "Restoring…" for up to 30 s between tries, and after 8 tries it said "Disconnected: could not attach" while still retrying. A second, smaller hole: a re-attach event (the daemon connection came back, a key, focus) that arrived while an attach was in flight was dropped. If that attach then failed, the view waited the full backoff.

Change (TerminalAttachMachine, the reducer every terminal view's attachment runs on):

  • A restored view that has not shown content yet retries at most every 2 s (restoreBackoffCap). It keeps saying "Restoring…" for about the same 90 s as before (quietRestoreReconnects), then offers Retry and backs off as before, so a daemon that never answers is not asked every 2 s for good.
  • An event that arrives while an attach is in flight is kept (reattachRequested). If that attach fails, the next one opens at once.

Views that already went live, hidden views and the non-restore path keep their old behavior.

Testing

Regression tests in Packages/macOS/CmuxNext/Tests/CmuxNextAppTests/TerminalRestoreAttachTests.swift, committed red first (efe6e37) and then fixed:

  • aRestoredViewKeepsRetryingWithinTwoSecondsUntilItFirstAttaches: about 90 s of failed attaches stay within 2 s of each other and still say Restoring; then the view goes live by itself.
  • theConnectionBackDuringAnAttachReattachesAtOnceWhenThatAttachFails
  • aRestoreThatNeverAttachesOffersRetryAndBacksOffAsBefore (review follow-up)

Focused command, fleet step: cmux-ci run --class light --script scripts/ci/package-test-lane.sh --arg=suite --arg=Packages/macOS/CmuxNext --arg=TerminalRestoreAttachTests,TerminalAttachRetryTests --ref <sha>, at the red commit efe6e37 (--red) and at the fix head 7791cdf. Results:

  • Red at efe6e37, step d22d2be53019945e16320275 on aws-m4pro-5: TerminalRestoreAttachTests failed, 2 of 2 tests and 69 issues. On the base the restore waits backoff step 38 and more, says Disconnected after 8 tries, and drops the in-flight reconnect.
  • Green at 7791cdf, step 1f6a32d544646acf5baf96db: all 3 TerminalRestoreAttachTests passed, and both TerminalAttachRetryTests still pass. The head 346a568 adds only the release-note file on top of that.

Observed / Not run

Observed in the cmux NEXT GUI: tagged builds driven only through their control socket and the bundled cmux CLI, launched with open -g so the user's focus never moved. Real Claude Code was idling in the visible pane. I used Quit and Keep Sessions, then relaunched with CMUX_NEXT_DEBUG_SLOW_ATTACH=10:200 (the debug seam that makes each terminal's first 10 attaches fail, as against a busy daemon). There was no input after the relaunch. Link states are from debug.surfaces:

Build Visible agent pane after relaunch
base 18611e5c (tag 23029-restore-waits-input-base), run 1 "Restoring…" from 3.6 s, "Disconnected: could not attach" from 89 s, live at 148 s
base, run 2 "Restoring…" from 3.7 s, "Disconnected" from 67 s, live at 122 s
fix 346a5681 (tag 23029-restore-pane-waits-input) "Restoring…" from 3.3 s, live at 19.2 s

Also run on the base: the repo's scripts/cmux-next/restore-journey.py --continuity --variants quit,reboot, plus plain quit and reboot without slow attaches. Every visible terminal came back live within 2.5 to 6.5 s, so the bug needs failing attaches to show. The journey's reboot terminal-agent rows fail outside a capture slot because its shell can't find the test CLIs; that is unrelated.

Not run:

  • The full CmuxNext package suite: only the two attach suites ran.
  • A reproduction against a real busy daemon: no local trigger made attaches fail on their own, so the GUI comparison uses the debug seam.
  • No screenshots: window.screenshot runs only for a person, and the banner's look is unchanged; only its timing changed.

Correctness review by a subagent found no bugs. It flagged one risk, fixed in 7791cdf: the 2 s cap now ends with the quiet period. It also raised two nits: a doc comment, now reworded, and the retry count carrying past going live, which is existing behavior that bounds a flapping link; the test no longer leans on it.

Changelog

Fixed: A restored terminal or agent pane that says "Restoring…" now comes back by itself, without a key press

Proof

The before/after link timelines above come from the real app in tagged builds.

🤖 Generated with Claude Code

austinywang and others added 4 commits October 21, 2026 00:08
…led attaches (red, #23029)

A restored pane said Restoring… and came back only when a key cut its
backoff short. These cases fail on the base: the restore backs off to
30 s and offers Retry after 8 tries, and a connection that comes back
while an attach waits for its replay is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…3029)

A restored terminal view whose attaches fail while the daemon is busy
backed off like any dropped link, 500 ms doubling to 30 s, still saying
Restoring…, and only a key press cut the wait short. A restored view
that has not shown content yet now retries every 2 s at most and keeps
saying Restoring… for about the same 90 s before it offers Retry.

An event that re-attaches (the daemon connection back, a key, focus)
that arrives while an attach is in flight is no longer dropped: if that
attach fails, the next one opens at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e Retry shows (review, #23029)

The 2 s cap now ends with the restore's quiet period (about 90 s): a
daemon that never answers is not asked every 2 s by every restored pane
for good. The test pins that, and no longer leans on the retry count a
restore carries past going live.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@austinywang austinywang added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 21, 2026
@austinywang

Copy link
Copy Markdown
Contributor Author

Correctness review (subagent), head 3008042 → resolved at 7791cdf

No correctness bugs found. The reviewer traced the reducer by hand. Both new tests fail on the base (efe6e37) and pass with the fix, and TerminalAttachRetryTests is unaffected because non-restore views take the base path.

The reviewer checked these paths and found no issue:

  • Every place reattachRequested is set or cleared: live, overflow, exitAfterReplay, processRevived, close.
  • No tight loop: disconnect always opens after setting .disconnected, and only outside events set the flag.
  • Hidden views re-attach only once, through an event they would get anyway.

Findings and resolution:

  1. Risk: the 2 s cap never expired for a view that never goes live, so a wedged daemon would get an attach every 2 s per restored pane for good. Fixed in 7791cdf: the cap applies only while the view is still in its about-90 s quiet period. After that Retry shows and the backoff grows as before. New test: aRestoreThatNeverAttachesOffersRetryAndBacksOffAsBefore.
  2. Nit: autoReconnects carries past going live. This is the base's existing behavior: it bounds a link that flaps between live and dropped. Kept. The test that leaned on it was removed.
  3. Nit: a doc comment said "nothing else may wake it". Reworded.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 346a568: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23145 -f ref=23029-restore-pane-waits-input -f platform=macos -f backend_mode=local

Not verified: macOS jobs did not run: compile and app tests were skipped on 346a5681c2; ci-status is green without them.

CI passes on 346a5681c2 (run 39563448139 attempt 1).
macOS jobs did not run: compile and app tests were skipped: Fast static checks skipped (the macos job needs both to succeed). Re-run CI once they pass, before merging.

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

Tour of 346a5681c2 (run)

Path-selected interactions

Interaction Changed paths
core Packages/macOS/CmuxNext/Sources/CmuxNextApp/DaemonTerminalIO.swift
Packages/macOS/CmuxNext/Sources/CmuxNextBridge/TerminalAttachMachine.swift
Packages/macOS/CmuxNext/Tests/CmuxNextAppTests/TerminalRestoreAttachTests.swift

Core UI (before): DEGRADED — 19 ok, 1 unavailable

tour

Video (mp4)

# Step Result Command
1 Launch ok
2 New workspace ok cmux workspace new --name Tour
3 Type in the terminal ok cmux workspace name:Tour screen current pane current tab current terminal current write --text "echo hello from the cmux-next tour\r"
4 A command reports progress: tab loading spinner ok cmux workspace name:Tour screen current pane current tab current terminal current write --text "clear; printf '\\033]9;4;3\\007'; sleep 900\r"
5 Split right ok cmux action run splitRight
6 Split down ok cmux action run splitDown
7 New terminal tab ok cmux action run newSurface
8 Browser tab ok cmux tab new-browser --url https://example.com
9 Hide the sidebar ok cmux action run toggleSidebar
10 Show the sidebar ok cmux action run toggleSidebar
11 Notification ok cmux notify --title Tour --body "Hello from CI"
12 Notifications panel ok cmux action run showNotifications
13 Agent chat ok cmux agent new-chat
14 Agent workspace ok cmux workspace new --name Agent
15 Agent starts ok cmux workspace name:Agent screen current pane current tab current terminal current write --text "clear; printf '\\033]7501;state=working\\033\\\\'; sleep 8; printf '\\033]7501;state=done\\033\\\\'; sleep 900\r"
16 An agent works: row working spinner ok cmux workspace new --name Notes
17 The agent finishes: done mark in the spinner's place ok
18 Session tree: keycaps on the first rows ok cmux settings set sidebar.sessionTree true
19 Session tree back off for later tours ok cmux settings set sidebar.sessionTree false
20 Profile menu unavailable: exited 1 (the menu shot needs a screen capture route) cmux action run sidebar.profileMenu
Screenshots (15)

1. Launch
Launch

2. New workspace
New workspace

3. Type in the terminal
Type in the terminal

4. A command reports progress: tab loading spinner
A command reports progress: tab loading spinner

5. Split right
Split right

6. Split down
Split down

7. New terminal tab
New terminal tab

8. Browser tab
Browser tab

9. Hide the sidebar
Hide the sidebar

11. Notification
Notification

12. Notifications panel
Notifications panel

13. Agent chat
Agent chat

16. An agent works: row working spinner
An agent works: row working spinner

17. The agent finishes: done mark in the spinner's place
The agent finishes: done mark in the spinner's place

18. Session tree: keycaps on the first rows
Session tree: keycaps on the first rows

### Core UI \(after\): DEGRADED — 19 ok, 1 unavailable

tour

Video (mp4)

# Step Result Command
1 Launch ok
2 New workspace ok cmux workspace new --name Tour
3 Type in the terminal ok cmux workspace name:Tour screen current pane current tab current terminal current write --text "echo hello from the cmux-next tour\r"
4 A command reports progress: tab loading spinner ok cmux workspace name:Tour screen current pane current tab current terminal current write --text "clear; printf '\\033]9;4;3\\007'; sleep 900\r"
5 Split right ok cmux action run splitRight
6 Split down ok cmux action run splitDown
7 New terminal tab ok cmux action run newSurface
8 Browser tab ok cmux tab new-browser --url https://example.com
9 Hide the sidebar ok cmux action run toggleSidebar
10 Show the sidebar ok cmux action run toggleSidebar
11 Notification ok cmux notify --title Tour --body "Hello from CI"
12 Notifications panel ok cmux action run showNotifications
13 Agent chat ok cmux agent new-chat
14 Agent workspace ok cmux workspace new --name Agent
15 Agent starts ok cmux workspace name:Agent screen current pane current tab current terminal current write --text "clear; printf '\\033]7501;state=working\\033\\\\'; sleep 8; printf '\\033]7501;state=done\\033\\\\'; sleep 900\r"
16 An agent works: row working spinner ok cmux workspace new --name Notes
17 The agent finishes: done mark in the spinner's place ok
18 Session tree: keycaps on the first rows ok cmux settings set sidebar.sessionTree true
19 Session tree back off for later tours ok cmux settings set sidebar.sessionTree false
20 Profile menu unavailable: exited 1 (the menu shot needs a screen capture route) cmux action run sidebar.profileMenu
Screenshots (15)

1. Launch
Launch

2. New workspace
New workspace

3. Type in the terminal
Type in the terminal

4. A command reports progress: tab loading spinner
A command reports progress: tab loading spinner

5. Split right
Split right

6. Split down
Split down

7. New terminal tab
New terminal tab

8. Browser tab
Browser tab

9. Hide the sidebar
Hide the sidebar

11. Notification
Notification

12. Notifications panel
Notifications panel

13. Agent chat
Agent chat

16. An agent works: row working spinner
An agent works: row working spinner

17. The agent finishes: done mark in the spinner's place
The agent finishes: done mark in the spinner's place

18. Session tree: keycaps on the first rows
Session tree: keycaps on the first rows

Tours are data in scripts/cmux-next/media/tours: add a step or a tour file to show your feature.

@blacksmith-sh

blacksmith-sh Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Found 28 test failures on Blacksmith runners:

Failures

Test View Logs
.github/workflows/
nightly-next-tick.yml: triggers on the display names ['cmux-next', 'cmux-tui artifacts'
] and declares no SOURCE_WORKFLOW_PATHS, so renaming any of those workflows stops this
one from ever triggering and nothing reports it
View Logs
main.IOSWiring.test_test_ios_macos_jobs_take_the_runner_jobs_pool/
test_test_ios_macos_jobs_take_the_runner_jobs_pool
View Logs
main.SideLanes.test_a_completed_failed_blacksmith_retry_is_not_watched_again/
test_a_completed_failed_blacksmith_retry_is_not_watched_again
View Logs
main.SideLanes.test_a_real_failure_on_cmux_next_attempt_two_is_not_rescued/
test_a_real_failure_on_cmux_next_attempt_two_is_not_rescued
View Logs
cmux-next.yml job swift-canary step 'Report the build fleet compile verdict' uses the t
oolchain (swift) before scripts/select-ci-xcode.sh runs
View Logs
debug info {'CMUX_SWIFT_SANITIZE': 'address', 'CMUX_SWIFT_TEST_DEBUG_INFO': 'none'}: 's
wift --version' lacks '-debug-info-format none'/
debug info {'CMUX_SWIFT_SANITIZE': 'address', 'CMUX_SWIFT_TEST_DEBUG_INFO': 'none'}: 's
wift --version' lacks '-debug-info-format none'
View Logs
debug info {'CMUX_SWIFT_SANITIZE': 'address', 'CMUX_SWIFT_TEST_DEBUG_INFO': 'none'}: 's
wift package resolve --package-path Packages/macOS/Pkg --scratch-path Packages/macOS/
Pkg/.build-sanitize-address' lacks '-debug-info-format none'
View Logs
debug info {'CMUX_SWIFT_SANITIZE': 'address', 'CMUX_SWIFT_TEST_DEBUG_INFO': 'none'}: wa
nt one build and two suites: ['swift package resolve --package-path Packages/macOS/
Pkg --scratch-path Packages/macOS/Pkg/
.build-sanitize-address', 'swift --version', 'swift build --build-tests -c debug -debug
-info-format none --sanitize=address --scratch-path Packages/macOS/Pkg/
.build-sanitize-address -Xlinker -no_compact_unwind --package-path Packages/macOS/
Pkg', 'swift test -c debug -debug-info-format none --sanitize=address --scratch-path Pa
ckages/macOS/Pkg/
.build-sanitize-address -Xlinker -no_compact_unwind --package-path Packages/macOS/
Pkg --skip-build --filter ATests', 'swift test -c debug -debug-info-format none --sanit
ize=address --scratch-path Packages/macOS/Pkg/
.build-sanitize-address -Xlinker -no_compact_unwind --package-path Packages/macOS/
Pkg --skip-build --filter BTests']
View Logs
debug info {'CMUX_SWIFT_SANITIZE': 'address'}: want one build and two suites: ['swift p
ackage resolve --package-path Packages/macOS/Pkg --scratch-path Packages/macOS/Pkg/
.build-sanitize-address', 'swift --version', 'swift build --build-tests -c debug --sani
tize=address --scratch-path Packages/macOS/Pkg/
.build-sanitize-address -Xlinker -no_compact_unwind --package-path Packages/macOS/
Pkg', 'swift test -c debug --sanitize=address --scratch-path Packages/macOS/Pkg/
.build-sanitize-address -Xlinker -no_compact_unwind --package-path Packages/macOS/
Pkg --skip-build --filter ATests', 'swift test -c debug --sanitize=address --scratch-pa
th Packages/macOS/Pkg/
.build-sanitize-address -Xlinker -no_compact_unwind --package-path Packages/macOS/
Pkg --skip-build --filter BTests']
View Logs
debug info {'CMUX_SWIFT_TEST_DEBUG_INFO': 'dwarf'}: want one build and two suites: ['sw
ift package resolve --package-path Packages/macOS/
Pkg', 'swift --version', 'swift build --build-tests -c debug --package-path Packages/
macOS/Pkg', 'swift test -c debug --package-path Packages/macOS/
Pkg --skip-build --filter ATests', 'swift test -c debug --package-path Packages/macOS/
Pkg --skip-build --filter BTests']
View Logs

...and 18 more test failures. View all on Blacksmith

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@austinywang austinywang added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 21, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator

Bead cx-y6s09 created for this PR by the cmux coordinator.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants