[teamleaderleo]

Closes gaps G4 and G5 of HQ #4877 ("Remote machines in NEXT: setup walkthrough, breaks, and recovery after a link drops"). Leo asked about this (ask-8fac5e5066, 2026-10-28): "after you've connected something remotely and whatever, that whole setup situation, like after we set it up, how good is it in terms of that whole, like, recovery flow, and, like, that kind of connection situation, and how reliable things are?"

From the issue:

G4. An SSH machine can show Connecting… forever. After the retry budget is spent nothing tries any more, but sshStatus still maps it to connecting (SidebarBridge+SSH.swift:47-51). The header's Reconnect sends only wake(.user), which does nothing while the link client is still inside its own redials (a guess).

G5 (to check). pr/26786's reconnect-on-use treats .failed (which includes sessionInUse) as reconnectable, and runs machine.reconnect with a user origin from the pane's automatic 2 s handshake retry. An open tab could keep taking over a session that is in use elsewhere.

What changes

G4: a reconnect that gave up reads Offline, and Reconnect starts a fresh connect

• DaemonConnection.observeRetriesSpent: the reconnect loop reports each failed attempt after its timed budget (10 tries, about 80 s) is spent. DaemonService.start(remote:) hooks it up (DaemonService+RetriesSpent.swift) to mark DaemonStore.retriesSpent. The next connect clears the mark, and so does a new attempt (beginConnectionAttempt). A late hop that lands while connected changes nothing.
• SidebarBridge.sshStatus maps a link that is connecting, connected or reconnecting, with its daemon not connected and its retries spent, to .offline, not .connecting. Offline is red, the header offers Reconnect, and reconnectsOnUse is true, so opening the chat reconnects it.
• Reconnect on a machine in that state is SSHReconnectStep.restart: SSHMachineSession.restart() stops the link client and the daemon connection, then connects again with a new retry budget, like Disconnect then Connect. Before, it was .nothing (link connected) or a wake that tried once.
• .restart joins base's SSHConnectionMachine.Status.reconnectStep (Forget and Devices Remove each take out both the SSH machine and its acpmux peer #28799 moved it to CmuxNextRemote). Since that move no longer offsets this PR's App lines, plans/cmux-next/arch-ratchet.json gets an app_lines exception of 24 lines.

G5: automatic paths never take over a session in use

• SSHConnectionMachine.waitsForUser: true while the gate is closed for sessionInUse, until a .user wake. SSHMachineLink mirrors it so it can be read without an actor hop.
• The pane's reconnect on use (AgentTabs+Wiring machineLink.reconnect) now runs machine.reconnect with origin .page, not .user. machine.reconnect passes its origin on to remote.reconnect. remote.reconnect counts only .user and .cli as a person's Reconnect, and SSHService.reconnect(_:byPerson:) does nothing on a session in use unless a person asked.
• SidebarBridge.paneUse reads an SSH session in use as .stuck, so the pane neither waits on it nor reconnects it. MachineLinkState.reconnectsOnUse(.failed) stays true for other failures, which can still be retried.

No collision with #28623 (webviews only: the error card while a chat's machine reconnects) or #28407 (Cloud statuses in MachineLinkState). Prior art: #26786 (merged; this PR fixes its G5) and #27060 (merged; first-connect recovery, whose needsRecovery and reconnectStep cases are unchanged).

Review fixes

• 81ca529: a server.busy refusal past the budget keeps a timed retry, so it doesn't mark retries spent (RetryPacer.waitsForEventOnly, with a test in BusyRefusalReconnectTests).
• 77564d2: a late retries-spent hop applies only while its connection is still the service's, and is not reported from a cancelled or closed loop. restart() keeps its teardown task, which the next restart, Disconnect and close() all cancel (aRestartedSessionThatClosesNeverConnectsAgain).
• a6efbba: an automatic endpoint attempt (.attemptStarted, sent before the gate check) no longer clears the in-use user-only gate.
• 09a1ccd: remote.connect, remote.newGroup and reconnectWorkspace pass ActionInvocation.isPersonsRequest (user or cli) as byPerson. SSHService.connect leaves a session in use alone for any other caller.

Follow-ups (P2, from the SHIP at 09a1ccd; owned by this lane, fix forward):

• An automated remote.connect with a different cmux-tui binary path replaces the saved session (cx-vuvc) before the in-use check, so the new session isn't gated. Check the saved session first.
• (P1, at c11edce; fixed in 3d0fa56) waitsForUser now reads the link's gate directly.
• (P2, at 3d0fa56) A connection that drops within 10 s of a successful handshake can enter the event-only wait without reporting retries spent. Report on every entry to the event-only wait.
• (P2, at 3d0fa56) The takeover check reads the gate, then queues .wake(.user)/.connect on the link actor; a .sessionInUse landing in between gets opened. Carry byPerson into the actor and check there.
• After retriesSpent is set, a wake that gets a busy refusal resumes timed retries but leaves the mark on, so the header reads Offline while it retries. Publish the transition back as well.

Tests

Tests/CmuxNextAppTests/SSHMachineLinkRecoveryTests.swift (new):

• aSpentReconnectReadsOfflineNotConnecting: the header status for each link state, with and without spent retries.
• reconnectAfterSpentRetriesIsAFreshConnect: SSHReconnectStep gives .restart.
• theStoreMarksSpentRetriesUntilTheNextConnect: the mark is cleared on connect, and a late hop is ignored.
• reconnectOnASpentSessionStartsANewAttemptAndClearsTheMark: the full SSHService.reconnect path.
• aSessionInUseWaitsForAPerson: waitsForUser across app-activation and user wakes.
• reconnectOnUseLeavesASessionInUseAndAPersonTakesItOver: paneUse == .stuck; byPerson: false starts no attempt, and byPerson: true does.

RemoteConnectRecoveryTests.reconnectDuringATransientInterruptionKeepsAnEstablishedDaemonsStartup is moved to SSHReconnectStep and its expectations are unchanged.

Command: swift test --filter 'SSHMachineLinkRecoveryTests|RemoteConnectRecoveryTests|SSHSessionInUseTests' (the CmuxNextAppTests suites in this PR's CI).

Observed / Not run

• Observed: scripts/cmux-next/prepush-gate.py passes (no new violations, and arch-ratchet shows no App growth). The namespace lint passes.
• Not run: no Swift toolchain on this lane VM, so I haven't compiled or run anything here. The compile and the tests come from this PR's CI, and I'll compare their results against base. No red/green run on base either: the new tests use APIs this PR adds (retriesSpent, waitsForUser, .restart), so on base they fail to compile rather than fail.

Risk

risk:normal. This changes the SSH reconnect paths.

• A person's Reconnect after spent retries now tears down and redials the link client. Before, it did nothing on a connected link.
• The pane's reconnect on use runs with origin .page. For SSH machines this only matters when the session is in use. Paired servers and Cloud machines ignore the origin (wake() and retry()).
• A CLI cmux machine reconnect still counts as a person.

UI

The sidebar SSH header reads Offline (red dot, with Reconnect) where it used to show Connecting… forever.

No capture: getting there takes a live SSH link whose daemon reconnect runs out of its 10 timed retries (about 80 s), and no scene or fixture can drive DaemonConnection's reconnect loop. The words and dot are the existing Offline header, which is unchanged.

Changelog

Fixed: an SSH machine whose connection gave up shows Offline with Reconnect instead of Connecting… forever, and an open chat no longer takes over a machine's session that is in use elsewhere.

Agent session: agent session

🤖 Generated with Claude Code

[blacksmith-sh]

Found 12 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmpnuw54ccx/summary-missing.json (live):/summary-missing.json (live):	View Logs
/tmp/tmpy70cpwod/summary-fail.json (live):/summary-fail.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/ test_download_prints_the_active_profile_only	View Logs
stubs: /System/Library/Frameworks/Security.framework/Versions/A/Security has no stub/ Security has no stub	View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/ CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_CFRetain']/ CoreFoundation, missing 1: ['_CFRetain']	View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/ CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_log2']/ CoreFoundation, missing 1: ['_log2']	View Logs

...and 2 more test failures. View all on Blacksmith

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Review (router-review, gpt-6-astra high fast): SHIP at 77564d2

SHIP at 77564d20968703168932a2011e8e21910fde7127

Visual verdict: no images supplied (range review); the visual verdict must come separately

1. [P2] [follow-up] The “session in use” protection is bypassed by `remote.connect` when it reuses an existing saved host. `SSHService.connect` calls `reconnect(saved)` with the new default `byPerson: true` ([SSHService.swift:162](tree/Packages/macOS/CmuxNext/Sources/CmuxNextApp/Remote/SSHService.swift:162)), while `RemoteHandlers` does not pass the invocation origin ([RemoteHandlers.swift:50](tree/Packages/macOS/CmuxNext/Sources/CmuxNextApp/Handlers/RemoteHandlers.swift:50)). A page or other non-person caller can therefore take over a session held by another daemon, contrary to G5. Propagate the origin or explicitly mark this path as non-person.

Follow-ups:

- [P2] [follow-up] Prevent non-person `remote.connect` reuse from calling `reconnect` with `byPerson: true`.

Earlier rounds' findings are fixed at this head: the busy refusal past the budget (81ca529), the late retries-spent hop and the closed session's restart (77564d2).

[teamleaderleo]

Review (codex gpt-6-astra high, next-review): SHIP at 77564d2

No blocking findings.

[cmux-bot]

cmux-bot approves 77564d2: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 3d0fa56: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28794 -f ref=lane/ssh-machine-link-recovery -f platform=macos -f backend_mode=local

Default changes

This pull request changes 3 defaults at 3d0fa5642f53 (what everyone gets without configuring anything):

• tunable-source.CmuxNextApp/Handlers/RemoteGroupHandlers.swift:bind: "fcf4c25cc5e4d1047d7ef433e49bf0dea5b08ba904fbf7a74affae2b9f7d5646" -> "d16717865137b5be752da3f1d4ccbef1c763e51e9b158581fd039f82a2a32ab7"
• tunable-source.CmuxNextApp/Handlers/RemoteHandlers.swift:bind: "d8f074d1e21ae2ae29d0929458239f998671c225503bb82c880a5dbd5ce5434e" -> "2585b4c6766bd93c200f65b6c54d82b838b2016d650f783df0df3acc0392ee3e"
• tunable-source.CmuxNextApp/Handlers/SSHTargetHandlers.swift:bind: "6334799836d31d8444e07d35f88ad80dc6ee4aa441b037f8fcfe059eab515ac4" -> "9c48d11c8a1da6b8220e33d0b8787581aebb9d263e4ad797457ee3de20158878"

Signed off on this commit by review (router-review, gpt-6-astra high fast).

Merge receipt: merged as a768273bf47b with these defaults changed, signed off by review (router-review, gpt-6-astra high fast).

Regenerate the snapshot with python3 scripts/ci/defaults_guard.py snapshot.

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4122680534501 (b6eb84916c)
• cmux NEXT nightly 1.0.0-nightly.4122811790801 (228b1fb697)
• cmux NEXT nightly 1.0.0-nightly.4122926028501 (e4a99b0dc7)

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (router-review, gpt-6-astra high fast): SHIP at 09a1ccd

SHIP at 09a1ccd4a5123a13d36a91d5a9902d2685f93cf4

Visual verdict: no images supplied (range review); the visual verdict must come separately

1. [P2] [follow-up] `Packages/macOS/CmuxNext/Sources/CmuxNextApp/Remote/SSHService.swift:186` — Check the existing session before replacing it. An automated `remote.connect` supplying a different binary path removes the blocked session at lines 179–183. This guard then checks a fresh session whose `waitsForUser` is false, allowing another connection attempt despite the person-only gate. Apply the check before closing/removing the saved session.

2. [P2] [follow-up] `Packages/macOS/CmuxNext/Sources/CmuxNextDaemon/Connection/DaemonConnection.swift:391` — Clear exhaustion when timed retries resume. After exhaustion sets `store.retriesSpent`, a network/app wake can receive a busy refusal and resume timed retries. Suppressing another notification leaves the existing flag true, so the sidebar still reports Offline and reconnect-on-use unnecessarily restarts the connection. Publish the transition back to active retries as well.

Static review completed; Swift tests could not run because Swift is unavailable.

Follow-ups:

- Check the saved session’s person-only gate before replacing it.
- Clear the exhausted state when busy/restoring responses resume timed retries.

[teamleaderleo]

Review (router-review, gpt-6-astra high fast): SHIP at c11edce

SHIP at c11edce515cad9c92dc27832f162b82f6125e044

Visual verdict: no images supplied (range review); the visual verdict must come separately

Focused Swift tests could not run because `swiftc` is unavailable in the review environment.

1. [P1][follow-up] `SSHMachineSession.waitsForUser` only consults the mirrored gate when `linkStatus == .failed` ([SSHMachineSession.swift:65](Packages/macOS/CmuxNext/Sources/CmuxNextApp/Remote/SSHMachineSession.swift:65)). `SSHMachineLink` updates its mirror before asynchronously publishing the status, so an automatic reconnect can observe the old `.connecting`/`.connected` status, bypass the person-only guard, and wake a session held by another daemon. Read the link’s mirror directly or update the session state atomically.

Follow-ups:

- [P1][follow-up] Close the status-publication race so reconnect-on-use cannot take over an in-use SSH session.

c11edce merges the base tip a3f620e (clears the crash-class base red, #29032); the PR's own files are unchanged from 09a1ccd.

[teamleaderleo]

Review (router-review, gpt-6-astra high fast): SHIP at 3d0fa56

SHIP at 3d0fa5642f53f175ad7b0296fab1a70b5ffab3f9
Visual verdict: no images supplied (range review); the visual verdict must come separately

1. [P2] [follow-up] `Packages/macOS/CmuxNext/Sources/CmuxNextDaemon/Connection/DaemonConnection.swift:430` — Exhaustion notification misses a short-lived recovery. A successful handshake clears `retriesSpent`, but the retry budget resets only after 10 healthy seconds. If the connection drops sooner, the loop can enter an indefinite event-only wait before reaching this callback. The sidebar stays Connecting, and Reconnect misses the new restart path. Notify when entering the event-only wait, including after a successful connection drops.

2. [P2] [follow-up] `Packages/macOS/CmuxNext/Sources/CmuxNextApp/Remote/SSHService.swift:224` — The takeover check races with the link actor. An automated reconnect can read `waitsForUser == false`, then the link reports `.sessionInUse` before the queued `.wake(.user)` or `.connect` executes. That queued event opens the user-only gate. Carry `byPerson` into the actor operation and check the gate atomically there.

Tests could not run: this Linux box has no Swift toolchain.

Follow-ups:

- #1: Report exhaustion whenever reconnect enters an event-only wait.
- #2: Enforce the takeover restriction inside the link actor.

3d0fa56 merges base c5b06d3 (base's #28799 moved the reconnect step to SSHConnectionMachine.Status.reconnectStep; this PR's .restart joins it), reads the link's in-use gate directly (the c11edce P1 follow-up), and adds an app_lines exception for the 24 App lines the PR's own move no longer pays for.