Skip to content

Stop SSH auto-reconnect when host is unreachable; add manual Reconnect control (#5734) - #5767

Merged
austinywang merged 13 commits into
mainfrom
issue-5734-ssh-reconnect-manual-control
Jun 10, 2026
Merged

austinywang merged 13 commits into
mainfrom
issue-5734-ssh-reconnect-manual-control

Conversation

@austinywang

@austinywang austinywang commented Jun 10, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #5734

Problem

Switching networks (plane wifi → hotspot → office, captive portals) drops SSH remote workspaces, and the auto-reconnect loop in WorkspaceRemoteSessionController retries indefinitely while the host is unreachable — scheduleReconnectLocked has no halt condition, and the user has no way to say "stop — I'll reconnect when I'm ready."

Pre-fix repro (local route-drop simulation)

Configured an SSH remote workspace against a connection-refused endpoint (nobody@127.0.0.1:49999) on a Debug build of the pre-fix HEAD and polled workspace.remote.status for 5 minutes:

[20:54:25] error | Remote daemon bootstrap failed: … Connection refused (retry 2 in 8s)
[20:55:43] error | … Connection refused (retry 5 in 60s)
[20:58:49] error | … Connection refused (retry 8 in 60s)   ← still climbing, never halts

Controller debug log shows remote.session.connect.begin retry=0…8 with 60s-capped exponential backoff and no bound.

Fix

Reconnect policy (built into the existing state machine, not a parallel one):

  • WorkspaceRemoteReconnectPolicy — a pure decision function: every scheduled retry kicks a quick reachability probe; consecutive unreachable probes build a streak, and at 3 the loop suspends instead of rescheduling. Reachable/indeterminate probes reset the streak, so transient blips (sleep/wake, wifi handoff) keep today's backoff behavior (Make Cloud VM SSH sessions resilient to sleep and reconnects #3776 unaffected).
  • WorkspaceRemoteHostReachabilityProbe — resolves the effective endpoint with ssh -G (honors ~/.ssh/config aliases, HostName overrides, and probes the first ProxyJump hop), then attempts a short-timeout TCP connect via NWConnection. ProxyCommand transports can't be probed directly and report indeterminate, which never suspends — the policy only halts on positive evidence of unreachability.
  • Suspension cancels the pending retry, logs remote.session.reconnect.suspended, and publishes the new WorkspaceRemoteConnectionState.suspended with a localized detail. Suspended workspaces are exempt from remote-workspace demotion so the configuration (and the reconnect affordance) survives the last terminal session dying.

Manual control (one shared action path — Workspace.reconnectRemoteConnection() — for every entrypoint):

  • Sidebar: the SSH row stays visible while suspended (status shows "Unreachable" with an explanatory tooltip) and gains an inline Reconnect button.
  • Workspace context menu: existing Reconnect/Disconnect items work for suspended workspaces (suspended ≠ connecting, so Reconnect stays enabled).
  • CLI: new cmux workspace reconnect / cmux workspace disconnect verbs calling the existing workspace.remote.reconnect / workspace.remote.disconnect socket methods. Targets a positional/--workspace handle, then the caller's workspace, then the selected one.
  • A sidebar status entry, sidebar log line, and notification ("SSH Reconnect Paused") surface the suspension when it happens.

On manual reconnect the existing configureRemoteConnection path preserves persistent PTY session identity, so terminals reattach to live remote sessions where the relay/cmuxd-remote layer supports it; when the remote session is gone, the existing ended-session banner ("falling back to a local shell…") reports it rather than silently spawning a fresh shell.

Two-commit red/green structure

  1. Commit 1 adds WorkspaceRemoteReconnectPolicy encoding the current never-suspend behavior plus tests asserting the desired halt-on-unreachable policy → the suspend assertions fail (verified locally: 3 failed / 3 passed).
  2. Commit 2 lands the policy change and the integration above → suite green.

Post-fix verification (local sshd + severable TCP proxy)

User-level sshd on 127.0.0.1:2299 behind a kill-able TCP proxy on :2300 (kill = route drop):

  • connect → state=connected, persistent PTY session listed
  • kill proxy → transport failure → 3 unreachable probes → state=suspended, retry loop stops (debug log goes quiet)
  • restart proxy → cmux workspace reconnect → state=connected, same persistent PTY session reattached

(Exact transcripts in the PR comments if needed.)

Scope notes

  • The per-pane ssh attach loop is already bounded (CMUX_SSH_RECONNECT_LIMIT, default 20) and prints attempts in-terminal; unchanged here. In-terminal reconnect UI is Show SSH reconnection attempts in terminal #1530; the persistent-remote-daemon architecture is Persistent SSH relay daemon that survives SSH disconnects #2696.
  • The shared WorkspaceRemoteProxyBroker retains its internal short retries; sustained transport failures escalate to the controller's re-bootstrap path, where this policy applies.
  • No new keyboard shortcuts (no KeyboardShortcutSettings changes needed).

Localization audit

New user-facing strings (remote.status.suspended, sidebar.remote.help.suspended, sidebar.remote.reconnect.button, sidebar.remote.reconnect.help, remote.state.suspended.detail, remote.statusEntry.suspended, remote.notification.suspendedTitle) are added to Resources/Localizable.xcstrings in en/ja/ko/uk, matching the locale coverage of their remote.status.* / sidebar.remote.* sibling families. The previously-uncataloged workspace context-menu reconnect/disconnect labels (contextMenu.{re,dis}connectWorkspace{,s}) were also added in the same four locales. CLI help/error text follows the existing unlocalized CLI conventions; docs/cli-contract.md is an English-only doc. No web message catalogs are affected.

🤖 Generated with Claude Code


View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.


Summary by cubic

Pause SSH auto-reconnect when an SSH host is unreachable and add a manual Reconnect control in the UI and CLI. This stops endless retry loops and lets users reconnect when ready (fixes #5734).

  • New Features

    • Reconnect policy: after 3 consecutive unreachable probes, suspend auto-retry; indeterminate probes never suspend; normal backoff stays for transient blips.
    • Reachability probe: resolves via ssh -G (honors aliases/HostName; probes first ProxyJump hop; ProxyCommand → indeterminate) and runs a short TCP check; includes an sshConfigFile seam for hermetic tests.
    • UI: show “Unreachable,” keep the SSH row visible while suspended, and add an inline Reconnect button; context-menu Reconnect/Disconnect stay enabled; suspension surfaces a status entry, log line, and notification; manual reconnect preserves persistent PTY sessions; suspended workspaces are not demoted.
    • CLI/docs: new cmux workspace reconnect / disconnect call workspace.remote.reconnect / workspace.remote.disconnect; with --window, target that window’s selected workspace (no CMUX_WORKSPACE_ID fallback); cmux workspace --help and docs/cli-contract.md updated; CLI help and subcommand errors localized (en/ja/ko/uk).
  • Bug Fixes

    • Probe: clear NWConnection stateUpdateHandler before cancel() to avoid a retain cycle and per-retry leaks; first-finisher latch relies on the serial probe queue.
    • Concurrency: run ssh -G endpoint resolution on a concurrent utility queue so simultaneous probes don’t block NW callbacks.
    • State: clear the suspended flag when the proxy becomes .ready so later failures can reschedule retries.

Written for commit e93aa48. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • New Features

    • Auto SSH reconnect can enter a suspended state after repeated unreachable probes; UI shows a “Reconnect” affordance and suspended status.
  • Connectivity

    • Added reachability probing to decide whether to continue retries or suspend automatic reconnects.
  • UI

    • Sidebar shows suspended status, updated help/detail text, and a contextual Reconnect button.
  • CLI

    • Added workspace subcommands: reconnect and disconnect; help/errors updated.
  • Tests

    • Added tests for reconnect policy, reachability probing, and snapshot behavior.
  • Localization

    • New localized strings for suspended/reconnect UI and notifications.

The SSH remote auto-reconnect loop retries forever while the host is
unreachable. Introduce the WorkspaceRemoteReconnectPolicy seam encoding
today's never-suspend behavior, plus tests asserting the desired
policy: suspend the loop after consecutive failed reachability probes
so the user controls when reconnection happens. The suspend assertions
fail (red) until the policy change lands in the follow-up commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jun 10, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
cmux Ready Ready Preview, Comment Jun 10, 2026 10:16pm
cmux-staging Building Building Preview, Comment Jun 10, 2026 10:16pm

@coderabbitai

coderabbitai Bot commented Jun 10, 2026 •

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds an SSH host reachability probe and a reconnect policy that suspends automatic reconnect after three consecutive unreachable probes; wires suspend state into workspace daemon, UI snapshot and sidebar affordance, CLI verbs, localization, tests, and Xcode project build entries.

Changes

SSH Remote Reconnect Feature

Layer / File(s) Summary
Core reconnect policy and tests
Sources/WorkspaceRemoteReconnectPolicy.swift, cmuxTests/WorkspaceRemoteReconnectPolicyTests.swift
Probe outcome enum, Decision/Evaluation model, evaluate(...) logic; unit tests cover reset/increment/suspend and reset-on-success behaviors.
SSH resolution and TCP probe
Sources/WorkspaceRemoteHostReachabilityProbe.swift, cmuxTests/WorkspaceRemoteReconnectPolicyTests.swift
Resolves effective SSH endpoint via ssh -G (hostname/port, ProxyJump first-hop), rejects ProxyCommand setups for direct probing, probes TCP using NWConnection with timeout; parsing helpers and async TCP tests (with BlockingTCPListener).
Workspace reconnect loop & daemon state
Sources/Workspace.swift
Tracks consecutive-unreachable probes, probe generation; evaluates reachability before retrying; suspends auto-reconnect on threshold, cancels pending retries, and publishes .suspended daemon state with dedicated handling.
Sidebar UI affordance & snapshot wiring
Sources/ContentView.swift, Sources/Sidebar/SidebarWorkspaceSnapshotRefreshPolicy.swift, cmuxTests/SidebarWorkspaceSnapshotRefreshPolicyTests.swift
Adds showsRemoteReconnectAffordance for .suspended, keeps SSH row visible, shows Reconnect button/help text, preserves the field in snapshot builder and test helper.
CLI verbs & docs
CLI/cmux.swift, docs/cli-contract.md
Adds workspace reconnect and workspace disconnect branches and runWorkspaceRemoteConnectionCommand helper; updates CLI messages and documents workspace verbs.
Localization
Resources/Localizable.xcstrings
Adds localized strings for suspended reconnect UI state, help text, status/notification, and context menu actions (en/ja/ko/uk).
Xcode project build integration
cmux.xcodeproj/project.pbxproj
Registers new probe/policy source files and tests in cmux/cmuxTests targets and workspace groups.

Sequence Diagram(s)

sequenceDiagram
  participant Scheduler as ReconnectScheduler
  participant Probe as WorkspaceRemoteHostReachabilityProbe
  participant Policy as WorkspaceRemoteReconnectPolicy
  participant Workspace as WorkspaceDaemon
  Scheduler->>Probe: schedule probe
  Probe->>Policy: WorkspaceRemoteHostProbeOutcome
  Policy->>Workspace: Decision (scheduleRetry / suspend)
  alt suspend
    Workspace->>Scheduler: cancel pending retries
    Workspace->>UI: publish .suspended state
  end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related issues

Poem

🐰 A rabbit hops along the SSH line,
Probes three times then pauses to mind,
If paths return, it hops anew,
If shadows stay, it waits for you—
Reconnect when the sky looks kind.


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (4 errors, 1 warning, 2 inconclusive)

Check name Status Explanation Resolution
Cmux Swift Blocking Runtime ❌ Error WorkspaceRemoteHostReachabilityProbe.swift introduces usleep polling in runSSHConfigResolution, violating swift-blocking-runtime.md which forbids sleep/usleep in shipped code. Replace usleep polling loop with a real signal mechanism like ProcessTermination notification, DispatchGroup, or async process wrapper for SSH process completion.
Cmux Swift Concurrency ❌ Error PR introduces custom DispatchQueue (probeQueue) and completion-handler APIs in new internal code where both caller/callee are under cmux control, violating the swift concurrency modernization guide. Refactor WorkspaceRemoteHostReachabilityProbe to expose async functions instead, wrapping NWConnection callbacks internally via withCheckedContinuation.
Cmux Full Internationalization ❌ Error 11 new reconnect/suspended string keys added with only 4 locales (en/ja/ko/uk) instead of all 19 supported locales in Localizable.xcstrings catalog. Add translations for missing 16 locales (ar/bs/da/de/es/fr/it/km/nb/pl/pt-BR/ru/th/tr/zh-Hans/zh-Hant) for all 13 new keys in Resources/Localizable.xcstrings.
Cmux Source Artifacts ❌ Error PR adds 26 files in .claude/ and .agents/ hidden scratch directories (.claude/worktrees/ explicitly listed in policy) without removing them or adding to .gitignore. Remove .claude/* and .agents/* from the diff, or add .claude/ and .agents/ patterns to .gitignore to exclude these hidden tool/scratch directories.
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Cmux Swift @Concurrent ❓ Inconclusive No result was produced after verification. Marking as INCONCLUSIVE. Re-run the check or adjust instructions to produce a final result.
Cmux Architecture Rethink ❓ Inconclusive No result was produced after verification. Marking as INCONCLUSIVE. Re-run the check or adjust instructions to produce a final result.
✅ Passed checks (14 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically summarizes the main change: stopping indefinite SSH auto-reconnect and adding manual reconnect controls, with issue reference.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Cmux Swift Actor Isolation ✅ Passed New pure enums (probe outcome, policy, probe) with static methods need no MainActor. Workspace is @MainActor; probe completions hop back to DispatchQueue safely. No isolation issues found.
Cmux Expensive Synchronous Load ✅ Passed The PR adds background probes on a utility DispatchQueue (not main actor) and uses SharedLiveAgentIndex.shared with nil-check fallback, following the policy's allowed patterns.
Cmux Cache Substitution Correctness ✅ Passed Snapshot is purely transient UI state (not persisted). The showsRemoteReconnectAffordance field is computed fresh from live @Published var remoteConnectionState on each render, never cached.
Cmux No Hacky Sleeps ✅ Passed Check scope is TypeScript/JavaScript/shell/non-Swift build scripts; all PR changes are Swift code, Xcode config, strings, and docs — outside scope and covered by separate Swift checks.
Cmux Algorithmic Complexity ✅ Passed All loops operate on bounded inputs (ssh -G ~30 lines, options ~5); no nested scans, batch rescans, or collection iteration in hot paths; per-workspace operations only.
Cmux Swift File And Package Boundaries ✅ Passed Files comply: ProbeOutcome (17), ReconnectPolicy (45), HostReachabilityProbe (269 lines, SSH probing). Workspace.swift +134 under 250 limit. No improper responsibility mixing.
Cmux Swift Logging ✅ Passed All new production code complies with swift-logging.md: new Sources files have no print/NSLog/Logger; Workspace.swift reconnect code adds no logging; CLI output uses print() which is allowed.
Cmux User-Facing Error Privacy ✅ Passed All user-facing messages use safe generic terms; probe error reasons confined to debugLog only. No vendor names, credentials, or implementation details exposed.
Cmux Swiftui State Layout ✅ Passed PR extends existing @Published remoteConnectionState enum with .suspended case, passes immutable snapshot values to views, proper button action closures, no violations of rules.
Cmux Swift Auxiliary Window Close Shortcuts ✅ Passed No new NSWindow, NSPanel, NSWindowController, or SwiftUI Window/WindowGroup classes added. Reconnect UI renders as a button in existing ContentView, not a standalone window.
Description check ✅ Passed The PR description comprehensively covers problem, fix, testing approach, verification, scope, and localization audit. All required sections from template are present and well-documented.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch issue-5734-ssh-reconnect-manual-control

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@greptile-apps

greptile-apps Bot commented Jun 10, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR stops the indefinite SSH auto-reconnect loop by adding a reachability probe (WorkspaceRemoteHostReachabilityProbe) that runs after each failed retry and a pure policy type (WorkspaceRemoteReconnectPolicy) that suspends the loop after three consecutive unreachable probes, then wires a manual Reconnect affordance across the sidebar, context menu, and CLI.

  • WorkspaceRemoteReconnectPolicy.evaluate correctly gates suspension on streak >= maxConsecutiveUnreachableProbes; reachable/indeterminate probes reset the streak so transient blips keep existing backoff behavior.
  • WorkspaceRemoteHostReachabilityProbe resolves the effective SSH endpoint via ssh -G (run on DispatchQueue.global to avoid serializing concurrent probes), handles ProxyJump first-hop and ProxyCommand indeterminate cases, clears the NWConnection state handler before cancel() to prevent the retain cycle, and uses a per-generation latch to discard stale outcomes.
  • State-machine reset is self-consistent: stopAllLocked, handleProxyBrokerUpdateLocked(.ready), and manual reconnect all clear reconnectSuspended + consecutiveUnreachableProbeCount + bump reachabilityProbeGeneration. All new user-facing strings are fully localized across en/ja/ko/uk.

Confidence Score: 5/5

The change is safe to merge: core policy logic, state-machine resets, and probe lifecycle are all correctly implemented, and the previously-flagged issues are resolved at HEAD.

The suspension threshold, generation-latch, retain-cycle fix, and reconnectSuspended reset on .ready are all present and correct. No new blocking logic issues or state-machine gaps were found beyond the already-addressed threads.

Sources/ContentView.swift — the safeHelp call passes the optional remoteWorkspaceSidebarText directly to String(format:locale:) rather than using a nil-coalesced value; safe today but fragile if conditions diverge.

Important Files Changed

Filename Overview
Sources/WorkspaceRemoteReconnectPolicy.swift New pure decision type: suspends after maxConsecutiveUnreachableProbes (3) consecutive unreachable probes; reachable/indeterminate resets the streak. Logic is correct at HEAD.
Sources/WorkspaceRemoteHostReachabilityProbe.swift New TCP probe with ssh -G endpoint resolution; runs on DispatchQueue.global to avoid serializing concurrent probes; NWConnection retain-cycle fix and first-finisher latch are correct; ProxyCommand and multi-hop ProxyJump handling is conservative (indeterminate).
Sources/WorkspaceRemoteHostProbeOutcome.swift Tiny new enum with three cases (reachable/unreachable/indeterminate); Equatable + Sendable; well-documented invariant that .indeterminate must never suspend the loop.
Sources/Workspace.swift Adds reconnectSuspended, consecutiveUnreachableProbeCount, and reachabilityProbeGeneration to the controller; resets all three in both stopAllLocked and the proxy .ready handler; demotion guard extended to exempt .suspended workspaces.
Sources/ContentView.swift Adds .suspended status text and sidebar Reconnect button; showsRemoteReconnectAffordance drives button visibility from snapshot. One P2: safeHelp call passes remoteWorkspaceSidebarText (String?) directly to String(format:locale:) rather than using a nil-coalesced value.
CLI/cmux.swift Adds reconnect and disconnect workspace subcommands; correctly skips CMUX_WORKSPACE_ID fallback when --window is given; error messages localized in en/ja/ko/uk.
Resources/Localizable.xcstrings All new user-facing strings present in en/ja/ko/uk, matching the locale coverage of adjacent string families.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[connection fails] --> B[scheduleReconnectLocked]
    B --> C{isStopping or reconnectSuspended?}
    C -- yes --> D[return early, no retry scheduled]
    C -- no --> E[arm reconnectWorkItem with backoff delay]
    E --> F[evaluateReconnectPolicyLocked]
    F --> G[bump reachabilityProbeGeneration, launch TCP probe on global utility queue]
    G -->|probe result| H{generation still current?}
    H -- no --> I[discard stale outcome]
    H -- yes --> J{reconnectWorkItem still pending?}
    J -- no --> I
    J -- yes --> K[WorkspaceRemoteReconnectPolicy.evaluate]
    K --> L{decision?}
    L -- scheduleRetry --> M[streak++ keep work item armed]
    L -- suspend streak >= 3 --> N[suspendAutoReconnectLocked]
    N --> O[cancel reconnectWorkItem, reconnectSuspended=true, publishState .suspended]
    O --> P[sidebar Reconnect button + notification]
    P --> Q{user clicks Reconnect?}
    Q -- yes --> R[reconnectRemoteConnection, new controller, all state reset]
    R --> A
    E -->|retryDelay expires| S[connectLocked]
    S -->|success| T[handleProxyBrokerUpdateLocked .ready, clears reconnectSuspended, bumps generation]
    S -->|failure| A
Loading

Reviews (9): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile

Comment thread Sources/WorkspaceRemoteReconnectPolicy.swift
…t control

Fixes #5734.

Every scheduled reconnect retry now kicks a quick reachability probe
(ssh -G endpoint resolution + short-timeout TCP connect; first
ProxyJump hop when present, indeterminate for ProxyCommand). Three
consecutive unreachable probes suspend the loop instead of retrying
indefinitely: the controller cancels the pending retry and publishes
the new .suspended connection state with a localized detail.

Manual control shares the existing Workspace.reconnectRemoteConnection
path: an inline sidebar Reconnect button on the (still visible) SSH
row, the workspace context menu, and new "cmux workspace reconnect" /
"cmux workspace disconnect" CLI verbs over the existing
workspace.remote.reconnect/disconnect socket methods. Suspended
workspaces are exempt from remote demotion, surface a status entry,
sidebar log line, and notification, and reattach to persistent PTY
sessions on manual reconnect where the relay layer supports it.

New strings localized in en/ja/ko/uk, matching their sibling
families; previously uncataloged context-menu reconnect/disconnect
labels added in the same locales.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ect-manual-control

# Conflicts:
#	Resources/Localizable.xcstrings
#	cmux.xcodeproj/project.pbxproj

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/cli-contract.md`:
- Line 98: Update the markdown table cell describing the workspace verbs to
escape the pipe characters inside the `--workspace <id|ref|index>` fragment so
it reads `--workspace <id\|ref\|index>`; locate the table row for the
`workspace` namespace (the cell mentioning verbs `list, create, close, rename,
select, reconnect, disconnect, group` and the `workspace reconnect`/`workspace
disconnect` text) and replace the unescaped `<id|ref|index>` with
`<id\|ref\|index>` to maintain table consistency (same pattern used elsewhere
for `--workspace`).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2498f341-232a-494e-b727-768586cf5667

📥 Commits

Reviewing files that changed from the base of the PR and between 8f6527a and 04857bb.

⛔ Files ignored due to path filters (1)
  • .github/swift-file-length-budget.tsv is excluded by !**/*.tsv
📒 Files selected for processing (11)
  • CLI/cmux.swift
  • Resources/Localizable.xcstrings
  • Sources/ContentView.swift
  • Sources/Sidebar/SidebarWorkspaceSnapshotRefreshPolicy.swift
  • Sources/Workspace.swift
  • Sources/WorkspaceRemoteHostReachabilityProbe.swift
  • Sources/WorkspaceRemoteReconnectPolicy.swift
  • cmux.xcodeproj/project.pbxproj
  • cmuxTests/SidebarWorkspaceSnapshotRefreshPolicyTests.swift
  • cmuxTests/WorkspaceRemoteReconnectPolicyTests.swift
  • docs/cli-contract.md
💤 Files with no reviewable changes (1)
  • Resources/Localizable.xcstrings

Comment thread docs/cli-contract.md
hhsw2015 pushed a commit to hhsw2015/cmux that referenced this pull request Jun 10, 2026
…ble + manual Reconnect (manaflow-ai#5734)

# Conflicts:
#	.github/swift-file-length-budget.tsv
#	ghostty
austinywang and others added 2 commits June 10, 2026 10:51
…ests

- "cmux workspace reconnect/disconnect" no longer falls back to the
  caller's CMUX_WORKSPACE_ID when an explicit --window is supplied, so
  the server resolves that window's selected workspace instead of a
  workspace from a different window (matches the rename command's
  convention).
- WorkspaceRemoteHostReachabilityProbe.resolveEndpoint gains an
  sshConfigFile test seam (ssh -F) so resolver tests pin /dev/null and
  stay hermetic against the ambient ~/.ssh/config; production callers
  keep honoring the user's real config.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment on lines +208 to +215
let deadline = Date().addingTimeInterval(sshResolveTimeout)
while process.isRunning, Date() < deadline {
usleep(20_000)
}
if process.isRunning {
process.terminate()
return nil
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Blocking poll loop on shared probeQueue thread

runSSHConfigResolution is dispatched onto probeQueue (the same queue used by probeTCP and its NWConnection callbacks). The usleep(20_000) busy-poll loop can block that thread for up to the full sshResolveTimeout (3 s). Any other probe dispatched to probeQueue during that window — including NWConnection state updates from a concurrent TCP probe — is stalled behind it. For users with multiple suspended SSH workspaces whose probes fire near-simultaneously, each probe can delay the next by a full 3 s sleep cycle.

The fix is to use process.terminationHandler (a real signal from the owning subsystem) and post back via a DispatchWorkItem cancel for the timeout side, instead of a polling sleep loop.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2fe454e — ssh -G resolution now runs on the global utility pool, so concurrent probes don't serialize on probeQueue and NWConnection callbacks stay unblocked; only the connection state updates and the timeout remain on the serial queue.

— Claude Code

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
Sources/WorkspaceRemoteHostReachabilityProbe.swift (1)

24-27: ⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Don't serialize every workspace probe behind one blocking queue.

runSSHConfigResolution can hold probeQueue for up to sshResolveTimeout, and that same serial queue also drives every other probe's NWConnection callbacks and timeout block. A single slow ssh -G therefore delays unrelated workspaces' probes past their configured deadlines, so suspension timing starts depending on queue backlog instead of the host being checked. Split the blocking resolution work off this shared queue, or make the queue concurrent and keep synchronization local to each probe.

Also applies to: 39-55, 196-219, 261-264

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Sources/WorkspaceRemoteHostReachabilityProbe.swift` around lines 24 - 27, The
serial probeQueue used by WorkspaceRemoteHostReachabilityProbe serializes long
blocking work (notably runSSHConfigResolution which can hold the queue for up to
sshResolveTimeout) and thus delays other probes' NWConnection callbacks and
timeout handlers; change the design so blocking SSH resolution does not run on
the shared serial queue — either make probeQueue concurrent and confine
synchronization to per-probe state, or create a separate DispatchQueue (e.g.,
sshResolutionQueue) and dispatch runSSHConfigResolution (and any blocking
file/exec work) onto that queue while keeping NWConnection callback and
short-lived probe state updates on the original probeQueue; update all uses
(including runSSHConfigResolution, NWConnection callback dispatches, and timeout
handlers) to ensure only non-blocking operations run on the shared probeQueue
and that per-probe locking/synchronization is local to each probe.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@Sources/WorkspaceRemoteHostReachabilityProbe.swift`:
- Around line 24-27: The serial probeQueue used by
WorkspaceRemoteHostReachabilityProbe serializes long blocking work (notably
runSSHConfigResolution which can hold the queue for up to sshResolveTimeout) and
thus delays other probes' NWConnection callbacks and timeout handlers; change
the design so blocking SSH resolution does not run on the shared serial queue —
either make probeQueue concurrent and confine synchronization to per-probe
state, or create a separate DispatchQueue (e.g., sshResolutionQueue) and
dispatch runSSHConfigResolution (and any blocking file/exec work) onto that
queue while keeping NWConnection callback and short-lived probe state updates on
the original probeQueue; update all uses (including runSSHConfigResolution,
NWConnection callback dispatches, and timeout handlers) to ensure only
non-blocking operations run on the shared probeQueue and that per-probe
locking/synchronization is local to each probe.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 8cbbe7c5-7c5f-4cd7-a3ce-e92642f5ce76

📥 Commits

Reviewing files that changed from the base of the PR and between 04857bb and f2cd329.

⛔ Files ignored due to path filters (1)
  • .github/swift-file-length-budget.tsv is excluded by !**/*.tsv
📒 Files selected for processing (3)
  • CLI/cmux.swift
  • Sources/WorkspaceRemoteHostReachabilityProbe.swift
  • cmuxTests/WorkspaceRemoteReconnectPolicyTests.swift

- probeTCP clears the NWConnection stateUpdateHandler before cancel so
  the handler/connection cycle can't leak one connection per backoff
  retry while the host stays reachable but bootstrap keeps failing.
- "cmux workspace --help" now lists the reconnect/disconnect verbs with
  their targeting rules and examples, keeping CLI discovery coherent
  with docs/cli-contract.md.
- Refresh the CLI/cmux.swift length budget for the help-text growth.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…be lock

- Restore the ghostty submodule pointer to origin/main's commit
  (34cbf18): the main-merge conflict resolution accidentally staged a
  stale local submodule checkout via git add -A. No ghostty change
  belongs to this PR.
- Move WorkspaceRemoteHostProbeOutcome into its own file so the policy
  file carries one top-level type (Aziz file-organization policy).
- probeTCP's first-finisher latch needs no NSLock: both finish paths
  (NWConnection state updates and the timeout) already run on the
  serial probeQueue, so plain queue confinement replaces the lock
  (Aziz concurrency policy).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@austinywang

Copy link
Copy Markdown
Contributor Author

Autoreview triage for the Aziz policy findings (canonical helper, --mode branch):

Fixed:

  • Two top-level types in WorkspaceRemoteReconnectPolicy.swift → WorkspaceRemoteHostProbeOutcome moved to its own file (a1cc202).
  • NSLock in probeTCP → removed; both finish paths already run on the serial probeQueue, so the first-finisher latch uses plain queue confinement (a1cc202).
  • NWConnection stateUpdateHandler retain cycle → handler cleared before cancel (6d34ec2).
  • Stale workspace --help → reconnect/disconnect documented (6d34ec2).
  • Ghostty submodule pointer accidentally staged during the main-merge → restored to origin/main's commit (a1cc202); no ghostty delta remains in this PR.

Rejected (with rationale):

  • Nested value types in the probe/policy files counted as extra major types: Endpoint/ResolvedSSHConfig/JumpSpec and Decision/Evaluation are the owning namespace's return/parsing vocabulary with a single consumer; the existing remote-connection API family models exactly this shape (WorkspaceRemoteProxyBroker.Lease/.Update, WorkspaceRemotePTYBridgeServer.Endpoint, SidebarWorkspaceSnapshotBuilder.Snapshot), and tests assert against the namespaced forms. Promoting them to top-level files would pollute the global namespace without an ownership win.
  • Convert SidebarWorkspaceSnapshotRefreshPolicyTests.swift to Swift Testing: this PR only adds one required constructor argument to that pre-existing XCTest suite (snapshot field ripple); converting the whole suite is out of scope for this change. The PR's own new tests use Swift Testing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread Sources/Workspace.swift
Comment on lines +7215 to 7217
consecutiveUnreachableProbeCount = 0
reachabilityProbeGeneration &+= 1
guard proxyEndpoint != endpoint else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 reconnectSuspended is not cleared when the proxy broker self-heals while the controller is already suspended. If WorkspaceRemoteProxyBroker succeeds through its own internal short retries during the suspended window, handleProxyBrokerUpdateLocked(.ready) fires, the workspace transitions to .connected, and reconnectSuspended stays true. The next connection failure calls scheduleReconnectLocked, which hits guard !reconnectSuspended and returns silently — no retry is scheduled and no UI affordance appears, leaving the workspace durably stuck in .error with no automatic or visible recovery path. Compare with stopAllLocked, which correctly resets all three counters together.

Suggested change
consecutiveUnreachableProbeCount = 0
reachabilityProbeGeneration &+= 1
guard proxyEndpoint != endpoint else {
consecutiveUnreachableProbeCount = 0
reconnectSuspended = false
reachabilityProbeGeneration &+= 1
guard proxyEndpoint != endpoint else {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2fe454e — the proxy .ready handler now clears reconnectSuspended alongside the retry/streak resets, so a connection established while suspended can't strand a later failure behind the suspended guard. (Today suspension implies no live broker lease, so the path was unreachable, but the state machine is now self-consistent either way.)

— Claude Code

austinywang and others added 2 commits June 10, 2026 14:03
…ect-manual-control

# Conflicts:
#	.github/swift-file-length-budget.tsv
#	Resources/Localizable.xcstrings
- Run ssh -G endpoint resolution on the global utility pool instead of
  the serial probeQueue so simultaneous probes from multiple suspended
  workspaces don't serialize behind its bounded blocking wait, and
  NWConnection callbacks stay unblocked (Greptile P1).
- Clear reconnectSuspended in the proxy .ready handler alongside the
  other policy resets so a connection established while suspended can't
  leave a later failure unable to reschedule retries (Greptile P1).
- Refresh the Workspace.swift length budget.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
austinywang and others added 2 commits June 10, 2026 14:34
The CLI target already localizes usage blocks (cli.claude-teams.usage
et al) and error strings (cli.error.*), so the new workspace
reconnect/disconnect help and the touched subcommand error messages
follow the same pattern: cli.workspace.usage,
cli.error.workspaceSubcommandRequired, and
cli.error.workspaceSubcommandUnknown, each with en/ja/ko/uk entries in
Localizable.xcstrings. This also corrects the PR's earlier audit note
that claimed CLI text was conventionally unlocalized.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ect-manual-control

# Conflicts:
#	.github/swift-file-length-budget.tsv
@austinywang

Copy link
Copy Markdown
Contributor Author

Autoreview triage, round 3:

Fixed (e462f1a / 2fe454e):

  • ssh -G endpoint resolution moved off the serial probeQueue onto the global utility pool so concurrent probes don't serialize and NWConnection callbacks stay unblocked.
  • The proxy .ready handler now clears reconnectSuspended alongside the other policy resets.
  • New workspace CLI help + touched subcommand error strings are localized (cli.workspace.usage, cli.error.workspaceSubcommand*) with en/ja/ko/uk catalog entries, matching the existing cli.claude-teams.usage / cli.error.* convention. This supersedes the earlier audit note that claimed CLI text was conventionally unlocalized — the CLI target does localize usage/error strings.

Rejected (with rationale):

  • Injectable probe seam + controller-level suspended-transition test: WorkspaceRemoteSessionController is a deliberately private, queue-confined state machine inside Workspace.swift; a deterministic controller test requires widening private members and neutralizing real ssh process spawns in beginConnectionAttemptLocked — a testability refactor beyond this fix's scope. The branching logic was extracted into WorkspaceRemoteReconnectPolicy precisely so the decision is exhaustively unit-tested; the wiring (3 unreachable probes → cancel retry → publish .suspended) is verified live on two builds with the logged trail remote.session.reachability streak=1→2→3 decision=suspend → remote.session.reconnect.suspended, plus end-to-end suspend/reconnect/reattach runs against a severable local sshd and a real remote Mac. Happy to do the seam as a follow-up if maintainers want controller-level coverage.

…ect-manual-control

# Conflicts:
#	Resources/Localizable.xcstrings
@austinywang
austinywang merged commit 8efa28b into main Jun 10, 2026
30 checks passed
austinywang added a commit that referenced this pull request Jun 11, 2026
…evert

The revert of PR #5767 also unwrapped three live user-facing workspace
CLI strings (subcommand-required error, unknown-subcommand error, and
the `workspace` usage text) back to bare literals, and deleted the
`workspace` row from the CLI contract docs. The `workspace` command and
its remaining verbs (list, create, close, rename, select, group) still
exist post-revert, so:

- Re-wrap the three strings in String(localized:) using the existing
  catalog keys, dropping the removed reconnect/disconnect verbs.
- Update en/ja/ko/uk catalog values for those keys to match.
- Restore the `workspace` contract row with the reduced verb set.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
austinywang added a commit that referenced this pull request Jun 11, 2026
* Revert PR 5767 remote reconnect suspension

This reverts commit 8efa28b.

* Keep workspace CLI strings localized and restore contract row after revert

The revert of PR #5767 also unwrapped three live user-facing workspace
CLI strings (subcommand-required error, unknown-subcommand error, and
the `workspace` usage text) back to bare literals, and deleted the
`workspace` row from the CLI contract docs. The `workspace` command and
its remaining verbs (list, create, close, rename, select, group) still
exist post-revert, so:

- Re-wrap the three strings in String(localized:) using the existing
  catalog keys, dropping the removed reconnect/disconnect verbs.
- Update en/ja/ko/uk catalog values for those keys to match.
- Restore the `workspace` contract row with the reduced verb set.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview – cmux — e93aa483 Deployed Jun 10, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SSH sessions don't persist across network switches; reconnect loop should stop when host unreachable and offer manual reconnect

1 participant