Skip to content

Stop reporting recovered remote reconnects as errors - #9664

Closed
austinywang wants to merge 3 commits into
mainfrom
feat-transient-remote-reconnect-noise
Closed

austinywang wants to merge 3 commits into
mainfrom
feat-transient-remote-reconnect-noise

Conversation

@austinywang

@austinywang austinywang commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Restarting cmux while a remote workspace is attached reported two failures for an outage that had already healed.

  • Terminal: cmux ssh-pty-attach exits with a retryable status when its bridge closes, and the CLI's top-level handler printed Error: ssh-pty-attach: bridge closed before remote PTY exit could be confirmed: remote daemon is not ready even though the persistent attach wrapper immediately retried and printed its own yellow "reattaching (attempt n/∞)" notice. The CLI now leaves reporting to the wrapper whenever the wrapper will retry the failure itself. Failures the wrapper will not retry, and every non-attach command, still print normally.
  • Sidebar: relay bounces publish daemon .error with details like Remote SSH relay unavailable; retrying in 2 seconds and always schedule a reconnect, but retraction-on-recovery only matched the "proxy-only" substrings, so those entries stayed red forever on a workspace that was Connected and working. Retraction now covers every daemon error once the daemon reports .ready — a ready daemon has by definition resolved the errors recorded before it. Outages the daemon never recovers from still persist, because retraction only runs on .ready.

Testing

  • cmuxTests/WorkspaceRemoteDaemonRecoveryTests.swift: a recovered relay bounce retracts its sidebar error (fails before this change); an outage that never reaches .ready keeps its entry. The pre-existing Recovered daemon-transport bounce leaves a permanent error on the workspace row #8917 proxy-bounce test still passes.
  • Live A/B on a real host (cmux-mac-mini), identical protocol on both builds — one fresh remote session, one app restart, no other cmux instance running:
    • Before (build without this change): sidebar keeps [remote-daemon] [error] Remote daemon error (cmux-mac-mini): Remote SSH relay unavailable; retrying in 2 seconds.
    • After (this branch): No log entries, daemon ready, workspace connected.
    • The duplicate red Error: line no longer appears in the terminal on a bridge close the wrapper retries.

Known gap (pre-existing, not from this change)

Reattaching a remote PTY after an app restart is itself unreliable, and reproduces on a build without this change: the control build ended the restored session with ^D, and the fixed build reported ssh-pty-attach: remote PTY attach failed. That is a separate defect in the reattach path; this PR only stops a recovered reconnect from being reported as a failure. Filed separately.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Stops showing errors for remote reconnects that recover on their own. The CLI suppresses duplicate errors for retryable ssh-pty-attach disconnects, and the sidebar clears resolved daemon errors once the daemon is ready and a remote terminal is active.

  • Bug Fixes
    • CLI: Skip the top-level error line when the attach wrapper will retry; other failures still print.
    • Sidebar: On .ready with an active remote terminal, retract all prior remote-daemon errors (including relay bounces); outages that never reach ready or lack a remote terminal keep their entries.

Written for commit d3ebc23. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Bug Fixes
    • Prevented duplicate error messages when SSH PTY attachment failures have already been reported.
    • Improved remote daemon recovery handling by removing resolved error indicators once the daemon is ready and active remote sessions are restored.
    • Preserved error reporting for unresolved connection, startup, and non-retryable failures.
    • Added recovery validation to ensure relay errors appear during outages, remain visible when recovery is incomplete, and clear after successful recovery.

Quitting or restarting cmux while a remote PTY is attached surfaced two
failures for an outage that healed itself seconds later: the terminal
printed a red "Error: ssh-pty-attach: bridge closed before remote PTY
exit could be confirmed: remote daemon is not ready", and the sidebar
kept a red "Remote daemon error" row on a workspace that was connected.

The CLI now leaves reporting to the persistent attach wrapper whenever
that wrapper will retry the failure itself; the wrapper already prints
its own reattaching notice, so the error line was duplicate and made a
successful reconnect look broken. Failures the wrapper will not retry
still print.

The sidebar now retracts every daemon error once the daemon reports
ready, instead of only the proxy-only subset. Relay bounces publish
details that say "retrying in 2 seconds" and always schedule a
reconnect, but did not match the proxy-only substrings, so they stayed
red forever. Errors the daemon never recovers from still persist,
because retraction only runs on ready.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 5, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The CLI suppresses duplicate errors from the persistent SSH PTY attach wrapper. Remote workspace readiness clears resolved remote-daemon errors when active remote terminals exist. Tests cover successful recovery and retained errors.

Changes

Remote SSH recovery

Layer / File(s) Summary
Suppress wrapper-reported attach failures
CLI/CMUXCLI+SSHPTYAttachBridge.swift, CLI/cmux.swift
The CLI identifies retryable SSH PTY attach failures handled by the wrapper and omits duplicate standard error output. Other errors retain existing reporting.
Clear resolved daemon errors
Sources/Workspace.swift, cmuxTests/WorkspaceRemoteDaemonRecoveryTests.swift
Workspace readiness clears remote-daemon error entries only when an active remote terminal exists. Tests cover cleared errors, incomplete recovery, and workspaces without remote terminals.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

Possibly related PRs


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name Status Explanation Resolution
Cmux Swift Package Boundaries ❌ Error Workspace.swift adds pure remote-daemon SidebarLogEntry filtering in the app target; CmuxSidebar already owns SidebarLogEntry and WorkspaceSidebarMetadataModel. Move source/level removal to CmuxSidebar, for example WorkspaceSidebarMetadataModel.clearLogEntries(source:level:). Keep only the active-session guard and ready-event composition in Workspace.
Docstring Coverage ⚠️ Warning Docstring coverage is 77.78% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (23 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Cmux Swift Actor Isolation ✅ Passed The new Workspace method is inside @MainActor final class Workspace; CLI additions are synchronous value-type code, and changed tests are @MainActor. No new isolation or Sendable violation is present.
Cmux Swift Blocking Runtime ✅ Passed The production diff adds only environment/error checks and synchronous log filtering; no new semaphore, wait, sleep, timer, polling, sync, or lock primitive appears. Test scaffolding is deterministic.
Cmux Browser Automation Off-Main ✅ Passed The PR diff from base to HEAD changes only cmuxTests/WorkspaceRemoteDaemonRecoveryTests.swift; it adds no browser socket, WebKit, AppKit, or automation routing changes.
Cmux Expensive Synchronous Load ✅ Passed The production diff adds only environment parsing and in-memory sidebar cleanup; it adds or moves no agent-history, transcript, JSON/JSONL, directory, or per-record synchronous load.
Cmux Cache Substitution Correctness ✅ Passed The diff does not replace a fresh persistence, history, undo, or snapshot read; it adds retry checks and removes in-memory sidebar log entries on .ready using an event-maintained session count.
Cmux No Hacky Sleeps ✅ Passed The complete PR diff changes only Swift production files and a Swift test file; it introduces no covered TypeScript, JavaScript, shell, or build/runtime-script sleeps.
Cmux Algorithmic Complexity ✅ Passed The new CLI logic is O(1); daemon cleanup performs one linear scan of bounded sidebar logs (max 500 entries) with no nested or per-target rescans.
Cmux Swift Concurrency ✅ Passed The PR adds synchronous error filtering and test setup only; the diff introduces no new Dispatch queues, Tasks, Combine state, completion-handler APIs, or fire-and-forget async work.
Cmux Swift @Concurrent ✅ Passed The PR adds only synchronous helpers and call-site logic; no changed nonisolated async or @concurrent declarations. Workspace cleanup remains @MainActor-isolated, which is allowed.
Cmux Swiftpm Lockfiles ✅ Passed The PR range changes only CLI, Sources/Workspace.swift, and recovery tests; it changes no Package.swift, Package.resolved, .gitignore, Xcode project, workflow, or dependency metadata.
Cmux Swift Logging ✅ Passed The PR adds no print, debugPrint, dump, NSLog, ad hoc diagnostics, Logger constants, or sensitive logging; it only suppresses CLI stderr and removes existing sidebar entries, with assertions in tests.
Cmux User-Facing Error Privacy ✅ Passed Production additions suppress duplicate output and clear existing daemon logs; they add no prohibited vendor, credential, token, header, or raw payload text.
Cmux Full Internationalization ✅ Passed Production changes add control flow and comments only; existing CLI/sidebar text is not introduced or worsened, and new strings are test fixtures or developer comments. No catalog/web locale update...
Cmux Swiftui State Layout ✅ Passed The Swift diff adds CLI and Workspace model logic only; it adds no SwiftUI state, layout readers, lazy/list store references, or render-time state writes.
Cmux Architecture Rethink ✅ Passed The diff adds no timing, locking, polling, observer, or side-channel repair. Workspace owns daemon-log cleanup, and comments define the ready-state invariant.
Cmux Swift Auxiliary Window Close Shortcuts ✅ Passed PASS: The PR changes CLI, Workspace, and test logic only; it adds or changes no user-visible NSWindow, NSPanel, controller, SwiftUI Window, or WindowGroup.
Cmux Source Artifacts ✅ Passed The PR changes only four existing Swift source/test files; no artifact directories or generated binary/log paths appear in the diff.
Cmux No Test Or Debug Seam In Production Source ✅ Passed Sources/Workspace.swift adds a product cleanup method called by applyRemoteDaemonStatusUpdate; no test/debug guard, test-named seam, visibility widening, or test-only accessor was added.
Cmux No Ambient Global State ✅ Passed The production diff adds instance methods on CMUXCLI and @MainActor Workspace only; it adds no top-level functions, mutable globals, static-only namespaces, or new singletons.
Title check ✅ Passed The title clearly describes the main change: recovered remote reconnects no longer appear as errors.
Description check ✅ Passed The description clearly covers the change, rationale, testing, behavior verification, and known scope gap, but omits template sections for the demo video, review trigger, and checklist.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat-transient-remote-reconnect-noise

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

active_terminal_sessions=0 on a workspace that still reports Connected
is the signature of a remote terminal that fell back to a local shell:
the sidebar row, the tab title, and the connection state all still name
the remote host while the prompt is local. Retracting daemon errors
there would erase the only visible sign the workspace is not remote, so
retraction now requires a live remote terminal session as well as a
ready daemon.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Sources/Workspace.swift`:
- Around line 2760-2768: Update clearResolvedRemoteDaemonSidebarArtifacts so
daemon error entries are removed only when an authoritative remote terminal
session is in the .connected state, not merely when
hasActiveRemoteTerminalSessions is true; alternatively, invoke the cleanup again
when RemoteTerminalLiveness records the later .connected transition.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 3d5c2510-359f-440f-8cbc-2ffd44bdf8ed

📥 Commits

Reviewing files that changed from the base of the PR and between 0716db1 and bc5d53a.

📒 Files selected for processing (2)
  • Sources/Workspace.swift
  • cmuxTests/WorkspaceRemoteDaemonRecoveryTests.swift

Comment thread Sources/Workspace.swift
Comment on lines +2760 to +2768
// A ready daemon only proves the management lane recovered. When the
// workspace has no remote terminal session, its terminal has fallen
// back to a local shell while the row still reads "Connected", and
// retracting here would erase the only sign that anything went wrong.
// Keep the errors visible until a remote terminal is actually alive.
guard hasActiveRemoteTerminalSessions else { return }
logEntries.removeAll { entry in
entry.source == "remote-daemon" && entry.level == .error
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

rg -n -C 8 --glob '*.swift' \
  'hasActiveRemoteTerminalSessions|activeRemoteTerminalSessionCount|activeRemoteTerminalSurfaceIds|trackRemoteTerminalSurface|markRemoteTerminalSessionLaunching|untrackRemoteTerminalSurface|applyRemoteDaemonStatusUpdate|clearResolvedRemoteDaemonSidebarArtifacts|clearProxyOnlyRemoteSidebarArtifacts' \
  Sources cmuxTests

Repository: manaflow-ai/cmux

Length of output: 50373


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

# Inspect focused declarations, lifecycle transitions, and cleanup call sites.
sed -n '1,220p' Sources/Workspace+RemoteTerminalLiveness.swift
printf '\n--- Workspace line ranges near declarations and helper ---\n'
sed -n '2650,2775p;5640,5652p;6050,6185p' Sources/Workspace.swift
printf '\n--- focused call sites ---\n'
rg -n -C 6 --glob '*.swift' \
  'clearResolvedRemoteDaemonSidebarArtifacts\(|clearProxyOnlyRemoteDaemonSidebarArtifacts\(|clearProxyOnlyRemoteSidebarArtifacts\(|markRemoteTerminalSessionLaunching\(|markRemoteTerminalSessionConnected\(|trackRemoteTerminalSurface\(|activeRemoteTerminalSessionCount =|hasActiveRemoteTerminalSessions' \
  Sources

Repository: manaflow-ai/cmux

Length of output: 50372


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

cat -n Sources/Workspace.swift | sed -n '6967,7005p'
printf '\n--- all Workspace calls to daemon/sidebar cleanup helpers ---\n'
rg -n -C 4 --glob '*.swift' \
  'clearResolvedRemoteDaemonSidebarArtifacts\(|clearProxyOnlyRemoteDaemonSidebarArtifacts\(\)|clearProxyOnlyRemoteSidebarArtifacts\(\)' Sources | sed -n '1,260p'
printf '\n--- active session count and surface phase update context ---\n'
cat -n Sources/Workspace.swift | sed -n '6660,6820p;6470,6535p'
printf '\n--- remote daemon status transition call sites ---\n'
rg -n -C 3 --glob '*.swift' 'remoteDaemonStatus \(.\)|WorkspaceRemoteDaemonStatus|remoteDaemonStatusUpdate|applyRemoteDaemonStatusUpdate' Sources

Repository: manaflow-ai/cmux

Length of output: 25510


Gate resolved daemon-error cleanup on a connected remote terminal, not just an active session count.

trackRemoteTerminalSurface adds the panel to activeRemoteTerminalSurfaceIds and increments activeRemoteTerminalSessionCount before markRemoteTerminalSessionLaunching records .launching. If remoteDaemonStatus changes to .ready before RemoteTerminalLiveness records .connected, clearResolvedRemoteDaemonSidebarArtifacts() removes daemon .error entries while no remote PTY is live. Use the authoritative .connected session state for this cleanup, or reapply cleanup on the later connect transition.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Sources/Workspace.swift` around lines 2760 - 2768, Update
clearResolvedRemoteDaemonSidebarArtifacts so daemon error entries are removed
only when an authoritative remote terminal session is in the .connected state,
not merely when hasActiveRemoteTerminalSessions is true; alternatively, invoke
the cleanup again when RemoteTerminalLiveness records the later .connected
transition.

Sources: Path instructions, MCP tools

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale-revisit Closed after 30+ days without activity; preserved for possible revisit or reopening.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants