Skip to content

Retry restored SSH after boot-time network failure - #9083

Merged
austinywang merged 43 commits into
mainfrom
issue-9067-ssh-boot-retry
Jul 30, 2026
Merged

austinywang merged 43 commits into
mainfrom
issue-9067-ssh-boot-retry

Conversation

@austinywang

@austinywang austinywang commented Jul 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • move initial foreground authentication into the persistent SSH reconnect state machine
  • apply the same phase ownership to CLI-created and app-restored SSH startup commands
  • preserve the existing retry limit, exponential backoff, signal cancellation, and terminal cleanup paths

Regression coverage

  • execute the restored SSH attach command against a fake SSH that first reports Network is unreachable with exit 255, then succeeds
  • verify authentication is attempted twice, backoff occurs, and PTY attach begins only after authentication succeeds
  • regression test is committed before the fix so the PR commit history demonstrates red/green behavior

Validation

  • git diff --check
  • focused CmuxFoundation SSH package tests: 26 passed across 3 suites, including HUP/INT/TERM during both attach and transient-auth backoff
  • repository guards passed: PBX test wiring, Package.resolved policy, workspace package grouping, and pbxproj normalization/check
  • tagged Debug build succeeded for issue-9067-ssh-boot-retry on commit 68a5a52632
  • focused app-host test target compiled successfully on the prior reviewed head, but Xcode timed out after 367 seconds while preparing the host runner; none of the selected SSH tests executed, so this is recorded as test infrastructure-blocked rather than a test assertion failure
  • earlier app-host attempts were interrupted by a compile watchdog and shared-host ENOSPC; neither produced a source compiler error
  • localization audit: no new user-facing strings; the existing retry text moved unchanged

Closes #9067


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Retries initial SSH foreground authentication inside the reconnect loop and fails closed on unknown errors, so boot-time network outages and server‑alive timeouts retry without hiding prompts. Unifies retry/backoff and signal handling across CLI and restored sessions using CmuxFoundation builders (SSHForegroundAuthenticationRetryPolicy, SSHRetryBackoffScriptBuilder, SSHPTYAttachRetryScriptBuilder) with Sonoma-compatible PTY capture and anchored, bounded cleanup; closes #9067.

  • Bug Fixes

    • Classified stderr under a PTY; mapped retryable transport exits to 254, kept 252 unclassified and non‑retryable, preserved 255 as permanent.
    • Moved foreground auth into the attach/startup retry loop; forwards signals, terminates auth process trees with escalation, and anchors/bounds authentication marker cleanup.
    • Shared interruptible backoff across startup and attach; preserves input and cancels sleeps promptly.
    • Cleared inherited SSH signal state before retry loops; retries server‑alive timeouts and standard OpenSSH transport exits.
  • Tests

    • Expanded coverage for failure classification, server‑alive timeouts, standard transport exits, fail‑closed cases, and bounded diagnostics/waits.
    • Added lifecycle tests for process‑tree termination on direct/restored signals, persistent foreground auth across CLI/restored sessions, and anchored marker cleanup.
    • Verified permission‑denied diagnostics, interruption of backoff by signals, and PTY input preservation during retry.

Written for commit a3573c8. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Bug Fixes

    • Improved SSH foreground authentication with more accurate transient vs. permanent failure detection and consistent retryable exit handling.
    • Refined reconnect-delay and retry-limit behavior, including correct behavior during reauthentication.
    • Updated signal/session cleanup so nested authentication subprocesses are terminated reliably without hanging.
    • Enhanced persistent PTY attach retry sequencing so reauthentication happens only when required.
  • Tests

    • Added/expanded integration and regression coverage for failure classification, process-tree termination, signal interruption lifecycles, and reconnect/backoff retry behavior.

@coderabbitai

coderabbitai Bot commented Jul 28, 2026 •

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Foreground SSH authentication now classifies transport and authentication failures, retries transient failures with bounded limits, and cleans up authentication subprocesses during signals and reconnects. Startup and PTY attach tests cover retry limits, signal exits, failure classification, and retry-loop ordering.

Changes

SSH foreground authentication retries

Layer / File(s) Summary
Authentication failure classification and policy
Packages/macOS/CmuxFoundation/Sources/..., Packages/macOS/CmuxFoundation/Tests/...
Adds SSHForegroundAuthenticationRetryPolicy, which classifies SSH diagnostics and maps exit status 255 to retryable, permanent, or unclassified statuses.
Initial SSH startup retry integration
CLI/CMUXCLI+SSHStartupScripts.swift, CLI/cmux.swift, cmuxTests/...
Applies the retry policy to generated authentication commands and updates startup-loop variables, traps, subprocess cleanup, and status-254 reconnect behavior.
Persistent PTY reauthentication lifecycle
Sources/SSHPTYAttachStartupCommandBuilder.swift, Packages/macOS/CmuxFoundation/Sources/..., Packages/macOS/CmuxFoundation/Tests/...
Generates shared PTY retry-loop logic, classifies foreground authentication failures, enforces retry limits, and handles signals and reconnect backoff.
Retry taxonomy and lifecycle regression coverage
cmuxTests/..., CLI/SSHPTYAttachExitCode.swift, cmux.xcodeproj/project.pbxproj
Updates retry-loop assertions and failure fixtures, exposes a startup helper for integration coverage, documents the shared retry taxonomy, and registers signal tests.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SSHStartupScript
  participant SSHForegroundAuthenticationRetryPolicy
  participant SSHForegroundAuth
  participant SSHPTYAttachRetryScriptBuilder
  SSHStartupScript->>SSHForegroundAuthenticationRetryPolicy: wrap foreground authentication
  SSHStartupScript->>SSHForegroundAuth: launch tracked authentication subprocess
  SSHForegroundAuth-->>SSHForegroundAuthenticationRetryPolicy: return diagnostics and status
  SSHForegroundAuthenticationRetryPolicy-->>SSHPTYAttachRetryScriptBuilder: classify mapped status
  SSHPTYAttachRetryScriptBuilder->>SSHForegroundAuth: retry transient failure
  SSHPTYAttachRetryScriptBuilder-->>SSHStartupScript: exit after limit or permanent failure
Loading

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (4 errors, 1 warning)

Check name Status Explanation Resolution
Cmux Swift Blocking Runtime ❌ Error FAIL: SSHPTYAttachRetryScriptBuilder adds a while :; do loop with sleep/wait, and SSHForegroundAuthenticationRetryPolicy adds a 10-iteration /bin/sleep 0.02 poll. Move retry/backoff and auth-tree cleanup off blocking shell loops into cancellation-aware Swift timers/state transitions or signal-driven callbacks.
Cmux No Hacky Sleeps ❌ Error New production shell cleanup code polls kill -0 and sleeps 0.02 in a loop to wait for auth PIDs, which is a race-masking fixed delay. Replace the grace-period polling with an event-driven/wait-based termination path or a cancellation-aware abstraction that avoids wall-clock sleeps.
Cmux Algorithmic Complexity ❌ Error SSHForegroundAuthenticationRetryPolicy.swift:16-21 does a recursive pgrep -P walk plus string membership scans of the growing PID list, making auth-tree cleanup O(n²) over processes. Replace the PID-string membership check with a one-pass traversal/visited set, or document a hard upper bound and benchmark if the tree is always tiny.
Cmux Full Internationalization ❌ Error New user-facing stderr copy is hardcoded in SSHPTYAttachRetryScriptBuilder and has no matching Resources/Localizable.xcstrings entry. Add a localized xcstrings key for the attach-retry message in every supported locale, then have the builder read it via a localized helper instead of embedding the literal.
Docstring Coverage ⚠️ Warning Docstring coverage is 3.57% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (20 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes address #9067 by retrying initial transient SSH failures inside the reconnect loop and adding coverage for recovery, backoff, and signal handling.
Out of Scope Changes check ✅ Passed The added code and tests stay focused on SSH retry, auth cleanup, and signal handling, with no clear unrelated feature work.
Cmux Swift Actor Isolation ✅ Passed Changed production code adds plain value-type builders; CMUXCLI isn’t @MainActor, and no new mutable Sendable reference types or service protocols appear.
Cmux Browser Automation Off-Main ✅ Passed The patch only changes SSH startup/retry code and tests; no browser.* commands, WebKit/AppKit routing, or socket-worker policy changes appear in the diff.
Cmux Expensive Synchronous Load ✅ Passed PR only changes SSH startup/retry script generation; diff scan found no added synchronous agent-history/session loads or main-actor coupling.
Cmux Cache Substitution Correctness ✅ Passed No persistence/history/undo/snapshot cache substitution appears in the touched Swift/JS/TS code; the patch only adds SSH auth retry and signal handling.
Cmux Swift Concurrency ✅ Passed The only changed Swift code generates shell-script strings; it adds no new Dispatch/Task/Combine/completion-handler async patterns.
Cmux Swift @Concurrent ✅ Passed Touched Swift code is sync shell-script generation/tests; scans found no @concurrent or nonisolated async changes, so the rule isn’t violated.
Cmux Swift Package Boundaries ✅ Passed The reusable retry/auth logic was extracted into CmuxFoundation; the remaining app-target changes are startup-command composition and glue.
Cmux Swiftpm Lockfiles ✅ Passed Diff only adds source/test files and a pbxproj source-file entry; no Package.swift, Package.resolved, .gitignore, or SwiftPM package-reference changes are present.
Cmux Swift Logging ✅ Passed No new Swift logging violations found: touched production files add shell-script strings only, and no new print/debugPrint/dump/NSLog/Logger use appears in runtime Swift.
Cmux User-Facing Error Privacy ✅ Passed The new raw SSH messages are only internal classifier patterns; no user-facing error, alert, or recovery copy was added or changed.
Cmux Swiftui State Layout ✅ Passed No SwiftUI views/layout/state were changed; the diff is confined to SSH auth retry policy code, with no ObservableObject/GeometryReader/lazy-row patterns.
Cmux Architecture Rethink ✅ Passed PASS: Retry/auth behavior is centralized in shared builders/policy, with documented invariants and no duplicate entrypoints or side-channel ownership; bounded sleeps are core backoff.
Cmux Swift Auxiliary Window Close Shortcuts ✅ Passed No diff hunks touched NSWindow/WindowGroup or cmuxAuxiliaryWindowIdentifiers; changes were SSH auth/retry logic and test fixtures only.
Cmux Source Artifacts ✅ Passed PASS: The PR only changes Swift source, tests, and pbxproj/config paths; no temp dirs, logs, caches, build output, or other artifact paths appear in the diff.
Cmux No Test Or Debug Seam In Production Source ✅ Passed No new DEBUG/test-only seams were added in production Sources; the new public builders are used by CLI/startup code, and the diff shows no seam-marker additions.
Cmux No Ambient Global State ✅ Passed The PR adds/extends instance-based builders; no new top-level API, mutable globals, or singleton/state-namespace patterns appear in the production diff.
Title check ✅ Passed The title clearly matches the main change: restoring SSH retries after an initial boot-time network failure.
Description check ✅ Passed The description covers summary and validation well, but it omits the template’s demo video, review trigger, and checklist sections.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch issue-9067-ssh-boot-retry

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@CLI/CMUXCLI`+SSHStartupScripts.swift:
- Around line 360-376: Update the foreground-auth retry flow in the generated
script around cmux_ssh_foreground_auth and cmux_ssh_auth_retry so transient
failures do not increment a shell-managed retry loop or enter the sleep-based
reconnect coordinator. Signal the persistent reconnect owner through the
existing lifecycle/network signaling mechanism, preserving terminal handling for
non-retryable failures and successful authentication.

In
`@Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHForegroundAuthenticationRetryPolicy.swift`:
- Around line 24-54: Pin the locale for the wrapped SSH/auth invocation so the
diagnostic text classified by transientFailurePattern and
permanentFailurePattern is deterministic. Update the nestedCommand construction
and execution path around nestedCommand and the classifier invocation to run the
wrapped command with LC_ALL=C (and the appropriate locale environment), while
preserving the existing regex classification behavior.
- Around line 57-72: The classifyingTransientFailure API currently allows
trailing commands to affect classification; constrain it to wrap only the
authentication command. Update its contract and implementation or, preferably,
ensure buildReusableForegroundAuthThenSSHPTYAttachStartupCommand terminates
successfully immediately after ssh -T ... true before appending
localCommandScript, preserving the local command’s independent exit status.
- Line 122: Update the SSH foreground authentication retry command around the
`/usr/bin/script` invocation to avoid the unsupported macOS `-F`/pipe flags. Use
only documented Apple `script(1)` behavior while still streaming classifier
output through `cmux_ssh_auth_classifier_fifo` and preserving the child
command’s exit status.

In
`@Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHForegroundAuthenticationRetryPolicyTests.swift`:
- Around line 8-118: Add coverage for the non-255 exit-status contract in the
SSHForegroundAuthenticationRetryPolicy tests by adding a test near the existing
status-mapping cases that runs exit 3 and asserts the wrapper preserves status 3
and creates no temporary files.

In `@Sources/SSHPTYAttachStartupCommandBuilder.swift`:
- Around line 95-96: Update the startup command construction around
cmux_ssh_attach_foreground_auth and cmux_ssh_attach_auth_pid to use the same
pending-signal handling as the CLI startup loop: record HUP/INT/TERM received
during authentication launch, assign the spawned process PID immediately, then
terminate that recorded PID before continuing when a pending signal exists,
preventing orphaned interactive authentication.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 2c7038f9-0502-4077-80bb-bdabde5373fb

📥 Commits

Reviewing files that changed from the base of the PR and between 64ec2bb and d7a7e60.

📒 Files selected for processing (12)
  • CLI/CMUXCLI+SSHStartupScripts.swift
  • CLI/cmux.swift
  • Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHForegroundAuthenticationRetryPolicy.swift
  • Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHForegroundAuthenticationRetryPolicyTests.swift
  • Sources/SSHPTYAttachStartupCommandBuilder.swift
  • cmux.xcodeproj/project.pbxproj
  • cmuxTests/CLINotifyProcessIntegrationRegressionTests.swift
  • cmuxTests/SSHDeepSleepReattachTests.swift
  • cmuxTests/SSHForegroundAuthenticationMarkerCleanupTests.swift
  • cmuxTests/SSHForegroundAuthenticationSignalTests.swift
  • cmuxTests/SSHPersistentPTYRetryLifecycleTests.swift
  • cmuxTests/SSHStartupSignalLifecycleTests.swift

Comment thread CLI/CMUXCLI+SSHStartupScripts.swift Outdated
Comment thread Sources/SSHPTYAttachStartupCommandBuilder.swift Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHForegroundAuthenticationRetryPolicyTests.swift (1)

151-159: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Replace the fixed three-second synchronization delay.

/bin/sleep 3 makes every run slower and can let the child exit before a loaded CI worker observes incremental classification. Block the fixture on test-controlled stdin after writing producer-ready, then release it through a Pipe only after observing transient\n.

As per coding guidelines, tests must use completion signals or deadline-bounded predicate polls rather than fixed-duration waits.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHForegroundAuthenticationRetryPolicyTests.swift`
around lines 151 - 159, Replace the fixed /bin/sleep 3 synchronization in
SSHForegroundAuthenticationRetryPolicyTests with test-controlled stdin blocking
after the fixture writes producer-ready. Use a Pipe to release the child only
after observing transient\n, and coordinate readiness/release with completion
signals or deadline-bounded predicate polling rather than fixed-duration waits.

Source: Coding guidelines

Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHForegroundAuthenticationRetryPolicy.swift (1)

130-131: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Close the launch-to-PID-registration signal race.

A signal between script ... & and cmux_ssh_auth_command_pid=$! makes the trap see no auth PID, so it exits without terminating the launched authentication process. Mirror the pending-signal/launching state handling used by Sources/SSHPTYAttachStartupCommandBuilder.swift and process any deferred signal immediately after recording $!.

As per path instructions, cancellation and cleanup ownership must derive from tracked lifecycle state without staleness windows.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHForegroundAuthenticationRetryPolicy.swift`
around lines 130 - 131, Update the launch/trap handling in
SSHForegroundAuthenticationRetryPolicy so signals arriving between starting the
authentication command and recording its PID are deferred rather than handled
with a missing PID. Mirror the pending-signal and launching-state lifecycle used
by SSHPTYAttachStartupCommandBuilder, record the launched process identifier
immediately, then process any deferred signal. Ensure cancellation and cleanup
derive from the tracked lifecycle state without stale ownership windows.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHPTYAttachRetryScriptBuilder.swift`:
- Line 60: Replace the inline blocking sleep in SSHPTYAttachRetryScriptBuilder’s
generated reconnect-delay logic with a lifecycle-aware, signal-interruptible
backoff mechanism so HUP, INT, and TERM terminate the wait immediately. Preserve
the configured delay and retry behavior for normal execution, including
transient authentication failures, and add a regression test proving signals
received during backoff trigger prompt cleanup.

---

Outside diff comments:
In
`@Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHForegroundAuthenticationRetryPolicy.swift`:
- Around line 130-131: Update the launch/trap handling in
SSHForegroundAuthenticationRetryPolicy so signals arriving between starting the
authentication command and recording its PID are deferred rather than handled
with a missing PID. Mirror the pending-signal and launching-state lifecycle used
by SSHPTYAttachStartupCommandBuilder, record the launched process identifier
immediately, then process any deferred signal. Ensure cancellation and cleanup
derive from the tracked lifecycle state without stale ownership windows.

In
`@Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHForegroundAuthenticationRetryPolicyTests.swift`:
- Around line 151-159: Replace the fixed /bin/sleep 3 synchronization in
SSHForegroundAuthenticationRetryPolicyTests with test-controlled stdin blocking
after the fixture writes producer-ready. Use a Pipe to release the child only
after observing transient\n, and coordinate readiness/release with completion
signals or deadline-bounded predicate polling rather than fixed-duration waits.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 14d26a2a-6a51-44aa-a896-c7310b05eea6

📥 Commits

Reviewing files that changed from the base of the PR and between d7a7e60 and 5cd8f4e.

📒 Files selected for processing (7)
  • CLI/SSHPTYAttachExitCode.swift
  • CLI/cmux.swift
  • Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHForegroundAuthenticationRetryPolicy.swift
  • Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHPTYAttachRetryScriptBuilder.swift
  • Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHForegroundAuthenticationRetryPolicyTests.swift
  • Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHPTYAttachRetryScriptBuilderTests.swift
  • Sources/SSHPTYAttachStartupCommandBuilder.swift

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHPTYAttachRetryScriptBuilderTests.swift`:
- Line 143: Remove the fixed Thread.sleep(forTimeInterval:) synchronization wait
from the SSH PTY attach retry test after confirming the marker exists. Send the
signal immediately after the existing real completion check, preserving the
test’s marker-based synchronization and retry assertions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 08280637-7acb-42b7-bcef-76eb2492c06b

📥 Commits

Reviewing files that changed from the base of the PR and between 5cd8f4e and 68a5a52.

📒 Files selected for processing (2)
  • Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHPTYAttachRetryScriptBuilder.swift
  • Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHPTYAttachRetryScriptBuilderTests.swift

@cursor

cursor Bot commented Jul 29, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
CLI/CMUXCLI+SSHStartupScripts.swift (1)

361-378: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Extract the auth-retry classification case-pattern into a shared policy helper.

Line 378's case "$cmux_ssh_status:$cmux_ssh_auth_established" in 254:*|\(authRetryPolicy.unclassifiedFailureExitStatus):1) ... *) ... duplicates, near-verbatim, the classification case-pattern already generated by SSHPTYAttachRetryScriptBuilder.lines(command:reauthenticates:) (254:*|\(authPolicy.unclassifiedFailureExitStatus):1) ... *) ...). The only differences are the variable-name prefix (cmux_ssh_ vs cmux_ssh_attach_) and the terminal action (break vs exit 255). This is the exact retry/classification state machine the PR is trying to make robust (issue #9067); if one copy is updated (e.g., a new status code, an off-by-one in the retry-limit check) and the other isn't, initial-connection and reattachment retry behavior will silently diverge.

Consider adding a small parameterized method on SSHForegroundAuthenticationRetryPolicy (e.g. taking the status/established/retry variable names and the "exceeded" action as a shell fragment) that both this file and SSHPTYAttachRetryScriptBuilder call, so the classification logic has one source of truth.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CLI/CMUXCLI`+SSHStartupScripts.swift around lines 361 - 378, Extract the
duplicated authentication retry classification state machine from the
reauthentication loop and
SSHPTYAttachRetryScriptBuilder.lines(command:reauthenticates:) into a
parameterized helper on SSHForegroundAuthenticationRetryPolicy. Have both
callers supply their variable prefixes and terminal exceeded action, while
preserving their existing retry-limit and status-handling behavior. Replace both
inline case-pattern implementations with this shared helper output.
CLI/cmux.swift (1)

10252-10328: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Eliminate the zsh-only lock implementation or keep the zsh interpreter wrapper.

zmodload zsh/system and zsystem flock only run when the foreground-auth prefix is executed by zsh; the code removes any explicit zsh invocation around this classified oneTimeCommand, so the lock path can fail early and drop the single-in-flight-auth invariant. Keep these commands behind a guard or restore the zsh wrapper.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CLI/cmux.swift` around lines 10252 - 10328, Ensure the locking commands in
buildReusableForegroundAuthThenSSHPTYAttachStartupCommand are executed under zsh
by restoring the explicit zsh interpreter wrapper for the classified
oneTimeCommand, or guard the zmodload/zsystem flock path to run only when zsh is
active. Preserve the single-in-flight-auth behavior and existing retry/status
handling.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmuxTests/SSHForegroundAuthenticationSignalTests.swift`:
- Around line 47-56: In the test flow surrounding childPIDFile and the result
assertions, validate result.timedOut and result.status before reading or
unwrapping the child PID file so timeout and early-exit diagnostics, including
stderr, are reported first. Keep the existing child-process cleanup via
Darwin.kill after the PID is successfully unwrapped.

In
`@Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHPTYAttachRetryScriptBuilderTests.swift`:
- Around line 258-273: Update the waitForFile helper to perform one final
file-content check after the process-running poll loop exits, before returning
false. Preserve the deadline-bounded polling behavior and return true when the
final read contains expectedContents, including when the process exits
immediately after writing the content.

---

Outside diff comments:
In `@CLI/cmux.swift`:
- Around line 10252-10328: Ensure the locking commands in
buildReusableForegroundAuthThenSSHPTYAttachStartupCommand are executed under zsh
by restoring the explicit zsh interpreter wrapper for the classified
oneTimeCommand, or guard the zmodload/zsystem flock path to run only when zsh is
active. Preserve the single-in-flight-auth behavior and existing retry/status
handling.

In `@CLI/CMUXCLI`+SSHStartupScripts.swift:
- Around line 361-378: Extract the duplicated authentication retry
classification state machine from the reauthentication loop and
SSHPTYAttachRetryScriptBuilder.lines(command:reauthenticates:) into a
parameterized helper on SSHForegroundAuthenticationRetryPolicy. Have both
callers supply their variable prefixes and terminal exceeded action, while
preserving their existing retry-limit and status-handling behavior. Replace both
inline case-pattern implementations with this shared helper output.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 945b6ce3-5b43-4e71-9563-541a26e4a0a0

📥 Commits

Reviewing files that changed from the base of the PR and between 68a5a52 and cba7f6c.

📒 Files selected for processing (9)
  • CLI/CMUXCLI+SSHStartupScripts.swift
  • CLI/cmux.swift
  • Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHForegroundAuthenticationRetryPolicy.swift
  • Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHPTYAttachRetryScriptBuilder.swift
  • Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHForegroundAuthenticationRetryPolicyTests.swift
  • Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHPTYAttachRetryScriptBuilderTests.swift
  • Sources/SSHPTYAttachStartupCommandBuilder.swift
  • cmuxTests/SSHForegroundAuthenticationMarkerCleanupTests.swift
  • cmuxTests/SSHForegroundAuthenticationSignalTests.swift

Comment on lines +47 to +56
let childPID = try XCTUnwrap(Int32(
String(contentsOf: childPIDFile, encoding: .utf8)
.trimmingCharacters(in: .whitespacesAndNewlines)
))
defer {
Darwin.kill(childPID, SIGKILL)
}

XCTAssertFalse(result.timedOut, result.stderr)
XCTAssertEqual(result.status, 130, result.stderr)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Assert the process result before unwrapping the child PID file.

If the wrapper times out or exits early, the child PID file is never written and XCTUnwrap at Line 47 fails first, hiding the result.timedOut/result.status diagnostics (including stderr) that explain why.

🧹 Proposed reorder
-        let childPID = try XCTUnwrap(Int32(
-            String(contentsOf: childPIDFile, encoding: .utf8)
-                .trimmingCharacters(in: .whitespacesAndNewlines)
-        ))
-        defer {
-            Darwin.kill(childPID, SIGKILL)
-        }
-
         XCTAssertFalse(result.timedOut, result.stderr)
         XCTAssertEqual(result.status, 130, result.stderr)
+        let childPID = try XCTUnwrap(Int32(
+            String(contentsOf: childPIDFile, encoding: .utf8)
+                .trimmingCharacters(in: .whitespacesAndNewlines)
+        ))
+        defer {
+            Darwin.kill(childPID, SIGKILL)
+        }
         XCTAssertTrue(
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
let childPID = try XCTUnwrap(Int32(
String(contentsOf: childPIDFile, encoding: .utf8)
.trimmingCharacters(in: .whitespacesAndNewlines)
))
defer {
Darwin.kill(childPID, SIGKILL)
}
XCTAssertFalse(result.timedOut, result.stderr)
XCTAssertEqual(result.status, 130, result.stderr)
XCTAssertFalse(result.timedOut, result.stderr)
XCTAssertEqual(result.status, 130, result.stderr)
let childPID = try XCTUnwrap(Int32(
String(contentsOf: childPIDFile, encoding: .utf8)
.trimmingCharacters(in: .whitespacesAndNewlines)
))
defer {
Darwin.kill(childPID, SIGKILL)
}
XCTAssertTrue(
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmuxTests/SSHForegroundAuthenticationSignalTests.swift` around lines 47 - 56,
In the test flow surrounding childPIDFile and the result assertions, validate
result.timedOut and result.status before reading or unwrapping the child PID
file so timeout and early-exit diagnostics, including stderr, are reported
first. Keep the existing child-process cleanup via Darwin.kill after the PID is
successfully unwrapped.

Comment on lines +258 to +273
private func waitForFile(
at url: URL,
containing expectedContents: String,
while process: Process,
timeout: TimeInterval
) -> Bool {
let deadline = Date().addingTimeInterval(timeout)
while process.isRunning, Date() < deadline {
let contents = (try? String(contentsOf: url, encoding: .utf8)) ?? ""
if contents.contains(expectedContents) {
return true
}
Thread.sleep(forTimeInterval: 0.01)
}
return false
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Re-check file contents after the poll loop exits.

while process.isRunning makes the helper return false as soon as the child exits, even when the expected content was already written. That is a live race for the queued-input assertion at Lines 239-245: cmux_test_attach writes input:queued-input and then returns 0, so the shell exits immediately afterward and the poll can observe isRunning == false before it ever reads the final contents. The sibling helper in cmuxTests/SSHStartupSignalLifecycleTests.swift performs a final read for this reason.

🧹 Proposed fix
         let deadline = Date().addingTimeInterval(timeout)
         while process.isRunning, Date() < deadline {
             let contents = (try? String(contentsOf: url, encoding: .utf8)) ?? ""
             if contents.contains(expectedContents) {
                 return true
             }
             Thread.sleep(forTimeInterval: 0.01)
         }
-        return false
+        let contents = (try? String(contentsOf: url, encoding: .utf8)) ?? ""
+        return contents.contains(expectedContents)
     }

As per path instructions, tests must use "deadline-bounded polls of real predicates rather than fixed-duration waits before assertions."

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
private func waitForFile(
at url: URL,
containing expectedContents: String,
while process: Process,
timeout: TimeInterval
) -> Bool {
let deadline = Date().addingTimeInterval(timeout)
while process.isRunning, Date() < deadline {
let contents = (try? String(contentsOf: url, encoding: .utf8)) ?? ""
if contents.contains(expectedContents) {
return true
}
Thread.sleep(forTimeInterval: 0.01)
}
return false
}
private func waitForFile(
at url: URL,
containing expectedContents: String,
while process: Process,
timeout: TimeInterval
) -> Bool {
let deadline = Date().addingTimeInterval(timeout)
while process.isRunning, Date() < deadline {
let contents = (try? String(contentsOf: url, encoding: .utf8)) ?? ""
if contents.contains(expectedContents) {
return true
}
Thread.sleep(forTimeInterval: 0.01)
}
let contents = (try? String(contentsOf: url, encoding: .utf8)) ?? ""
return contents.contains(expectedContents)
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHPTYAttachRetryScriptBuilderTests.swift`
around lines 258 - 273, Update the waitForFile helper to perform one final
file-content check after the process-running poll loop exits, before returning
false. Preserve the deadline-bounded polling behavior and return true when the
final read contains expectedContents, including when the process exits
immediately after writing the content.

Source: Path instructions

@cursor

cursor Bot commented Jul 29, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHForegroundAuthenticationRetryPolicy.swift`:
- Around line 89-110: Replace the timer-driven poll-and-sleep and repeated PID
rescans in the foreground-auth cancellation path with supervision through a
single owner/process group and explicit child reaping. Update the surrounding
SSHForegroundAuthenticationRetryPolicy teardown flow so cancellation delegates
lifecycle management to that owner, removes the fixed grace loop and escalation
coordination, and preserves reliable cleanup without timing-based
synchronization.

In
`@Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHForegroundAuthenticationRetryPolicyTests.swift`:
- Around line 186-201: Bound all process completion in
SSHForegroundAuthenticationRetryPolicyTests: at lines 186-201 replace
waitUntilExit() with a deadline-bounded helper before reading the child PID,
capture or asynchronously drain stderr and report diagnostics on timeout, and
terminate lingering children during cleanup; at lines 282-283 apply the same
bounded completion to the diagnostic-state test; at lines 311-313 update run(_:)
to avoid unbounded pipe-to-EOF reads and process waits while preserving real
completion or deadline-bounded predicate polling.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 0981831e-6107-4a35-8bb5-fe94d0a5b938

📥 Commits

Reviewing files that changed from the base of the PR and between cba7f6c and ef5be87.

📒 Files selected for processing (3)
  • Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHForegroundAuthenticationRetryPolicy.swift
  • Packages/macOS/CmuxFoundation/Tests/CmuxFoundationTests/SSHForegroundAuthenticationRetryPolicyTests.swift
  • cmuxTests/SSHForegroundAuthenticationSignalTests.swift

Comment on lines +89 to +110
cmux_ssh_auth_tree_grace_attempt=0
while [ "$cmux_ssh_auth_tree_grace_attempt" -lt 10 ]; do
cmux_ssh_auth_tree_has_survivor=0
for cmux_ssh_auth_tree_pid in $cmux_ssh_auth_tree_initial_pids; do
if /bin/kill -0 "$cmux_ssh_auth_tree_pid" >/dev/null 2>&1; then
cmux_ssh_auth_tree_has_survivor=1
break
fi
done
if [ "$cmux_ssh_auth_tree_has_survivor" -eq 0 ]; then exit 0; fi
/bin/sleep 0.02
cmux_ssh_auth_tree_grace_attempt=$((cmux_ssh_auth_tree_grace_attempt + 1))
done

cmux_ssh_auth_tree_pids=
for cmux_ssh_auth_tree_pid in $cmux_ssh_auth_tree_initial_pids; do
if /bin/kill -0 "$cmux_ssh_auth_tree_pid" >/dev/null 2>&1; then
cmux_ssh_collect_auth_process_tree "$cmux_ssh_auth_tree_pid"
fi
done
for cmux_ssh_auth_tree_pid in $cmux_ssh_auth_tree_pids; do
/bin/kill -KILL "$cmux_ssh_auth_tree_pid" >/dev/null 2>&1 || true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Remove timer-driven cleanup escalation from the signal path.

This adds a fixed 200 ms poll-and-sleep wait to every foreground-auth cancellation before escalation. Model the authentication subprocess under one supervised owner/process group with explicit reaping instead of using timed rescans to coordinate teardown.

As per coding guidelines, do not introduce timing, blocking, or polling repair paths to paper over lifecycle or shared-state races.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@Packages/macOS/CmuxFoundation/Sources/CmuxFoundation/SSHForegroundAuthenticationRetryPolicy.swift`
around lines 89 - 110, Replace the timer-driven poll-and-sleep and repeated PID
rescans in the foreground-auth cancellation path with supervision through a
single owner/process group and explicit child reaping. Update the surrounding
SSHForegroundAuthenticationRetryPolicy teardown flow so cancellation delegates
lifecycle management to that owner, removes the fixed grace loop and escalation
coordination, and preserves reliable cleanup without timing-based
synchronization.

Source: Coding guidelines

@cursor

cursor Bot commented Jul 29, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@cursor

cursor Bot commented Jul 29, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@cursor

cursor Bot commented Jul 30, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SSH workspace never retries after boot-time "Network is unreachable" failure

1 participant