Skip to content

Cloud legacy link: bound and log every start step, show the failing step with Retry - #33567

Merged
teamleaderleo merged 7 commits into
feat-cmux-nextfrom
leo/cloud-legacy-link-hang
Nov 3, 2026
Merged

teamleaderleo merged 7 commits into
feat-cmux-nextfrom
leo/cloud-legacy-link-hang

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Cloud start on the legacy link (what production builds use) could wait 3 minutes and then fail with "didn't connect" and no reason. In the report: "Nightly has no wg hub or cmux-tui remote connect process for any Cloud machine, and it logs no link error ... It took 182.8 s, which matches the 6×30 s connect wait."

Root cause. Every legacy link joins one shared tunnel-hub start (POST /api/vm/tunnel enroll, then spawn the hub). That start had no overall bound. The enroll's token read sits outside the API client's own deadline, and withDeadline waits for cancelled children. So if one start stalled, every link waiting on it stalled too, with nothing spawned and nothing logged. That matches the report: no hub and no remote connect for any machine, and no error.

Fix

  • Each legacy link step has its own deadline and log lines: attach-endpoint 45 s, hub 120 s (enroll 75 s), approve 75 s, remote connect 75 s. The new CloudLinkStart.swift uses SandboxDeadline.race, which answers at the bound even when the work ignores cancellation. Logs (cloud.link, notice level) cover start, step began/done, pid spawned, link up, superseded, and failed at after N s: .
  • A failure records its step and reason on the session (legacyLinkFailure). A definitive one (signed out, plan required, 4xx, missing binary, hub error) ends the 180 s wait at once. A timeout or transport error triggers one redial (retryNow), and the wait goes on.
  • The "Moving this chat to Cloud…" toast now says " didn't connect: failed ()." and offers Retry, which reruns the start. Its duration is 30 s with Retry and 12 s without. The toast then ends.
  • attach() redials a live legacy machine that has no connection.
  • The new strings are translated in all locales.

The app-server link path and classic Cloud are untouched.

Risk. The main risk is in the legacy Cloud start: the deadlines could cut off a slow but healthy start. The bounds sit well above measured times (attach-endpoint about 0.6 s, enroll about 1.7 s, link up about 1 s on a production account).

Tests. CloudLinkStartTests cover: the deadline firing on work that ignores cancellation; the step and reason surviving; the definitive classification; pass-through; and progress state. Compile check is green for the app and cmux-tui at the head.

Read-back. A production-auth build on a fleet Mac signed in and brought up about 20 legacy links, each logged per step ("link up in 1.1 s"). One machine failed and the log names it: "failed at remote-connect after 20.3 s: remote connection did not become ready". A Cloud start created its machine and copied the folder over the legacy link.

Related: SSH host-key failure filed as #33204.

Agent session: agent session

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes the legacy Cloud link start stalling silently: one stalled shared tunnel-hub start blocked every link for minutes, then failed with "didn't connect" and no reason.

  • Each link step (attach-endpoint, hub, approve, remote connect) now runs under its own deadline with step-level log lines, so a start that doesn't come up names the step and reason.
  • A definitive failure (signed out, plan required, 4xx, missing binary, hub error) ends the 180 s wait at once; a timeout or transport error triggers one redial.
  • The "didn't connect" toast now names the failing step and offers Retry, which reruns the start; a new wait or Retry clears the earlier attempt's failure so a stale one can't answer a fresh wait.
  • attach() redials a live legacy machine that has no connection.
  • The app-server link path and classic Cloud are untouched.

Written for commit 6d44327. Summary will update on new commits.

View guided diff Turn on auto-fix

teamleaderleo and others added 6 commits November 2, 2026 22:42
…failed connect says the step and why

The NEXT nightly's legacy Cloud link start went silent: no hub, no
remote connect, no log line, and the Cloud chat start gave up after
3 minutes with "didn't connect". Each step of a legacy link start
(attach-endpoint, the WireGuard hub, approve, remote connect) now runs
under its own deadline with a notice-level began/done/failed line, and
a failure keeps its step. A Cloud start's wait ends at once on a refusal
and its toast says the step and reason with Retry; within the wait a
legacy link's backoff is woken every 30 s; a manual connect retries a
live legacy machine that has no connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3bab30503f0b9a7059e5
…the step deadlines

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3bab30503f0b9a7059e5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3bab30503f0b9a7059e5
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 6d44327: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr33567 -f ref=leo/cloud-legacy-link-hang -f platform=macos -f backend_mode=local

Shipped in:

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/41619934635 at 092736d failed: cmux-next fleet Release compile.
Those jobs last passed at 8cc02ce. This pull request is one of 5 merged since: #33578, #33567, #33411, #33407, #33405.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at df24848

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 2. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at df24848
Visual verdict: no images; Leo tries the build

  1. [P2] [follow-up] CloudMove.swift:304: A previous definitive failure immediately ends a fresh retry. retryNow() leaves legacyLinkFailure populated until connection succeeds, so even after resolving the refusal, Retry reports the old error before the new attempt answers. Clear or version the failure when explicitly retrying, and apply the same fix to CloudStartSession.connected. Add a failure → retry → success test.

  2. [P2] [follow-up] CloudStartChat.swift:79: Supply a screenshot of the failure toast with Retry, including a long step/reason, to verify readability and button visibility.

Follow-ups:

  • Scope definitive failures to the attempt being awaited.
  • Capture the failure toast with Retry.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at 6d44327

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 1. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at 6d44327
Visual verdict: no images; Leo tries the build

Reviewed deadline cancellation, retry state, failure propagation, and localization placeholders. No blocking findings; Swift tests could not run in this box.

  1. [P2] [follow-up] Packages/macOS/CmuxNext/Sources/CmuxNextApp/Cloud/CloudStartChat.swift:82: No screenshots cover the changed failure toast. Verify long failure reasons and the Retry button in the build.

Follow-ups:

  • [P2] [follow-up] Verify the failure toast’s text and Retry button visually.

lawrencecchen added a commit that referenced this pull request Nov 3, 2026
…(arch app_lines +74 from #33567) (#33730)

#33567 (e8fc118) added 74 CmuxNextApp lines, 13 of them in
CloudMoveStrings (the failed link step's text) and the rest in the Cloud move,
start and session code that shows it. The text belongs with the Cloud UI
strings: CloudMoveStrings moves whole to CmuxNextCloudUI (a public struct,
as CloudPortStrings), and its 29 keys move from the App's Cloud.xcstrings to
CmuxNextCloudUI's Cloud.xcstrings, every locale unchanged. The seven App files
and three App tests that use it import CmuxNextCloudUI. CmuxNextApp -104.
@lawrencecchen

Copy link
Copy Markdown
Contributor

Swift logic follow-up (tip watcher swift-logic row, STATUS 13027 "SWIFT LOGIC CULPRITS"; lead order).

This PR raised swift_logic_lines by +269 at its merge commit e8fc118: scripts/cmux-next/swift-logic-lines.py before and after the merge, with the same table (the tip's, c6be943). The largest declarations of the rise:

  • struct CloudLinkStartError in Packages/macOS/CmuxNext/Sources/CmuxNextCloud/Link/CloudLinkStart.swift: +57 lines counted as Swift logic (no row in plans/cmux-next/swift-logic-classes.tsv)
  • actor CloudMachineLink in Packages/macOS/CmuxNext/Sources/CmuxNextCloud/Link/CloudMachineLink.swift: +45 lines counted as Swift logic (its row counts it as logic)
  • enum CloudLinkStep in Packages/macOS/CmuxNext/Sources/CmuxNextCloud/Link/CloudLinkStart.swift: +32 lines counted as Swift logic (no row in plans/cmux-next/swift-logic-classes.tsv)
  • class CloudLinkProgress in Packages/macOS/CmuxNext/Sources/CmuxNextCloud/Link/CloudLinkStart.swift: +31 lines counted as Swift logic (no row in plans/cmux-next/swift-logic-classes.tsv)

plans/cmux-next/layer-ownership.md says Rust owns every state, list and rule and Swift is the platform shell. Please move the new logic of this PR in a follow-up: to Rust (the crate that owns the area; the owner lists in STATUS 12147-12165 name the target per area), or at least out of the app into its package. If a declaration above is platform code (AppKit, Core Animation, OS calls, strings, tokens, DEBUG tooling), reply here and the table gets a PLATFORM row with your reason instead.

The swift-logic row is a report, not a gate: nothing blocks on this comment.

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 6d44327f Deployed Nov 3, 2026 by teamleaderleo via cmux-tui tree preflight #105362
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants