Skip to content

fix: recover interrupted Cloud vm run creates - #16221

Merged
teamleaderleo merged 3 commits into
mainfrom
fix/cloud-vm-run-idempotency-15644
Oct 1, 2026
Merged

teamleaderleo merged 3 commits into
mainfrom
fix/cloud-vm-run-idempotency-15644

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #15644

cmux vm run used a fresh idempotency key for every pool-machine create and only recorded pool membership after the create response. If the process was interrupted, timed out, or lost its socket after the backend accepted the request, the machine finished provisioning but became invisible to the local pool. The next run then sent a new key and could create another paid machine.

This persists one pool-create idempotency key under ~/.cmuxterm, protected by an advisory lock. Concurrent routers converge on the same key; transport failures and vm_create_in_progress preserve it for recovery, while definite backend failures clear it. The key is cleared only after the created machine is recorded in the pool, so a local persistence failure can still be retried against the same backend create.

Validation:

  • swiftc -parse on the changed CLI and regression test
  • scripts/sync-test-wiring --check
  • scripts/lint-pbxproj-test-wiring.sh
  • git diff --check
  • Regression covers retry key reuse after an unknown outcome and fresh-key allocation after completion.

Native app/CLI execution was not run on the MacBook Air.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Closes #15644. cmux vm run used a fresh idempotency key for every pool-machine create, so an interrupted, timed-out, or dropped-socket run after the backend accepted the request left a provisioned machine invisible to the local pool, and the next run created another paid machine.

This persists one pool-create idempotency key under ~/.cmuxterm, guarded by an advisory lock. Concurrent routers converge on the same key; transport failures and vm_create_in_progress keep it for recovery, while definite backend failures clear it. The key clears only after the created machine is recorded in the pool, so a local persistence failure can retry against the same backend create. A regression test, run against an isolated temporary store, covers key reuse after an unknown outcome and fresh-key allocation after completion.

Written for commit 74b8871. Summary will update on new commits.

Review in cubic

@coderabbitai

coderabbitai Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: b1c42751-be05-4bfc-9c89-a6a0d2c73fd6

📥 Commits

Reviewing files that changed from the base of the PR and between 8840999 and 74b8871.

📒 Files selected for processing (1)
  • cmuxTests/CLIVMTransferTests.swift
 ____________________________________________
< Forcing bugs to surrender unconditionally. >
 --------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@teamleaderleo teamleaderleo added bug Something isn't working S2: major A crash, hang, lost state, broken connection, or a regression on a path people use area: cloud Cloud machines and workspaces, relay transport area: cli The cmux CLI, cmux-tui, the socket API and SDKs difficulty:3 Systems: multiple components or a runtime lifecycle help wanted Nobody on the team is working on this and we would take a patch labels Sep 30, 2026
@teamleaderleo
teamleaderleo enabled auto-merge (squash) September 30, 2026 21:35
@github-actions

Copy link
Copy Markdown
Contributor

CI failure attribution

CI failed on cff70ed21c (run 36775477178 attempt 2): 1 code.

Job Verdict Why
macos / macOS compile admission code a compile error
Matched log lines
macos / macOS compile admission: /tmp/cmux-ci/src/Sources/CmuxConfig.swift:2816:51: error: missing argument for parameter 'actionReferenceID' in call

Not re-run automatically: macos / macOS compile admission is not a machine failure.

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's.

Co-Authored-By: Codex <noreply@openai.com>
@teamleaderleo
teamleaderleo merged commit be10c64 into main Oct 1, 2026
32 of 34 checks passed
@teamleaderleo
teamleaderleo deleted the fix/cloud-vm-run-idempotency-15644 branch October 1, 2026 14:36
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Merge receipt for 74b8871106, merged 2026-10-01 14:36:23 UTC

  • Not verified at merge: ci-status (not reported), CI fast guards (in progress), GhosttyKit release check (in progress), guards (6) (in progress)
  • Verified: Fast static checks, Web complexity, web-validation
  • Skipped by policy: admission-placement, browser, Claude wrapper regressions, Dogfood build #​${{ github.event.pull_request.number }}, full-suite-coverage, remote-daemon, suite-coverage, swift-package-tests, web, web-build, web-database-tests, web-tests
  • Full suite: runs on main after merge.

Labeled merged-unverified: if main breaks near this merge, look here first.

@github-actions github-actions Bot added the merged-unverified A judging check was not green at merge; see the merge receipt comment label Oct 1, 2026
rustybret pushed a commit to rustybret/bmux that referenced this pull request Oct 1, 2026
877df6f docs: add Kiro and Hermes to Feed matrix (manaflow-ai#16106)
b5aa76c Re-land Cloud machine stale status punctuation (manaflow-ai#16010)
4fdd903 Merge pull request manaflow-ai#15345 from manaflow-ai/fix-v2-startup-local-route-impl
73ae480 fix(cli): list browser JSON flags in help (manaflow-ai#16098)
2bb742d fix(agents): preserve HTML-like Codex auto-naming messages (manaflow-ai#15984)
0d1b8dd fix(web): restore the seats-follow-membership copy the dashboard port dropped (manaflow-ai#16265)
be10c64 fix: recover interrupted Cloud vm run creates (manaflow-ai#16221)
28742a8 docs: document agent session recovery command (manaflow-ai#16108)
bcaf5e4 test: resolve the temp root before comparing rerun xcconfig paths (manaflow-ai#16372)
86230a5 Add built-in custom sidebar templates (manaflow-ai#15931)
5ccac53 fix(session): discard persisted listening ports on restore (manaflow-ai#12436)
882b6fc fix(flags): extend the fourteen reviewBy dates that expire on 2026-10-02 (manaflow-ai#15922)
45914ae docs: document Kimi session restore (manaflow-ai#16107)
2f574d6 Treat aborted Codex turns as terminal monitor events
f59f37d Remove stale Cloud reorder indicator on cleanup
eba8488 Fix drag cleanup and message inbox fixture
0d2b312 Include CLI error type in CLI tests
5955b67 Include CLI error type in app target
94acff9 Repair stale macOS test compile references
0117647 Fix isolated test window geometry cleanup
4b7b837 Fix canonical identity and accent color compile errors
538aaf6 Normalize instance tags and fix browser restoration compile
2c33e92 Fix post-merge compile blockers
5ea8219 Merge remote-tracking branch 'origin/main' into fix-v2-startup-local-route-impl
825e0d3 Fail closed when release gate V2 origin is unset
c21b0a8 Publish warmup recovery and preserve workspace IDs
f66904b Allow full relay rollover soak to finish
2ca91b5 Harden forget cleanup and Codex iteration proof
b908dde Bound endpoint warmup retries and retry on foreground
859ca92 Retry endpoint warmup after transient failures
97e5a7f Cancel timed out endpoint warmup
c5caca7 Serialize snapshot encoding in persistence actor
c589ff8 Bound Codex workload log polling
40aa04e Harden startup identity cache and snapshot invalidation
f06f502 Use shared deadline scheduler for endpoint warmup
2fa7297 Validate combined host status before reuse
5386126 Scope paired Mac cache before reconnect reuse
a34c7ee Bound endpoint warmup timeout independently of cancellation
b834f86 Bound cached endpoint warmup and diagnostics
00959bf Revalidate scope after cached snapshot restore
1b444a0 Keep attach tickets in memory during retries
6985f9e Repair cached runtime reconciliation and warmup retries
08c8047 Make explicit snapshot deletion win races
4c78ecf Preserve v2 runtime ownership through auth and soak
b1f7a18 Await workspace snapshot removal
0583749 Move snapshot restore off the main actor
7ea8285 Persist workspace snapshots off the main actor
07f7c13 Bound snapshot maintenance and relay gate timing
6b68c8c Keep real usage terminals available for verification
2f5c012 Normalize hidden Mac snapshot identities
5c5388e Stabilize real usage release gate lifecycle
344cac1 fix: bound cached startup and redact snapshots
1ce9bc8 fix: preserve and prune cached workspace snapshots
1887e53 fix: close release gate workload hazards
27fba71 Merge main (0398322) into fix-v2-startup-local-route-impl
d9724bc test: compile the vm ready poll policy into cmuxCLITests
beadd57 test: drive hook state recovery through the bundled CLI
c3b97e1 Use shared queue drain helper after compile repair
a11fc3c test: restore cmuxTests compile on main
c5a9d15 Keep close tab test queue drain compatible with main
07b9624 Hit test the measured text line in inline link regression
6c9f137 Merge main (a66a8bb) into fix-v2-startup-local-route-impl
4cbdeea Measure inline link view before hit testing
82618ef Await actor epoch during cached warmup
59cbda2 Fix cached endpoint warmup self lifetime
a06061a Keep release gate on approved runner route
1c10a44 Leave staging gate outside protected environments
8c4cfb5 Use staging environment for staging release gates
1db5b80 Pin manual gate to an available trusted Mac
b934ccf Route manual gate through selected owned pool
9ecd5aa Allow trusted owned runner for manual gate
5afde77 Use supported hosted macOS label for manual gate
2359026 Allow hosted runner for manual Iroh acceptance gate
01b0f64 Make release-gate evidence durable and ordered
c7cb704 Bind real-use replay to the requested workspace
5bd6005 Require real Codex workload evidence
35c7723 Measure app-side foreground readiness
e52cb3e Recover backup pairing when only demo row is cached
849ed89 Harden v2 startup state and workspace targeting
02bb470 Bind iOS e2e checks to the Codex workspace
76178d0 Document complete iOS e2e driver contract
908e2ee Merge green main into v2 startup branch
185be6c Make iOS e2e driver open a workspace
45f19d0 Keep IROH gate independent of Ghostty helper fetches
5f67eea Allow full relay gate to publish its verdict
9574395 Measure release gate rows at UIKit visibility
f3928a3 fix: persist every complete workspace snapshot
c5cb772 fix: measure cached workspace rows before Iroh readiness
c84fb67 ci: install axe before iOS Iroh gate
451933a Retain foreground state during snapshot reconciliation
eb1aa35 Preserve local demo workspace ownership
14dc577 Upload real Iroh usage evidence
41659c1 Add real Codex and background coverage to Iroh gate
5696ef1 Close workspace snapshot scope races
cb9d421 Prevent stale workspace snapshots from resurfacing
4f7f4b5 Restore scoped workspace snapshots before paired Mac load
a93af42 Give real relay gate cleanup time
61a88a2 Fix workspace snapshot initializer ordering
8bc596e Verify relay renewal at its real lifetime
99ca1d8 Cache scoped workspace rows during v2 startup
5aeab39 Enforce sub-2.5-second workspace startup gate
4fe8ac5 Warm cached v2 state before auth bootstrap
bf90a77 fix mobile attach parsing with CLI diagnostics
f7db0fa Merge remote-tracking branch 'origin/main' into fix-v2-startup-local-route-impl
1a91f7c test: keep workspace creation override signatures current
db62464 test: pass remote workspace flag through overrides
241ba19 test: avoid asserting startup cache implementation details
1e87a6e fix simulator launch environment propagation
6a17189 fix: keep release gate on workspace list
fa8f1c1 fix: retry stored Mac after auth restore
2bbb2ef fix: remove merge artifact from mobile shell
0ba4041 Merge remote-tracking branch 'origin/main' into fix-v2-startup-local-route-impl
2b64a40 test: preserve simulator startup diagnostics in release gates
73bd646 fix: avoid speculative duplicate Iroh dials
814ebe7 perf: reuse hydrated pairing cache during startup
27c401b Revert "perf: publish startup workspace rows before pairing persistence"
c146010 perf: publish startup workspace rows before pairing persistence
582713d fix: drain parallel iroh preconnect task
b980721 perf: preconnect iroh before first authenticated request
a396428 fix: use v2 identity for combined host status
5901f7d test: preserve admitted identity in combined startup status
5c504e8 fix: match cached host status in combined workspace response
9fac2b6 perf: build combined host proof from live service
0cf1b27 perf: combine startup workspace and host status
1ece6fc test: persist onboarding before cached launch
6372c0c test: measure cached pairing release-gate startup
73bc8cf test: reject injected credentials during cached startup verification
792a561 Revert "perf: reuse hydrated pairing snapshot during startup"
c9a6b58 Revert "perf: start cached pairing dial during startup hydration"
93d0db5 Revert "fix: bind optional cached pairing row correctly"
12ef215 fix: bind optional cached pairing row correctly
16cd25d perf: start cached pairing dial during startup hydration
d8a9c75 perf: reuse hydrated pairing snapshot during startup
4f6f58b fix: dial cached route before startup backup refresh
7814fd2 test: prove startup dials during backup refresh

# Conflicts:
#	.github/workflows/iroh-release-gate.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: cli The cmux CLI, cmux-tui, the socket API and SDKs area: cloud Cloud machines and workspaces, relay transport bug Something isn't working difficulty:3 Systems: multiple components or a runtime lifecycle help wanted Nobody on the team is working on this and we would take a patch merged-unverified A judging check was not green at merge; see the merge receipt comment S2: major A crash, hang, lost state, broken connection, or a regression on a path people use

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cloud: cmux vm run orphans a paid pool machine when the create call is interrupted

1 participant