Repository navigation
fix: recover interrupted Cloud vm run creates - #16221
Conversation
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Repository: manaflow-ai/cmux/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
All contributors have signed the CLA ✍️ ✅ |
CI failure attributionCI failed on
Matched log linesNot re-run automatically: Written by |
Co-Authored-By: Codex <noreply@openai.com>
|
Merge receipt for
Labeled |
877df6f docs: add Kiro and Hermes to Feed matrix (manaflow-ai#16106) b5aa76c Re-land Cloud machine stale status punctuation (manaflow-ai#16010) 4fdd903 Merge pull request manaflow-ai#15345 from manaflow-ai/fix-v2-startup-local-route-impl 73ae480 fix(cli): list browser JSON flags in help (manaflow-ai#16098) 2bb742d fix(agents): preserve HTML-like Codex auto-naming messages (manaflow-ai#15984) 0d1b8dd fix(web): restore the seats-follow-membership copy the dashboard port dropped (manaflow-ai#16265) be10c64 fix: recover interrupted Cloud vm run creates (manaflow-ai#16221) 28742a8 docs: document agent session recovery command (manaflow-ai#16108) bcaf5e4 test: resolve the temp root before comparing rerun xcconfig paths (manaflow-ai#16372) 86230a5 Add built-in custom sidebar templates (manaflow-ai#15931) 5ccac53 fix(session): discard persisted listening ports on restore (manaflow-ai#12436) 882b6fc fix(flags): extend the fourteen reviewBy dates that expire on 2026-10-02 (manaflow-ai#15922) 45914ae docs: document Kimi session restore (manaflow-ai#16107) 2f574d6 Treat aborted Codex turns as terminal monitor events f59f37d Remove stale Cloud reorder indicator on cleanup eba8488 Fix drag cleanup and message inbox fixture 0d2b312 Include CLI error type in CLI tests 5955b67 Include CLI error type in app target 94acff9 Repair stale macOS test compile references 0117647 Fix isolated test window geometry cleanup 4b7b837 Fix canonical identity and accent color compile errors 538aaf6 Normalize instance tags and fix browser restoration compile 2c33e92 Fix post-merge compile blockers 5ea8219 Merge remote-tracking branch 'origin/main' into fix-v2-startup-local-route-impl 825e0d3 Fail closed when release gate V2 origin is unset c21b0a8 Publish warmup recovery and preserve workspace IDs f66904b Allow full relay rollover soak to finish 2ca91b5 Harden forget cleanup and Codex iteration proof b908dde Bound endpoint warmup retries and retry on foreground 859ca92 Retry endpoint warmup after transient failures 97e5a7f Cancel timed out endpoint warmup c5caca7 Serialize snapshot encoding in persistence actor c589ff8 Bound Codex workload log polling 40aa04e Harden startup identity cache and snapshot invalidation f06f502 Use shared deadline scheduler for endpoint warmup 2fa7297 Validate combined host status before reuse 5386126 Scope paired Mac cache before reconnect reuse a34c7ee Bound endpoint warmup timeout independently of cancellation b834f86 Bound cached endpoint warmup and diagnostics 00959bf Revalidate scope after cached snapshot restore 1b444a0 Keep attach tickets in memory during retries 6985f9e Repair cached runtime reconciliation and warmup retries 08c8047 Make explicit snapshot deletion win races 4c78ecf Preserve v2 runtime ownership through auth and soak b1f7a18 Await workspace snapshot removal 0583749 Move snapshot restore off the main actor 7ea8285 Persist workspace snapshots off the main actor 07f7c13 Bound snapshot maintenance and relay gate timing 6b68c8c Keep real usage terminals available for verification 2f5c012 Normalize hidden Mac snapshot identities 5c5388e Stabilize real usage release gate lifecycle 344cac1 fix: bound cached startup and redact snapshots 1ce9bc8 fix: preserve and prune cached workspace snapshots 1887e53 fix: close release gate workload hazards 27fba71 Merge main (0398322) into fix-v2-startup-local-route-impl d9724bc test: compile the vm ready poll policy into cmuxCLITests beadd57 test: drive hook state recovery through the bundled CLI c3b97e1 Use shared queue drain helper after compile repair a11fc3c test: restore cmuxTests compile on main c5a9d15 Keep close tab test queue drain compatible with main 07b9624 Hit test the measured text line in inline link regression 6c9f137 Merge main (a66a8bb) into fix-v2-startup-local-route-impl 4cbdeea Measure inline link view before hit testing 82618ef Await actor epoch during cached warmup 59cbda2 Fix cached endpoint warmup self lifetime a06061a Keep release gate on approved runner route 1c10a44 Leave staging gate outside protected environments 8c4cfb5 Use staging environment for staging release gates 1db5b80 Pin manual gate to an available trusted Mac b934ccf Route manual gate through selected owned pool 9ecd5aa Allow trusted owned runner for manual gate 5afde77 Use supported hosted macOS label for manual gate 2359026 Allow hosted runner for manual Iroh acceptance gate 01b0f64 Make release-gate evidence durable and ordered c7cb704 Bind real-use replay to the requested workspace 5bd6005 Require real Codex workload evidence 35c7723 Measure app-side foreground readiness e52cb3e Recover backup pairing when only demo row is cached 849ed89 Harden v2 startup state and workspace targeting 02bb470 Bind iOS e2e checks to the Codex workspace 76178d0 Document complete iOS e2e driver contract 908e2ee Merge green main into v2 startup branch 185be6c Make iOS e2e driver open a workspace 45f19d0 Keep IROH gate independent of Ghostty helper fetches 5f67eea Allow full relay gate to publish its verdict 9574395 Measure release gate rows at UIKit visibility f3928a3 fix: persist every complete workspace snapshot c5cb772 fix: measure cached workspace rows before Iroh readiness c84fb67 ci: install axe before iOS Iroh gate 451933a Retain foreground state during snapshot reconciliation eb1aa35 Preserve local demo workspace ownership 14dc577 Upload real Iroh usage evidence 41659c1 Add real Codex and background coverage to Iroh gate 5696ef1 Close workspace snapshot scope races cb9d421 Prevent stale workspace snapshots from resurfacing 4f7f4b5 Restore scoped workspace snapshots before paired Mac load a93af42 Give real relay gate cleanup time 61a88a2 Fix workspace snapshot initializer ordering 8bc596e Verify relay renewal at its real lifetime 99ca1d8 Cache scoped workspace rows during v2 startup 5aeab39 Enforce sub-2.5-second workspace startup gate 4fe8ac5 Warm cached v2 state before auth bootstrap bf90a77 fix mobile attach parsing with CLI diagnostics f7db0fa Merge remote-tracking branch 'origin/main' into fix-v2-startup-local-route-impl 1a91f7c test: keep workspace creation override signatures current db62464 test: pass remote workspace flag through overrides 241ba19 test: avoid asserting startup cache implementation details 1e87a6e fix simulator launch environment propagation 6a17189 fix: keep release gate on workspace list fa8f1c1 fix: retry stored Mac after auth restore 2bbb2ef fix: remove merge artifact from mobile shell 0ba4041 Merge remote-tracking branch 'origin/main' into fix-v2-startup-local-route-impl 2b64a40 test: preserve simulator startup diagnostics in release gates 73bd646 fix: avoid speculative duplicate Iroh dials 814ebe7 perf: reuse hydrated pairing cache during startup 27c401b Revert "perf: publish startup workspace rows before pairing persistence" c146010 perf: publish startup workspace rows before pairing persistence 582713d fix: drain parallel iroh preconnect task b980721 perf: preconnect iroh before first authenticated request a396428 fix: use v2 identity for combined host status 5901f7d test: preserve admitted identity in combined startup status 5c504e8 fix: match cached host status in combined workspace response 9fac2b6 perf: build combined host proof from live service 0cf1b27 perf: combine startup workspace and host status 1ece6fc test: persist onboarding before cached launch 6372c0c test: measure cached pairing release-gate startup 73bc8cf test: reject injected credentials during cached startup verification 792a561 Revert "perf: reuse hydrated pairing snapshot during startup" c9a6b58 Revert "perf: start cached pairing dial during startup hydration" 93d0db5 Revert "fix: bind optional cached pairing row correctly" 12ef215 fix: bind optional cached pairing row correctly 16cd25d perf: start cached pairing dial during startup hydration d8a9c75 perf: reuse hydrated pairing snapshot during startup 4f6f58b fix: dial cached route before startup backup refresh 7814fd2 test: prove startup dials during backup refresh # Conflicts: # .github/workflows/iroh-release-gate.yml
Closes #15644
cmux vm runused a fresh idempotency key for every pool-machine create and only recorded pool membership after the create response. If the process was interrupted, timed out, or lost its socket after the backend accepted the request, the machine finished provisioning but became invisible to the local pool. The next run then sent a new key and could create another paid machine.This persists one pool-create idempotency key under
~/.cmuxterm, protected by an advisory lock. Concurrent routers converge on the same key; transport failures andvm_create_in_progresspreserve it for recovery, while definite backend failures clear it. The key is cleared only after the created machine is recorded in the pool, so a local persistence failure can still be retried against the same backend create.Validation:
swiftc -parseon the changed CLI and regression testscripts/sync-test-wiring --checkscripts/lint-pbxproj-test-wiring.shgit diff --checkNative app/CLI execution was not run on the MacBook Air.
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by cubic
Closes #15644.
cmux vm runused a fresh idempotency key for every pool-machine create, so an interrupted, timed-out, or dropped-socket run after the backend accepted the request left a provisioned machine invisible to the local pool, and the next run created another paid machine.This persists one pool-create idempotency key under
~/.cmuxterm, guarded by an advisory lock. Concurrent routers converge on the same key; transport failures andvm_create_in_progresskeep it for recovery, while definite backend failures clear it. The key clears only after the created machine is recorded in the pool, so a local persistence failure can retry against the same backend create. A regression test, run against an isolated temporary store, covers key reuse after an unknown outcome and fresh-key allocation after completion.Written for commit 74b8871. Summary will update on new commits.