Skip to content

cmux vm run: remove the guest's sync and pull archives on every exit path - #29722

Merged
teamleaderleo merged 6 commits into
feat-cmux-nextfrom
lane/cloud-vm-29704
Nov 1, 2026
Merged

cmux vm run: remove the guest's sync and pull archives on every exit path#29722
teamleaderleo merged 6 commits into
feat-cmux-nextfrom
lane/cloud-vm-29704

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

cmux vm run now removes its archives on the guest on every exit path, failures included, and says so when it couldn't remove them from a kept VM.

From #29704: "cmux vm run removes the sync archive on the guest only after a successful extraction, and ignores a failed removal of the pull archive (/tmp/cmux-pull.tgz and its .rc). With --keep (pause instead of delete), a failed extraction or cleanup leaves up to 64 MiB (sync) or 256 MiB (pull) of the user's files on the paused VM. Fix: remove both archives on every path, failure included (a trap in the guest script), and report a failed removal when the VM is kept."

How:

  • The unpack script sets trap 'rm -f /tmp/cmux-run.tgz' EXIT before tar -xzf, so the archive goes whether tar succeeds or fails.
  • The run records when an archive may be on the guest: the first sync upload, or the start of the pull. Once one may be, it runs rm -f /tmp/cmux-run.tgz /tmp/cmux-pull.tgz /tmp/cmux-pull.tgz.rc before it pauses or deletes the VM. That covers a failed upload, a failed unpack, a failed pull and an interrupt. The old pull-only cleanup, which ignored its own result, is replaced by this step.
  • When the VM is kept and the removal fails, stderr says so and gives the command that finishes the job (cmux vm exec <id> -- 'rm -f ...'). The run's exit status is unchanged.

Fixes #29704

Changelog

  • Fixed: cmux vm run --keep no longer leaves the synced or pulled archive of your files on the paused VM when an upload, unpack or pull fails, and it reports a cleanup it could not finish.

Tests

tests/lifecycle.rs, against a mock API with a kept (paused) VM:

  • a_kept_run_removes_a_partly_uploaded_sync_archive: the upload fails, and the archives are removed once before the pause.
  • a_kept_run_removes_a_sync_archive_that_failed_to_unpack: tar fails. The unpack script carries the trap, and the archives are removed once.
  • a_kept_run_reports_pull_archives_it_could_not_remove: the pull fails, then the removal fails. stderr reports it with the command to run, and the VM is still paused.

Focused command (in workers/cmux-vm/cli): cargo test --locked -p cmux-vm --test lifecycle a_kept_run. Red with run.rs at the test commit ac70722 (all 3 fail). Green at the head 5c28ed0.

Observed / Not run

  • Observed on the CI Linux pool at 5c28ed0 (run 41244659217): the red and green above, cargo test --locked -p cmux-vm (all suites pass), and cargo clippy --locked -p cmux-vm --all-targets -- -D warnings (clean).
  • Not run: a real VM. The guest-side change is the trap ... EXIT in the unpack script, plus one rm -f exec before the VM is paused or deleted.

No capture: CLI only, no UI change.

🤖 Generated with Claude Code
Agent session: agent session

teamleaderleo and others added 2 commits October 31, 2026 21:31
@teamleaderleo @claude
cmux vm run: test that a kept run removes its guest archives on failure
ac70722 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-75b7a03d1a2084cf16af
@teamleaderleo @claude
cmux vm run: remove the guest's sync and pull archives on every exit …
5f49d29 
…path

The sync archive was removed only after a successful unpack, and a failed
removal of the pull archive was ignored, so a kept (paused) VM could hold up
to 64 MiB (sync) or 256 MiB (pull) of the user's files. The unpack now
removes its archive on exit (a trap), and once anything may be staged the run
removes both archives before it ends the VM, failure and interrupt included,
reporting a failed removal on a kept VM with the command that finishes it.
Fixes #29704.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-75b7a03d1a2084cf16af
@teamleaderleo teamleaderleo added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@github-actions

github-actions Bot commented Oct 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 95f231b: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr29722 -f ref=lane/cloud-vm-29704 -f platform=macos -f backend_mode=local

Shipped in:

teamleaderleo and others added 2 commits October 31, 2026 22:21
@teamleaderleo @claude
cmux vm run: fence and stop guest writers before removing the archives
26410b6 
An interrupted run drops its in-flight exec, but the guest's upload or pull
pack can still be running or not yet started, so it could reopen an archive
after the cleanup, or hold a removed one open. Writers now refuse once a fence
file exists; the cleanup makes it, then in rounds kills any process holding an
archive open and removes them. The hint for a failed removal on a kept VM now
resumes it first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-75b7a03d1a2084cf16af
@teamleaderleo @claude
cmux vm run: writers register before the fence check, and the cleanup…
4658065 
… stops them

A single fence check let a writer that had passed it open its archive after
the cleanup. Each upload and pull now runs in its own process group,
registers itself, then checks the fence; the cleanup makes the fence, then
kills every registered group until none is left, then removes the archives.
Whichever comes second sees the other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-75b7a03d1a2084cf16af
@teamleaderleo @claude
cmux vm run: register the archive readers too
95f231b 
The unpack and the pull's chunk reads hold an archive open as well, so a
reader left behind by an interrupted run kept a removed archive's bytes on a
kept VM. They now register like the writers, and the unpack removes its
archive explicitly once tar ends (its own EXIT trap would have replaced the
registration's).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-75b7a03d1a2084cf16af
@cursor

cursor Bot commented Oct 31, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

teamleaderleo commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, next-review): SHIP at 95f231b

SHIP at 95f231b

Visual verdict: no images; Leo tries the build

  1. [P2] [follow-up] run.rs:711: The recovery command only unlinks archives. If cleanup failed before stopping registered processes, resuming and running rm can leave archive bytes held open. Reproduced with a registered upload: rm succeeds while its deleted archive remains open. The recovery command should also fence and stop registered processes.

Direct shell checks passed for interruption, delayed writers, and repeated cleanup. Rust tests could not run because Cargo is unavailable.

Follow-ups:

  • Make the recovery command perform the full fenced cleanup.

cmux-bot[bot]
cmux-bot Bot approved these changes Oct 31, 2026
View reviewed changes

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves 95f231b: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@teamleaderleo
teamleaderleo merged commit 9cf58be into feat-cmux-next Nov 1, 2026
222 of 223 checks passed
@teamleaderleo
teamleaderleo deleted the lane/cloud-vm-29704 branch November 1, 2026 00:10
teamleaderleo added a commit that referenced this pull request Nov 1, 2026
@teamleaderleo @claude
Merge feat-cmux-next into lane/cloud-vm-29707
1ee2ae8 
The --pull size check now measures the fenced pull command that #29722
sends (pull_command), not the bare pack script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-75b7a03d1a2084cf16af
teamleaderleo added a commit that referenced this pull request Nov 1, 2026
@teamleaderleo @claude
Merge feat-cmux-next into the token lock and unreadable-token branch
00dd1a0 
lifecycle.rs: both sides added tests at the end (revoke_removes_an_unreadable_stored_token
here, #29722's kept-run archive tests on base); both are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-75b7a03d1a2084cf16af
@teamleaderleo teamleaderleo mentioned this pull request Nov 4, 2026
Closed

This branch was previously deployed

3 inactive deployments
artifacts — 95f231b2 Deployed Oct 31, 2026 by teamleaderleo via add the Windows daemon to the tree #91864
sentry-dsyms — 95f231b2 Deployed Oct 31, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #91864
ci-routing-agent — 95f231b2 Deployed Oct 31, 2026 by teamleaderleo via cmux-tui tree preflight #91864
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cmux-bot cmux-bot[bot] cmux-bot[bot] approved these changes

Assignees

No one assigned

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@teamleaderleo