[teamleaderleo]

Leo (2026-11-03), on the sync speed-ups: "yeah let's do alllll of those things! :DDD". This is the content-addressed blob store from that list.

What it does

cmux link sync push --store PLATFORM|default (and start-chat copy --store, which the app passes for a personal account on non-production auth) sends a folder through the user's sync store on the cmux platform instead of byte by byte over the link:

1. The Mac hashes each file it has to write. Files up to 4 MiB are one blob keyed by their sha256; bigger files are cut by content-defined chunking (FastCDC with normalized chunking, 512 KiB min / 2 MiB average / 8 MiB max), so an edit in the middle changes only the chunks around it.
2. It asks the store which blobs it lacks (POST /v1/sync/blobs/missing), uploads only those (PUT /v1/sync/blobs/{sha}, 16 at once, each piece re-read and re-hashed before it goes), then uploads the plan (itself a blob: every file's path, mode, size, sha256 and chunks).
3. It mints a read-only grant (POST /v1/sync/grants, csg_…, one hour) and sends the machine blob.fetch.start {store:{origin, grant}, plan}. The machine never sees the account credential.
4. The machine daemon (new capability blob-fetch-v2, ops blob.fetch.start / .status / .cancel / .forget) fetches the plan and blobs from the store, 16 files at once, verifies each chunk's and each file's sha256, and writes through the same Roots sandbox and PendingWrite staging as fs.write (atomic rename, permissions set before commit). A path outside the roots is refused and named.
5. The Mac reads each file's revision back and records it in the sync manifest as before.

The direct link copy stays the fallback and is unchanged:

• a machine without blob-fetch-v2, no --store, or a store that fails before the machine writes: everything goes over the link.
• a file the machine refused: that file goes over the link.
• less than 1 MiB to write (a typical one-file edit): over the link, since the store's round trips cost more than the bytes (store.reason: "small").

Blobs are keyed per user (v1/u/<user>/<sha[0..2]>/<sha>), so identical files are stored once per user, a second machine costs the Mac only the plan, and a push cut off partway is resumed by the next push re-planning: the store already has the blobs, so the Mac uploads nothing again.

Server side: manaflow-ai/cmux-platform#746 (merged; the routes are live in development only, so staging and production answer 404 and the push falls back to the link).

What I took from Mutagen

• Content is staged and verified by digest, then renamed into place atomically (Mutagen's staging + transition). Here: PendingWrite plus a per-chunk and whole-file sha256 check.
• The receiver pulls what it lacks instead of the sender pushing everything (Mutagen's rsync-style stage from the receiving end).
• The sync state (our manifest) is kept separate from the transfer.

Where it differs: Mutagen has two ends; this puts a store in the middle, so a third end (another machine, a resumed machine) costs the Mac nothing. Rsync deltas inside a changed chunk are a possible follow-up.

Performance

Same harness for both columns: cargo test -p cmux-tui --test link_sync_store -- --nocapture (a fake platform and a fake machine daemon running the real FsService and BlobFetchService, loopback). Repo: 600 files of 2-24 KiB, a vendored 200 KiB file twice, a 24 MiB asset, committed. "Bytes from the Mac" is bytes over the link plus bytes uploaded to the store. Run on the Linux pool (fast checks).

metric	before (direct link copy)	after (store)	delta	harness
first sync, bytes from the Mac	89,610,640 B	66,955,373 B (134,865 link + 66,820,508 store)	-25.3%	link_sync_store, run 41809330696
second machine, bytes from the Mac	89,610,640 B	235,291 B (134,865 link + 100,426 store)	-99.7%	link_sync_store, run 41809330696
one-file edit, bytes from the Mac	131,459 B	131,123 B (over the link: under 1 MiB)	-0.3%	link_sync_store, run 41809330696
1 byte edited in a 24 MiB file, bytes from the Mac	33,682,014 B	3,115,369 B (126,805 link + 2,988,564 store)	-90.8%	link_sync_store, run 41809330696

Wall time on loopback, where bandwidth is free, is about the same: first sync 1.08 s before and 1.17 s after, second machine 1.07 s and 1.09 s. On a real link, the bytes are what it pays for. The link bytes in each store push are the folder listings and the manifest. Before the 1 MiB floor, a one-file edit took 401 ms through the store and 103 ms over the link, so small writes stay on the link. Numbers are measured on this branch merged with feat-cmux-next, which sends git loose objects as one pack (#34537).

Tests

• blob_fetch unit tests: a chunked plan written with its permissions, each file's revision from its commit, and a rerun after a machine-side edit; a blob that does not match its hash fails and writes nothing; a path outside the roots is named and not written; a plan file outside the job's folder fails the plan; bad origins, grants and targets refused before any fetch; starts past the running cap refused; a blob body past its size cut off; a symlink at the path refused; a job held mid-fetch writes nothing after a newer start or a cancel under its folder; a cancel names every job kept there until it is forgotten; only the four ops are frames.
• sync_cli::store unit tests: chunks cover the file within bounds; an edit in the middle changes at most two pieces; the same bytes always cut the same way.
• link_sync_store: the end-to-end test above, plus a store that refuses the credential still lands every file over the link, and a lost blob.fetch.start answer (with the link) fails the push as unsettled, then the next push fences the job and lands the whole tree. A machine that advertises the pre-release blob-fetch-v1 (no cancel) gets a plain link copy, with or without --store. A push whose fetch a newer push cancelled stops without a link copy. A file only a failed push's fetch wrote is deleted by the next push whose tree drops it.
• The link gate tests admit the four ops and refuse an unknown blob.fetch.* one.

Review findings fixed in this PR: #34788 (each written file's revision comes from its commit), #34789 (no direct link fallback while the machine's fetch may still be running), #34790 (blob bodies are cut off past their size). Also fixed from review: at most 4 live fetch workers (blob.busy), a cancelled one counting until its thread ends; a symlink at a target path is refused; and a fence: every job names its folder (target), a start and the new blob.fetch.cancel cancel each running job under an overlapping folder and answer only once none of them can commit (a commit and its cancelled check share one lock), and every push to a machine serving blob-fetch-v2 fences first. The capability is blob-fetch-v2: pre-release builds of this branch advertised blob-fetch-v1 without cancel, and a Mac treats those machines as serving no store. So a retry with a changed plan, a push under the 1 MiB floor, or a link fallback after a lost start answer is never overwritten by an older job. A start answer that is lost stops the job, or fails the push when the stop cannot be confirmed. A push whose fetch was cancelled by a newer push stops (blob.superseded) instead of copying over the link. What a cancelled or failed fetch committed is in no manifest, so the cancel answers every job kept under the folder, the next push reads what each wrote and counts those files as its own at that revision (deleting any its tree dropped), and it forgets the jobs once its manifest is saved (blob.fetch.forget). Still open as follow-ups: #34791, #34861.

Risk

Medium. New daemon ops behind a capability and the existing fs gate; a new Mac-side path behind --store. Every failure falls back to the existing link copy. Only NEXT uses this code.

Why this is one PR (about 1,900 lines with tests): the machine op and the Mac client are two ends of one wire format (the plan), and the end-to-end test that measures it needs both.

Follow-ups: pack small files into one blob; a machine-side chunk cache; per-user quota and GC (cmux-platform#751); presigned direct-to-R2 uploads.

Changelog

Added: Folder sync to a Cloud machine can go through your sync store: only content the store lacks is uploaded, and a second machine or a resumed sync moves almost nothing from your Mac.

Agent session: agent session

🤖 Generated with Claude Code

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review b043228: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr34604 -f ref=feat/sync-blob-store -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4188039004101 (3e464b08ff)
• cmux NEXT nightly 1.0.0-nightly.4190094575701 (036f8aa538)
• cmux NEXT nightly 1.0.0-nightly.4191131679101 (e01bbae5c9)

[blacksmith-sh]

Found 1 test failure on Blacksmith runners:

Failure

Test	View Logs
test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job _names_its_warm_sources)/ test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job _names_its_warm_sources)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Review (codex gpt-6.1-sol xhigh, next-review, since=20261104T022515Z): BLOCK at 7669a36

BLOCK at 7669a36

Visual verdict: no images; Leo tries the build

1. [P1] [blocking] blob_fetch.rs:329 — Completed-job reuse breaks subsequent pushes. For one file above 1 MiB, push content A, then B, then revert and push A. The third push reuses A’s completed job without writing anything. finish_deferred then records B’s revision alongside A’s hash, so the push reports success and future pushes skip the incorrect content. Scope deduplication to a particular push’s retries or revalidate completed results.

2. [P1] [blocking] blob_fetch.rs:496 — An abandoned fetch can overwrite a completed retry. Delay job A’s blob response, terminate its CLI, change the source to B, and complete another push before releasing A. The detached worker survives and its unconditional overwrite commits A over B, potentially after the chat starts editing. Prevent overlapping jobs from committing stale content, including when falling back to direct writes.

3. [P1] [blocking] blob_fetch.rs:554 — The response body bypasses the size limits when Content-Length is absent. An accepted HTTPS store can return an arbitrarily large chunked response; response.bytes() buffers everything before checking size or digest, allowing daemon memory exhaustion. Enforce the requested size while streaming and reject excess bytes immediately.

Rust tests could not run because Rust and Cargo are unavailable.

Follow-ups:

• [P2] [follow-up] CloudStartChat.swift:151 — No after images were supplied; verify the changed Cloud copy flow in the build.

[teamleaderleo]

Review (subagent claude-opus-5-5 medium): SHIP at ba62e65
Visual verdict: unchanged by this diff

The first review, at 7669a36, blocked on one issue: blob.fetch.start returned the earlier finished job for a repeated plan, so a file edited on the machine could be recorded as pushed without being written. Fixed: a repeated plan now reuses a job only while that job is still running, and a test covers the case. The rest was checked: the origin check (redirects off, http only in debug builds, loopback only), grant handling, the Roots sandbox, PendingWrite staging, the digest and size checks, the link-copy fallback, and that the access token stays on the Mac.

Fleet tests at this tree pass: blob_fetch, link_sync_store, sync_cli and link_sync, with no clippy findings in the files this PR changes. The app compile check passed at 7669a36; the Swift files have not changed since.

Follow-ups: filed as issues and linked below.

[teamleaderleo]

Review (codex, router-review): SHIP at ba62e65

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 1. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at ba62e65

Visual verdict: no images; Leo tries the build

• [P2] [follow-up] [category:reliability] [finding:F1] [outcome:pending] fs_wire::try_handle only dispatches blob frames when the raw line contains "blob.fetch.. A valid JSON frame using an escaped command such as {"cmd":"blob.\u0066etch.start"} passes frame_command and the gate, but is never answered and the client waits until timeout. Dispatch should rely on the parsed command rather than this substring guard.

Follow-ups:

• [finding:F1] Fix blob frame dispatch for valid JSON escaped command strings.

[cmux-bot]

cmux-bot approves ba62e65: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (subagent opus medium, correctness): SHIP at 2c52d77

Covers the feat-cmux-next merge resolution (store path routed through the parallel write pipeline of #35149/#35138) and the panic-ratchet indexing fix, on top of Codex gpt-6-astra's SHIP at 6df0a6e. Fleet at this head: link_sync_store 6, sync_cli 33, clippy clean; CI clippy (changed cmux-tui packages) passes. Remaining reds are nightly-guards (release-notary, Guard status), base reds.

Follow-ups: direct_files counts a symlink-skipped file; under-1 MiB store-path writes go one at a time instead of in parallel; progress bytes can step back after a fetch.

[teamleaderleo]

Review (codex gpt-6-astra high, router-review): BLOCK at 2c52d77

BLOCK at 2c52d77

Visual verdict: no images; Leo tries the build

• [P1] [blocking] [category:security] [finding:F1] [outcome:pending] Blob fetch can write through machine symlinks. The check at [blob_fetch.rs:677] only inspects the final path once, then PendingWrite::begin follows the final and every parent symlink; a symlink such as target/src -> target/other or a race after the check redirects the atomic rename into another folder. Direct sync explicitly rejects these paths. Use no-follow descriptor resolution (begin_no_follow) and handle the refusal.

Follow-ups: none

[teamleaderleo]

Review (codex, router-review): SHIP at 6f50c80

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 2. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at 6f50c80

Visual verdict: no images; Leo tries the build

• [P1] [follow-up] [category:reliability] [finding:F1] [outcome:pending] fetch_served converts a successful identify response with missing/malformed capabilities into Some(false) (sync_cli/store.rs:83-86), so the push skips fencing and may write over an older blob-fetch job. Treat unknown capability data as None and fence conservatively.
• [P2] [follow-up] [category:correctness] [finding:F2] [outcome:pending] retire_finished evicts unforgotten jobs after 32 entries (blob_fetch.rs:556-572). If a client dies after a fetch commits but before saving its manifest, enough later abandoned jobs can erase the recovery record; a subsequent push can leave fetched files that its tree dropped.

Follow-ups:

• F1: fence when identify lacks a valid capability list.
• F2: retain or persist recovery metadata until abandoned jobs are safely reconciled.

[cmux-bot]

cmux-bot approves 6f50c80: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[teamleaderleo]

Review (codex, router-review): SHIP at b043228

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 2. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at b043228

Visual verdict: no images; Leo tries the build

• [P2] [follow-up] [category:correctness] [finding:F1] [outcome:pending] start-chat copy --store still calls seed_cli::push (start_chat_cli.rs:877), which returns immediately for a successful Git seed (seed_cli.rs:332). Therefore GitHub-backed copies silently bypass the new store path.

• [P1] [follow-up] [category:reliability] [finding:F2] [outcome:pending] When identify reports no blob-fetch-v2, the push skips fencing (sync_cli.rs:813). A pre-release blob-fetch-v1 job may still be running and can commit stale files after this direct link push, overwriting it; the fallback needs to wait for or otherwise neutralize such jobs.

Follow-ups:

• F1: Make start-chat copy --store explicitly select the store path, or document and test seed precedence.
• F2: Safely handle active v1 jobs before direct fallback.

[cmux-bot]

cmux-bot approves b043228: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[lawrencecchen]

Hi Leo. This PR (merged directly) made two tip reds on feat-cmux-next:

1. arch-ratchet: [app_lines] CmuxNextApp Swift lines: +3. The store-copy rule and its token read were added to CmuxNextApp/Cloud/CloudStartChat.swift. base red: arch app_lines +3 from #34604 #35695 moves them to CmuxNextCloud: CloudStartCore().copiesThroughStore(account:productionAuth:), with a test, and CloudAuth.startChatStoreAccess(account:). Both App sites now call it, including the same expression in CloudStartChat+Machine.swift. Behavior does not change. Please put new Cloud start rules in CmuxNextCloud and not in CmuxNextApp. The arch ratchet has no margin.
2. A link error in compile job 71199dd4810df1fc88e1f2a6: an old startChatCopy symbol from a stale CloudBringBackTests.swift.o. This is a fleet build-cache defect, not your code. A rerun on the same commit (e2a717de3840d84d4ba387cd) passed.