[teamleaderleo]

"like how fast can we just have it, like hot and incrementally built?... reusing as much as you can" (Leo)

What

#24994 turned on Xcode's compilation cache (CAS) for the cmux NEXT DEV Release compile, but the cache's keys hold the absolute source and DerivedData paths, and both were per runner instance (RUNNER_NAME + workspace, release-derived-data-path.sh). DEV compiles land on 30+ runner instances (cmuxNN-glaeda-K, several per host), so every compile since has been cold: the compile job's median is still 10.8 min over the last 12 DEV publishes (measure_next_build.py --dev-chain --limit 12).

This gives each mini ONE build root for the DEV Release compile, so the keys match whichever instance builds:

• scripts/cmux-next/compile-dev-release.sh (the DEV step now calls it): on a glaeda mini with the cache on, it takes the host lock, clones this checkout to ~/Library/Caches/cmux-next-ci/dev-daily-release/host/src with scripts/ci/canonical-build-root.sh (the fixed-root copy that the owned catch-up and compile admission use: the old tree is removed first, then an APFS clone, with an rsync --delete fallback, .git included, plus the cmux-tui tree and web bundles that earlier steps put in the checkout), checks the clone's HEAD is the checked-out commit, and builds there into one DerivedData per host (release-derived-data-path.sh dev-daily-release Release --host → .../host/Release/xcode-<toolchain>, with the same toolchain pruning and 12 GiB bound). The checkout's acpmux path is remapped into the tree.
• scripts/cmux-next/host_build_lock.py: one build per host, taken without waiting. The lock is flock(2) on host/lock through fd 9, which the wrapper and every child it starts (the clone, xcodebuild, ditto) inherit. The kernel holds the lock until the last of them exits and drops it however they end. So a SIGKILLed wrapper whose xcodebuild still runs keeps the root until that xcodebuild is gone, and no stale lock is ever left behind. There is no pid file to judge (this changed after the first Codex review). The file's content only names the holder, for a busy caller's log.
• Fallback, never a wait: a busy lock, a lock error or a failed tree setup builds right away the old way (per runner instance from the checkout, still with the cache). No job waits on a host, and no host is reserved.
• Unchanged: cache keys stay Xcode CAS content hashes (the path is fixed only so they match), the build still starts from an empty DerivedData that keeps only CompilationCache.noindex and SourcePackages (ci(nightly): start the shipped Release build from empty DerivedData #21761), the ci(cmux-next): DEV Release compile reuses a content-keyed compilation cache, not kept build products #24994 cache-failure retry, the 3 GiB limit and the prune after each build. The switch-off is the same: CMUX_NEXT_DEV_COMPILATION_CACHE=0 also turns off the host root.
• Content-keyed whole-app reuse (ci(cmux-next): key the compiled DEV Release app by its content #24963/ci(cmux-next): reuse the compiled DEV Release app when its inputs did not change #24964/fix(ci): an unset CI_NEXT_DEV_PRODUCT_REUSE leaves DEV product reuse on #25075) runs before this and is untouched. Its key changes once, because this workflow file is part of it. The key stays as it is: the build path is "where", not "what". Before this PR the path already varied per runner and was never part of the key.

Tests

• New tests/test_cmux_next_dev_release_host_root.py (run in the checks job) runs compile-dev-release.sh against stub compilers. It covers:
  • two runner instances build at one host src and DerivedData;
  • the tree is exactly the new commit (a file from an earlier checkout is gone);
  • a build that starts while another holds the root builds per instance at once and leaves that lock in place;
  • a SIGKILLed wrapper whose compile still runs keeps the root until the compile exits, and the next build then takes it;
  • a lock file left behind is not held;
  • a failed setup builds per instance and releases the lock;
  • with the cache off, or off a mini, nothing touches the host root;
  • a failed compile keeps its status and releases the lock;
  • the lock CLI holds one holder at a time, and a child keeps the lock after its shell is killed.
• tests/test_cmux_next_release_build_retention.py runs the DEV step through the new script. Its per-instance assertions are unchanged.

Risk

Medium, CI only, and it touches only the DEV Release compile (nightly and stable are untouched).

• Failure mode: a mis-set-up host tree. It fails closed to the per-instance build, and the step summary says which build root it used and why.
• Disk: one more CoW tree plus one DerivedData per host, bounded by the existing caps. The old per-instance DerivedData stay, for the fallback.

Signals

• Success: two DEV compiles on the same host, back to back, where the second's xcodebuild time is well below the cold time. I'll measure after landing with the same harness and report.
• Failure: the step log or summary shows instance build root (host tree setup failed) on a mini, or a DEV compile on a host that has already built the cache is no faster.

Agent session: cosmic-gray

🤖 Generated with Claude Code

[teamleaderleo]

Owned by claude-sr2/incremental-build on cosmic-gray · session

[blacksmith-sh]

Found 9 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmp8_wvvgdq/summary-missing.json (live):/summary-missing.json (live):	View Logs
/tmp/tmph06g48hr/summary-fail.json (live):/summary-fail.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
bench OTHER_SWIFT_FLAGS is 'OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_BENCH_HOOKS -DCMUX_BL OCKING_CHECKS'/ bench OTHER_SWIFT_FLAGS is 'OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_BENCH_HOOKS -DCMUX_BL OCKING_CHECKS'	View Logs
bench OTHER_SWIFT_FLAGS is 'OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_BENCH_HOOKS -DCMUX_BL OCKING_CHECKS'/ bench OTHER_SWIFT_FLAGS is 'OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_BENCH_HOOKS -DCMUX_BL OCKING_CHECKS'	View Logs
convention violations found (ERROR lines above)./ convention violations found (ERROR lines above).	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

cmux-next guards / nightly-guards / workflow-guard-tests / release-notary is red on 17 of 20 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[teamleaderleo]

Bead cx-z19nv created for this PR by the cmux coordinator.

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 65ec818: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr25147 -f ref=incremental-build-pr5-host-cache -f platform=macos -f backend_mode=local

web and size checks: green

[teamleaderleo]

Review (codex gpt-6-astra xhigh): SHIP at 65ec818

[github-actions]

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40255417638 at ebbefce failed: cmux-next Release compile (fleet fast check).
Those jobs last passed at b14d855. This pull request is one of 3 merged since: #24272, #24289, #25147.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

[teamleaderleo]

base-red-watch: cmux-next / cmux-next fast tier went red on feat-cmux-next in https://github.com/manaflow-ai/cmux/actions/runs/40256330328 (first red at ab42ace, last green at c7c6530). Suspect: #25147 (commit ebbefce). If it is yours, fix forward on feat-cmux-next.

[teamleaderleo]

base-red-watch: cmux-next / cmux-next fast tier went red on feat-cmux-next in https://github.com/manaflow-ai/cmux/actions/runs/40256330328 (first red at ab42ace, last green at c7c6530). Suspect: #25147 (commit ebbefce). If it is yours, fix forward on feat-cmux-next.

[teamleaderleo]

base-red-watch: cmux-next / cmux-next fast tier went red on feat-cmux-next in https://github.com/manaflow-ai/cmux/actions/runs/40256330328 (first red at ab42ace, last green at c7c6530). Suspect: #25147 (commit ebbefce). If it is yours, fix forward on feat-cmux-next.