[teamleaderleo]

What

build-nightly-app on nightly-next: both dedicated build slots on each of the nightly Macs now build from one fixed root per Mac instead of each slot's own workspace, and the nightly-next compile cache limit goes from 3 GiB to 10 GiB when its volume has room.

All logic is in scripts; nightly.yml is unchanged (still under its size cap).

Why (measured on 20 nightly-next runs, 41455121554 to 41485637444)

• The cache keys differed per slot. Xcode's compile cache keys hold the absolute source, DerivedData and package-checkout paths. Each slot built in its own workspace into its own kept DerivedData and SourcePackages, so the same module (Atomics) had 4 keys, one per slot. Prefix mapping on Xcode 26.6 doesn't remove that.
• The slots pushed each other's entries out of a small cache. All slots on a Mac share one CAS, capped at COMPILATION_CACHE_LIMIT_SIZE=3 GiB. The CAS starts a new generation whenever a build ends above half the limit (1.5 GiB), and keeps only one generation behind it. One build is about 3.3 GB, so every build rotates, and two builds by other slots (other keys) empty a slot's entries. Each slot builds only about every 4th tip, so its entries were nearly always gone. Every run compiled all ~138 Swift module tasks (SwiftCompile (138 tasks) | 1600 s CPU in run 41485637444). The cache replayed only about 311 tasks, the same set every run.
• "clean" was almost always chosen. 18 of 19 runs chose mode=clean, because Package.swift or an import or #if line changed since that slot's last build. That build was 3-4 promotions back, so Package.swift (FFI re-pins, new targets: 12 changes this morning) had nearly always changed.

Change

• scripts/ci/nightly-steps/app-host-build-root.sh (sourced by app-build-release.sh): on a slot that keeps build state (the dedicated nightly-next builder), it takes flock on the root's lock and never waits for it (scripts/cmux-next/host_build_lock.py, as the DEV and PR release compiles already do). Then:
  • It clones the checkout to ROOT/src (canonical-build-root.sh). Before that, it fetches the root's last-built commit into the checkout, so the clean rules can diff against it even when the other slot built it.
  • It builds into ROOT/DerivedData, with packages in ROOT/SourcePackages (seeded once from the slot's own) and XCBUILD_FILE_SYSTEM_MODE=checksum-only.
  • The root path is the same on every nightly Mac, so keys also match across Macs.
• nightly_build_state.py prepare-host / finish-host: the same clean rules as before (no rule relaxed), judged against the Mac's last build. On success, while the lock is held, it marks the build done and clones the products into the workspace's build-universal. Later steps change the app bundle there, never in the root. The slot's own kept DerivedData is dropped, so there is one DerivedData per Mac instead of two.
• Fallback: a held lock, a lock error or a failed setup builds on the slot's own state exactly as before, at once. A cold-measurement run (cold_cache) never takes the root.
• Compile cache limit: 10 GiB for nightly-next when the CAS volume has at least 60 GiB free; otherwise 3 GiB, as before. Other tracks keep 3 GiB. 10 GiB means a generation rotates at 5 GiB, so about three builds' entries survive. That stays under glaeda's hourly prune, which empties a local CAS past 20 GiB. app-bound-compilation-cache.sh bounds a kept (linked) cache at 12 GiB instead of 5, so a full one isn't deleted.

Is a full clean needed on a Package.swift or import change?

Yes; I kept that rule. Those are the changes behind the audited stale autolink and force-load objects. A clean build is only slow when the compile cache misses. An empty DerivedData plus a warm cache whose keys match replays every unchanged task: 102 s with 4,192 of 4,192 hits, against 600 s at another path. So the fix is matching keys and a cache big enough to keep them, not relaxing the clean rules. Per Mac (every 2nd tip instead of every 4th), the diff window behind each build is also about half as long, so more builds stay incremental.

Shared compile cache: 0 hits (finding, not fixed here)

The nightly reads the fleet cache node (COMPILATION_CACHE_REMOTE_SERVICE_PATH is set; the build uploads outputs). It can't hit, because the only trusted writer fills a different build. The writer's catch-up fill compiles the feat-cmux-next tip's Release compile from the next-release root, with no nightly defines. The nightly builds:

• from other paths (this PR moves it to a fixed nightly-next root, the same on both nightly Macs);
• with OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_NO_PASSWORD_IMPORT -DCMUX_NO_BROWSER_DATA_IMPORT -DCMUX_BLOCKING_CHECKS, which apply to every Swift target, packages included;
• with AppIcon-Nightly and an emptied browser-sources.json.

Every Swift key differs from the writer's. Getting hits takes a writer fill of the nightly-next recipe at this PR's root path; that is a follow-up on the fleet side. Separately, other jobs that share a Mac's CAS with a smaller limit can still rotate it early.

Tests

• tests/test_nightly_build_state.py: new HostBuildRoot cases (two slot checkouts, real scripts, stubbed Xcode):

  • both slots share one root, and slot 2 builds incrementally on slot 1's DerivedData although its own checkout never held slot 1's commit;
  • a structure change since the Mac's last build is clean;
  • an unfinished build makes the next one clean;
  • a held lock falls back to the slot's own tree;
  • the root is off unless the slot keeps build state.

  30/30 pass.

• tests/test_ci_fleet_compilation_cache.py 9/9 and tests/test_ci_fleet_compilation_cache.sh pass.

• tests/test_workflow_file_size.py passes.

• shellcheck is clean.

• tests/test_nightly_universal_build.sh fails the same way on base ("the short nightly decide gate …").

Performance

Before: the 20 runs above, Build step (Build nightly app (Release)) only. After: to be filled in from the first nightly-next runs once this is on nightly-next. Branch dispatches don't reach the nightly Macs, because placement there requires refs/heads/nightly-next.

Metric	Before	After	Delta	Harness or run link
Build step s, nightly Mac A (median of 9)	587 (383 to 634)	pending		41485637444, 41457449250
Build step s, nightly Mac B (median of 10)	583 (304 to 1304)	pending		41480007529, 41471126151
mode, Mac A	9 clean / 0 incremental	pending		same runs
mode, Mac B	9 clean / 1 incremental	pending		same runs
cache replay hits / local misses per build (both Macs)	about 311 / 45-165 (all ~138 Swift module compiles run)	pending		job logs, note: replay cache hit / local cache miss
Blacksmith minutes per day	0	0	0	no routing change

Warm sources: the nightly Mac's kept DerivedData and package checkouts (now one per Mac), and the local compile cache on the Mac (10 GiB). The fleet store is read but can't hit yet (above).

Agent session: agent session

🤖 Generated with Claude Code

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Nightly-next builds no longer recompile all ~138 Swift module tasks every run: both dedicated build slots on each Mac now build from one fixed root per Mac with a 10 GiB compile cache, so Xcode's cache keys match across slots and survive between runs.

Details

• Each slot used to build in its own workspace with its own kept DerivedData and packages; cache keys embed those absolute paths, so one module had a different key per slot, the 3 GiB cache rotated on every build (one build is about 3.3 GB), and each slot compared against its own last build several tips back — 18 of 19 runs were clean builds.
• The Release build step now locks a per-Mac root with a flock it never waits for, clones the checkout into ROOT/src, and builds into shared ROOT/DerivedData with packages in ROOT/SourcePackages, applying the same clean rules against the Mac's last build. The root path is identical on both nightly Macs, so keys match across them too.
• A held lock, lock error, or failed setup falls back to the slot's own state as before; cold-measurement runs never take the root. The cache limit is 10 GiB when the CAS volume has at least 60 GiB free, else 3 GiB, and a kept linked cache is bounded at 12 GiB instead of 5.
• The fleet compile cache still can't hit: the only trusted writer fills a different build recipe, so hits from the shared store are a fleet-side follow-up, not fixed here.

^{Written for commit b30606e. Summary will update on new commits.}

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 9c61686: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr32514 -f ref=lane/nightly-warm-build-path -f platform=macos -f backend_mode=local

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4150738990701 (069f8c0811)
• cmux NEXT nightly 1.0.0-nightly.4150891005901 (2fdcbbae7e)
• cmux NEXT nightly 1.0.0-nightly.4150807093801 (069f8c0811)
• cmux NEXT nightly 1.0.0-nightly.4151067571301 (546cb0bf70)
• cmux NEXT nightly 1.0.0-nightly.4151196080201 (8d6a5624a6)
• cmux NEXT nightly 1.0.0-nightly.4151284660301 (22fd89a839)
• cmux NEXT nightly 1.0.0-nightly.4151418814301 (b899f6b4f0)

[blacksmith-sh]

Found 2 test failures on Blacksmith runners:

Failures

Test	View Logs
test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job _names_its_warm_sources)/ test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job _names_its_warm_sources)	View Logs
the short nightly decide gate must use the pooled Linux route with a Blacksmith fallbac k/ the short nightly decide gate must use the pooled Linux route with a Blacksmith fallbac k	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Review (codex, router-review): SHIP at 9c61686

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

Follow-ups: 2. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at 9c61686

Visual verdict: no images; Leo tries the build

Focused nightly state and fleet-cache tests pass (39 tests).

1. [P1][follow-up] app-build-release.sh:95 launches run-with-helper-prebuild.py, whose Popen calls do not pass fd 9. A forced cancellation can release the host lock while the detached Xcode build still writes the shared root, allowing the next slot to race it. Pass the lock fd through all build descendants or avoid detaching them.

2. [P2][follow-up] app-bound-compilation-cache.sh:5-11 only inspects build-universal/CompilationCache.noindex. After finish-host, that path is replaced by a product-only copy while the kept cache remains at HOST_ROOT/CompilationCache, so the new 12 GiB bound never runs for host-root builds.

Follow-ups:

• [P1] Preserve the host lock through cancellation and detached build descendants.
• [P2] Bound the host root’s kept compilation cache.

[cmux-bot]

cmux-bot approves 9c61686: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[teamleaderleo]

Review (codex gpt-6-astra high, sweep, via the review router; full diff from merge base 8db9104 to this head): SHIP at 9c61686

[P2] [follow-up] scripts/ci/nightly_build_state.py:459 - finish_host replaces the workspace’s build-universal symlink with a products-only directory, so the later cache-bound step cannot prune the host CAS; it also leaves each slot’s orphaned CompilationCache, allowing stale generations to accumulate and eventually exhaust the runner disk - preserve a cache symlink for pruning or run the bound step against the host root, and remove slot CAS directories with their DerivedData.

[teamleaderleo]

Review (codex gpt-6.1-sol xhigh, carried): SHIP at b30606e

This is a carry of the SHIP at 9c61686 (#32514 (comment)). b30606e is cmux-bot's clean base merge with no hand resolution (git show --cc is empty), and the PR's diff against base is the same 6 files, +360/-12.

Follow-ups (to be filed): [P1] keep the host lock held through cancellation and detached build descendants; [P2] bound the host root's kept compilation cache.

[teamleaderleo]

Performance after merge (first nightly-next builds at the per-host build root):

metric	host	before	after	delta	run
Build nightly app (Release) step	nightly Mac A	610 s	347 s	−263 s (−43%)	before 41485637444, after 41508070938 (job 125496716000)
Compile mode	nightly Mac A	clean, in a per-slot DerivedData	clean ("no earlier build on this builder"), in the host build root	first build at the root	same
Compile-cache replay hits / misses	nightly Mac A	349 / 167	300 / 13	misses −154	same
Compile-cache limit	nightly Mac A	3 GiB	10 GiB (107 GB free on the cache volume)	+7 GiB	same
Build nightly app (Release) step	nightly Mac B	617 s	pending (next run on that host)		before 41484155495

This was still a clean build, the first on the new root, but nearly every cached task now replays. The second build on the same host should be incremental whenever no Package.swift or import change lands in between. I'll fill in the Mac B row from the next nightly that builds there.

🤖 Generated with Claude Code