[teamleaderleo]

Leo:

The last build took 37.5 minutes end to end: 18.8 minutes compiling and 15 minutes signing and publishing.

is it as warmed/sharded/skipped of redundancies as possible tho? and also the signing/publishing don't require 30 cores

This PR is the warm half. The signing/publishing cores are another PR.

Leo-ask: ask-217810e1b3

What was wrong

build-nightly-app takes the warm path (kept DerivedData in the Mac's host build root, no R2 restores) only when
nightly_build_state.py prepare nightly-next writes a mode. For a *-glaeda-nightly-only-N runner it did so
only with NIGHTLY_ONLY_BUILDER=true, which holds only when pick-nightly-build-runner read the runners and
offered glaeda-nightly-only-m4pro-builder. That read (GET /actions/runners with the routing App) got HTTP 403
(rate limit) in most recent runs, so the picker fell back to the plain glaeda-nightly-only-m4pro label, mode
stayed empty, and the job restored the compilation cache and the Swift packages from R2 and compiled colder.
Four of the last six nightly-next runs went that way.

run	runner	mode	Restore Xcode compilation cache	Cache Swift packages	Build nightly app (Release)	job
41570134320	cmux7s-mac-mini-glaeda-nightly-only-1 (builder label)	incremental	skipped	skipped	562 s	1071 s
41586556429	cmux7s-mac-mini-glaeda-nightly-only-5 (plain label)	none	83 s	185 s	413 s	1064 s

Across the six runs, cold builds took 413-1567 s in the build step, plus about 270-320 s of restores.

The fix

dedicated() now covers every nightly-only runner, without the builder flag, and NIGHTLY_ONLY_BUILDER is
gone from the workflow and from nightly_build_state.py. The picker still offers the builder label when it
can read the runners; it only stops deciding whether the build is warm.

Why concurrent slots on one mini are safe:

• The Release build takes the Mac's one host build root (app-host-build-root.sh,
  /Users/Shared/cmux-build-fleet/xcode/nightly-next) under a flock that is never waited for
  (host_build_lock.py); fd 9 is inherited by xcodebuild, so the lock lives as long as the build does.
• A slot that finds the lock held builds in its own kept tree,
  ~/Library/Caches/cmux-nightly-next/<runner>/<sha256(workspace, toolchain)>. That tree is keyed by the runner
  name, so no two slots share one, and its own incremental.json checkpoint keeps the same clean rules.
  The slot's next host-root build deletes it (finish_host), so a mini does not keep one DerivedData per slot.
• mark-built only marks a checkpoint under the slot's own runner directory, and only when prepare linked
  build-universal there.
• The workflow's mode conditions are unchanged: with a mode, the R2 restores, the package sanitize and the
  cache save are skipped, and mark-built runs (tests/test_nightly_trusted_routing.py passes unchanged).

Disk: each nightly-only slot now also keeps its notices crate cache (bounded at 8 GiB by notices_cache), and
a fallback build leaves a slot's DerivedData until that slot's next host-root build.

Design surface: CI only, no UI.

Performance

metric	before	after	delta	harness or run link
R2 restores in build-nightly-app (compilation cache + Swift packages), plain-label run	268 s (83 + 185, run 41586556429)	0 s (skipped, as in builder-label run 41570134320)	-268 s	job step timings, nightly.yml build-nightly-app
nightly-next runs that built cold because the runner read failed	4 of 6	0 expected	-4	last six nightly-next runs
Build nightly app (Release), cold vs warm	413-1567 s cold	~550 s warm	varies by tip	same runs

The "after" column is the measured builder-label run, not a run of this head: the first nightly after the land
gives the real number.

Observed / Not run

Observed:

• tests/test_nightly_build_state.py (30 tests, the two new ones fail on base and pass here),
  tests/test_nightly_build_runner.py, tests/test_nightly_trusted_routing.py, tests/test_ci_warm_sources.py,
  tests/test_nightly_dedicated_mini.py, tests/test_workflow_file_size.py, tests/test_ci_runs_on_fallback.py
  pass locally.
• scripts/cmux-next/prepush-gate.py: no new violations.

Not run:

• A nightly at this head. The next nightly-next run after the land is the check: its
  "Start Release DerivedData empty" log should print a clean or incremental mode on any nightly-only runner.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Fixes nightly-only nightly-next builds running cold after the runner picker failed to read the runners.

The picker's runner read returned HTTP 403 in four of the last six nightly-next runs, so those jobs took the plain glaeda-nightly-only-m4pro label, restored the compilation cache and Swift packages from R2 (~270–320 s), and compiled from empty DerivedData. Every *-glaeda-nightly-only-K runner now keeps warm state (dedicated() covers all of them), and NIGHTLY_ONLY_BUILDER is removed from the workflow, nightly_build_state.py, and tests.

Concurrent slots on one Mac stay apart:

• The Release build takes the Mac's single host build root under a never-waited flock.
• A slot that finds the lock held builds in its own kept tree keyed by its runner name; that tree is dropped by the slot's next host-root build.
• mark-built only writes a checkpoint under the slot's own runner directory.

CI-only; no migration or rollout steps.

^{Written for commit c084316. Summary will update on new commits.}

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review c084316: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr33348 -f ref=lane/nightly-warm-nightly-only-slots -f platform=macos -f backend_mode=local

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4161078434801 (732ad010ea)
• cmux NEXT nightly 1.0.0-nightly.4161217392901 (1a7e2c1865)

[teamleaderleo]

Review (codex gpt-6-astra high, sweep, via the review router; full diff from merge base 52393e1 to this head): SHIP at c084316

No P1/P2 findings.