[teamleaderleo]

Leo, 2026-11-03: "is it as warmed/sharded/skipped of redundancies as possible tho?"

In reference run 41570134320 (attempt 1), build-nightly-app took 17.9 min. The Release compile was 9.4 min of that. Three costs ran in series around it, and none of them depends on the compile. A sample of 15 nightly-next runs from 2026-11-02/03 shows how often each one happens.

Changes

1. Web bundles come from R2 when the hand-off is missing.
   • What happened: build-nightly-inputs waited up to 9.9 min for a runner. When it was late, the app built the bundles itself, 36-112 s, in 4 of 15 runs (106 s in the reference run).
   • Change: the app job now restores the bundles that feat-cmux-next pushes publish to R2 under the same source key. This is the restore build-nightly-inputs already does (8 s in the reference run), followed by build-web-bundles.sh --from.
   • Order: the hand-off artifact first, then R2, then a local build.
2. The cmux-tui client downloads during the compile while the tree is still publishing.
   • What happened: the prefetch added in ci(nightly): take the cmux-tui download and the light-job placement wait off build-nightly-app's tail #32501 runs only when the tree is already published. The tree was pending in 10 of 15 runs. In those runs "Bundle the cmux-tui client" took 45-139 s, against 15-33 s when the prefetch ran (113 s in the reference run, 78 s of it downloading).
   • Change: a new step, app-prefetch-pending-cmux-tui.sh, runs the same await-tree-client.sh wait and the same install into the client cache in the background.
   • The bundle step still resolves the commit and installs it itself. It joins the prefetch (180 s while pending, since the prefetch is at most one 20 s poll behind). The install reuses a cached slice only when its sha256 matches the attested manifest. A failed prefetch is stopped and its cache is dropped, as before.
3. A change under a package's Tests/ tree no longer wipes the kept DerivedData.
   • What happened: the reference run compiled clean because an import line changed in Packages/Shared/CmuxMessagesLab/Tests/.../HomeAgentMessageAvatarTests.swift.
   • Change: Packages/*/*/Tests/ holds test targets and support targets that only tests link. The app scheme's Release build compiles none of them, so nightly_build_state.py now skips that tree for the import and structure rules.
   • New check: test_package_tests_trees_feed_only_test_targets reads every Package.swift and fails if a product, or a non-test target outside Tests/, depends on a target there. A local mutation confirmed it catches that case.
   • Measured effect: none. This happened once in the sample (the reference run). Incremental and clean compiles had similar medians (about 731 s vs 664 s); compile time followed the runner and its compilation-cache misses more than the mode. So this only removes a wrong clean build. It does not shorten the compile.

Whether the compile was warm: in the reference run the compilation cache was warm (434 hits, 162 misses), with kept package checkouts and the host build root. Only DerivedData started clean, for the test-file reason in item 3. The other clean builds in the sample had legitimate reasons: a Package.swift change, an import change in Sources/, or a first build on a slot.

Routing, admission and runner labels are unchanged; no job or step names a runner. nightly.yml is 489,211 bytes, under the 490,000 cap. The compile step's text is unchanged, so the incremental-state key is the same.

Performance

These numbers are a projection from the step breakdown, not from a run after this change; the first nightly-next run after it lands will give the real number.

metric	before	after	delta	harness or run link
build-nightly-app wall, reference run	1071 s (17.9 min)	~885 s (~14.7 min), projected	about -186 s	run 41570134320 attempt 1, per-step times
"Build the web bundles (cmux-next)" when the hand-off is late (4/15 runs)	36-112 s (106 s in ref)	~10 s (R2 restore 8 s + install)	-26 to -102 s	15 nightly-next runs, 2026-11-02/03
"Bundle the cmux-tui client" with the tree pending (10/15 runs)	45-139 s, median 51 s (113 s in ref)	15-33 s (what the prefetched case takes)	about -35 s median, -93 s in ref	same 15 runs
Release compile	562 s in ref (clean on a test-file import)	unchanged or incremental	not claimed	see item 3

Tests

Run with system python3:

• Passed: test_workflow_file_size, test_nightly_step_coverage, test_ci_warm_sources, test_nightly_build_runner, test_ci_nightly_off_blacksmith, test_nightly_dedicated_mini, test_ci_runs_on_fallback, test_supplychain_security, test_ci_self_hosted_guard.sh, test_nightly_build_state (32), test_nightly_helpers_beside_release_build, test_nightly_next_web_bundles_order, test_nightly_chief_cache_repair, test_nightly_background_steps, test_ci_owned_build_state and the other nightly tests that name these scripts.
• Fail the same way on base: test_nightly_next_arm64_only.py (it needs lipo) and test_nightly_universal_build.sh (its "nightly preparation must run on Blacksmith Linux" assertion).

Prior art: #32501 (the companion prefetch this extends), #33348 and #32514 (warm host build root), #32455 (Sentry off the tail).

Agent session: agent session

🤖 Generated with Claude Code

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 3834a9f: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr33544 -f ref=lane/nightly-app-parallel -f platform=macos -f backend_mode=local

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4166004947701 (abcde7ea04)
• cmux NEXT nightly 1.0.0-nightly.4166228559901 (03b4a57138)
• cmux NEXT nightly 1.0.0-nightly.4166378385701 (b3553220e0)
• cmux NEXT nightly 1.0.0-nightly.4166527817101 (72aa23abec)

[teamleaderleo]

Review (codex gpt-6.1-sol xhigh): SHIP at 24a34ef

Review (codex gpt-6.1-sol xhigh, router-review job 13d67fff555c035e3cc16a479c8acdf6): SHIP at 24a34ef

Visual verdict: no images; Leo tries the build

1. [P2] [follow-up] .github/workflows/nightly.yml:2259: the R2 restore runs before the hand-off artifact is tried, so when the hand-off installs, the R2 download is thrown away. Try the hand-off first and gate the R2 restore on its failure.

Follow-ups:

• #33568 Skip the R2 restore when the hand-off installs successfully.

(This carries the router-review receipt: its queued post was dropped.)

[blacksmith-sh]

Found 1 test failure on Blacksmith runners:

Failure

Test	View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/ test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Review (codex, router-review): SHIP at 24a34ef

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

Follow-ups: 1. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at 24a34ef
Visual verdict: no images; Leo tries the build

1. [P2] [follow-up] .github/workflows/nightly.yml:2259: R2 restoration runs before checking the hand-off artifact. When the hand-off succeeds, the R2 download is discarded, adding unnecessary work to the app’s critical path. Attempt the hand-off first and gate R2 restoration on its failure.

Follow-ups:

• Skip the R2 restore when the hand-off installs successfully.

[teamleaderleo]

Review (codex gpt-6.1-sol high, sweep, via the review router; full diff from merge base 58ec39f to this head): SHIP at 4ff6d01

No P1/P2 findings.

[teamleaderleo]

Review (codex, router-review): SHIP at 3834a9f

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

Follow-ups: 1. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at 3834a9f
Visual verdict: no images; Leo tries the build

1. [P2] [follow-up] nightly.yml:2259: Restore from R2 only after the hand-off misses validation. Currently every app build reaches the R2 download before checking the artifact, even when that artifact is ready. The successful hand-off then discards the restored bundles, adding roughly 8 seconds to the critical path using the PR’s measurement; slow R2 responses delay it further. Download and validate the hand-off first, then conditionally restore from R2.

Follow-ups:

• Skip the R2 restore when the hand-off is usable.

[teamleaderleo]

Review (codex gpt-6-astra high, sweep, via the review router; full diff from merge base e491514 to this head): SHIP at c83fc91

No P1/P2 findings.