[teamleaderleo]

Leo, 2026-10-26: "And I thought we were incremental. You keep saying, yeah, we're gonna do the incremental building. Nobody has come back with the incremental building results." Earlier: "like how fast can we just have it, like hot and incrementally built?... reusing as much as you can".

What this does (2 of 2, on top of #25729)

The cmux NEXT Nightly build (nightly.yml build-nightly-app, nightly-next push runs) now reuses its compiled, unsigned app when nothing it is built from has changed. The DEV lane already does this (#24963, #24964). This change shares that code instead of copying it.

• Key. The key is job_input_fingerprint.py nightly-release-product. It uses the DEV product's closure: the whole tree minus docs, tests and other products. It also keys:

  • the whole nightly.yml;
  • every decide output the build reads (identity, icon, arches, browser-import flags, password_readers_out);
  • Xcode, the SDK, the arch, the bundled cmux-tui tree key and a recipe.

  The fingerprint job's name is part of the key, so a nightly key never equals a DEV key.

• Records. Reuse runs only for nightly-next push builds that are not fast, build-only or notary tests. A record is taken only from this repository's own nightly.yml push run on nightly-next whose build-nightly-app finished successfully. The archive's SHA-256 is checked against the record. Records and archives use their own names (cmux-nightly-next-release-product-*, cmux-nightly-next-release-archive-*). A DEV record is never read as a nightly product, and the reverse never happens either. Pull-request runs cannot write any of this.

• On a hit. The app and its dSYMs are unpacked into build-universal/Build/Products/Release. Every step that only feeds xcodebuild is skipped: web bundles, Rust helpers, Chief, mobile host, CEF fetch, notices, xcodebuild, Iroh drop, Computer Use helper, strip. These steps still run on a hit:

  • the password-import proof;
  • the SSH daemon assets (per-run build number);
  • the CEF report;
  • Sentry;
  • the unsigned-app archive and dSYM handoff.

  The version, build number and CMUXCommit are stamped later, in build-sign-notarize-nightly, so the compiled products name no run. Signing, notarization and publication always run fresh.

• Recording. After a successful build, the products are packed and recorded with a 2-day retention. A build missing the Chromium engine or the remote browser host (both optional inputs) is never recorded.

• Misses. Every problem counts as a miss with a reason, and the job compiles as before. That includes:

  • an API error or the deadline (240 s, no gh wait);
  • a digest mismatch;
  • an unsafe tar member;
  • a build commit whose nightly.yml differs from the running one.

  An unpack is staged and renamed into place, so xcodebuild never merges into a partial app. vars.CI_NIGHTLY_NEXT_PRODUCT_REUSE=0 switches the lookup off.

Baseline (before)

Measured with the same harness (measure_next_build.py --workflow nightly.yml --branch nightly-next --event push --limit 10 --steps). The xcodebuild seconds and cache counts come from each job's log.

run	sha	runner	build step (min)	xcodebuild (s)	cache hit/miss	commits since previous	Swift files
40380339203	8b3b189	cmuxs-mac-mini-5-glaeda-3	13.0	779	364/136	1	0
40378565252	c2276c4	cmux8s-mac-mini-glaeda-2	9.9	596	364/136	6	7
40372260966	1b4a676	cmux14-glaeda-2	11.2	672	364/136	3	0
40370780830	3ce8dfd	cmux12s-mac-mini-glaeda-3	8.1	488	415/85	2	0
40370046046	a9d1434	cmux10s-mac-mini-glaeda-5	8.7	524	415/85	14	13
40357982361	3660113	cmuxs-mac-mini-3-glaeda-5	8.7	524	447/53	11	13
40351269110	61ea34b	cmux9s-mac-mini-glaeda-5	10.3	621	415/85	428	579
40203748160	c0cb858	cmux14-glaeda-5	7.7	461	366/134	16	49
40197613419	8f6a3f3	cmuxs-mac-mini-6-glaeda-5	7.8	468	404/96	30	14
40197090763	09a881f	cmux13s-mac-mini-glaeda-2	8.6	514	0/500	-	-

• Median build step: 8.7 min. The build-nightly-app job takes 15.6-28.2 min. Landing to publish: median 88.8 min.
• Every build is WMO Release, arm64, on a different owned runner instance, and all 134 Swift module compiles run every time.

Risk

Medium. The change touches the shipped nightly's build path, but it is all-or-nothing and content-keyed:

• On a miss (the default until a record exists), every step runs exactly as before.
• On a hit, the shipped bytes are the app that an earlier successful nightly-next build compiled from identical inputs, after the same post-processing.
• Main's nightly and RC builds are untouched, because the lookup runs only for track nightly-next.
• A wrong hit would ship an app compiled from other inputs. The tests check the key's coverage (each decide flag, the toolchain, nightly.yml, app sources) and that the skipped steps read no decide output or repository variable outside the key.

Signals

• Success: a nightly-next run's step summary says nightly-next Release product: **hit** and the build step is skipped. Its later jobs sign, notarize and publish as usual.
• Failure: a run whose lookup says hit but whose later checks fail. To switch reuse off, set vars.CI_NIGHTLY_NEXT_PRODUCT_REUSE=0.

Tests

• tests/test_nightly_release_product_reuse.py (new) covers:
  • the pack/unpack round trip (symlinks, hard links, dSYMs);
  • tar safety: .. and absolute paths, symlink chains, case-insensitive aliasing, members under a link, duplicates, oversized PAX headers, member and byte bounds;
  • a build missing CEF is never recorded;
  • the running-workflow check, and a miss clearing the products directory;
  • workflow wiring: every decide flag reaches the key, skipped steps read nothing unkeyed, and only nightly.yml uses the lane.
• Key and namespace tests are in the first PR (tests/test_nightly_release_product_key.py).
• Updated tests/test_gh_budget_wait.py (the lookup makes no gh call) and .github/nightly-step-coverage.json.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Owned by an agent session · session

[teamleaderleo]

Review (Codex gpt-6-astra xhigh, independent): SHIP at 3d69dd5

The final round on both stacked heads (#25729 at 4c16c22, this PR at 3d69dd5): SHIP, no findings.

[teamleaderleo]

Bead cx-axgd6 created for this PR by the cmux coordinator.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (Codex gpt-6-astra xhigh, independent): SHIP at fc99f07

Re-review after the rebase onto feat-cmux-next (background helper builds, the Linux-built inputs, the early Chromium fetch). No blocking findings: helper starts and joins are gated together, ungated steps need no skipped outputs, packing follows helper and notices bundling and the strip, and PR runs cannot reach the record lookup or upload.

[blacksmith-sh]

Found 8 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmp9_16e4qq/summary-missing.json (live):/summary-missing.json (live):	View Logs
/tmp/tmpkrmnkmm5/summary-fail.json (live):/summary-fail.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
test_the_nightly_is_covered (main.NightlyStepCoverageTests.test_the_nightly_is_cove red)/ test_the_nightly_is_covered (main.NightlyStepCoverageTests.test_the_nightly_is_cove red)	View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl ights_validate)/ test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl ights_validate)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review d870732: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr25731 -f ref=incremental-build-pr8b-nightly-reuse -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 7c2a29c to compare; do not merge until green; see lane-rules

[teamleaderleo]

Review (Codex gpt-6-astra xhigh, independent): SHIP at d870732

After three more rounds on the archive checks: they now fold names as APFS does (letter case and Unicode normalization), walk each symlink by exact and by folded name, and refuse collisions before extraction. The Linux CI failure of test_nightly_release_product_reuse.py (case-sensitive volume) is fixed, and the test passes on macOS and on Linux. Codex found no concrete escape or overwrite.

[teamleaderleo]

cmux-next guards / nightly-guards / workflow-guard-tests / release-notary is red on 11 of 18 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[teamleaderleo]

cmux-next / cmux-next checks (god files, concurrency, crash safety, l10n) › Checks job structure is red on 13 of 19 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[github-actions]

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4045227375101 (53f8928b83)
• cmux NEXT nightly 1.0.0-nightly.4045248345901 (ab883bf956)
• cmux NEXT nightly 1.0.0-nightly.4045402417302 (9b898d61ab)
• cmux NEXT nightly 1.0.0-nightly.4045474536502 (a6e613294e)
• cmux NEXT nightly 1.0.0-nightly.4048747891701 (2c7dda4036)
• cmux NEXT nightly 1.0.0-nightly.4048947256901 (273534101b)
• cmux NEXT nightly 1.0.0-nightly.4049353932801 (e7cbb89fab)