[teamleaderleo]

Leo: "it is unacceptable to have nightlies >30min out of date."

What is slow

When a landing changes the cmux-tui tree key, its NEXT nightly could not start until that tree was compiled, published and marked. Measured on today's feat-cmux-next pushes (cmux-tui artifacts runs 40913738554, 40918933039 and 40921878495):

step after the landing	ends at
the four build / build <target> legs	13-15 min
publish to R2 (commit-addressed, attested manifest)	21-22 min
publish tree-addressed cmux-tui	26 min
tree-commit marker	27-28 min

The nightly itself then takes 24-40 min (median about 30; nightly.yml runs 40868602625 to 40920248555). So these landings published 52-58 min after they landed. About a quarter of landings change the key.

The app does not need the tree to compile. In a Release build, "Bundle cmux-tui" is in pin mode, and "Bundle acpmux" only makes bin/acpmux an alias of bin/cmux. The tree's client is installed after xcodebuild, in "Bundle the cmux-tui client". On the dedicated builder that compile takes a median 2.1 min.

What changes

Main's promote guard already accepts such a commit when its nightly.yml carries NIGHTLY_NEXT_TREE_AT_INSTALL: v1 (#27498, merged). This PR adds that line and the nightly that honours it. The client is fetched by tree, never rebuilt: the same attested manifest, through the same install-cmux-tui-client.sh command.

• resolve-nightly-cmux-tui-client. A new step, scripts/ci/nightly-steps/resolve-tree-pending.sh, handles the nightly-next tip:
  • If its tree is a clean 404 (verify-key exit 1), the tip is built as it is: tree_pending=true, plus its same-key first-parent candidates, oldest first.
  • A published tree, or a read failure, leaves the decision to the existing resolver, which is unchanged.
• build-nightly-app compiles at once:
  • "Bundle acpmux" gets a stand-in input file. The phase reads only $CMUX_NEXT_ACPMUX_BIN.ref, which is absent in both paths.
  • "Bundle the cmux-tui client" first runs scripts/cmux-next/await-tree-client.sh. Once the tree is published, it takes the commit that verify-key names, exactly as today. Before that, it takes the first candidate whose commit-addressed manifest has been uploaded (the manifest is written last, after the attestation).
  • It fails fast after two checks with no active cmux-tui-artifacts run for the key, and otherwise after 45 min.
  • The step outputs the installed commit.
• await-nightly-tree-marker (new job, Linux, read-only). publish-nightly needs it whenever the build was tree_pending. The build commit's tree-commit marker must exist, name the build's key, and name the installed client's commit as publishedBy. This is G5's check, moved into the run. prepare-nightly-variant checks the bundled client against that installed commit.
• request-nightly-next.sh requests such a tip only when three things hold:
  • it is the feat-cmux-next tip;
  • its nightly.yml has the line;
  • a cmux-tui-artifacts run for its key is active.
• Unchanged:
  • the non-pending path, which runs every published tree exactly as today;
  • the unsigned resume, fast and build-only paths;
  • recovery;
  • signing, notarizing and publishing, and their environments.
• To keep nightly.yml under its size limit, two step bodies moved to scripts/ci/nightly-steps/ and some comments in the touched jobs got shorter.

Byte-identity proof

Both paths ran on one fleet Mac (an AWS M4 Pro, macOS 27.0.1, Xcode 26.6) for one commit, 5828634. Its tree b00b2f91b8e0 was published by c230bb5, the oldest of 14 same-key candidates. Each path starts from the same skeleton app and runs the real scripts:

• A (today): the Release "Bundle cmux-tui" and "Bundle acpmux" phases with the fetched companion; then resolve-newest-published; then install-cmux-tui-client.sh, strip-release-bundle.sh, write-cmux-tui-version.sh and the capability check.
• B (this PR): the same phases with the stand-in input; then await-tree-client.sh, with the tree hidden by a curl wrapper that answers 404 for tree/<key>/, as before publication; then the same install.

Results (full log kept with the run; key lines):

A installs the client of c230bb58a50cf8b35ee7de9f1130bff840c8e459
B (tree hidden) installs the client of c230bb58a50cf8b35ee7de9f1130bff840c8e459
B (tree published) picks c230bb58a50cf8b35ee7de9f1130bff840c8e459
--- files: 19 in A, 19 in B
IDENTICAL: diff -r
IDENTICAL: paths, modes, sha256, link targets
IDENTICAL: codesign -dvvv and lipo -archs of every Mach-O
SAME COMMIT: c230bb58a50cf8b35ee7de9f1130bff840c8e459

path in cmux.app	A and B
Contents/Resources/bin/cmux	sha256 6cd30a2d…eb10, universal (x86_64 arm64), ad hoc, Identifier=cmux
bin/cmux-app-host, cmux-browser-host, cmux-cloud, cmux-holder, cmux-tui-host	same sha256 in both, universal, ad hoc
bin/acpmux, bin/cmux-tui	symlinks to cmux in both
bin/acpmux.version	8f7cf0e0…9bdd in both (the stand-in input changes nothing)
bin/cmux-tui.version, daemon-capabilities.json, cmux-tui-ssh/manifest.json	same sha256 in both

The compiled app is not in this diff because xcodebuild is unchanged. Its only new input is the stand-in path above, which no phase copies. The fleet Mac has no gh, so both paths installed with --allow-unattested. CI's install still verifies the manifest's build-provenance attestation, and that code is unchanged.

Tests

• scripts/cmux-next/tests/nightly-tree-at-install.test.sh (new, 14 cases, run in cmux-next.yml's script tests) covers:
  • await-tree-client: published tree; a manifest uploaded mid-wait; oldest-first candidates; fail fast with no publisher; the limit; bad usage;
  • the marker gate: match; another publisher; another tree; a late marker; no marker;
  • the pending step: pending; published; an unverifiable tree.
• scripts/cmux-next/tests/request-nightly-next.test.sh, new cases for a building tip:
  • without the line, it is not requested;
  • with no active run, it waits and says why;
  • with both, it is requested;
  • a commit behind the tip never is.
• tests/test_nightly_next_recovery.py: publish-nightly's if and needs.
• .github/nightly-step-coverage.json: the three new steps.
• Passing locally: tests/test_nightly_step_coverage.py, tests/test_workflow_file_size.py (nightly.yml 489,822 bytes, limit 490,000), tests/test_nightly_cmux_tui_resolve.py, tests/test_nightly_stop_if_publish_doomed.py and 63 other nightly/CI workflow tests. Six others fail the same way on base here, because of this host's sparse checkout or missing macOS tools: test_ci_nightly_push_throttle, test_nightly_next_arm64_only, test_nightly_next_guard, test_nightly_next_identity, test_nightly_next_live_feed_check, test_nightly_universal_build.sh (same failing line).

Review fixes (second head)

• publish-nightly's "Select the build to publish" takes this run's own build only when it never waited for its tree, or when await-nightly-tree-marker passed (OWN_TREE). A recovered build can no longer be selected while an unmarked own build is skipped. The job's if is unchanged.
• await-nightly-tree-marker also runs for an unsigned resume, against the commit the resumed app installed.
• Recovery refuses a run whose await-nightly-tree-marker failed, was cancelled, has not finished or cannot be read (scripts/ci/nightly-steps/recover-tree-marker-ok.sh).
• Before the newest-50 listing, await-tree-client.sh reads each candidate's own artifacts runs (pin-cmux-tui.sh runs-state), so a crowded listing never reads as "no publisher".
• New tests:
  • a candidate's own active run counts as a publisher;
  • recovery's marker check for ok, skipped, none, failed, running and unreadable;
  • an unmarked own build is never the source, and a recovered one is chosen instead.

Performance

metric	before	after	delta	harness or run link
tree-changing landing to its nightly run created	27-28 min (marker) + window	about 1-5 min (tick + window)	about -23 min	cmux-tui artifacts runs 40913738554, 40918933039, 40921878495 (jobs API)
tree-changing landing to the client install starting	27-28 min + about 10 min	about 19-21 min (manifest upload)	about -17 min	same runs; build-nightly-app median 10.2 min on the dedicated builder (13 runs, 10-29)
tree-changing landing to publish	52-58 min	about 38-40 min, to be measured on the first landings after merge	about -15 min	same harness as #27179's table; filled in after merge

The remaining critical path is the manifest upload (about 20 min) plus the post-build chain (about 18 min); both have their own PRs.

Timers: await-tree-client.sh polls the CDN every 20 s for at most 45 min, and ends early when the client is ready or when two run checks find no publisher. await-tree-marker.sh polls every 20 s for at most 50 min, inside a 60-min job. Both are bounded and end with the step.

Risk:

• If the tree build fails after promotion, its nightly fails at install and publishes nothing, and the feed keeps its last build.
• If two same-key builds race and the other one wins the tree, the marker names a different publisher. That build publishes nothing, and the next build takes the published tree.

🤖 Generated with Claude Code

Agent session: agent session

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review c32a395: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27531 -f ref=dep-fleet/nightly-tree-at-install -f platform=macos -f backend_mode=local

web and size checks: green

[blacksmith-sh]

Found 9 test failures on Blacksmith runners:

Failures

Test	View Logs
--from must install a complete prebuilt tree/ --from must install a complete prebuilt tree	View Logs
/tmp/tmpbh51slbi/summary-missing.json (live):/summary-missing.json (live):	View Logs
/tmp/tmps8wa_tg7/summary-fail.json (live):/summary-fail.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/ test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)	View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl ights_validate)/ test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl ights_validate)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Bead cx-dwdbys created for this PR by the cmux coordinator.

[teamleaderleo]

Review (codex gpt-6-astra high, next-review): BLOCK at 18c51bd

• [P1] .github/workflows/nightly.yml:5918 — Recovery can bypass the marker gate. If the current build passes signing and verification but fails await-nightly-tree-marker, an accepted recovered build still allows publish-nightly to run. The unchanged source selector prefers the current build when both are accepted (tests/test_nightly_next_recovery.py:348–353), so it publishes the client the marker gate rejected. Include marker eligibility in source selection before allowing the current build to win.

Tests could not run in this read-only, partial review snapshot.

[teamleaderleo]

Owned by an agent session · session

[teamleaderleo]

Stage measurements for the NEXT nightly, for this lane's freshness work. Source: first attempts of the last 16 nightly.yml nightly-next push runs, 2026-10-29 21:42Z to 00:04Z.

stage	typical	range	note
landing → nightly-next push (Release compile, tick, promote)	~27 min	17-37	1674720: compile green 23:42Z, promoted 23:58Z (16 min after green)
pick-nightly-build-runner	4.8 min	0.1-4.9	4.8 whenever the dedicated builder is busy (#27676)
build-nightly-app	15 min	7-18	cmux7s nightly slot, mini-3, 12s, 14
prepare variant + signer-app	5 min	4-5
signer-dmg-image	5.5 min	4.5-7.7
signer-dmg (notarize)	3.5 min	2.6-4.2
verify app + DMG	4 min	2-5
publish-nightly	1.3 min
push → publish	~37 min	34-44

So land → publish is about 60-75 min even with no queue: about 27 min before promotion plus about 37 min in the run. Since 23:43Z the base Release compile has also been red (00d6d54, #26517 × #27223). Fix: #27722.

Side effect: in every run, sign-nightly-identity-legacy and generate-nightly-deltas sit queued with no runner for 30+ min after publish, and then get cancelled. So each run stays queued, and the feed-watch starvation guard (is_active_delivery_run) counts it as an active delivery long after publication.

Agent session: agent session

[teamleaderleo]

Superseded by #28197, a backlog-drain carry that keeps this PR's commits and their authorship, merges current base and keeps the marker-eligible source selection. Closing this one in its favour.