Skip to content

ci(nightly): take the cmux-tui download and the light-job placement wait off build-nightly-app's tail - #32501

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-app-build-tail
Nov 2, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-app-build-tail

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

build-nightly-app takes about 17 min, and the path from landing to the NEXT feed needs to fit in 30 min. After the Release compile, the job ends with two waits on the feed's critical path. This PR moves both off it. The compile itself is unchanged.

What changes

  • cmux-tui client prefetch. "Bundle the cmux-tui client" downloaded the resolved client and its companions after the compile. Over 16 nightly-next runs on 2026-11-02 the step took 40-121 s (median 50 s), and the download was 31-108 s of that. "Fetch the resolved cmux-tui tree companions" now starts install-cmux-tui-client.sh in the background (background_build.py) on a scratch bundle while the app compiles. It writes into the same CMUX_TUI_CLIENT_CACHE and uses the same manifest URL, expected commit and capability checks. The bundle step joins it and then runs its own install, which reuses a cached slice only if its sha256 matches the attested manifest. A failed or unfinished prefetch is stopped and its cache is deleted, so the install then downloads everything as it does today. The prefetch runs only when the bundle step would also run and the tree is already published. Its token goes into the prefetch command only, not into the pin-cmux-tui.sh fetch in the same step.
  • Light-job placement in the background. "Place the light jobs on a free mini" waits up to 180 s for idle nightly-only runners and was the job's last step: 3-188 s, mean 66 s (70 s in run 41485637444, 188 s in run 41482932276). It now starts before "Archive unsigned nightly app" (light-placement.sh start), so its wait runs during the archive, the uploads and the browser-data proof (about 100-230 s). "Collect the light-job placement" (id: place-light, still the last step before the always() cleanup) joins it and copies its outputs. The light jobs read steps.place-light exactly as before. If the placement fails, the light jobs take Blacksmith as before, and "Stop the helper builds still running" ends the placement if the job fails first.
  • One comment block moves from nightly.yml into browser-data-proof-start.sh to keep nightly.yml under its 490,000-byte cap.

Performance

The before numbers come from run logs. The after numbers are estimates, not measurements.

metric before after (estimate) delta (estimate) run link
Bundle the cmux-tui client median 50 s (40-121 s, 16 runs) about 15 s (attestation check, install from cache, strip, capability checks) about -35 s median, up to -100 s 41485637444 (64 s), 41474606044 (105 s)
Place the light jobs on a free mini (on path) mean 66 s (3-188 s, 16 runs) about 1 s (join; the wait overlaps 100-230 s of archive, uploads and proof) about -65 s mean, up to -187 s 41485637444 (70 s), 41482932276 (188 s)
build-nightly-app job median 1118 s (857-2207 s, 16 runs) about 1020 s about -100 s 41485637444 (1013 s)

Blacksmith minutes per day: unchanged. The light jobs use the same placement logic, and the placement now observes capacity about 2-3 min earlier.

Warm sources: none change. Both cuts remove waits after the compile. "Build nightly app (Release)" took 383-1304 s over the same runs, and its kept DerivedData and compilation cache work as before.

Tests

  • tests/test_nightly_trusted_routing.py: new tests run light-placement.sh start and finish against a stand-in placement. They check that the outputs and summary reach the collecting step and that a failed placement fails it. They also check the step order: the start step comes before the archive and the collect step is last before cleanup. A structural test checks that the prefetch uses the bundle step's cache, commit, manifest and capability checks, that it runs only where the bundle step runs, and that the join comes before the install and drops the cache on failure. 47 tests pass.
  • Pass: test_ci_portable_job_placement.py (updated to the start step's id), test_nightly_step_coverage.py (new step registered), test_workflow_file_size.py (489,915 bytes), test_nightly_background_steps.py, test_ci_nightly_off_blacksmith.py, test_nightly_build_state.py, test_ci_resolve_cmux_tui_client_commit.sh, test_ci_self_hosted_guard.sh, test_nightly_delivery_concurrency.py, test_nightly_signer_jobs.py.
  • test_nightly_universal_build.sh fails on the current base with the same message ("the short nightly decide gate must use the pooled Linux route with a Blacksmith fallback"), so that failure is not from this PR.

No started nightly run is cancelled by this change or while landing it.

Agent session: agent session

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Takes two waits off build-nightly-app's critical path so the path from landing to the NEXT feed fits its 30-minute window. The Release compile is unchanged; the estimated job median drops from about 1118 s to about 1020 s.

Cmux-tui client prefetch

  • "Fetch the resolved cmux-tui tree companions" now starts install-cmux-tui-client.sh in the background on a scratch bundle while the app compiles, using the same cache, manifest URL, expected commit, and capability checks as the install.
  • "Bundle the cmux-tui client" joins the prefetch and installs from cache, reusing slices only when sha256 matches the attested manifest; a failed or unfinished prefetch is stopped and the install downloads as before.

Light-job placement

  • The placement now starts in the background before archiving and is collected as the last step, so its wait for an idle mini overlaps the archive, uploads, and browser-data proof.
  • The light jobs read the collecting step exactly as before, and a failed placement still falls back to Blacksmith.

Written for commit cbf707c. Summary will update on new commits.

View guided diff Turn on auto-fix

…ait off build-nightly-app's tail

build-nightly-app ended with two waits on the feed's critical path:

- "Bundle the cmux-tui client" downloaded the resolved client and its
  companions after the compile: 40-121 s, median 50 s, of which 31-108 s
  was the download (16 nightly-next runs, 2026-11-02). The tree companions
  step now starts install-cmux-tui-client.sh in the background on a scratch
  bundle, beside the compile, into the cache the bundle step reads; the
  bundle step joins it and then installs from the cache, which it reuses
  only for slices whose sha256 matches the attested manifest. A failed
  prefetch is stopped and its cache dropped, so the install downloads as
  before.
- "Place the light jobs on a free mini" waited up to 180 s for idle
  nightly-only runners as the job's last step: 3-188 s, mean 66 s. It now
  starts in the background before the archive and is collected at the end,
  so the wait overlaps the archive, the uploads and the browser-data proof.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review cbf707c: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr32501 -f ref=lane/nightly-app-build-tail -f platform=macos -f backend_mode=local

Shipped in:

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 2, 2026
@blacksmith-sh

blacksmith-sh Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Found 6 test failures on Blacksmith runners:

Failures

Test View Logs
main.SharedRunnerStateTests.test_all_requested_picker_workflows_receive_repository_
context/test_all_requested_picker_workflows_receive_repository_context
View Logs
test_all_requested_picker_workflows_receive_repository_context (main.SharedRunnerSt
ateTests.test_all_requested_picker_workflows_receive_repository_context)/
test_all_requested_picker_workflows_receive_repository_context (main.SharedRunnerSt
ateTests.test_all_requested_picker_workflows_receive_repository_context)
View Logs
test_nightly_keeps_a_margin_under_the_limit (main.WorkflowFileSizeTest.test_nightly
_keeps_a_margin_under_the_limit)/
test_nightly_keeps_a_margin_under_the_limit (main.WorkflowFileSizeTest.test_nightly
_keeps_a_margin_under_the_limit)
View Logs
test_no_workflow_adds_a_single_size_linux_job (main.WorkflowTests.test_no_workflow_
adds_a_single_size_linux_job)/
test_no_workflow_adds_a_single_size_linux_job (main.WorkflowTests.test_no_workflow_
adds_a_single_size_linux_job)
View Logs
test_no_workflow_falls_back_to_blacksmith_outside_manaflow_ai (main.ForkRunnerRouti
ngTests.test_no_workflow_falls_back_to_blacksmith_outside_manaflow_ai)/
test_no_workflow_falls_back_to_blacksmith_outside_manaflow_ai (main.ForkRunnerRouti
ngTests.test_no_workflow_falls_back_to_blacksmith_outside_manaflow_ai)
View Logs
the short nightly decide gate must use the pooled Linux route with a Blacksmith fallbac
k/
the short nightly decide gate must use the pooled Linux route with a Blacksmith fallbac
k
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at cbf707c

Reviewer: codex gpt-6-astra high, fast tier.

SHIP at cbf707c

Visual verdict: no images; Leo tries the build.

Checked the changed workflow/scripts and 102 targeted unittest cases; all passed.

Follow-ups: none

@teamleaderleo teamleaderleo removed the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 2, 2026
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 2, 2026
@teamleaderleo
teamleaderleo merged commit f2e5284 into feat-cmux-next Nov 2, 2026
184 of 198 checks passed
@teamleaderleo
teamleaderleo deleted the lane/nightly-app-build-tail branch November 2, 2026 18:16
teamleaderleo added a commit that referenced this pull request Nov 2, 2026
…ghtly.yml back under its size cap (#32513)

#32490, #32501 and #32498 together took nightly.yml to 493,218 bytes, past the
490,000-byte cap in tests/test_workflow_file_size.py. Move "Build and sign the
release notes" and "Embed channel provisioning profile" into
scripts/ci/nightly-steps/, the same way earlier steps were moved; the second
runs under bash -e like its inline block did. The workflow expanded by
tests/nightly_workflow_text.py is byte-identical to base. nightly.yml is now
485,819 bytes.

Base-red-job: test_workflow_file_size (nightly.yml over its 490,000-byte cap)


Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo added a commit that referenced this pull request Nov 2, 2026
…ness PRs (#32526)

- Validate pull request macOS pool choice: #32501 moved the light-job
  placement's portable_job_placement.py call into
  scripts/ci/nightly-steps/light-placement.sh, so the step that carries
  CI_RUNNER_STATE no longer names the picker in its run text. The leak
  check now also reads the nightly-steps script a step calls; a step
  whose script runs no picker still fails.
- Validate supply-chain security boundaries: #32455's sentry-dsyms-nightly
  job takes SENTRY_AUTH_TOKEN with no environment. It now has the same
  environment expression build-nightly-app gave the in-job Sentry upload
  it replaced; no environment or its protection changes.
- Validate test determinism gate: #32498's test asserted on elapsed wall
  time. It now asserts the failure names the stalled range, which shows
  the fetch ended on its own bound.

Base-red-job: Validate pull request macOS pool choice
Base-red-job: Validate supply-chain security boundaries
Base-red-job: Validate test determinism gate


Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — cbf707c4 Deployed Nov 2, 2026 by teamleaderleo via cmux-tui tree preflight #102211
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant