Skip to content

ci(nightly): the primary preparation takes fleet Linux while a runner idles there - #32511

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-prepare-fleet-linux
Nov 2, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-prepare-fleet-linux

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

prepare-nightly-variant-primary takes fleet Linux first: the GCP Actions runner pool (cmux-fleet-linux-x64). It falls back to the owned nightly Mac, then Blacksmith, as it does today.

Why

The job only moves and reads files: download, thin with llvm-lipo, check, tar, upload. On a dedicated nightly Mac its median is 202 s over 24 runs (signing-input download 49 s, upload 85 s, both slow on the Mac network). On Linux it is 45 s. #31463 moved it to Blacksmith Linux alone, and #31636 reverted that because Blacksmith's 16 vCPU label queued nightly jobs for 5-14 min. The fleet pool doesn't have that queue.

Design

  • build-nightly-app gets a new step, place-fleet, right after "Collect the light-job placement" and before the always() cleanup. It runs the existing scripts/ci/portable_job_placement.py with FLEET_LINUX=cmux-fleet-linux-x64, REQUIRE_LIVE_CAPACITY=1 and JOBS=LINUX_ONLY=prepare-nightly-variant-primary. This is the same live-capacity pick cmux-tui-artifacts.yml's daemon tests use. It names the label only when the runner census shows an idle Linux runner beyond the label's queue, so a job sent there starts at once. It never waits, so it adds no time to the build job. It reuses the light placement's existing routing token and adds no permission or secret.
  • The primary's runs-on order is: fleet Linux from light_linux_runner, then the light placement's mini (unchanged), then the runner set's Blacksmith member (unchanged). All of it is gated on light_attempt like the mini, so a re-run takes Blacksmith.
  • With no idle fleet runner, the job goes to the Mac at once instead of waiting about 3 minutes for a start. A wait would hold the nightly build Mac idle on the feed's critical path, and that costs more than the difference between Linux and the Mac.
  • The fallback leg prepare-nightly-variant is unchanged: it still reuses the primary's artifact or prepares on its mini/Blacksmith route.
  • The Linux-only steps already exist and ran on Blacksmith Linux until ci(nightly): the all-nightly reservation's idle runners take nightly-next's light jobs #31410. On a fresh VM without llvm, the lipo step installs it with apt. The job runs no macOS tool (tests/test_ci_nightly_off_blacksmith.py checks that).
  • scripts/ci/runner_routing_table.py now checks that a nightly light job reads this fleet output before the mini.

Size

On base, nightly.yml was 490,126 bytes, 126 over the 490,000 cap that test_workflow_file_size enforces. The lipo shim moves to scripts/ci/nightly-steps/put-llvm-lipo-on-path.sh, which brings the file to 489,720 bytes.

Also

The fat Mach-O check in "Verify nightly binary architectures" now fails when file is missing or errors, instead of reading every file as not fat. A Linux runner without file installs it first.

The placement's census read stays inline rather than in the background with the light placement. It takes a few seconds, and reading at the end of the build keeps the idle-runner check as fresh as possible for the job that starts right after.

Note

The fleet Actions pool reports no runners online right now (its provisioning grant is pending). Until it has idle runners, place-fleet names nothing and every run takes today's route.

Performance

metric before after delta harness or run link
prepare-nightly-variant-primary run time, nightly Mac route 177 s / 245 s (median 202 s over 24 runs) ~60-90 s on fleet Linux (estimate: 45 s Linux median plus llvm install on a fresh VM) about -110 to -140 s on the signing path before: 41488515134, 41485637444; Linux reference: 41492012856 (44 s), 41490937436 (45 s)
Blacksmith minutes per day for this job unchanged unchanged or lower (fleet takes the job ahead of Blacksmith) 0 or less same runs
build-nightly-app added time n/a one census read, no wait (a few seconds) ~+3 s step place-fleet

Tests

  • bash tests/test_ci_self_hosted_guard.sh (pass)
  • python3 tests/test_nightly_trusted_routing.py: new PrimaryPreparationOnFleetLinux covers fleet, then mini, then Blacksmith, plus re-run and fork
  • python3 tests/test_ci_nightly_off_blacksmith.py
  • python3 tests/test_nightly_step_coverage.py
  • python3 tests/test_workflow_file_size.py: fails on base, passes here
  • python3 tests/test_ci_portable_job_placement.py: new test runs the step's own env through decide() for idle, busy, offline, unread and queued fleet runners
  • python3 tests/test_runner_routing_table.py and python3 scripts/ci/runner_routing_table.py check (output unchanged from base)
  • tests/test_runs_on_labels.py, tests/test_nightly_signer_jobs.py, tests/test_ci_runner_set.py
  • tests/test_nightly_universal_build.sh fails the same way on base ("the short nightly decide gate ..."). That failure isn't from this PR.

Agent session: agent session

🤖 Generated with Claude Code

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 2, 2026
… idles there

prepare-nightly-variant-primary only moves and reads files. Its median is 202 s
on a dedicated nightly Mac (signing-input download 49 s, upload 85 s) against
45 s on Linux. #31463 moved it to Blacksmith Linux alone and #31636 reverted
that when Blacksmith's 16 vCPU label queued nightly jobs 5-14 min.

build-nightly-app's new step place-fleet (after "Collect the light-job placement",
before the always() cleanup) runs portable_job_placement.py
with FLEET_LINUX=cmux-fleet-linux-x64 and REQUIRE_LIVE_CAPACITY=1 for this one
job: it names the fleet label only while the runner census shows an idle Linux
runner beyond the label's queue, so the job starts at once, and it never waits.
Otherwise the job keeps today's route: the light placement's mini, then the
runner set's Blacksmith member. A re-run takes Blacksmith, as before. The
fallback leg prepare-nightly-variant is unchanged.

The lipo shim moves to scripts/ci/nightly-steps/put-llvm-lipo-on-path.sh to
keep nightly.yml under its size cap (it was 126 bytes over on base).
runner_routing_table.py now checks that a nightly light job reads the fleet
output before the mini.

The fat Mach-O check in "Verify nightly binary architectures" now fails when
file(1) is missing or errors (a fresh Linux VM installs it first), instead of
reading every file as not fat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7
@teamleaderleo
teamleaderleo force-pushed the lane/nightly-prepare-fleet-linux branch from b6d9a7c to 791fe8b Compare November 2, 2026 18:30
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 791fe8b: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr32511 -f ref=lane/nightly-prepare-fleet-linux -f platform=macos -f backend_mode=local

Shipped in:

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at 791fe8b

Both points of the earlier BLOCK are fixed. The head merges cleanly, and on the merged tree the six listed unittest modules and test_ci_self_hosted_guard.sh pass. The updated test still requires place-fleet and then the always() cleanup to come last. /usr/bin/file ships with macOS, so the stricter check can't fail a Mac run. Re-runs and the fallback leg are unchanged.

Follow-ups:

  • On fleet Linux, run apt-get update -qq before apt-get install file, or bake file into the image.

@teamleaderleo
teamleaderleo merged commit 241f476 into feat-cmux-next Nov 2, 2026
116 of 123 checks passed
@teamleaderleo
teamleaderleo deleted the lane/nightly-prepare-fleet-linux branch November 2, 2026 18:33
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, sweep, via the review router; full diff from merge base f6b340f to this head): SHIP at b6d9a7c

No P1/P2 findings.

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 791fe8bb Deployed Nov 2, 2026 by teamleaderleo via cmux-tui tree preflight #102272
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant