Repository navigation
ci(nightly): the primary preparation takes fleet Linux while a runner idles there - #32511
Conversation
… idles there prepare-nightly-variant-primary only moves and reads files. Its median is 202 s on a dedicated nightly Mac (signing-input download 49 s, upload 85 s) against 45 s on Linux. #31463 moved it to Blacksmith Linux alone and #31636 reverted that when Blacksmith's 16 vCPU label queued nightly jobs 5-14 min. build-nightly-app's new step place-fleet (after "Collect the light-job placement", before the always() cleanup) runs portable_job_placement.py with FLEET_LINUX=cmux-fleet-linux-x64 and REQUIRE_LIVE_CAPACITY=1 for this one job: it names the fleet label only while the runner census shows an idle Linux runner beyond the label's queue, so the job starts at once, and it never waits. Otherwise the job keeps today's route: the light placement's mini, then the runner set's Blacksmith member. A re-run takes Blacksmith, as before. The fallback leg prepare-nightly-variant is unchanged. The lipo shim moves to scripts/ci/nightly-steps/put-llvm-lipo-on-path.sh to keep nightly.yml under its size cap (it was 126 bytes over on base). runner_routing_table.py now checks that a nightly light job reads the fleet output before the mini. The fat Mach-O check in "Verify nightly binary architectures" now fails when file(1) is missing or errors (a fresh Linux VM installs it first), instead of reading every file as not fat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7
b6d9a7c to
791fe8b
Compare
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
Shipped in:
|
|
Review (subagent claude-opus-5-5 medium): SHIP at 791fe8b Both points of the earlier BLOCK are fixed. The head merges cleanly, and on the merged tree the six listed unittest modules and test_ci_self_hosted_guard.sh pass. The updated test still requires place-fleet and then the always() cleanup to come last. /usr/bin/file ships with macOS, so the stricter check can't fail a Mac run. Re-runs and the fallback leg are unchanged. Follow-ups:
|
prepare-nightly-variant-primarytakes fleet Linux first: the GCP Actions runner pool (cmux-fleet-linux-x64). It falls back to the owned nightly Mac, then Blacksmith, as it does today.Why
The job only moves and reads files: download, thin with llvm-lipo, check, tar, upload. On a dedicated nightly Mac its median is 202 s over 24 runs (signing-input download 49 s, upload 85 s, both slow on the Mac network). On Linux it is 45 s. #31463 moved it to Blacksmith Linux alone, and #31636 reverted that because Blacksmith's 16 vCPU label queued nightly jobs for 5-14 min. The fleet pool doesn't have that queue.
Design
build-nightly-appgets a new step,place-fleet, right after "Collect the light-job placement" and before the always() cleanup. It runs the existingscripts/ci/portable_job_placement.pywithFLEET_LINUX=cmux-fleet-linux-x64,REQUIRE_LIVE_CAPACITY=1andJOBS=LINUX_ONLY=prepare-nightly-variant-primary. This is the same live-capacity pickcmux-tui-artifacts.yml's daemon tests use. It names the label only when the runner census shows an idle Linux runner beyond the label's queue, so a job sent there starts at once. It never waits, so it adds no time to the build job. It reuses the light placement's existing routing token and adds no permission or secret.runs-onorder is: fleet Linux fromlight_linux_runner, then the light placement's mini (unchanged), then the runner set's Blacksmith member (unchanged). All of it is gated onlight_attemptlike the mini, so a re-run takes Blacksmith.prepare-nightly-variantis unchanged: it still reuses the primary's artifact or prepares on its mini/Blacksmith route.apt. The job runs no macOS tool (tests/test_ci_nightly_off_blacksmith.pychecks that).scripts/ci/runner_routing_table.pynow checks that a nightly light job reads this fleet output before the mini.Size
On base,
nightly.ymlwas 490,126 bytes, 126 over the 490,000 cap thattest_workflow_file_sizeenforces. The lipo shim moves toscripts/ci/nightly-steps/put-llvm-lipo-on-path.sh, which brings the file to 489,720 bytes.Also
The fat Mach-O check in "Verify nightly binary architectures" now fails when
fileis missing or errors, instead of reading every file as not fat. A Linux runner withoutfileinstalls it first.The placement's census read stays inline rather than in the background with the light placement. It takes a few seconds, and reading at the end of the build keeps the idle-runner check as fresh as possible for the job that starts right after.
Note
The fleet Actions pool reports no runners online right now (its provisioning grant is pending). Until it has idle runners,
place-fleetnames nothing and every run takes today's route.Performance
place-fleetTests
bash tests/test_ci_self_hosted_guard.sh(pass)python3 tests/test_nightly_trusted_routing.py: newPrimaryPreparationOnFleetLinuxcovers fleet, then mini, then Blacksmith, plus re-run and forkpython3 tests/test_ci_nightly_off_blacksmith.pypython3 tests/test_nightly_step_coverage.pypython3 tests/test_workflow_file_size.py: fails on base, passes herepython3 tests/test_ci_portable_job_placement.py: new test runs the step's own env throughdecide()for idle, busy, offline, unread and queued fleet runnerspython3 tests/test_runner_routing_table.pyandpython3 scripts/ci/runner_routing_table.py check(output unchanged from base)tests/test_runs_on_labels.py,tests/test_nightly_signer_jobs.py,tests/test_ci_runner_set.pytests/test_nightly_universal_build.shfails the same way on base ("the short nightly decide gate ..."). That failure isn't from this PR.Agent session: agent session
🤖 Generated with Claude Code