Skip to content

ci(nightly): keep the primary preparation off the light-job placement - #31463

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-prepare-primary-linux
Nov 1, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-prepare-primary-linux

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #31410. It placed all three light jobs on the dedicated nightly Macs. The two small ones belong there. prepare-nightly-variant-primary doesn't: it downloads the whole unsigned app, scans every file for the browser-data import proof, and uploads the prepared app. On a dedicated nightly Mac that cost more than the 3-minute placement wait #31410 removed.

metric before (Blacksmith route, run 41358657711) with #31410 (dedicated Mac, run 41360732074) after this PR delta harness
prepare-nightly-variant-primary duration 57 s 4 min 45 s (download 59 s, browser-data proof 99 s, upload 62 s) about 57 s (Blacksmith route, no placement wait) about -3.8 min against #31410 job and step timings
light-job placement 186 s under 15 s under 15 s none step timings

Change: the placement step's JOBS drops prepare-nightly-variant-primary, so it is never offered and its runs-on takes its Blacksmith route at once. Its placement-aware runs-on is unchanged.

Tests: tests/test_nightly_trusted_routing.py (updated), tests/test_ci_portable_job_placement.py, tests/test_ci_nightly_off_blacksmith.py and tests/test_runner_routing_table.py pass.

Agent session: agent session

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Keeps prepare-nightly-variant-primary off the dedicated nightly Mac placement so it takes its faster Blacksmith route.

#31410 put all three light jobs on dedicated nightly Macs. That made prepare-nightly-variant-primary take 4.8 min (download, browser-data import proof, upload) versus 57 s on its Blacksmith route, wiping out the 3-minute placement wait the placement removed. This PR drops it from the placement step's JOBS list — it's no longer offered, so it neither waits nor lands on a Mac — while the two small jobs stay on the reservation and its placement-aware runs-on is unchanged. Updated the placement test to match.

Written for commit 974d7b2. Summary will update on new commits.

View guided diff Turn on auto-fix

prepare-nightly-variant-primary moves the whole app twice and scans every file
of it. Placed on a dedicated nightly Mac by #31410 it took 4.8 min (run
41360732074: download 59 s, browser-data proof 99 s, upload 62 s) against 57 s
on its Blacksmith route (run 41358657711), which undid the 3 min #31410 saved.
The two tiny light jobs stay on the reservation; the primary is no longer
offered, so it neither waits nor lands on a Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 974d7b2: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr31463 -f ref=lane/nightly-prepare-primary-linux -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 73f6764 to compare; do not merge until green; see lane-rules

Shipped in:

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at 974d7b2

  • The primary's runs-on can no longer match a placed job, so it resolves to its Blacksmith route, and the wait loop no longer waits on it.
  • prepare-nightly-variant's fallback leg is unchanged.
  • Passing: nightly_trusted_routing (44), nightly_off_blacksmith (21), runner_routing_table (33), portable_job_placement.
  • test_ci_self_hosted_guard.sh fails the same way on base.

Follow-ups:

  • [follow-up] The primary's runs-on still has a light_mini term it can no longer reach, so test_ci_nightly_off_blacksmith counts it as having a fleet route. Add it to BLACKSMITH_LINUX_ONLY_DEBT, or make the test check that the job is actually offered in place-light JOBS.
  • [follow-up] When the primary fails, a fallback prepare placed on a mini runs the full slow preparation there.
  • [follow-up] Once a fleet Linux runner exists, re-measure the primary there. The 99 s browser-data proof is worth speeding up on its own.

@teamleaderleo
teamleaderleo merged commit 1d6ff66 into feat-cmux-next Nov 1, 2026
154 of 165 checks passed
@teamleaderleo
teamleaderleo deleted the lane/nightly-prepare-primary-linux branch November 1, 2026 23:52
teamleaderleo added a commit that referenced this pull request Nov 2, 2026
…lacement (#31463)" (#31636)

This reverts commit 1d6ff66.

Blacksmith's 16-vCPU Linux label is queueing nightly jobs 5-14 min (runs 41372372224,
41373656988, 41374257749: prepare-nightly-variant-primary, attest-nightly-remote-daemon and
pick-nightly-sign-runner queued there). With #31463 the primary preparation had only that route.
On an idle all-nightly runner it takes about 4.8 min, but it starts at once.


Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo added a commit that referenced this pull request Nov 2, 2026
… idles there

prepare-nightly-variant-primary only moves and reads files. Its median is 202 s
on a dedicated nightly Mac (signing-input download 49 s, upload 85 s) against
45 s on Linux. #31463 moved it to Blacksmith Linux alone and #31636 reverted
that when Blacksmith's 16 vCPU label queued nightly jobs 5-14 min.

build-nightly-app's new step place-fleet (after "Collect the light-job placement",
before the always() cleanup) runs portable_job_placement.py
with FLEET_LINUX=cmux-fleet-linux-x64 and REQUIRE_LIVE_CAPACITY=1 for this one
job: it names the fleet label only while the runner census shows an idle Linux
runner beyond the label's queue, so the job starts at once, and it never waits.
Otherwise the job keeps today's route: the light placement's mini, then the
runner set's Blacksmith member. A re-run takes Blacksmith, as before. The
fallback leg prepare-nightly-variant is unchanged.

The lipo shim moves to scripts/ci/nightly-steps/put-llvm-lipo-on-path.sh to
keep nightly.yml under its size cap (it was 126 bytes over on base).
runner_routing_table.py now checks that a nightly light job reads the fleet
output before the mini.

The fat Mach-O check in "Verify nightly binary architectures" now fails when
file(1) is missing or errors (a fresh Linux VM installs it first), instead of
reading every file as not fat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7
teamleaderleo added a commit that referenced this pull request Nov 2, 2026
… idles there (#32511)

prepare-nightly-variant-primary only moves and reads files. Its median is 202 s
on a dedicated nightly Mac (signing-input download 49 s, upload 85 s) against
45 s on Linux. #31463 moved it to Blacksmith Linux alone and #31636 reverted
that when Blacksmith's 16 vCPU label queued nightly jobs 5-14 min.

build-nightly-app's new step place-fleet (after "Collect the light-job placement",
before the always() cleanup) runs portable_job_placement.py
with FLEET_LINUX=cmux-fleet-linux-x64 and REQUIRE_LIVE_CAPACITY=1 for this one
job: it names the fleet label only while the runner census shows an idle Linux
runner beyond the label's queue, so the job starts at once, and it never waits.
Otherwise the job keeps today's route: the light placement's mini, then the
runner set's Blacksmith member. A re-run takes Blacksmith, as before. The
fallback leg prepare-nightly-variant is unchanged.

The lipo shim moves to scripts/ci/nightly-steps/put-llvm-lipo-on-path.sh to
keep nightly.yml under its size cap (it was 126 bytes over on base).
runner_routing_table.py now checks that a nightly light job reads the fleet
output before the mini.

The fat Mach-O check in "Verify nightly binary architectures" now fails when
file(1) is missing or errors (a fresh Linux VM installs it first), instead of
reading every file as not fat.


Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 974d7b2d Deployed Nov 1, 2026 by teamleaderleo via cmux-tui tree preflight #97905
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant