[teamleaderleo]

HOLD: superseded by #30085, #30126 and #30173 on feat-cmux-next; do not land.

Leo, 2026-10-31: "FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT"

The ask, as it came to this lane: "NEXT nightly Linux jobs still land on Blacksmith, because cmux never requests the GCP label cmux-fleet-linux-x64. Rulebook: nightly Linux jobs run on fleet Linux first, Blacksmith only as stall-aware overflow after about 3 min, and a CI test fails a nightly runs-on that can resolve to Blacksmith before the fleet."

What

This builds on decide's early placement (#29864) and, since this PR's first head, on #30085 (controller admission for the four early controls) and #30158 (credentialed controls on the validated ephemeral set). It adds no second picker.

• portable_job_placement.py: when FLEET_GROUP_LABEL and FLEET_LINUX_GROUP are set, it also lists the org runner group cmux-fleet-linux on attempt 1. That group holds HQ's ephemeral GCP runners (build-fleet/swift-linux-builder/gha-runners/README.md). The label counts as live only when the group has an online idle runner carrying it and fewer than 12 busy ones (HQ's MAX_VMS is 16, less a nightly's burst). Only then does a new fleet_linux output name it. Otherwise the output is empty and the step does not wait for the label. That covers an unread group, a group with no runners (today: the grant is not in place), no idle runner, a full pool, and a re-run.
• pr_runner_pool.GitHub.runners(group=...) can list any org runner group. Its default is unchanged.
• nightly.yml: decide's place-early-mac step (the credentialed pick-nightly-build-runner, MAC_FIRST, so it never takes the GCP label itself) reports fleet_linux. The 9 secret-free nightly Linux jobs outside ci(nightly): admit Linux controls on fleet before paid overflow #30085's admission read github.run_attempt == 1 && needs.decide.outputs.fleet_linux first, then their existing route. Those jobs are build-nightly-remote-daemon, both prepare-nightly-variant jobs, the visual-gate verdict and its report, await-nightly-tree-marker, nightly-identity-legacy-published, and report/close-nightly-failure. Jobs that hold release or production credentials keep ci(nightly): validate trusted sets for credentialed controls #30158's validated ephemeral set: GCP runners are not on the signing trust list, and this PR does not change that list. The signer jobs (ci(nightly): signer jobs take GitHub-hosted Ubuntu, Blacksmith 4 vCPU on a re-run, never 16 or 32 #29886/ci(nightly): route credentialed signers through trusted Linux set #30081) are not touched.
• tests/test_ci_nightly_off_blacksmith.py: the new test_nightly_linux_jobs_offer_the_fleet_before_blacksmith fails any nightly Linux job whose runs-on reaches a Blacksmith label before its first fleet route. The exceptions are the jobs named in FLEET_LINUX_EXCEPTIONS, each with a reason: signer route, signing publisher, credentialed set, promotion dispatch, never runs. A job reading fleet_linux must hold no secret and no environment. BLACKSMITH_LINUX_ONLY_DEBT becomes exactly those exceptions, down from 28 entries.
• tests/test_ci_portable_job_placement.py: covers the group cases (live, unread, empty, busy, offline, unlabelled, full, re-run) and main reading the group.

Demand: HQ's gha_runners.py reads queued jobs from the workflows listed in workflows.json. Its *nightly*.yml pattern already covers nightly.yml.

Live state of the GCP label

The warm floor is in place: cmuxterm-hq#5806 (687d2838) keeps 2 idle runners always on, on c4d-standard-8 (n4d-standard-8 as fallback). A runner that takes a job is replaced on the next tick. With idle runners there, this placement's idle-runner check can see the pool.

What's left for the fleet route is the grant. Every fleet-reconciler-gha-runners run (HQ run 41259961054, 01:10Z) logs workload identity exchange failed (grant not in place: build-fleet/swift-linux-builder/gha-runners/grant.sh); no runner VM made. Until a GCP project owner runs grant.sh, the group has no runners, fleet_linux stays empty, and these jobs route exactly as they do today, on Blacksmith.

Risk

This changes the CI routing of 9 secret-free nightly Linux jobs. Each new term is gated on attempt 1 and on a live read of the runner group, and a failure falls through to the previous route. Two follow-ups are filed: #30122 (late jobs reuse decide's early check) and #30123 (the GCP read depends on CI_PR_POOL_OWNED). test_workflow_file_size and test_ci_owned_pool_sources are red on base the same way. Compared by test name against base, the touched suites gain no red.

Performance

Blacksmith minutes per day for the 9 nightly jobs that can now take the GCP label (HQ build-fleet/data/summary/summary.json, runtime_by_job week 2026-W44, mean x n / 7 over the blacksmith | <job> rows):

metric	before	after	delta	harness or run link
Blacksmith Linux minutes/day, the 9 jobs this PR routes	~102	~102 until the grant; ~0 while the GCP group has idle runners	0 now, up to -102	HQ summary.json runtime_by_job 2026-W44
Wait on a GCP label that starts nothing	n/a	0 s (offered only with an idle runner)	0	tests/test_ci_portable_job_placement.py StallAwareFleetLinux

Timers, observers and loops: none added. The group read happens inside #29864's existing bounded loop (180 s cap, 60 s read deadline).

Agent session: agent session

🤖 Generated with Claude Code

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review d00049a: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr30114 -f ref=lane/fleet1-nightly-fleet-linux -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base a30fe2b; does not block

[blacksmith-sh]

Found 9 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmp6cmmpj2o/summary-missing.json (live):/summary-missing.json (live):	View Logs
/tmp/tmprn9vc3rj/summary-fail.json (live):/summary-fail.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
convention violations found (ERROR lines above)./ convention violations found (ERROR lines above).	View Logs
test_every_part_of_bun_run_check_runs_in_the_warn_branch (main.WebviewsWarnSplitCov ersCheck.test_every_part_of_bun_run_check_runs_in_the_warn_branch)/ test_every_part_of_bun_run_check_runs_in_the_warn_branch (main.WebviewsWarnSplitCov ersCheck.test_every_part_of_bun_run_check_runs_in_the_warn_branch)	View Logs
test_no_runner_picker_returns_a_hosted_mac (main.NoHostedMacInNightlies.test_no_run ner_picker_returns_a_hosted_mac)/ test_no_runner_picker_returns_a_hosted_mac (main.NoHostedMacInNightlies.test_no_run ner_picker_returns_a_hosted_mac)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Review (codex gpt-6.1-sol xhigh, router-review): SHIP at d00049a

SHIP at d00049a

Visual verdict: no images; Leo tries the build

1. [P1] [follow-up] fleet_linux is checked only once in decide, then reused by downstream jobs that may start much later. If the GCP pool fills or goes offline meanwhile, jobs such as build-nightly-remote-daemon still resolve to cmux-fleet-linux-x64; GitHub will queue them and cannot fall back to Blacksmith. See nightly.yml:203 and nightly.yml:3045.

2. [P2] [follow-up] The new GCP route is unintentionally disabled whenever CI_PR_POOL_OWNED is off: the routing token needed by read_fleet is minted only under that condition at nightly.yml:787, despite the GCP configuration at nightly.yml:810. A live GCP pool is therefore ignored in this supported configuration.

Focused routing suites passed: 350 tests.

Follow-ups:

• [P1] Refresh or reserve GCP capacity for downstream jobs instead of reusing one early liveness snapshot.
• [P2] Decouple GCP token minting from CI_PR_POOL_OWNED.

[teamleaderleo]

Review (codex gpt-6.1-sol xhigh, router-review): SHIP at 822637d

Follow-ups:

• [P1] late jobs reuse decide's early GCP check: already filed as ci(nightly): late nightly Linux jobs reuse decide's early GCP liveness check #30122.
• [P2] screenshots: not applicable, this is a CI routing change with no UI.

[teamleaderleo]

Closing as superseded. While this waited to land, feat-cmux-next got its own fleet route for these jobs:

• ci(nightly): admit Linux controls on fleet before paid overflow #30085: the early nightly controls get a controller-reserved lease on cmux-fleet-linux-x64. Blacksmith is used only when there is a matching 180 s no-start receipt.
• ci: route light nightly tail controls without stale fleet placement #30126: the late light tail jobs no longer reuse a stale prebuild placement. They take declared light Linux on attempt 1. That fixes the P1 in both reviews here (ci(nightly): late nightly Linux jobs reuse decide's early GCP liveness check #30122).
• Use live fleet or declared hosted Linux for light nightly controls #30173: unproven fleet readiness overflows instead of queueing, and a new test refuses any unverified cmux-fleet-linux-x64 route in nightly runs-on.

This PR's fleet_linux output is an unverified route of exactly that kind, so it would undo #30173's design. The GCP route still waits on the same thing: a GCP project owner running build-fleet/swift-linux-builder/gha-runners/grant.sh. The HQ warm floor is in place (cmuxterm-hq#5806).

[teamleaderleo]

Closed for asks-fleet-1: superseded on base by #30085, #30126 and #30173.