[teamleaderleo]

Leo, 2026-10-31: "i dont like doing display tests anymore". Leo, 2026-10-31: "FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT". Rulebook: "Nightly freshness" and "No pins".

What was wrong

The NEXT nightly fell more than 100 landings behind (NOW.md, 04:35Z). Every nightly-next run after f8c1ccb (published 04:14Z) crawled. Blacksmith's 32 vCPU size waited 8 to 15 minutes per job from about 03:45Z: the overflow probe at 03:55Z had 32 vCPU start after 11 minutes and 16 vCPU after 3 s. 17 nightly jobs named blacksmith-32vcpu-ubuntu-2404 literally:

• both pickers (pick-nightly-build-runner, pick-nightly-sign-runner)
• signer-app, signer-app-notary and signer-dmg
• every publish job

The run chains about ten of these one after another, so each stall added up. The display Mac visual gate (visual-gate-nightly) also took 37 minutes of f8c1ccb's critical path.

The main cause: every run skipped its own build

In resolve-nightly-cmux-tui-client, "Leave the build slot to a newer nightly-next push" sets superseded=true whenever nightly-next has moved past the run's commit. Promotions came about every 6 minutes. A run took 12 minutes or more of Linux queue to reach that step. So every push run from 04:28Z to 05:23Z found itself superseded and built nothing (runs 41170916165 and 41171540012 log "nightly-next moved from ... the newer push's run builds it"), and no run ever built. The step dates from #21917, when pushes shared one build slot. NEXT runs are run-scoped now, and the rulebook says a newer tip builds alongside a running one. publish-nightly already publishes latest-wins against the feed (publish-recheck-recovered.sh, --superseded-ok), so builds running side by side are safe. The skip is now opt-in: NIGHTLY_NEXT_SUPERSEDE_SKIP=1.

What this changes

• Those 17 jobs take the expression the rest of the nightly's Linux jobs already use: vars.LINUX_RUNNER == 'blacksmith-16vcpu-ubuntu-2404' && 'blacksmith-16vcpu-ubuntu-2404' || 'blacksmith-32vcpu-ubuntu-2404'. It only ever yields one of two literal Blacksmith labels, never the variable's value. So a job that holds a secret still cannot follow a variable onto a reused machine (signing rule 7).

• route-visual-gate-nightly runs only when NIGHTLY_NEXT_VISUAL_GATE names a mode. Unset now means off, so no nightly waits on a display Mac. Setting the variable to report, enforce or auto turns the gate back on.

• The signing guard (tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py, rule 7) now accepts vars.X == 'L' && 'L' || when L is an ephemeral label. The variable is only compared, and the job runs on the literal. A self-check covers the accepted form and the two forms it still refuses. The guard's 4 remaining failures (sign-nightly-identity-legacy and generate-nightly-deltas reading CI_NIGHTLY_NEXT_BUILD_RUNNER and CI_SIDE_LANE_RUNNER) are on base since a7b6a61, in lines this PR leaves alone.

• tests/test_nightly_legacy_publish_linux.py expects the new expression.

Done by hand at the same time (mitigation)

• LINUX_RUNNER = blacksmith-16vcpu-ubuntu-2404 (04:50Z).
• NIGHTLY_NEXT_VISUAL_GATE = off (05:02Z).

Risk

CI routing only, no product code. If the 16 vCPU size stalls instead, setting LINUX_RUNNER back moves every nightly Linux job together. Not in this PR, as follow-ups: fleet-first Linux routing for the nightly (no GitHub runner on our own Linux hosts exists today) and a CI test for it.

🤖 Generated with Claude Code

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review c5a0a56: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28961 -f ref=ci/nightly-next-linux-follow-runner -f platform=macos -f backend_mode=local

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4117757429001 (cbd1fc5db2)
• cmux NEXT nightly 1.0.0-nightly.4117797037301 (c9ae20d5c2)
• cmux NEXT nightly 1.0.0-nightly.4117948839501 (e43764d3e1)
• cmux NEXT nightly 1.0.0-nightly.4118042477101 (133d5a9bd7)
• cmux NEXT nightly 1.0.0-nightly.4118089936901 (7975c7a9e0)
• cmux NEXT nightly 1.0.0-nightly.4118163201202 (05730c30a0)
• cmux NEXT nightly 1.0.0-nightly.4118400110201 (c904112cd1)

[teamleaderleo]

Owned by an agent session · session

[teamleaderleo]

Review (codex, router-review): SHIP at eba48ff

Follow-ups: 1. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at eba48ff

Visual verdict: Not applicable; workflow-only change with no UI diff.

1. [P2][follow-up] .github/workflows/nightly.yml:5790 skips the visual route when NIGHTLY_NEXT_VISUAL_GATE is unset, but recovery still resolves an unset variable as auto via visual-gate-read-verdicts.sh. After auto enforcement begins, a pending recovery with an older failed verdict can still be refused even though unset is now intended to mean off. Align recovery and the mode documentation with the new opt-in behavior.

Follow-ups:

• [P2][follow-up] Make unset visual-gate configuration consistently mean off for recovery, or document and preserve the intentional exception.

[teamleaderleo]

Review (codex gpt-6-astra high, router-review): BLOCK at 8db2130

BLOCK at 8db2130

Visual verdict: Not applicable; CI workflow only.

1. [P1] [blocking] The new runner expression at .github/workflows/nightly.yml:7345 breaks the repository’s release-notary CI guard. tests/test_nightly_legacy_publish_linux.py:65 still requires the legacy publisher to remain on the literal 32-vCPU label, and CI invokes that test in .github/workflows/ci-guards.yml:1617. Running python3 tests/test_nightly_legacy_publish_linux.py fails immediately, so this PR cannot pass its required checks.

[blacksmith-sh]

Found 3 test failures on Blacksmith runners:

Failures

Test	View Logs
test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_ a_picked_signing_label_has_a_rerun_alternative)/ test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_ a_picked_signing_label_has_a_rerun_alternative)	View Logs
test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes t_no_runs_on_chain_ends_on_the_reserved_side_label)/ test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes t_no_runs_on_chain_ends_on_the_reserved_side_label)	View Logs
test_publish_runs_on_blacksmith_linux (main.Workflow.test_publish_runs_on_blacksmit h_linux)/ test_publish_runs_on_blacksmith_linux (main.Workflow.test_publish_runs_on_blacksmit h_linux)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (codex gpt-6-astra high fast, router-review): SHIP at c5a0a56

No findings. The routing, publication, concurrency, signer and security checks all passed. The four signing-guard failures reproduce on base, and the visual-gate fixture failure is in files this PR leaves alone.

Follow-ups: none (an earlier follow-up from eba48ff is #28968)