[teamleaderleo]

Leo: "we just need the nightlies bro"

Problem

Run 41203691245 was the first nightly-next tip that included #29366, after about three hours with no tip build. Its pick-nightly-build-runner got a 403 rate limit on /actions/runners:

nightly build runner: owned: none (owned runners could not be read live, so no owned label is waited on)
nightly build runner: route: blacksmith-6vcpu-macos-26

build-nightly-app then sat queued on blacksmith-6vcpu-macos-26 and nothing started it. Meanwhile the nightly slot pool (glaeda-nightly-std-xcode-26.6, now on nine minis after HQ#5641) was ready to take it.

Change

nightly_build_runner.py: when the runners cannot be read (the token's API budget is spent, or there is no routing token), owned becomes the first watched owned label from OWNED_LABELS (unread_owned_route()). That is the nightly slot pool, a label many runners carry and the rescue watches and moves. With no watched owned label the pick still falls back to Blacksmith. Fleet first: Blacksmith is capped, paid overflow since 2026-10-30.

Tests

tests/test_nightly_build_runner.py: the three tests that expected Blacksmith on an unread read now expect the owned pool, and a new test checks that with no watched label the pick still takes Blacksmith. All 36 pass. The related routing suites (cascade, sign runner, trusted routing, train) pass too. test_ci_signing_owned_mac's hook-list failure is the base red that #29332 fixes.

Risk: low. It changes only the case where the picker could not read the runners.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Nightly app builds now queue on the owned slot pool when the runner picker can't read the runners, instead of a Blacksmith macOS label that may sit stalled.

Fixes run 41203691245, where a 403 rate limit on /actions/runners queued the first tip build after a three-hour red on blacksmith-6vcpu-macos-26 while the nightly pool had nine minis ready. When runners can't be read (API budget spent or no routing token) and the caller opts in, the picker uses the first watched owned label — a many-runner pool the rescue watches and moves. Only nightly.yml's build pick opts in, and only while the CI_PR_POOL_OWNED kill switch is on; the train picker and verify picks still take Blacksmith on an unread pool. With no watched owned label the pick falls back to Blacksmith as before, and the owned-pool rescue now also recognizes the nightly slot label.

^{Written for commit c3f86c3. Summary will update on new commits.}

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review c3f86c3: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr29401 -f ref=dep-fleet-3/nightly-picker-owned-when-unread -f platform=macos -f backend_mode=local

[blacksmith-sh]

Found 1 test failure on Blacksmith runners:

Failure

Test	View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/ test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Review (codex gpt-6.1-sol xhigh, router-review): BLOCK at 3ff482c

BLOCK at 3ff482c

Visual verdict: no images; Leo tries the build

1. [P1] [blocking] The fallback can select glaeda-nightly-*, but the rescue does not recognize that label as an owned pool. nightly_build_runner.py:319-327 accepts it, while owned_pool_rescue.py:457-462 does not. A busy or unread nightly slot can therefore remain queued without rescue.

2. [P1] [blocking] This shared picker also serves nightly-next-train.yml:230-250, which has no rescue workflow or watch marker. An unread runner response now sends a busy train compile to its owned label instead of Blacksmith, potentially consuming the train deadline.

3. [P1] [blocking] Unread owned is copied to the Ghostty helper at nightly_build_runner.py:338-343, and verify mode also uses it, but owned_pool_rescue.py:465-476 only judges the app job and trusted labels. Side-labeled helper or verify jobs can remain queued after the app itself starts.

4. [P1] [blocking] The fallback ignores the owned-pool/rescue switches. With CI_PR_POOL_OWNED=0 or rescue disabled, configured labels can still be emitted at nightly_build_runner.py:319-327; nightly.yml:1885 consumes that output, while the rescue is disabled by ci-owned-pool-rescue.yml:146. The documented Blacksmith fallback can become an unmonitored self-hosted queue.

[teamleaderleo]

Review (codex gpt-6.1-sol xhigh, router-review): BLOCK at 3ff482c

BLOCK at 3ff482c

Visual verdict: no images; Leo tries the build

1. [P1] [blocking] The CI_PR_POOL_OWNED kill switch is bypassed at nightly_build_runner.py:319. When the switch is off, the workflow intentionally provides no routing token, so runners is None; this branch then emits the first owned label from OWNED_LABELS anyway. build-nightly-app consumes that output, while the rescue exits immediately when the switch is off. Disabled pools can therefore receive queued jobs instead of using Blacksmith.

2. [P1] [blocking] This global fallback changes the nightly train picker too at nightly_build_runner.py:319. nightly-next-train.yml invokes this script but has no owned-pool rescue marker; its builder only retries a completed refused job. A rate-limited runner read now leaves a busy train compile queued on an owned label until the 120-minute train cap, whereas it previously used Blacksmith.

3. [P1] [blocking] The label this change specifically prefers, glaeda-nightly-std-xcode-26.6, is accepted by unread_owned_route() at nightly_build_runner.py:222, but owned_pool_rescue.job_pool() does not recognize nightly-slot labels. If every slot runner is busy or offline when the API is unread, the resulting job is not rescued and can remain queued for the full job timeout.

[teamleaderleo]

Review (codex, router-review): SHIP at c3f86c3

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 1. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at c3f86c3

Visual verdict: no images; Leo tries the build

1. [P1] [follow-up] The fallback flag ignores the rescue kill switch. When CI_OWNED_POOL_RESCUE=0, the rescue workflow is disabled by its condition in ci-owned-pool-rescue.yml:146, but nightly.yml:1837 still sets UNREAD_OWNED_FALLBACK=1 whenever CI_PR_POOL_OWNED=1. An unread picker will then queue on the owned label with no rescue to move it, potentially stalling indefinitely. Include the rescue switch in this expression or disable the fallback when rescue is off.

Focused routing/rescue suites passed: 283 tests via unittest.