[teamleaderleo]

The fast tier could fall through to an unavailable side label when placement found no idle Mac. It now uses a current-attempt owned allocation or the Blacksmith cascade. Bot attempt 2 honors its freshly allocated owned runner. GUI and session routes also require an online runner.

The shared label_full_or_dead adapter reads per-label gates from the pool artifact or CI_FLEET_GATES, checks each label timestamp, and detects saturated owned pools, stale/missing readings after a three-minute wait, Blacksmith cap/backlog, and 30-minute no-start outages. The cascade preserves queued nightly work's priority. fleet_jobs, gateway_runner, and the client filter remain available for the separate fleet-client implementation; full Mac jobs retain their Mac routes.

Rescue discovers unmarked side-lane runs, recovers missing cmux-next PR associations, and detects queued jobs even when a sibling failed. Active runs at an open PR's current head remain intact: GitHub cannot retry them while active, and retrying a downstream job would retain upstream placement. The watcher reports that limitation; completed runner refusals retain the supported failed-job retry. This change improves admission and detection but does not claim to relocate already queued active PR jobs.

Request: "Fix the class, not one job." Recovery constraint: "Only re-dispatches, no cancels of an open PR's run at its head."

Design: https://github.com/manaflow-ai/cmuxterm-hq/issues/5123#issuecomment-6515215902

Placement reads have a 90-second deadline that request error handlers cannot swallow. A timeout clears partial outputs and emits a default Mac placement; checkout, token minting, and watch upload are bounded within the job deadline. The observed #28460 placement had already completed its script and cleanup before GitHub reported a timeout, so this hardening does not claim to repair that runner acknowledgement failure.

When every admissible Mac target is excluded and no fleet client is enabled, the fast tier preserves the base owned queue and still runs. It neither skips required tests nor bypasses nightly/outage exclusions onto Blacksmith. #28391 adds client-only fleet execution for this deferred case after this PR lands.

Validation at 7d085f9774ae25f43b89b1954f7d10fb80203985: all 525 affected tests pass on the fleet (https://github.com/manaflow-ai/cmux/actions/runs/41114380935), and independent gpt-6-astra router review returned SHIP with no blocking findings. Expired Blacksmith outage/capacity data is tested independently of fresh owned gates, and placement carries the observation clock consistently through its admission checks.

Previously compared base warnings: release-notary's DigestTests.test_the_repository_highlights_validate and SigningHooksInStepWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check failed identically on head run 41080623573 and base run 41071719011 (test_reds.py: own=false). The unrelated action-export-area job hit its five-minute timeout during checkout before running checks; a retry has been requested.

The routing, health adapter, and regression tests stay together because workflow placement and rescue share the admission contract. The fleet client is a separate execution mechanism in #28391.

Risk: admission depends on fleet health freshness; missing readings favor overflow once a queue has waited three minutes. Live-run recovery is limited by the no-cancellation constraint and Actions' retry API.

Agent session: agent session

[blacksmith-sh]

Found 6 test failures on Blacksmith runners:

Failures

Test	View Logs
main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/ test_download_prints_the_active_profile_only	View Logs
stubs: /System/Library/Frameworks/Security.framework/Versions/A/Security has no stub/ Security has no stub	View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/ CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_CFRetain']/ CoreFoundation, missing 1: ['_CFRetain']	View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/ CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_log2']/ CoreFoundation, missing 1: ['_log2']	View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/ test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)	View Logs
test_the_ten_first_party_gpl_crates_are_named_exactly (main.AppCrateLicenseGate.tes t_the_ten_first_party_gpl_crates_are_named_exactly)/ test_the_ten_first_party_gpl_crates_are_named_exactly (main.AppCrateLicenseGate.tes t_the_ten_first_party_gpl_crates_are_named_exactly)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review d9f7c20: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28249 -f ref=lane/fast-tier-rescue-28140 -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4115755828101 (7c0adaf184)
• cmux NEXT nightly 1.0.0-nightly.4115843934401 (8b3fa55ef5)
• cmux NEXT nightly 1.0.0-nightly.4115962701801 (631d310eb9)

[teamleaderleo]

Review (codex-review-box/gpt-6-astra high, lane-place): BLOCK at 93cd38d

BLOCK

1. tree/scripts/ci/owned_pool_rescue.py:1371-1385 calls POST /actions/jobs/{job_id}/rerun while the workflow is still running and the job is queued. The same module documents that GitHub rejects job reruns for an in-progress run (around line 1041), so this path will fail and leave the stuck job queued instead of moving it.

2. tree/scripts/ci/owned_pool_rescue.py:1289-1294 only recovers missing pull_requests for WATCH_RUN_ID dispatches. The normal workflow_run path and sweeper call target_from_event directly (1473), so a cmux-next run with an omitted PR list is discarded as “does not name exactly one pull request” and never reaches the new rescue logic.

[teamleaderleo]

Review (codex-review-box/gpt-6-astra high, lane-place): BLOCK at bf35c37

BLOCK

1. [P1] scripts/ci/owned_pool_rescue.py:1566 — PR recovery runs outside the adoption error handler. If the commit-to-PR lookup exhausts its retries, the exception terminates the entire sweeper, including unrelated watches. Reproduced locally with a failing lookup. Catch failures per run and allow later adoption retries.

2. [P2] scripts/ci/owned_pool_rescue.py:699 — READ_ERRORS misses RuntimeError, which the snapshot downloader raises for HTTP download failures. Such failures terminate the watch instead of treating health as unavailable and rescuing the queued job. Reproduced with artifact download failed (410).

3. [P2] .github/workflows/cmux-next.yml:2661 — The routing change leaves two CI guard tests failing: tests/test_ci_next_macos_cascade.py:1144 and tests/test_ci_side_lane_placement.py:1166. The 323-test run reports both failures. Update the expectations and placement fixtures to match the intended fast-tier behavior.

[teamleaderleo]

Review (independent Codex gpt-6.1-sol, high, via the review router; full diff from merge base 97133a4 to this head): BLOCK at be77727.

[P2] .github/workflows/cmux-next.yml:2668 - Bot attempt 2 ignores freshly assigned owned capacity. rerun_main() assigns an idle mini through owned_jobs, emits no runner, and clears retry_runner. The new owned branch excludes bot retries, so fast-tier falls through to Blacksmith even when that pool is stalled and a mini is available. Route freshly assigned owned jobs to the selected side label on bot attempt 2.

[teamleaderleo]

Review (codex, router-review): SHIP at 9161731

Independent gpt-6-astra review: SHIP. No blocking findings or non-blocking notes. All 530 tests across the five affected test modules passed; routing, timeout fallback, and active-PR preservation match the stated behavior.

[teamleaderleo]

Review (codex, router-review): SHIP at d9f7c20

Independent full-range review 0eda1f3..d9f7c20. Reviewer: "SHIP ... No blocking findings. Focused CI tests passed: 530 tests." Visual verdict N/A: CI-only change.