[teamleaderleo]

Problem

The build-fleet gateway jobs ran on vars.CI_COMPILE_CHECK_GATEWAY (glaeda-side-light-xcode-26.6) unconditionally:

• cmux-next app compile (build fleet)
• cmux-next Swift canary (build fleet)
• the push's cmux-next Release compile (fleet fast check)

At 06:00Z on 2026-10-26 that label had 5 jobs running and 44 queued, the oldest for 3457 s. cmux17, 18, 21 and 24 were reserved, and austin-0's side runner was deleted. PR #24155's compile jobs 121392651085, 121406074285 and 121421789999 queued there and never ran; each newer push cancelled the last.

Nothing moved them:

• next_macos_cascade.py places only fast-tier, swift-canary, cmux-scheme-compile and release-compile.
• The owned-pool rescue re-runs a job onto the same runs-on, which here was a fixed variable.
• The janitor's unclaimed-job mover skips cmux-next owned legs.

Fix

macos-placement now judges the gateway (side_lane_placement.gateway_route()) from the runners API it already reads. It takes:

1. The gateway label, while it has an idle runner for each gateway job this run starts (GATEWAY_JOBS), beyond those this run's own racing Mac jobs take (gateway_taken()). That count is the side jobs in CASCADE_RUNS, minus the Mac jobs a gateway job stands in for, plus a routed fast tier.
2. Otherwise the same label on a wider class (compatible_labels(): the std side minis), on the same condition.
3. Otherwise nothing.

A gateway job runs only on a gateway_runner that placement wrote in the same run attempt (gateway_attempt). In every other case the job skips and its Mac fallback runs at once. That covers no idle runner, unreadable runners, a re-run that reuses an earlier attempt's placement, a skipped placement, and an older base script:

• app-compile-fleet: release-compile runs (it already runs on no fleet verdict). For a PR it joins CASCADE_RUNS (GATEWAY_CASCADE), so it takes an idle owned mini first and then a free Blacksmith slot.
• swift-canary-fleet: the Mac swift-canary runs, and is already in the cascade.
• release-compile-fast (push): release-compile runs on the push's Mac route; release-compile-fast-admission skips. Before, release-compile waited for the fast check to leave the gateway queue.

The bound is the placement time, about a minute: after that a gateway job either runs on a runner seen idle or its Mac fallback runs. The idle reading is a snapshot, so a concurrent run can still take that runner first. That is the one remaining way a gateway job can wait.

Guard test

tests/test_cmux_next_checks_workflow.py::BuildFleetGatewayJobsFallBack is table-driven over the workflow's jobs.

• Any job whose runs-on names the gateway (CI_COMPILE_CHECK_GATEWAY, a side-light label or gateway_runner) must be in its FALLBACKS table.
• Across attempts and placement readings (idle on light, idle on std, none idle, no reading, stale attempt), the job runs only on an idle reading from the same attempt, and runs-on is that runner.
• Otherwise its Mac fallback's if is true and the fallback is in the cascade (CASCADE_RUNS plus GATEWAY_CASCADE, and next_macos_cascade.CASCADE_JOBS).
• Mutation check: dropping the guard from swift-canary-fleet, or adding a new job with runs-on: ${{ vars.CI_COMPILE_CHECK_GATEWAY }}, fails it.

Risk

• The gateway jobs now start after macos-placement, which does one runners read plus the cascade's read, in under 2 min.
• When the fleet is tight, a PR's Release compile or a push's Release compile runs on an owned mini or on Blacksmith macOS instead of waiting for the fleet controller. That follows the routing order.
• The first run of this PR uses the base's placement script, which writes no gateway_attempt, so its gateway jobs skip and the Mac fallbacks run.
• Rollback is a revert.

Tests

• tests/test_ci_side_lane_placement.py::GatewayRoute:
  • routing: idle own label, spill to std side, nothing idle, unread runners
  • per-job and racing-job counts
  • cascade dedupe
  • main() end to end, including gateway_attempt on attempt 2
• tests/test_cmux_next_checks_workflow.py: the guard test above, plus the app compile's if/runs-on and the placement env (GATEWAY_JOBS, GATEWAY_CASCADE) across same-repo, fork, full-ci, push and no-native.
• On cmux15, over the placement, workflow, cascade, fork-routing, Xcode-selection and supply-chain test files: the same 20 failures at base and at head. They come from files missing in the partial copy, plus the existing blacksmith-2vcpu-ubuntu-2404 fork-routing red. Head has 13 more passing tests.

Agent session: cosmic-gray

🤖 Generated with Claude Code

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review f2ece72: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr25360 -f ref=deputy-2/fleet-gateway-placement -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 5182c31 to compare; do not merge until green; see lane-rules

[teamleaderleo]

Owned by an agent session · session

[teamleaderleo]

Bead cx-p7nz4 created for this PR by the cmux coordinator.

[blacksmith-sh]

Found 7 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmp9q8_uj5b/summary-missing.json (live):/summary-missing.json (live):	View Logs
/tmp/tmppsx2c3jh/summary-fail.json (live):/summary-fail.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
convention violations found (ERROR lines above)./ convention violations found (ERROR lines above).	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

cmux-next / cmux-next checks (god files, concurrency, crash safety, l10n) › Script tests (pin-cmux-tui, embed-cef, ensure-cef store) is red on 11 of 12 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[teamleaderleo]

cmux-next fast gates / post cmux-next/fast-gates is red on 12 of 14 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[teamleaderleo]

high-signal review link / link is red on 13 of 15 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[github-actions]

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4030749448501 (b781e3b4a0)
• cmux NEXT nightly 1.0.0-nightly.4031371403702 (1dc93b2657)