[teamleaderleo]

Leo, 2026-10-28: "I hate pins. Like, just pin the set of the stuff that isn't hilariously slow."

Cause

On 2026-10-31, GhosttyKit release check (job 124413283263, run 41158893861, PR #28749) was queued from 02:18:12Z to 02:59:09Z on blacksmith-4vcpu-ubuntu-2404. During that time, the 16 and 32 vCPU sizes were starting jobs.

Its runs-on was ... || vars.LINUX_RUNNER || 'blacksmith-4vcpu-ubuntu-2404'. That reads like a live source, but it is one size:

• LINUX_RUNNER holds a single label, blacksmith-4vcpu-ubuntu-2404, since 2026-10-29.
• The overflow switch (cloud_overflow_switch.py) changes it only when every probed Blacksmith size stops starting jobs. A stall of one size changes nothing.

Why no test caught it:

• tests/test_ci_runner_set.py (the "no pins" test) exists only on main. Pull requests into feat-cmux-next run this branch's ci.yml.
• That test counts vars.LINUX_RUNNER as a live source anyway.

Fix

• ci.yml, ghosttykit-release-check: after the fork branches, the job reads fromJSON(vars.RUNNER_POOLS).linux, the Linux pool that runner_pools.py keeps on a live member (now blacksmith-32vcpu-ubuntu-2404, then 16 vCPU). main's cmux-tui-spec and required-checks-drift already read it this way.

• New tests/test_ci_linux_single_size.py (registered in linux-guard) counts, per workflow, the jobs whose Linux runner is LINUX_RUNNER plus one literal with no picker, matrix, input or pool ahead of it. That is 259 jobs in 137 workflows. The count can only shrink:

  • a new single-size job fails the test;
  • a workflow that drops to fewer such jobs fails until its count is lowered, so a moved job can't come back.

  It also checks that the GhosttyKit job reads the pool.

• test_ci_repo_variable_defaults.py and test_runner_label_policy.py take main's RUNNER_POOLS allowances verbatim:

  • a pool read after the fork pull request branch needs no literal fallback;
  • RUNNER_POOLS is a JSON map, so it is not a runs-on label variable.

Red/green

tests/test_ci_linux_single_size.py with the old ci.yml fails 2 tests: the GhosttyKit job is single-size, and ci.yml has 8 single-size jobs against its count of 7. On this branch it passes.

These pass as single files:

• test_ci_runs_on_values.py
• test_ci_blacksmith_linux_failover.py
• test_ci_fork_runner_routing.py
• test_runs_on_labels.py
• test_ci_linux_guard_routing.py
• test_ci_merge_queue_required_checks.py
• test_runner_label_policy.py
• test_ci_test_execution_registry.py

These fail exactly as they do on the base, test by test:

• test_ci_repo_variable_defaults.py (three nightly.yml lines)
• test_ci_change_areas.py (test_repository_helpers_route_by_where_they_run)
• test_ci_runs_on_fallback.py (two signing and side-label tests)
• test_ci_self_hosted_guard.sh (check_no_self_hosted_fleet_runners)

Other pins found (follow-ups, not in this PR)

• 259 jobs in 137 workflows on this branch read LINUX_RUNNER as one size. The most are in backend.yml (12), cmux-next-guards.yml (10), sdk-release-cut.yml (9), ci.yml (7) and cmux-vm.yml (7). They move a workflow at a time, each with its count lowered here.
• The queue janitor's own sweep job (ci-queue-janitor.yml on main) picks from LINUX_RUNNER, which holds 4 vCPU. On 2026-10-31 it waited 16 minutes on the same stalled label (job 124419676183).

🤖 Generated with Claude Code

Agent session: agent session

[teamleaderleo]

Owned by an agent session · session

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 319368f: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28858 -f ref=dep-fleet-3--ghosttykit-runner-pool -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4118163201202 (05730c30a0)
• cmux NEXT nightly 1.0.0-nightly.4118400110201 (c904112cd1)
• cmux NEXT nightly 1.0.0-nightly.4118583484401 (7bda8ef95e)
• cmux NEXT nightly 1.0.0-nightly.4118647226102 (5da935675f)
• cmux NEXT nightly 1.0.0-nightly.4118757870202 (49199c2f17)

[blacksmith-sh]

Found 3 test failures on Blacksmith runners:

Failures

Test	View Logs
test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_ a_picked_signing_label_has_a_rerun_alternative)/ test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_ a_picked_signing_label_has_a_rerun_alternative)	View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/ test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)	View Logs
test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes t_no_runs_on_chain_ends_on_the_reserved_side_label)/ test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes t_no_runs_on_chain_ends_on_the_reserved_side_label)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}