ci: one routing source, RUNNER_POOLS, and the reconciler that keeps it live

.github/runners.json

@@ -52,5 +52,72 @@
     "manaflow-ai": "blacksmith"
   },
 
-  "default_fleet": "hosted"
+  "default_fleet": "hosted",
+
+  "pools": {
+    "$comment": [
+      "The one routing source (scripts/ci/runner_pools.py, docs/ci-runners.md \"Runner pools\").",
+      "Each pool is a class of work and its members in order. The RUNNER_POOLS repository variable",
+      "holds each pool's live member; a job reads fromJSON(vars.RUNNER_POOLS).<pool> and names no",
+      "label itself (tests/test_ci_runner_set.py). runner_pools.py moves a pool to its next member",
+      "when the queue janitor or the overflow probe sees the current one start nothing.",
+      "Members are ephemeral only: owned Macs come first through a picker's output, never a pool.",
+      "Only a pool marked hosted may name a GitHub-hosted label, and it says why."
+    ],
+    "linux": {
+      "why": "Every Linux job: the runner set's Blacksmith sizes (Leo, 2026-10-28: \"just pin the set of the shit that isn't hilariously slow\")",
+      "members": [
+        "blacksmith-32vcpu-ubuntu-2404",
+        "blacksmith-16vcpu-ubuntu-2404"
+      ]
+    },
+    "linux_arm64": {
+      "why": "Native ARM64 Linux package checks",
+      "members": [
+        "blacksmith-4vcpu-ubuntu-2404-arm"
+      ]
+    },
+    "windows": {
+      "why": "Windows builds and tests",
+      "members": [
+        "blacksmith-4vcpu-windows-2025"
+      ]
+    },
+    "macos_15": {
+      "why": "The macOS 15 image (its default SDK), after any owned-Mac picker",
+      "members": [
+        "blacksmith-6vcpu-macos-15"
+      ]
+    },
+    "macos_26": {
+      "why": "The macOS 26 image, after any owned-Mac picker",
+      "members": [
+        "blacksmith-6vcpu-macos-26",
+        "blacksmith-12vcpu-macos-26"
+      ]
+    },
+    "macos_26_large": {
+      "why": "macOS 26 work that needs the 12 vCPU, 48 GB SKU",
+      "members": [
+        "blacksmith-12vcpu-macos-26",
+        "blacksmith-6vcpu-macos-26"
+      ]
+    },
+    "provenance": {
+      "why": "npm and PyPI provenance publishing and artifact attestations",
+      "hosted": true,
+      "hosted_why": "npm provenance, PyPI trusted publishing and artifact attestations accept only GitHub's OIDC from a GitHub-hosted runner",
+      "members": [
+        "ubuntu-24.04"
+      ]
+    },
+    "macos_intel": {
+      "why": "The Intel macOS leg of the dispatch-only compatibility suite",
+      "hosted": true,
+      "hosted_why": "Blacksmith has no Intel macOS image; GitHub's macos-15-intel is the only x86_64 Mac",
+      "members": [
+        "macos-15-intel"
+      ]
+    }
+  }
 }

.github/workflows/ci-cloud-overflow-probe.yml

@@ -137,6 +137,21 @@ jobs:
             MACOS_RUNNER_TESTS=${{ vars.MACOS_RUNNER_TESTS }}
         run: python3 scripts/ci/cloud_overflow_switch.py
 
+      # The one routing source (docs/ci-runners.md "Runner pools"): point each pool in
+      # RUNNER_POOLS at its first member that is not dead, from the probe's per-label health
+      # and the queue janitor's snapshot. Runs whatever the switch decided, with its token.
+      - name: Point the runner pools at live members
+        if: always() && steps.switch.outcome != 'skipped'
+        env:
+          GH_TOKEN: ${{ github.token }}
+          GH_REPO: ${{ github.repository }}
+          SWITCH_TOKEN: ${{ steps.switch-token.outputs.token }}
+          DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && (inputs.dry_run || inputs.probe_label != '') && 'true' || 'false' }}
+          RUNNER_POOLS: ${{ vars.RUNNER_POOLS }}
+          # Read live through the API first; this is the job-start value, the fallback.
+          CI_CLOUD_LABEL_HEALTH: ${{ vars.CI_CLOUD_LABEL_HEALTH }}
+        run: python3 scripts/ci/runner_pools.py
+
       # Last, in its own step so the summary above is already saved. If the
       # switch did not observe every probe start (one may still be queued),
       # cancel this run even when the switch failed before it could write its

.github/workflows/ci-guards.yml

@@ -150,6 +150,10 @@ jobs:
         if: ${{ matrix.group == 'ci' }}
         run: python3 tests/test_ci_cloud_overflow_switch.py
 
+      - name: Validate the runner pools and their reconciler
+        if: ${{ matrix.group == 'ci' }}
+        run: python3 tests/test_ci_runner_pools.py
+
       - name: Validate fast checks fail over with the overflow switch
         if: ${{ matrix.group == 'ci' }}
         run: python3 tests/test_ci_fast_checks_failover.py

.github/workflows/ci-repo-variables.yml

@@ -28,6 +28,8 @@ on:
       - scripts/ci/runner_label_policy.py
       - scripts/ci/runs_on_values.py
       - scripts/ci/signing_runner_variables.py
+      - scripts/ci/runner_pools.py
+      - .github/runners.json
   workflow_dispatch:
 
 permissions:
@@ -88,6 +90,14 @@ jobs:
             WINDOWS_RUNNER=${{ vars.WINDOWS_RUNNER }}
         run: python3 scripts/ci/check_repo_variables.py
 
+      # RUNNER_POOLS, the one routing source (scripts/ci/runner_pools.py): every pool set, each to
+      # one of its declared members. A missing pool fails every job that reads it.
+      - name: Check the runner pools
+        if: ${{ !cancelled() }}
+        env:
+          RUNNER_POOLS: ${{ vars.RUNNER_POOLS }}
+        run: python3 scripts/ci/runner_pools.py --check
+
       # No runs-on waits on one machine through a variable's value
       # (scripts/ci/runs_on_values.py). The workflow text passes
       # `vars.X || <pool>` however X is set, and a set X never falls through,
@@ -135,6 +145,7 @@ jobs:
             MACOS_RUNNER_PR=${{ vars.MACOS_RUNNER_PR }}
             MACOS_RUNNER_TESTS=${{ vars.MACOS_RUNNER_TESTS }}
             NIGHTLY_NEXT_TRAIN_RUNNER=${{ vars.NIGHTLY_NEXT_TRAIN_RUNNER }}
+            RUNNER_POOLS=${{ vars.RUNNER_POOLS }}
             WINDOWS_RUNNER=${{ vars.WINDOWS_RUNNER }}
         run: |
           set -euo pipefail

.github/workflows/cmux-tui-spec.yml

@@ -22,7 +22,7 @@ permissions:
 
 jobs:
   inventory:
-    runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository && 'blacksmith-4vcpu-ubuntu-2404' || vars.LINUX_RUNNER || 'blacksmith-4vcpu-ubuntu-2404' }}
+    runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository && 'blacksmith-32vcpu-ubuntu-2404' || fromJSON(vars.RUNNER_POOLS).linux }}
     timeout-minutes: 5
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

.github/workflows/required-checks-drift.yml

@@ -48,7 +48,7 @@ concurrency:
 jobs:
   reconcile:
     name: Reconcile required checks with GitHub
-    runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository && 'blacksmith-4vcpu-ubuntu-2404' || vars.LINUX_RUNNER || 'blacksmith-4vcpu-ubuntu-2404' }}
+    runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository && 'blacksmith-32vcpu-ubuntu-2404' || fromJSON(vars.RUNNER_POOLS).linux }}
     timeout-minutes: 10
     steps:
       - name: Check out the reconciliation

docs/ci-runners.md

@@ -970,6 +970,73 @@ caller.
 Hosted/isolated fallback remains available when the shared host refuses local
 admission or is draining, pressured, or unavailable.
 
+## Runner pools
+
+Leo, 2026-10-28: "I hate pins. Like, just pin the set of the shit that isn't
+hilariously slow. Like most of what GitHub has is hilariously slow. We can't
+be using them."
+
+One routing source. `.github/runners.json` declares each pool once under
+`pools`: a class of work and its members, in order. The `RUNNER_POOLS`
+repository variable holds each pool's live member as one JSON object, and a job
+reads it instead of naming a label:
+
+```yaml
+runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || fromJSON(vars.RUNNER_POOLS).linux }}
+```
+
+A picker's output still comes first where a run needs its own choice (an owned
+Mac, warm affinity); the pool is what follows it, never a literal label.
+`fromJSON()` of an unset `RUNNER_POOLS` fails the workflow at once, which is
+the point: a missing route is a red run, not a job queued forever on a label
+nobody serves.
+
+| Pool | Members (first is preferred) |
+| --- | --- |
+| `linux` | Blacksmith 32 vCPU, Blacksmith 16 vCPU |
+| `linux_arm64` | Blacksmith 4 vCPU ARM64 |
+| `windows` | Blacksmith 4 vCPU Windows |
+| `macos_15` | Blacksmith macOS 15 |
+| `macos_26` | Blacksmith macOS 26 (6, then 12 vCPU) |
+| `macos_26_large` | Blacksmith macOS 26 (12, then 6 vCPU) |
+| `provenance` | GitHub-hosted `ubuntu-24.04`: npm and PyPI provenance and artifact attestations accept only GitHub's OIDC |
+| `macos_intel` | GitHub-hosted `macos-15-intel`: Blacksmith has no Intel Mac (the dispatch-only compatibility leg) |
+
+Every member outside the two hosted pools is an ephemeral Blacksmith label, so
+a job reading a pool never lands on a reused machine, whoever started the run.
+Owned Macs come first through a picker job's output (`simple_pool_picker.py`,
+`side_lane_placement.py`, the nightly pickers) in the routing order: owned
+Macs, then Blacksmith macOS. The pool is what follows the picker. Jobs that
+hold signing secrets keep their literal ephemeral label for now and move
+through the stall-aware signing picker, not a pool.
+
+`scripts/ci/runner_pools.py` is the one writer, and it only chooses. The
+overflow probe's `watch` job runs it every 10 minutes, after the switch, with
+the token the switch already uses (the `manaflow-glaeda-route` App; no new
+credential). It reads per-label start evidence: the probe's per-label health
+(`CI_CLOUD_LABEL_HEALTH`, every Linux member probed), and the queue janitor's
+snapshot (a Blacksmith label whose oldest queued job waited 10 minutes, the
+same stall rule as `simple_pool_picker.blacksmith_stalled()`, and the
+dead-label entries once #27114 lands). It marks those labels dead, points each
+pool at its first unmarked member, and writes `RUNNER_POOLS` only on a change.
+A mark expires after 30 minutes unless seen again; a label seen starting a job
+is cleared at once. With every member marked, the pool keeps its first. A
+malformed `RUNNER_POOLS` is rewritten from these declarations, never crashed
+on. Jobs already queued on a dead member keep that label (GitHub binds it when
+it creates the job); moving them is the queue janitor's dead-label re-dispatch
+(with #27114), whose re-run reads the pool's new member.
+With no evidence at all (no probe health yet, no snapshot) it marks nothing
+dead: it fails closed and never retires a member on a guess. A pool not marked
+`hosted` never resolves to a GitHub-hosted label, whatever is dead.
+`ci-repo-variables.yml` runs `runner_pools.py --check` every 15 minutes, so a
+hand-edited `RUNNER_POOLS` that drops a pool or names a non-member goes red.
+`tests/test_ci_runner_pools.py` checks the declared pools and the reconciler
+against a recorded janitor snapshot.
+
+The first jobs on the pool are `cmux-tui-spec.yml` and
+`required-checks-drift.yml`; the rest of the workflows follow by group, and
+`tests/test_ci_runner_set.py` lands with the last group.
+
 ## Blacksmith outage: the overflow switch
 
 This is the only Blacksmith-outage lever. The 2026-09-29 manual variable