ci: route focused tests to one front door, and half of them to 12 vCPU

.github/workflows/perf-activation.yml

@@ -179,7 +179,7 @@ jobs:
 
           # Guard against a stale/poisoned Swift-package cache: resolve can
           # report success without binary artifacts materialized. Matches the
-          # check test-e2e.yml and test-depot.yml already run; this workflow
+          # check test-e2e.yml and test-macos-suite.yml already run; this workflow
           # now restores from the same shared `spm-` namespace they do.
           for attempt in 1 2 3; do
             if xcodebuild -project cmux.xcodeproj -scheme cmux -configuration Debug \

.github/workflows/test-e2e.yml

@@ -37,6 +37,7 @@ on:
           - auto
           - blacksmith-6vcpu-macos-15
           - blacksmith-6vcpu-macos-26
+          - blacksmith-12vcpu-macos-26
           - blacksmith-6vcpu-macos-latest
           - tart-canary
           - tart-dual

.github/workflows/test-depot.yml → .github/workflows/test-macos-suite.yml

@@ -1,4 +1,11 @@
-name: Run macOS tests
+# A whole macOS test target, compiled cold on the repository's pool.
+#
+# For one test, or a few, use `./scripts/run-e2e.sh <selector> --ref <sha>`
+# instead. It runs the same app-host tests through test-e2e.yml, reuses a run
+# already in flight at that commit, refuses a selector that already failed
+# there, and batches several selectors against one compile. This workflow does
+# none of that, so a single-test dispatch here pays a full cold build.
+name: Run macOS test suite
 run-name: >-
   macOS tests @ ${{ inputs.ref || github.ref_name }} ·
   unit=${{ inputs.skip_unit_tests && 'off' || inputs.unit_test_suites || 'all' }} ·
@@ -42,7 +49,7 @@ on:
         default: false
         type: boolean
       unit_test_suites:
-        description: "Comma-separated unit suites, or Suite/testName (XCTest) / Suite/testName() (Swift Testing) for one test; empty runs the full unit target"
+        description: "Comma-separated unit suites; empty runs the full unit target. For one test use ./scripts/run-e2e.sh, which reuses compiles and in-flight runs"
         required: false
         default: ""
       test_filter:
@@ -62,8 +69,8 @@ jobs:
 
   tests:
     needs: resolve-ref
-    # Depot is retired for cmux CI. Keep this reusable test lane on the
-    # repository's normal macOS runner and do not expose a provider override.
+    # Keep this lane on the repository's normal macOS runner and do not
+    # expose a provider override.
     runs-on: ${{ vars.MACOS_RUNNER_TESTS || 'blacksmith-6vcpu-macos-15' }}
     timeout-minutes: ${{ inputs.unit_test_suites != '' && 35 || 20 }}
     steps:

docs/ci-runners.md

@@ -24,7 +24,7 @@ gh variable list --repo manaflow-ai/cmux
 | `LINUX_ARM64_RUNNER` | native ARM64 package entrypoint verification | `ubuntu-24.04-arm` | `ubuntu-24.04-arm` |
 | `MACOS_RUNNER_15` | the macOS 15 default: `macos-compile-admission`, `app-host-unit-tests`, nightly helper and test-cache jobs, `iroh-release-gate.yml` streamed validation | `blacksmith-6vcpu-macos-15` | `blacksmith-6vcpu-macos-15` |
 | `MACOS_RUNNER_PR` | **pull-request** macOS jobs only, in `ci-macos.yml`, `cli-pipe-regressions.yml`, `terminal-hang-diagnostics.yml`, `ci.yml` (`claude-wrapper`) and `nightly.yml` (`refresh-test-compilation-cache`) | unset (see "Lanes" below) | `blacksmith-6vcpu-macos-15` |
-| `MACOS_RUNNER_TESTS` | the manual test-debugging lanes: `test-e2e.yml` and `test-depot.yml` | unset (see "Lanes" below) | `blacksmith-6vcpu-macos-26` for `test-e2e.yml`, `blacksmith-6vcpu-macos-15` for `test-depot.yml` |
+| `MACOS_RUNNER_TESTS` | the manual test-debugging lanes: `test-e2e.yml` and `test-macos-suite.yml` | unset (see "Lanes" below) | `blacksmith-6vcpu-macos-26` for `test-e2e.yml`, `blacksmith-6vcpu-macos-15` for `test-macos-suite.yml` |
 | `MACOS_RUNNER_DUAL_XCODE` | `swift-package-tests` (SDK 15 release helper, then SDK 26 package tests) on **every** event, pull requests included | `blacksmith-6vcpu-macos-15` | `blacksmith-6vcpu-macos-15` |
 | `MACOS_RUNNER_26` | the macOS 26 image: compatibility jobs, `release.yml` and nightly sign/notarize, the disk-heavy `release-build` universal app, and the nightly compilation-cache warmer | `blacksmith-6vcpu-macos-26` | `blacksmith-6vcpu-macos-26` |
 | `MACOS_RUNNER_26_LARGE` | the larger macOS 26 machine: changed-revision universal Nightly app builds | `blacksmith-12vcpu-macos-26` | `blacksmith-12vcpu-macos-26` |
@@ -74,12 +74,16 @@ the same cost profile or the same urgency.
 - **Pull requests** resolve through `MACOS_RUNNER_PR` first. Unset means
   Blacksmith. PR runs are cancelled on supersession by design, so they are the
   wrong place to spend elastic paid capacity.
-- **Manual test debugging** (`test-e2e.yml`, `test-depot.yml`) resolves through
+- **Manual test debugging** (`test-e2e.yml`, `test-macos-suite.yml`) resolves through
   `MACOS_RUNNER_TESTS`, and deliberately does **not** follow `MACOS_RUNNER_15`.
   Re-running one test to chase a flake should never reach for paid capacity.
   Both fallbacks stay on Blacksmith for that reason; `test-e2e.yml` falls back
   to macOS 26 because the macOS 15 pool's queue-to-start p90 was 83 min against
   1.0 min on 26, measured over 60 dispatches on 2026-09-22/23.
+  `scripts/run-e2e.sh` then sends commits whose SHA ends in an odd hex digit
+  to `blacksmith-12vcpu-macos-26`, so the two instance sizes are compared on
+  real focused-run traffic. It splits only that free default: a
+  `MACOS_RUNNER_TESTS` value naming any other pool is used unchanged.
 
 `MACOS_RUNNER_PR` does not move a lane on its own. The two images carry
 different Xcodes -- the `macos-15` image ships `/Applications/Xcode_26.3.app`

docs/ci/workflow-inventory.md

@@ -37,7 +37,7 @@ The required checks on `main` (ruleset "main: block force-push or delete") are `
 - `web-complexity-trusted.yml` and `cla-policy-guard.yml` together ran about 27,000 times. That is roughly 3x the PR events, because `pull_request_target` also fires on `edited`. Do not drop `edited` blindly: it also carries base-branch changes.
 - `web-complexity.yml` and `web-complexity-trusted.yml` are an intentional pair: an untrusted diagnostic plus the trusted required check. Not a duplicate.
 - `ci-status-fallback.yml` has been a dispatch-only `echo` since 09-18. The 3,697 runs this week came from before that change.
-- `test-depot.yml` is named "Run macOS tests", and its own comment says Depot is retired. It is dispatched alongside `test-e2e.yml`.
+- `test-macos-suite.yml` (formerly `test-depot.yml`) is named "Run macOS test suite". It runs a whole test target cold; focused runs belong on `test-e2e.yml` through `scripts/run-e2e.sh`.
 
 **Other problems found along the way:** `release.yml` has not succeeded since 08-03 (both runs this week failed), and `iroh-release-gate.yml` not since 08-14. `tmux-corpus.yml` is `disabled_manually`. `ci-macos-compat.yml` fails actionlint on the unknown label `macos-15-intel`, and `test-ios.yml:21` fails actionlint with an empty string.
 
@@ -55,7 +55,7 @@ Sorted by estimated runner minutes. Trigger abbreviations: pr = pull_request, pr
 | `cmux-tui-sdks.yml` | push pr dispatch (paths) | blacksmith | 453 | 235 / 82 / 0 / 132 | 7,271 | 2026-09-22 | lawrencecchen #9215 | Lawrence Chen 2026-09-16 | 1 docs/tests |  |
 | `test-e2e.yml` | dispatch | blacksmith/warp/macos | 959 | 270 / 531 / 0 / 158 | 6,730 | 2026-09-22 | lawrencecchen #778 | Leo 2026-09-21 | 6 docs/tests | 55% fail |
 | `cmux-tui.yml` | dispatch | blacksmith/macos/gh-ubuntu | 114 | 37 / 47 / 0 / 30 | 5,696 | 2026-09-22 | lawrencecchen #7710 | Austin Wang 2026-09-12 | 1 wf refs; 5 docs/tests |  |
-| `test-depot.yml` | call dispatch | blacksmith/warp/macos | 348 | 93 / 213 / 0 / 42 | 5,335 | 2026-09-22 | lawrencecchen #710 | Leo 2026-09-21 | 2 docs/tests | name says Depot, which is retired; 61% fail |
+| `test-macos-suite.yml` | call dispatch | blacksmith/warp/macos | 348 | 93 / 213 / 0 / 42 | 5,335 | 2026-09-22 | lawrencecchen #710 | Leo 2026-09-21 | 2 docs/tests | renamed from test-depot.yml; 61% fail, mostly single-test dispatches that belong on run-e2e.sh |
 | `reload-build.yml` | dispatch | blacksmith/macos | 276 | 107 / 71 / 0 / 98 | 4,689 | 2026-09-22 | lawrencecchen #6354 | Leo 2026-09-21 | 1 wf refs; 3 docs/tests |  |
 | `cla-policy-guard.yml` | pr_target | gh-ubuntu | 13,584 | 12,311 / 1,272 / 0 / 1 | 2,445 | 2026-09-22 | lawrence703 #11387 | Lawrence Chen 2026-09-02 | REQUIRED: CLA policy guard; 2 docs/tests |  |
 | `remote-daemon.yml` | pr push dispatch (paths) | blacksmith/warp/macos | 189 | 116 / 7 / 0 / 65 | 2,178 | 2026-09-22 | austinywang #12720 | austinpower1258 2026-09-15 | none |  |

scripts/ci/dispatch-focused-test.py

@@ -29,11 +29,19 @@
     "auto",
     "blacksmith-6vcpu-macos-15",
     "blacksmith-6vcpu-macos-26",
+    "blacksmith-12vcpu-macos-26",
     "blacksmith-6vcpu-macos-latest",
     "tart-canary",
     "tart-dual",
     "tart-small",
 )
+# Half of all commits compile on the large macOS 26 SKU, so the two sizes are
+# compared on real focused-run traffic rather than one benchmark. The split is
+# keyed on the commit, not drawn at random: every dispatch at one commit lands
+# on one pool, which is what in-flight reuse, the failed-selector refusal and
+# the product contract all match on.
+SMALL_RUNNER = "blacksmith-6vcpu-macos-26"
+LARGE_RUNNER = "blacksmith-12vcpu-macos-26"
 SELECTOR = re.compile(
     r"(?:(?:cmuxTests|cmuxUITests)/)?"
     r"[A-Za-z_][A-Za-z0-9_]*(?:/[A-Za-z_][A-Za-z0-9_]*(?:\(\))?)?"
@@ -209,6 +217,17 @@ def default_runner() -> str | None:
     return literal.group(1) if literal else None
 
 
+def routed_runner(commit: str, default: str | None) -> str | None:
+    """The pool an unpinned dispatch at `commit` runs on.
+
+    Only the free default is split. A repository variable naming any other
+    pool is an admin decision, and it wins unchanged.
+    """
+    if default == SMALL_RUNNER and int(commit[-1], 16) % 2:
+        return LARGE_RUNNER
+    return default
+
+
 def attempts(
     runs: list[dict], commit: str, selector: str, runner: str | None = None
 ) -> list[dict]:
@@ -391,12 +410,14 @@ def main() -> int:
     if args.ref is None and commit != requested_ref:
         raise ValueError("GitHub revision differs from local HEAD; push the intended commit first")
 
+    # Which pool this dispatch will actually land on. None means the answer
+    # could not be established, and the in-flight guards below stay silent
+    # rather than compare against a runner they guessed.
+    pinned = args.runner not in (None, "auto")
+    runner = args.runner if pinned else routed_runner(commit, default_runner())
+
     if not args.force:
         history = recent_dispatches()
-        # Which pool this dispatch will actually land on. None means the
-        # answer could not be established, and the in-flight guards below stay
-        # silent rather than compare against a runner they guessed.
-        runner = args.runner if args.runner not in (None, "auto") else default_runner()
 
         if runner is not None:
             # An identical dispatch is already answering this exact question on
@@ -471,6 +492,8 @@ def main() -> int:
     }
     if args.runner is not None:
         fields["runner"] = args.runner
+    if not pinned and runner == LARGE_RUNNER:
+        fields["runner"] = runner
     command = ["gh", "workflow", "run", WORKFLOW, "--repo", REPO]
     if args.workflow_ref:
         command.extend(["--ref", args.workflow_ref])

skills/cmux-testing/references/local-vs-ci-validation.md

@@ -31,7 +31,9 @@ Dispatch through the wrapper. It pins the exact pushed commit, carries a `dispat
 
 **Compile the test target locally before dispatching.** One focused run costs 10-20 macOS runner-minutes, and it compiles the whole tree before it runs anything, so the most common red result on a feature branch is a Swift compile error rather than a test failure. The `cmux-unit` command above catches those in a fraction of the time and without a runner.
 
-**The default runner is `blacksmith-6vcpu-macos-26`.** `--runner` overrides it. Over 60 consecutive dispatches (2026-09-22/23) the macOS 15 pool queued for a median 2.4 min but a p90 of 83 min and a worst case of 178 min, while macOS 26 queued 0.3 min median / 1.0 min p90; execution time on 26 ran about 4 min longer. Pick `blacksmith-6vcpu-macos-15` explicitly only when the question is specifically about macOS 15 behavior, and expect to wait for it.
+**The wrapper picks the runner; leave `--runner` off.** Commits whose SHA ends in an odd hex digit compile on `blacksmith-12vcpu-macos-26`, the rest on `blacksmith-6vcpu-macos-26`, so the two sizes are compared on real traffic. Every dispatch at one commit lands on the same pool, which is what lets the wrapper reuse a run already in flight there. Over 60 consecutive dispatches (2026-09-22/23) the macOS 15 pool queued for a median 2.4 min but a p90 of 83 min and a worst case of 178 min, while macOS 26 queued 0.3 min median / 1.0 min p90. Pass `--runner blacksmith-6vcpu-macos-15` only when the question is specifically about macOS 15 behavior, and expect to wait for it.
+
+**Do not dispatch `test-macos-suite.yml` for one test.** It runs a whole test target, compiles cold every time, and has none of the wrapper's reuse or refusal. A single-test dispatch there costs about 20 macOS runner-minutes for an answer the wrapper would share.
 
 **Do not re-dispatch the same selector at the same commit.** A focused run's result is a property of the commit; repeating it reprints the same failure at full cost. The wrapper now refuses a selector that already failed at that commit and points at the earlier run; read that run, fix the branch, push, and dispatch the new commit. `--force` exists for the rare case where you know the failure was infrastructure.
 

tests/test_ci_app_host_test_output.py

@@ -14,10 +14,10 @@
 
 ROOT = pathlib.Path(__file__).resolve().parents[1]
 SCRIPT = ROOT / "scripts/ci/classify-app-host-test-output.py"
-TEST_DEPOT_WORKFLOW = ROOT / ".github/workflows/test-depot.yml"
-TEST_DEPOT_RUN_UNIT_TESTS = next(
+SUITE_WORKFLOW = ROOT / ".github/workflows/test-macos-suite.yml"
+SUITE_RUN_UNIT_TESTS = next(
     step["run"]
-    for step in yaml.safe_load(TEST_DEPOT_WORKFLOW.read_text(encoding="utf-8"))["jobs"]["tests"]["steps"]
+    for step in yaml.safe_load(SUITE_WORKFLOW.read_text(encoding="utf-8"))["jobs"]["tests"]["steps"]
     if step.get("name") == "Run unit tests"
 )
 SPEC = importlib.util.spec_from_file_location("classify_app_host_test_output", SCRIPT)
@@ -339,7 +339,7 @@ def test_full_suite_keeps_nonzero_app_host_exit_red_with_clean_summary(self) ->
                 "TEST_RESULTS_ROOT": str(root / "results"),
             }
             completed = subprocess.run(
-                ["bash", "-c", TEST_DEPOT_RUN_UNIT_TESTS],
+                ["bash", "-c", SUITE_RUN_UNIT_TESTS],
                 cwd=root,
                 env=environment,
                 capture_output=True,
@@ -376,7 +376,7 @@ def test_selected_suite_requires_positive_summary_and_keeps_log_artifact(self) -
                     "FAKE_TEST_MODE": mode,
                 }
                 completed = subprocess.run(
-                    ["bash", "-c", TEST_DEPOT_RUN_UNIT_TESTS],
+                    ["bash", "-c", SUITE_RUN_UNIT_TESTS],
                     cwd=root,
                     env=environment,
                     capture_output=True,
@@ -410,7 +410,7 @@ def test_selected_single_test_passes_its_selector_and_names_its_log(self) -> Non
             fake_runner.chmod(0o755)
             results = root / "results"
             completed = subprocess.run(
-                ["bash", "-c", TEST_DEPOT_RUN_UNIT_TESTS],
+                ["bash", "-c", SUITE_RUN_UNIT_TESTS],
                 cwd=root,
                 env={
                     **os.environ,
@@ -453,7 +453,7 @@ def test_selector_rejects_anything_but_suite_or_suite_slash_test(self) -> None:
                 )
                 fake_runner.chmod(0o755)
                 completed = subprocess.run(
-                    ["bash", "-c", TEST_DEPOT_RUN_UNIT_TESTS],
+                    ["bash", "-c", SUITE_RUN_UNIT_TESTS],
                     cwd=root,
                     env={
                         **os.environ,

tests/test_ci_manual_macos_package_cache.py

@@ -10,7 +10,7 @@
 import yaml
 
 ROOT = Path(__file__).resolve().parents[1]
-STEPS = yaml.safe_load((ROOT / '.github/workflows/test-depot.yml').read_text())['jobs']['tests']['steps']
+STEPS = yaml.safe_load((ROOT / '.github/workflows/test-macos-suite.yml').read_text())['jobs']['tests']['steps']
 
 
 def step(name):

tests/test_ci_reusable_workflow_permissions.py

@@ -479,7 +479,7 @@ def test_reader_ignores_lookalike_text_and_accepts_github_yaml_shapes() -> None:
     "reload-build.yml": {
         "build": "ref: ${{ needs.resolve-ref.outputs.sha }}",
     },
-    "test-depot.yml": {
+    "test-macos-suite.yml": {
         "tests": "ref: ${{ needs.resolve-ref.outputs.sha }}",
     },
     "test-e2e.yml": {

tests/test_run_e2e.py

@@ -117,6 +117,41 @@ def test_default_runner_keeps_the_workflow_default(self):
         self.assertEqual(result.returncode, 0, result.stderr)
         self.assertNotIn("runner", self.dispatch())
 
+    def test_odd_commits_compile_on_the_large_sku(self):
+        # REMOTE_HEAD ends in b, so it is routed; HEAD ends in a, so it is not.
+        result = self.launch("cmuxTests/ExampleTests", "--ref", "topic/fix")
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertEqual(self.dispatch()["runner"], "blacksmith-12vcpu-macos-26")
+
+    def test_an_explicit_runner_is_never_rerouted(self):
+        result = self.launch(
+            "cmuxTests/ExampleTests", "--ref", "topic/fix",
+            "--runner", "blacksmith-6vcpu-macos-26",
+        )
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertEqual(self.dispatch()["runner"], "blacksmith-6vcpu-macos-26")
+
+    def test_an_admin_runner_variable_is_never_split(self):
+        result = self.launch(
+            "cmuxTests/ExampleTests", "--ref", "topic/fix",
+            LAUNCHER_VARIABLES=json.dumps([
+                {"name": "MACOS_RUNNER_TESTS", "value": "blacksmith-6vcpu-macos-15"},
+            ]),
+        )
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertNotIn("runner", self.dispatch())
+
+    def test_a_routed_commit_reuses_its_in_flight_run_on_the_large_sku(self):
+        result = self.launch(
+            "cmuxTests/ExampleTests", "--ref", "topic/fix",
+            LAUNCHER_PRIOR_RUNS=self._live(
+                commit=REMOTE_HEAD, runner="blacksmith-12vcpu-macos-26",
+            ),
+        )
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertIn("reusing that run", result.stdout)
+        self.assertFalse((self.root / "dispatch.json").exists())
+
     def test_invalid_runner_is_rejected_before_github_access(self):
         result = self.launch("cmuxTests/ExampleTests", "--runner", "macos-15")
         self.assertNotEqual(result.returncode, 0)