ci: route focused tests to one front door, and half of them to 12 vCPU

[teamleaderleo]

Summary

One front door for focused tests. test-depot.yml was named for a retired provider, and agents used it to run one test at a time. It had 32 dispatches on 2026-09-23. Each one was a cold build of about 20 minutes, with none of the reuse, refusal or batching that run-e2e.sh does. This PR:

• renames it to test-macos-suite.yml ("Run macOS test suite")
• says at the top of the file and in its selector input that a focused test goes through ./scripts/run-e2e.sh
• says the same thing in the cmux-testing skill, which is where agents read it

Half of focused runs on 12 vCPU. run-e2e.sh now sends every commit whose SHA ends in an odd hex digit to blacksmith-12vcpu-macos-26. That runs both instance sizes on real traffic instead of a one-off benchmark. The split is keyed on the commit, not random, so every dispatch at one commit lands on the same pool. That matters because in-flight reuse, the failed-selector refusal and the app-host product contract all match on the runner label. Only the free 6 vCPU default is split. An explicit --runner, or a MACOS_RUNNER_TESTS value that names another pool, is used unchanged.

Why

At 23:37 UTC the macOS 26 pool had 10 jobs running and 48 queued, with the oldest waiting 20 min. Most of the load was single-test dispatches paying for a full compile.

Trade-off

The runner label is part of the app-host product contract. So a product built on one size can't be reused by a run on the other. It still works within a commit, because the split is per commit. Across commits it's a real cost, but a small one: only 1 of 23 recent e2e builds I checked actually took the short path.

Open question this measures

It's not yet clear whether Blacksmith caps macOS capacity in jobs or in vCPUs. If the cap is in jobs, 12 vCPU is a free speedup. If it's in vCPUs, each large run takes two slots. Build-step durations split by pool, and queue depth, will tell us which.

Testing

• tests/test_run_e2e.py: 51 pass, including 4 new tests: odd commit is routed, explicit runner is never rerouted, an admin variable is never split, a routed commit reuses its in-flight run on the large SKU. Negative control: with the routing disabled, 2 of the new tests fail.
• linux-guard lane: exit 0, 434 passing
• actionlint is clean on both workflows

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Renames test-depot.yml, named for a retired CI provider, to test-macos-suite.yml and steers one-off test dispatches to run-e2e.sh. The wrapper reuses an in-flight compile at the same commit and refuses a selector that already failed there, where the suite workflow pays a cold full-tree build every time.

• Focused runs whose commit SHA ends in an odd hex digit now compile on blacksmith-12vcpu-macos-26, so the two instance sizes are compared on real traffic.
• The split keys on the commit, keeping every dispatch at one commit on one pool so the reuse and refusal guards still match.
• Only the free 6 vCPU default is split; an explicit --runner or a MACOS_RUNNER_TESTS value naming another pool is used unchanged.
• Docs and the cmux-testing skill now point focused runs at the wrapper, and test-e2e.yml lists the 12 vCPU runner as an option.

^{Written for commit 71f645c. Summary will update on new commits.}

[coderabbitai]

Warning

Review limit reached

Next included review available in 20 seconds.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: ddaa47eb-f987-4d9a-8eec-a5efc6bd4cb8

📥 Commits

Reviewing files that changed from the base of the PR and between 1ba90a1 and 71f645c.

📒 Files selected for processing (11)

• .github/workflows/perf-activation.yml
• .github/workflows/test-e2e.yml
• .github/workflows/test-macos-suite.yml
• docs/ci-runners.md
• docs/ci/workflow-inventory.md
• scripts/ci/dispatch-focused-test.py
• skills/cmux-testing/references/local-vs-ci-validation.md
• tests/test_ci_app_host_test_output.py
• tests/test_ci_manual_macos_package_cache.py
• tests/test_ci_reusable_workflow_permissions.py
• tests/test_run_e2e.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[teamleaderleo]

Independent subagent review at e5a034c: no blocking defects.

• Rename: no remaining references to test-depot.
• Guards: the in-flight reuse and failed-selector refusal still match. The refusal stays unnarrowed across all pools for unpinned dispatches.
• 12 vCPU label: accepted by every runner guard.
• Xcode and cache keys: neither depends on vCPU count, so both sizes share one cache.

Addressed: the stale Depot note in the workflow-inventory.md row.

Accepted as-is:

• Routing needs repo variable access: routing happens only when gh variable list succeeds. Otherwise every dispatch goes to 6 vCPU. That's the safe direction, because a run should never override an admin value it can't read.
• Merge order: main's test-e2e.yml doesn't list the 12 vCPU choice yet, so a routed dispatch made before merge is rejected by the dispatch API. Merging this PR resolves it.

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.