Skip to content

docs(testing): describe how run-e2e.sh actually picks the runner #14620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
teamleaderleo merged 2 commits into from
Sep 25, 2026
Merged

docs(testing): describe how run-e2e.sh actually picks the runner #14620

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1ace245
docs(testing): describe how run-e2e.sh actually picks the runner
teamleaderleo Sep 25, 2026
c901c24
docs(testing): note in-flight reuse across pools
teamleaderleo Sep 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion skills/cmux-testing/references/local-vs-ci-validation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Dispatch through the wrapper. It pins the exact pushed commit, carries a `dispat

**Compile the test target locally before dispatching.** One focused run costs 10-20 macOS runner-minutes, and it compiles the whole tree before it runs anything, so the most common red result on a feature branch is a Swift compile error rather than a test failure. The `cmux-unit` command above catches those in a fraction of the time and without a runner.

**The wrapper picks the runner; leave `--runner` off.** Commits whose SHA ends in an odd hex digit compile on `blacksmith-12vcpu-macos-26`, the rest on `blacksmith-6vcpu-macos-26`, so the two sizes are compared on real traffic. Every dispatch at one commit lands on the same pool, which is what lets the wrapper reuse a run already in flight there. Over 60 consecutive dispatches (2026-09-22/23) the macOS 15 pool queued for a median 2.4 min but a p90 of 83 min and a worst case of 178 min, while macOS 26 queued 0.3 min median / 1.0 min p90. Pass `--runner blacksmith-6vcpu-macos-15` only when the question is specifically about macOS 15 behavior, and expect to wait for it.
**The wrapper picks the runner; leave `--runner` off.** `auto` goes through `scripts/ci/e2e_runner_pool.py`, the rule pull request CI uses: an owned Mac with a free slot first, then `blacksmith-12vcpu-macos-26` while it has headroom, then `blacksmith-6vcpu-macos-26`, then the shorter queue. A dispatch that names a Blacksmith pool skips idle owned Macs and waits in that pool's queue, which can run to an hour or more when many branches are testing at once (76 minutes on 6vcpu macOS 26 on 2026-09-25). Leaving it off also lets the wrapper attach to an identical run already in flight at that commit on any of those pools; a named `--runner` only reuses a run on that pool. Pass `--runner blacksmith-6vcpu-macos-15` only when the question is specifically about macOS 15 behavior, and expect to wait for it.

**Do not dispatch `test-macos-suite.yml` for one test.** It runs a whole test target, compiles cold every time, and has none of the wrapper's reuse or refusal. A single-test dispatch there costs about 20 macOS runner-minutes for an answer the wrapper would share.

Expand Down
Loading