ci: overflow E2E to the 12vcpu macOS 26 pool only when the 6vcpu pool is backed up by teamleaderleo · Pull Request #14132 · manaflow-ai/cmux

teamleaderleo · 2026-09-24T04:08:39Z

The 12vcpu macOS 26 pool (blacksmith-12vcpu-macos-26) is reserved first for release and nightly builds (see nightly.yml's build job). This PR's first version sent every odd commit's E2E run there however busy it was, so E2E could keep the pool backed up ahead of a release.

runner: auto in test-e2e.yml now stays on blacksmith-6vcpu-macos-26. It overflows to the 12vcpu pool only when the 6vcpu pool is backed up and the 12vcpu pool has room:

at least CI_E2E_OVERFLOW_MIN_QUEUED (default 4) other E2E runs in flight on 6vcpu;
no release or nightly run in flight, and nothing queued on 12vcpu;
fewer than CI_E2E_OVERFLOW_MAX_LARGE_RUNNING (default 2) E2E runs on 12vcpu.

Anything uncertain stays on 6vcpu: an API error, a possibly truncated listing, or an invalid threshold. CI_E2E_LARGE_POOL_OVERFLOW=0 turns overflow off. An explicit runner input, or a MACOS_RUNNER_TESTS naming another pool, still wins.

The decision costs at most two GET requests: one page each of in-progress and queued runs, through the queue janitor's client. The GITHUB_TOKEN budget (about 1000 requests an hour) is shared by every workflow, so the decision estimates demand from run titles and workflow names instead of listing each run's jobs. The 6vcpu figure is therefore E2E's own demand on the pool, not the pool's whole job queue. A UI-started auto run that overflowed is still titled 6vcpu, because run-name cannot read job outputs. The new runner job holds actions: read; everything else stays read-only.

scripts/run-e2e.sh applies the same rule from the same script, names the pool it chose, and reads the queue only when it is about to dispatch. This also replaces the commit-parity 50/50 split #14067 added to the dispatcher. Because the pool is no longer a function of the commit, its in-flight reuse and overlap refusal match a run on either macOS 26 pool for an unpinned dispatch, so a commit that overflowed is not dispatched twice.

Validation

tests/test_run_e2e.py (80 passed) covers the rule and its thresholds, the kill switch, fail-safe on errors, a call-counting fake client (≤2 requests, 1 when 12vcpu is busy), the launcher end to end against a fake gh (overflow, cross-pool reuse and refusal), the workflow step script, and the job permissions. The new tests fail against the previous head.
Also passed: test_ci_queue_janitor.py, test_ci_e2e_compilation_cache.py, test_ci_repo_variable_defaults.py, the other suites that parse test-e2e.yml, and test_ci_self_hosted_guard.sh.
Not verified: a live Actions run of the new runner job, how well the default threshold of 4 tracks real 6vcpu backlog, and actionlint (not available where this was prepared).

🤖 Generated with Claude Code

https://claude.ai/code/session_012pAcDGiHaibDaMU4CPvXAP

github-actions · 2026-09-24T04:08:52Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

A direct test-e2e.yml dispatch with runner=auto lands every commit on blacksmith-6vcpu-macos-26; only run-e2e.sh applies the commit-keyed split to blacksmith-12vcpu-macos-26. These tests expect the workflow to resolve the pool itself with the dispatcher's rule, run build/test and the Tart checks on that label, and honour a CI_E2E_LARGE_POOL_SPLIT=0 kill switch in both places. They fail until the workflow does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012pAcDGiHaibDaMU4CPvXAP

A dispatch with runner=auto resolved to blacksmith-6vcpu-macos-26 for every commit unless it came through run-e2e.sh, which alone routed odd commits to blacksmith-12vcpu-macos-26. E2E builds queued for hours on the 6vcpu pool while the 12vcpu pool sat idle. A new Linux `runner` job resolves the pool after resolve-ref, with the rule now in scripts/ci/e2e_runner_pool.py: an explicit runner wins; else MACOS_RUNNER_TESTS, else the 6vcpu pool; and when that is the 6vcpu pool, a resolved commit whose last hex digit is odd goes to the 12vcpu pool. CI_E2E_LARGE_POOL_SPLIT=0 turns the split off. build, test, their Tart identity checks and CMUX_PRODUCT_RUNNER read the job's output. dispatch-focused-test.py imports the same function and reads the same switch, and test-macos-suite.yml passes the switch to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012pAcDGiHaibDaMU4CPvXAP

coderabbitai · 2026-09-24T04:09:50Z

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Warning

Review limit reached

Next included review available in 11 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 155a6815-e08e-42d4-988c-3620c0384c6d

📥 Commits

Reviewing files that changed from the base of the PR and between 5b646b7 and 31dd5a6.

📒 Files selected for processing (7)

.github/workflows/test-e2e.yml
.github/workflows/test-macos-suite.yml
scripts/ci/dispatch-focused-test.py
scripts/ci/e2e_runner_pool.py
tests/test_ci_e2e_compilation_cache.py
tests/test_ci_self_hosted_guard.sh
tests/test_run_e2e.py

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

Commit to this branch
Create a new PR

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…d up The 12vcpu macOS 26 pool is reserved first for release and nightly builds. The parity split sent every odd commit's E2E run there however busy it was. `auto` now stays on the 6vcpu pool and overflows only when at least CI_E2E_OVERFLOW_MIN_QUEUED (default 4) other E2E runs are in flight on 6vcpu, no release or nightly run is in flight, nothing is queued on 12vcpu, and fewer than CI_E2E_OVERFLOW_MAX_LARGE_RUNNING (default 2) E2E runs are on it. Any API error, a possibly truncated listing, or an invalid threshold stays on 6vcpu, and CI_E2E_LARGE_POOL_OVERFLOW=0 (renamed from CI_E2E_LARGE_POOL_SPLIT) turns overflow off. The decision costs at most two GET requests (one page each of in-progress and queued runs), through the queue janitor's client, and attributes demand from run titles and workflow names rather than per-run job listings, so dozens of dispatches an hour stay well inside the shared GITHUB_TOKEN budget. The runner job gains actions: read. run-e2e.sh applies the same rule, names the pool it chose, and reads the queue only when it is about to dispatch. Because the choice is no longer a function of the commit, its in-flight reuse and overlap refusal now match a run on either macOS 26 pool for an unpinned dispatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012pAcDGiHaibDaMU4CPvXAP

auth-refresh-tests.yml now runs on the dual-Xcode runner on main, so it keeps main's routing and leaves MACOS_RUNNER_TESTS. test-e2e.yml picks its runner in a separate job since #14132, so its hosted-route checks read that job's output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EduXdN9PKnGsMQztJK7WeE

Keeps both sides of dispatch-focused-test.py: main's runner-pool overflow and 400-character concurrency-group check (#14067, #14104, #14132), and this branch's per-definition dispatch history and selector normalization. test-depot.yml is now test-macos-suite.yml (#14075); the UI filter guard followed the rename and its test reads the new file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…st (#14225) * ci: route E2E by the pull request pool rule, 12vcpu first #14132 kept an unpinned E2E run on 6vcpu macOS 26 unless four other E2E runs waited there and 12vcpu was idle, holding 12vcpu back for release and nightly builds. Every Blacksmith pool is sponsored, so that reserve only cost E2E time. e2e_runner_pool.py now reads the queue janitor's macos-pool-load snapshot and calls pr_runner_pool.decide(): the first pool in CI_PR_POOL_ORDER with fewer than CI_PR_POOL_MAX_QUEUED jobs queued and no queued release or nightly job, limited to the macOS 26 pools (12vcpu, then 6vcpu). E2E runs created since the snapshot count on the pool their title names; pull request runs replay through their own rule over the whole order. Errors, a stale snapshot, or CI_E2E_LARGE_POOL_OVERFLOW=0 keep the 6vcpu default. The dispatcher reads the same snapshot through `gh api`. decide() gains `placed` (runs whose pool is known) and `choose_from` (limit the final pick); pull request behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: keep E2E pool choice fail-safe and honor pull request routing off in the replay Review follow-ups: a malformed snapshot (a timestamp with no zone) raised outside the fail-safe and would fail the runner job, so decide() now runs inside it. When the snapshot's copied settings show pull request routing off (kill switch or another lane), replayed pull request runs count on their lane instead of spreading from 12vcpu. The macOS 15 spill test now uses a queue where replaying over macOS 26 alone gives a different answer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

teamleaderleo requested a review from lawrencecchen as a code owner September 24, 2026 04:08

teamleaderleo enabled auto-merge (squash) September 24, 2026 04:08

teamleaderleo and others added 2 commits September 24, 2026 04:09

teamleaderleo force-pushed the claude/confident-sagan-p9isc9 branch from dce1fcc to 1565251 Compare September 24, 2026 04:09

teamleaderleo disabled auto-merge September 24, 2026 04:10

teamleaderleo changed the title ~~ci: split test-e2e.yml's auto runner across both macOS 26 pools~~ ci: overflow E2E to the 12vcpu macOS 26 pool only when the 6vcpu pool is backed up Sep 24, 2026

teamleaderleo merged commit 05a30ca into main Sep 24, 2026
49 of 50 checks passed

This was referenced Sep 24, 2026

ci: never let a focused test run pass after executing zero tests #14053

Merged

ci: route unpinned E2E runs by the pull request pool rule, 12vcpu first #14225

Merged

Navigation Menu

Repository navigation

ci: overflow E2E to the 12vcpu macOS 26 pool only when the 6vcpu pool is backed up - #14132

ci: overflow E2E to the 12vcpu macOS 26 pool only when the 6vcpu pool is backed up#14132
teamleaderleo merged 3 commits into
mainfrom
claude/confident-sagan-p9isc9

teamleaderleo commented Sep 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 24, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading

Review limit reached

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

teamleaderleo commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

teamleaderleo commented Sep 24, 2026 •

edited

Loading

github-actions Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading