Skip to content

ci: run the owned side lanes on side runners, not root runners - #14431

Merged
teamleaderleo merged 2 commits into
mainfrom
ci/side-runner-routing
Sep 25, 2026
Merged

teamleaderleo merged 2 commits into
mainfrom
ci/side-runner-routing

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

What

On an owned pick, the two picker-routed side lanes (Claude wrapper regressions in ci.yml and the macOS tests in remote-daemon.yml) now take the pool's side label, glaeda-side-<class>-xcode-<version>, instead of the pool label.

  • pr_runner_pool.py writes a new side_runner output. It is set only when the pick has a root count (root_runner is set) and CI_OWNED_POOL_SLOTS gives the pool more machines than root runners (std 36 vs 16 root, light 4 vs 2), so a side lane never waits on a label no runner carries.
  • ci.yml exposes it as macos_pr_side_runner and passes pr_side_runner to remote-daemon.yml. Both lanes take it before the pool label on attempt 1 and on the rescue's attempt 2, mirroring how the root jobs take pr_root_runner.
  • The self-hosted guard (tests/test_ci_self_hosted_guard.sh) allows the new output only through that one route.

Why

The pool label is carried by root and side runners alike, so GitHub handed these light jobs to a root runner about half the time. From 06:30 to 09:00Z on 2026-09-25, 11 of 21 pool-label side-lane jobs ran on a root runner, 3,300 s of root-runner time, while the root label was the saturated one (16/16 busy with jobs queued). On cmux7s and cmux9s, which have a single root, a wrapper job there blocked the mini's only compile.

What this does not change (measured, same window)

  • Compile admission, the app-host shards, tests-build-and-lag and cli-product-tests stay on the root label. The consumers are not warm-tree jobs, but glaeda gives them the producer's canonical root on whatever mini they land on (#filePath is baked into the product, restore takes it with glaeda-canonical-root take --wait), and the shards and lag also hold the mini's one gui token. The root label is what keeps root users per mini within its roots; on a side runner they would wait or be refused, not run sooner.
  • The side runners sit idle mostly because a mini running two compiles has used all 4 capacity units (compile costs 2). Moving jobs there cannot add capacity on those minis.
  • The picker's root reservation is no longer forcing first attempts to Blacksmith after ci: let a run queue a round behind busy pools instead of rolling over #14401: 50 of 50 first-attempt PR admissions and 31 of 31 first-attempt shards since 07:00Z ran on owned roots. The Blacksmith macOS shards (17) were attempt 2+ re-runs after hook refusals, mostly "the gui token is taken" (two root runners per mini, one gui token). That is a glaeda hook question, not routing, and is left for a follow-up.

Related: #14411 routes swift-package-tests as a third owned side lane through pr_runner; once both land it can take pr_side_runner the same way.

Repo variables

None changed.

Tests

  • python3 -m unittest tests.test_ci_pr_runner_pool (154 OK), plus test_ci_change_areas, test_ci_owned_pool_rescue, test_ci_fork_runner_routing, test_runner_label_policy, test_seed_derived_data, test_run_e2e, test_reuse_app_host_products: OK
  • bash tests/test_ci_self_hosted_guard.sh: 38 PASS
  • actionlint on ci.yml, ci-macos.yml, remote-daemon.yml: clean

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Routes the owned Claude wrapper and remote-daemon macOS test lanes to side runners instead of root runners, and fixes the queue janitor to count side-label jobs out of a run's root-runner share.

Previously these light lanes took the pool label, which root runners also carry, landing on a root runner about half the time (11 of 21 in a measured window) and blocking a compile or product consumer off that mini's root. The picker now emits side_runner (glaeda-side-<class>-xcode-<version>) when the pick has a root count and CI_OWNED_POOL_SLOTS leaves machines beyond the root runners; both lanes take it before the pool label on attempt 1 and on the rescue's attempt 2. The janitor's marker_peaks now subtracts side-label jobs too, so runs with side lanes no longer reserve up to 2 extra root runners.

  • Compile admission, app-host shards, tests-build-and-lag, and cli-product-tests stay on the root label because they need the canonical root and the mini's gui token.
  • The self-hosted guard permits the new output only through its changes output path.
  • Without a root count or spare machines, the pool label is kept and nothing changes.

Written for commit f64c0c8. Summary will update on new commits.

Review in cubic

The Claude wrapper and remote daemon macOS tests are light jobs with no
canonical root, but on an owned pick they took the pool label, which the
root runners carry too. From 06:30 to 09:00Z on 2026-09-25, 11 of 21 such
jobs landed on a root runner (3,300 s of root-runner time), keeping a
compile or product consumer off that mini's root; on cmux7s and cmux9s,
which have one root, they blocked the mini's only compile.

The picker now writes side_runner, the pool's glaeda-side-* label, when the
pool has a root count and CI_OWNED_POOL_SLOTS leaves it machines beyond its
root runners. Both side lanes take it before the pool label on attempt 1
and on the rescue's attempt 2. Without a root count nothing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 5 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 3450b013-12b3-4596-977e-0c69f1d2c075

📥 Commits

Reviewing files that changed from the base of the PR and between 2fd9ac4 and f64c0c8.

📒 Files selected for processing (6)
  • .github/workflows/ci.yml
  • .github/workflows/remote-daemon.yml
  • scripts/ci/pr_runner_pool.py
  • scripts/ci/queue_janitor.py
  • tests/test_ci_pr_runner_pool.py
  • tests/test_ci_self_hosted_guard.sh

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

marker_peaks took a run's root-runner share as its marker peak less its
jobs on the pool label. Side lanes on the side label were not subtracted,
so each run with side lanes reserved up to 2 more root runners than it
uses. Count jobs on the pool's side label too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo
teamleaderleo merged commit aaefd83 into main Sep 25, 2026
63 checks passed
@teamleaderleo
teamleaderleo deleted the ci/side-runner-routing branch September 25, 2026 09:31
teamleaderleo added a commit that referenced this pull request Sep 25, 2026
… bounds

- ci.yml passes macos_pr_side_runner to ci-macos.yml, and swift-package-tests
  takes pr_side_runner before the pool label on attempt 1 and the rescue's
  attempt 2, like the Claude wrapper and remote daemon lanes (#14431), so it
  never holds a mini's root runner.
- pr_runner_pool: a full suite with release_build false places the package
  lane as a third side lane, a peak of 12. SIDE_LANES is 3, MAX_RUN_JOBS 12,
  and FULL_RUN (unknown routing) carries the package lane. The replay charge
  stays 3 (admission plus the two usual side lanes).
- Tests: the release SDK lane, self-hosted guard and change-area checks read
  the new runs-on and Xcode conditions; picker tests cover the 12 peak and the
  side-runner wiring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo added a commit that referenced this pull request Sep 25, 2026
… bounds

- ci.yml passes macos_pr_side_runner to ci-macos.yml, and swift-package-tests
  takes pr_side_runner before the pool label on attempt 1 and the rescue's
  attempt 2, like the Claude wrapper and remote daemon lanes (#14431), so it
  never holds a mini's root runner.
- pr_runner_pool: a full suite with release_build false places the package
  lane as a third side lane, a peak of 12. SIDE_LANES is 3, MAX_RUN_JOBS 12,
  and FULL_RUN (unknown routing) carries the package lane. The replay charge
  stays 3 (admission plus the two usual side lanes).
- Tests: the release SDK lane, self-hosted guard and change-area checks read
  the new runs-on and Xcode conditions; picker tests cover the 12 peak and the
  side-runner wiring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo added a commit that referenced this pull request Sep 25, 2026
…ease helper (#14411)

* ci: run swift-package-tests on owned minis when the run builds no Release helper

swift-package-tests never reached the owned Macs: all 4 runs in the last two
hours were on blacksmith-6vcpu-macos-15. Its only hard need for macOS 15 is
the SDK 15 Release Ghostty helper, which runs only on a full suite with
release_build. Every other run (a package change under the compile-only
policy) is plain `swift test`, which glaeda already classes as light.

- pr_runner_pool: a `swift-package` side lane in run_plan when the lane runs
  and builds no helper (package_lane_owned(); an unknown release_build counts
  as a helper build). ci.yml passes RUN_SWIFT_PACKAGES and RUN_RELEASE_BUILD.
- ci-macos.yml: swift-package-tests takes the picked owned label on attempt 1
  (and the rescue's attempt 2 after a refusal) of a same-repository pull
  request whose owned_jobs names it, with the lane's Xcode; everything else
  keeps the macOS 15 pool and pin. The rescue already covers it (CI marker).
- docs/ci-runners.md: the lane, plus a table of every macOS job and whether
  it may take an owned Mac.
- Tests: picker placement and wiring, dual-Xcode guard, Xcode pin tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: put swift-package-tests on the side label and count it in the run bounds

- ci.yml passes macos_pr_side_runner to ci-macos.yml, and swift-package-tests
  takes pr_side_runner before the pool label on attempt 1 and the rescue's
  attempt 2, like the Claude wrapper and remote daemon lanes (#14431), so it
  never holds a mini's root runner.
- pr_runner_pool: a full suite with release_build false places the package
  lane as a third side lane, a peak of 12. SIDE_LANES is 3, MAX_RUN_JOBS 12,
  and FULL_RUN (unknown routing) carries the package lane. The replay charge
  stays 3 (admission plus the two usual side lanes).
- Tests: the release SDK lane, self-hosted guard and change-area checks read
  the new runs-on and Xcode conditions; picker tests cover the 12 peak and the
  side-runner wiring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant