Skip to content

ci(e2e): queue E2E runs for the owned pool within CI_PR_POOL_QUEUE_ROUNDS - #14640

Merged
teamleaderleo merged 2 commits into
mainfrom
ci-e2e-picker-rounds
Sep 25, 2026
Merged

teamleaderleo merged 2 commits into
mainfrom
ci-e2e-picker-rounds

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Follow-up to #14630 (same cause, E2E side). e2e_runner_pool.py read no CI_PR_POOL_QUEUE_ROUNDS, so an E2E run took an owned pool only when its peak was free, with every in-flight run's whole future peak (committed) counted as taken. Pull request CI dropped that rule in #14401. On the 2026-09-25 12:34:57 snapshot it read 43 of 32 std machines taken while 8 ran, so every owned pool looked full.

What

  • test-e2e.yml passes --queue-rounds "$POOL_QUEUE_ROUNDS" (vars.CI_PR_POOL_QUEUE_ROUNDS; "" is the default 1 round, 0 the kill switch, omitted is 0). dispatch-focused-test.py reads the same variable (unset is the default), and test-macos-suite.yml and main-regression-bisect.yml pass it through as CMUX_CI_PR_POOL_QUEUE_ROUNDS.
  • The rounds decide only whether an owned pool takes the run. When none does, e2e_runner_pool.decide() picks Blacksmith by E2E's unchanged headroom rule (first free machine, then the shorter queue), so the Blacksmith behavior and its tests stay the same.
  • owned_pool_rescue.py gives test-e2e.yml runs the same queue_seconds() allowance as ci.yml and test-ios.yml runs.

Tests

  • Replaying the real 12:34:57 snapshot: rounds 0 or omitted stays on Blacksmith ("every pool is full"). With 2 rounds and 13 replayed PR runs, the run joins an owned root queue within 20 min. With 22 or 40 replayed runs the Blacksmith pick is the same as rounds 0.
  • python3 tests/test_ci_pr_runner_pool.py 161 OK, tests/test_ci_owned_pool_rescue.py 92 OK, tests/test_run_e2e.py 101 OK, plus workflow_run_sources, fork routing and the self-hosted guard.

Design note from review: an E2E build or test job holds a root runner up to about 45 min, longer than the compile-length round the queue estimate assumes. So on the 2-runner glaeda-root-light pool, a queued E2E run can delay PR admission for longer than the rounds suggest. Setting CI_PR_POOL_QUEUE_ROUNDS to 0 turns this off.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Makes E2E runs queue for owned macOS pools within CI_PR_POOL_QUEUE_ROUNDS, matching pull request CI. Previously E2E used the kill-switch rule, which counted every in-flight run's whole future peak as taken now, so owned pools frequently looked full and runs fell back to Blacksmith.

  • The rounds decide only whether an owned pool takes the run; when none does, the Blacksmith pick stays on the unchanged headroom rule.
  • The workflows and dispatch launcher now pass CI_PR_POOL_QUEUE_ROUNDS through: "" is the default of 1 round, 0 turns the feature off, and an omitted flag keeps the old behavior.
  • owned_pool_rescue.py gives E2E runs the same queue allowance as ci.yml and test-ios.yml runs.
  • An E2E job holds a root runner up to about 45 minutes, longer than the round the queue estimate assumes, so set the variable to 0 to turn this off.

Written for commit 2670287. Summary will update on new commits.

Review in cubic

teamleaderleo and others added 2 commits September 25, 2026 09:18
…UNDS

e2e_runner_pool.py read no queue rounds, so an E2E run took an owned pool
only when its peak was free counting every in-flight run's whole future peak
(the janitor's `committed`), the rule pull request CI dropped in #14401. The
2026-09-25 12:34:57 snapshot read 43 of 32 std machines taken while 8 ran.

test-e2e.yml, and the dispatch launcher (dispatch-focused-test.py, with
test-macos-suite.yml and main-regression-bisect.yml passing the variable),
now pass CI_PR_POOL_QUEUE_ROUNDS. The rounds decide only whether an owned
pool takes the run; with no owned room the Blacksmith pick is the headroom
rule's, unchanged. The rescue gives test-e2e.yml runs the same queue
allowance as CI and test-ios.yml runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo
teamleaderleo merged commit 611eeac into main Sep 25, 2026
@teamleaderleo
teamleaderleo deleted the ci-e2e-picker-rounds branch September 25, 2026 13:22
@github-actions

Copy link
Copy Markdown
Contributor

Merge receipt for 2670287f91, merged 2026-09-25 13:22:08 UTC

  • Not verified at merge: ci-status (not reported), Web complexity (not reported), web-validation (not reported)
  • Full suite: runs on main after merge.

Labeled merged-unverified: if main breaks near this merge, look here first.

@github-actions github-actions Bot added the merged-unverified A judging check was not green at merge; see the merge receipt comment label Sep 25, 2026
@coderabbitai

coderabbitai Bot commented Sep 25, 2026

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: a8031be8-b32a-49a4-b126-93c6590fc899

📥 Commits

Reviewing files that changed from the base of the PR and between 27b8cbc and 2670287.

📒 Files selected for processing (9)
  • .github/workflows/main-regression-bisect.yml
  • .github/workflows/test-e2e.yml
  • .github/workflows/test-macos-suite.yml
  • scripts/ci/dispatch-focused-test.py
  • scripts/ci/e2e_runner_pool.py
  • scripts/ci/owned_pool_rescue.py
  • tests/test_ci_owned_pool_rescue.py
  • tests/test_ci_pr_runner_pool.py
  • tests/test_run_e2e.py
 ______________
< Bugs begone! >
 --------------
  \
   \   \
        \ /\
        ( )
      .( o ).
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

rustybret pushed a commit to rustybret/bmux that referenced this pull request Sep 25, 2026
60d8ac1 Land ejc3's isolated defaults test from manaflow-ai#12598 (manaflow-ai#14504)
e926f24 fix: resolve the terminal Copy guard through the Command-aware keyboard layout (manaflow-ai#10872) (manaflow-ai#13015)
0df2946 Fix startup-race crash in v2RefreshKnownRefs against a half-restored session (manaflow-ai#2751) (manaflow-ai#9627)
611eeac ci(e2e): queue E2E runs for the owned pool within CI_PR_POOL_QUEUE_ROUNDS (manaflow-ai#14640)
27b8cbc ci: seed the Swift package cache from main pushes (manaflow-ai#14638)
be46dba ci: stop E2E from saving an unresolved Swift package cache (manaflow-ai#14632)
2486e99 ci: run-e2e.sh --wait asks glaeda-gh instead of polling GitHub (manaflow-ai#14622)
088034b ci(ios): queue test-ios runs for the owned pool within CI_PR_POOL_QUEUE_ROUNDS (manaflow-ai#14630)

# Conflicts:
#	.github/workflows/main-regression-bisect.yml
#	.github/workflows/perf-activation.yml
#	.github/workflows/seed-derived-data.yml
#	.github/workflows/test-e2e.yml
#	.github/workflows/test-ios.yml
#	.github/workflows/test-macos-suite.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merged-unverified A judging check was not green at merge; see the merge receipt comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant