Repository navigation
ci(runner pools): one pool per Blacksmith Linux size, so a stalled size moves by declaration - #32393
Conversation
…cos_latest A job measured at one size reads that size's pool (linux_2, linux_8, linux_16, linux_32) instead of naming the size's label, so a size that stops starting jobs moves to its next member through runner_pools.py, with no YAML edit. The reconciler fills every declared pool on its next pass; feat-cmux-next's workflows start reading the new keys only after that. Leo-ask: ask-973a915bea Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-04cad32e0d917ebc8a49
|
All contributors have signed the CLA ✍️ ✅ |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-04cad32e0d917ebc8a49
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
 |
| Layer / File(s) | Summary |
|---|---|
Declare runner pools .github/runners.json |
Adds Linux pools for 2, 4, 8, 16, and 32 vCPU workloads with ordered fallback members. Adds a macOS-latest pool with macOS 26 as its fallback. |
Document and validate runner pools docs/ci-runners.md, tests/test_ci_runner_pools.py |
Documents size-based Linux pool selection and pool member ordering. Adds checks for Linux pool ordering and updates the held-state fixture. |
Estimated code review effort: 2 (Simple) | ~8 minutes
Change: Feature
Suggested reviewers: lawrencecchen
Merge Risk: 
49653
-
The new pools are declared, but workflows do not yet use the size-specific keys. Mark that migration as pending so operators do not expect size-pool failover to reroute existing jobs. This is a bounded documentation issue.
Pre-merge checks: 
-
Check name Status Explanation Resolution Docstring Coverage Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat… Write docstrings for the functions missing them to satisfy the coverage threshold. -
Check name Status Explanation Title check The title names the main change: one runner pool per Blacksmith Linux size, so a stalled size can fail over by declaration. The changeset adds size-specific pools (linux_2, linux_4, linux_8, linux_16,… Description check The description follows the template structure. It has a Summary that states the problem (literal single-size labels strand jobs when a size stalls) and the resulting behavior (pools with preferred an… Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Cmux Cloud Persistent Session And Early Input PASS: The pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff adds CI runner-pool declarations, documentation, and tests. It does not …Cmux Swift Actor Isolation The pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The authoritative diff contains no Swift files or production Swift changes, so the Swif…Cmux Swift Blocking Runtime The pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The authoritative diff contains no Swift files or Swift runtime synchronization changes…Cmux Browser Automation Off-Main PASS: The pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. It does not changeSources/TerminalController.swift, `ControlCommandExecutionPo…Cmux Expensive Synchronous Load The pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The reviewed diff contains no Swift files or production agent-history loading changes, …Cmux Cache Substitution Correctness PASS: The pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff contains no production Swift, TypeScript, or JavaScript changes, and no …Cmux No Hacky Sleeps PASS: The pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. It introduces no TypeScript, JavaScript, shell, or production runtime delay. The …Cmux Algorithmic Complexity PASS: The PR changes only runner configuration, documentation, and Python test code. It adds fixed-size pool member declarations and test assertions; it does not add or modify production Swift, TypeSc… Cmux Swift Concurrency The pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff contains no Swift code, so it does not introduce or expand any legacy Swift co…Cmux Swift @ConcurrentThe pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff contains no Swift files or Swift code, so the@concurrentcheck is not appli…Cmux Swift Package Boundaries PASS. The pull request changes only runner configuration, CI documentation, and Python tests. It introduces no Swift production changes, so the Swift package-boundary rule does not apply. Cmux Swiftpm Lockfiles PASS: The PR changes only runner configuration, CI documentation, and runner-pool tests. It changes no SwiftPM package, Xcode project, .gitignore, workflow, dependency declaration, or `Package.resol…Cmux Swift Logging The pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff contains no Swift files and adds or changes no logging statements. The Swift l…Cmux User-Facing Error Privacy The diff changes internal CI runner configuration, CI documentation, and tests only. It adds Blacksmith pool names and routing details, but the rule explicitly allows provider names in internal CI too… Cmux Full Internationalization The diff changes only .github/runners.json, CI operational documentation, and tests. The added pool names, runner labels, comments, andwhyfields are literal CI/configuration tokens, and `docs/ci…Cmux Swiftui State Layout PASS. The pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff contains no Swift or SwiftUI changes, so the SwiftUI state/layout failur…Cmux Architecture Rethink PASS: The pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff contains no Swift architecture changes, so the Swift architectural rethi…Cmux Swift Auxiliary Window Close Shortcuts PASS. The reviewed diff changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. It contains no Swift changes and no standalone cmux-owned window code, so the au…Cmux Source Artifacts PASS. The PR changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff adds runner configuration, durable documentation, and test coverage. It adds no lo…Cmux No Test Or Debug Seam In Production Source PASS: The pull request changes only .github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. It changes no Swift file under a productionSources/path, so it introduces no …Full details: Title check
-
Explanation
The title names the main change: one runner pool per Blacksmith Linux size, so a stalled size can fail over by declaration. The changeset adds size-specific pools (linux_2, linux_4, linux_8, linux_16, linux_32) and macos_latest in .github/runners.json, with matching docs and tests. The title does not mention macos_latest or the ephemeral-only linux_4 pool, but those are secondary to the main change.
Full details: Description check
-
Explanation
The description follows the template structure. It has a Summary that states the problem (literal single-size labels strand jobs when a size stalls) and the resulting behavior (pools with preferred and fail-over members, moved by declaration). It has a Testing section that names the commands run and the results. It has a Changelog-style entry only implicitly; the template asks for a Changelog section (none for internal-only CI changes) and a Proof section, and neither appears. The checklist is also omitted. The Summary and Testing content is substantive, so the description is mostly complete. The missing Changelog and Proof sections are non-critical for an internal CI change, but the author should add a Changelog line of 'none' and a Proof line explaining why no screenshots apply.
Resolution
Add a Changelog section with 'none' because this is an internal CI change. Add a Proof section with one line stating that no user-visible change exists (CI only). Delete checklist lines that do not apply. Consider adding a '### Observed / Not run' split under Testing, as the template asks for bug-fix PRs.
Full details: Cmux Cloud Persistent Session And Early Input
-
Explanation
PASS: The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff adds CI runner-pool declarations, documentation, and tests. It does not change Cloud terminal creation, cmux-tui transport, manual renderer admission, input routing, snapshots, authentication, revision fences, or leases.
Full details: Cmux Swift Actor Isolation
-
Explanation
The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The authoritative diff contains no Swift files or production Swift changes, so the Swift actor isolation check is not applicable.
Full details: Cmux Swift Blocking Runtime
-
Explanation
The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The authoritative diff contains no Swift files or Swift runtime synchronization changes, so this check is not applicable.
Full details: Cmux Browser Automation Off-Main
-
Explanation
PASS: The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. It does not changeSources/TerminalController.swift,ControlCommandExecutionPolicy.swift, browser socket commands, worker routing, or policy tests. Therefore it does not introduce or worsen the specified browser automation off-main violations.
Full details: Cmux Expensive Synchronous Load
-
Explanation
The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The reviewed diff contains no Swift files or production agent-history loading changes, so this check is not applicable.
Full details: Cmux Cache Substitution Correctness
-
Explanation
PASS: The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff contains no production Swift, TypeScript, or JavaScript changes, and no persistence, history, undo, or snapshot cache substitution.
Full details: Cmux No Hacky Sleeps
-
Explanation
PASS: The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. It introduces no TypeScript, JavaScript, shell, or production runtime delay. The changed test fixture contains no new sleep or timer.
Full details: Cmux Algorithmic Complexity
-
Explanation
PASS: The PR changes only runner configuration, documentation, and Python test code. It adds fixed-size pool member declarations and test assertions; it does not add or modify production Swift, TypeScript, JavaScript, shell, or runtime algorithms covered by the complexity rules. No prohibited collection scans, rescans, joins, or hot-path sorting/filtering are introduced.
Full details: Cmux Swift Concurrency
-
Explanation
The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff contains no Swift code, so it does not introduce or expand any legacy Swift concurrency pattern.
Full details: Cmux Swift `@Concurrent`
-
Explanation
The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff contains no Swift files or Swift code, so the@concurrentcheck is not applicable.
Full details: Cmux Swiftpm Lockfiles
-
Explanation
PASS: The PR changes only runner configuration, CI documentation, and runner-pool tests. It changes no SwiftPM package, Xcode project,
.gitignore, workflow, dependency declaration, orPackage.resolvedfile, so the SwiftPM lockfile conditions do not apply.
Full details: Cmux Swift Logging
-
Explanation
The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff contains no Swift files and adds or changes no logging statements. The Swift logging rule therefore has no applicable violation.
Full details: Cmux User-Facing Error Privacy
-
Explanation
The diff changes internal CI runner configuration, CI documentation, and tests only. It adds Blacksmith pool names and routing details, but the rule explicitly allows provider names in internal CI tools, operator diagnostics, docs, and tests. No changed user-facing error, alert, API body, command output, recovery copy, secret, credential, token, or payload dump has a concrete path to a cmux end user.
Full details: Cmux Full Internationalization
-
Explanation
The diff changes only
.github/runners.json, CI operational documentation, and tests. The added pool names, runner labels, comments, andwhyfields are literal CI/configuration tokens, anddocs/ci-runners.mdis operational documentation not shown to end users. No Swift UI text, app catalog, web UI text, locale registry, or message entries changed.
Full details: Cmux Swiftui State Layout
-
Explanation
PASS. The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff contains no Swift or SwiftUI changes, so the SwiftUI state/layout failure conditions do not apply.
Full details: Cmux Architecture Rethink
-
Explanation
PASS: The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff contains no Swift architecture changes, so the Swift architectural rethink criteria do not apply.
Full details: Cmux Swift Auxiliary Window Close Shortcuts
-
Explanation
PASS. The reviewed diff changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. It contains no Swift changes and no standalone cmux-owned window code, so the auxiliary-window close-shortcut rule does not apply.
Full details: Cmux Source Artifacts
-
Explanation
PASS. The PR changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. The diff adds runner configuration, durable documentation, and test coverage. It adds no logs, screenshots, recordings, temporary or cache directories, dependency checkouts, build output, or broad scratch directories.
Full details: Cmux No Test Or Debug Seam In Production Source
-
Explanation
PASS: The pull request changes only
.github/runners.json,docs/ci-runners.md, andtests/test_ci_runner_pools.py. It changes no Swift file under a productionSources/path, so it introduces no production test or debug seam.
-
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 2 functions across 1 files. (2 skipped: 2 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
Finishing Touches
-
Generate docstrings
-
- Commit to this branch
- Create a new PR
Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
…ember Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-04cad32e0d917ebc8a49
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-04cad32e0d917ebc8a49
|
Found 1 test failure on Blacksmith runners: Failure
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Probably yours: CI failed on
Matched log linesNot re-run automatically: Written by
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
|
Review (subagent claude-opus-5-5 medium): SHIP at 4965324 Checked: runner_pools.py spec/parse/reconcile with the six new pools, linux_4 vs feat's linux_ephemeral / LINUX_RUNNER mirror / spill / overflow switch, the dead-mark fixture change, macos_latest failover. Ran test_ci_runner_pools (42 OK), runner_label_policy, runner_set, capability_resolver, fork_runner_routing, simple_pool_picker, self_hosted_guard.sh. Follow-ups: probe/janitor evidence for the 2, 8 and 32 vCPU Linux labels (so those pools move on a stall); linux_4 in the queue-aware spill; fold linux_4 and feat's linux_ephemeral into one when that reaches main. |
…eading the pools Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-04cad32e0d917ebc8a49
|
Review (subagent claude-opus-5-5 medium): SHIP at c80b577 The only change since the reviewed head 4965324 reverts docs/ci-runners.md (the pool table moves to the feat-cmux-next PR), so this PR is CI data only: .github/runners.json and tests/test_ci_runner_pools.py. test_ci_runner_pools passes. Follow-ups: as in the receipt above. |
|
recheck |
github-actions
Bot
added
the
merged-unverified
Leo-ask: ask-973a915bea
Leo, 2026-10-28: "Where the f***ing are the generalized solution to all these problems that these incessant pins?"
475 runs-on lines on feat-cmux-next end in a literal single-size Blacksmith label (265 at 4 vCPU, 61 at macOS 26, 57 at 8 vCPU, 42 at 2 vCPU, 29 at 32 vCPU, 9 at 16 vCPU and others). A re-run cannot move a literal label, so a stalled size strands every job that names it. On 2026-10-28 that happened to blacksmith-4vcpu-ubuntu-2404 in nightly.yml.
This PR is step 1. It declares one pool per Blacksmith Linux size, plus
macos_latest, in.github/runners.jsonpools:linux_2linux_8linux_16linux_32macos_latestrunner_pools.pyis unchanged. On its next pass it writes each new key into RUNNER_POOLS, and it moves a key off a member the probe or the janitor marks dead. The follow-up PRs on feat-cmux-next replace each literal last resort withfromJSON(vars.RUNNER_POOLS).<size pool>. A fork pull request keeps its literal fork branch, because a fork's run gets no repository variables. Those PRs land only after this one has merged and the variable carries the keys.Changed:
.github/runners.jsonadds the five pools.docs/ci-runners.mdupdates the pool table and fixes thelinuxrow, which now lists 4 vCPU first.tests/test_ci_runner_pools.pychecks that each Linux size has its own pool with a member to fail over to.Tests:
tests/test_ci_runner_pools.py(42 OK), test_ci_runner_set, test_ci_queue_janitor, test_ci_simple_pool_picker and test_ci_starved_labels all pass.Performance
Agent session: agent session
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
Declares one pool per Blacksmith Linux size plus
macos_latestin.github/runners.json, so a stalled size moves its jobs by declaration instead of stranding them on a literal label.Hundreds of
runs-onlines end in a literal single-size Blacksmith label; a re-run cannot move a literal label, so when a size stops starting jobs every job that names it is stuck. This addslinux_2,linux_4,linux_8,linux_16,linux_32, andmacos_latest, each with a first (preferred) member and a fail-over member.macos_latestfails over to the macOS 26 image.linux_4is ephemeral-only: it serves secret-bearing and write-token jobs that must not ride the sharedlinuxpool.runner_pools.pyis unchanged — it writes the new keys intoRUNNER_POOLSon its next pass, and follow-up PRs replace literal labels withfromJSON(vars.RUNNER_POOLS)lookups.linux_4shares the 4 vCPU first slot.docs/ci-runners.mdpool table update moves to the same follow-up change that starts reading the pools.Written for commit c80b577. Summary will update on new commits.
Summary by CodeRabbit