[teamleaderleo]

Leo, 2026-10-31: "i assume blacksmith prob cut us off cuz we were using like $10k a day so", "y'know, we'll have to be more self-reliant and size the stuff better", and "i mean the linux things cant our minis just run them" (rulebook: leo-workspace#1867).

What changes

The jobs below need neither Linux nor Blacksmith. Their runner is now picked in this order:

1. fleet Linux: vars.LINUX_RUNNER, when it names one of our own Linux runners (not a Blacksmith label or a GitHub-hosted image);
2. a free owned mini: the side label (vars.CI_SIDE_LANE_RUNNER), or the same label on a wider class. The mini must be idle at placement time, after subtracting the jobs the janitor snapshot already shows queued on it. A mini takes nothing while any nightly or release job is queued on its pool family (reserved_queued), so nightly jobs still come first;
3. their existing Blacksmith label, with its CI_CLOUD_OVERFLOW_SAVED failover. It stays the last || term.

Job	Workflow	Order
cmux-tui build aarch64-apple-darwin, build x86_64-apple-darwin	cmux-tui-artifacts.yml → cmux-tui-build-package.yml	free mini first, built natively on the Mac path (the path the Linux build's macOS fallback already uses), then the Linux cross build on Blacksmith 32 vCPU. Both targets or neither.
cmux-next checks	cmux-next.yml	fleet Linux → free mini → Blacksmith 16 vCPU
cmux-next CI guard group	cmux-next-guards.yml	fleet Linux → free mini → Blacksmith 32 vCPU
nightly-guards / workflow-guard-tests / release-notary	cmux-next-guards.yml → ci-guards.yml (new optional input guard_tests_runner)	fleet Linux → free mini → vars.LINUX_RUNNER / Blacksmith 4 vCPU
tooling-tests	cmux-next-guards.yml	fleet Linux → free mini → Blacksmith 4 vCPU (16 vCPU under the overflow switch)

• scripts/ci/portable_job_placement.py: the decision. It reuses side_lane_placement.side_route(), pr_runner_pool.live_owned_free() and the janitor snapshot. Anything uncertain places nothing on a mini: runners it can't read, no snapshot or a stale one, a person's re-run, attempt 3 or later, no side label, or the AWS pool. Each job then keeps its Blacksmith route.
• A portable-placement job in cmux-next.yml and in cmux-next-guards.yml runs the base branch's copy of the script, the way macos-placement does, using the routing App's runner read with cmux-bot as the fallback. Every step fails open. The placed jobs read its outputs only in the attempt that placed them (placed_attempt), so a re-run of failed jobs never reuses a stale mini.
• In cmux-tui-artifacts, tree-preflight makes the same call, all or none, and passes darwin_mac_runner to the build. darwin_builder and the tree key stay linux, so the 32 vCPU cross-compile shadow does not start. A re-run takes macos_retry_runner, as the Mac path always has.
• On a mini, a job runs scripts/ci/reset-stale-submodules.sh right after checkout, because a reused workspace keeps the previous job's submodule checkouts. The guard group's --jobs "$(nproc)" becomes getconf _NPROCESSORS_ONLN, since macOS has no nproc.
• Routing-table guard (scripts/ci/runner_routing_table.py, .github/runners.json): new class portable-checks ("needs": "any"). A job in that class fails the check unless its runs-on takes portable placement's fleet Linux output, then its mini output, and ends on a Blacksmith Linux label. mini_runner counts as a macOS route, so these jobs are listed.
• tests/test_ci_portable_job_placement.py (registered in tests/test-execution.toml, run in the ci guard group) covers the order, the nightly hold, queued jobs taking idle runners first, the std spill, all-or-none, every uncertain case, the output format and the wiring in each workflow. The existing runs-on tests are updated: test_feat_ci_guard_group.py, test_cmux_next_checks_workflow.py (checks and the placement job now count as able to run on a mini and get the submodule reset), and test_ci_linux_single_size.py (counts lowered: ci-guards 6→5, cmux-next-guards 10→9).

Performance

These are Blacksmith vCPU-minutes per day that move off Blacksmith whenever a mini is free (or all of them, once vars.LINUX_RUNNER names fleet Linux). The coordinator measured the "before" column on 2026-10-31. The "after" column is the most that can move. How much actually moves depends on how often minis are idle, and each placement job's summary records it.

metric	before (Blacksmith vCPU-min/day)	after (Blacksmith vCPU-min/day, minis free)	delta	harness or run link
cmux-tui build aarch64-apple-darwin	349k	0 (mini)	−349k	coordinator's 2026-10-31 Blacksmith usage read
cmux-tui build x86_64-apple-darwin	339k	0 (mini)	−339k	same
cmux-next checks	264k	0 (fleet Linux / mini)	−264k	same
cmux-next CI guard group	221k	0 (fleet Linux / mini)	−221k	same
release-notary guards, tooling-tests	not measured (4 vCPU, under 1 min each)	0	small	coordinator's 2026-10-31 Blacksmith usage read
total	≥1,173k	0 at full mini availability	up to −1,173k/day	coordinator's 2026-10-31 Blacksmith usage read

Placement adds one 2 vCPU job of about 15 s per run, on fleet Linux when LINUX_RUNNER names it.

Risk

risk:normal, CI routing only. Two risks:

• A check that behaves differently on macOS. The guard group runs head and base on the same machine, so a red that comes from the machine shows up as a base red. checks, tooling-tests and release-notary have no base comparison: a macOS-only red there would turn a PR red only when it lands on a mini. The probe run (below) runs them on a mini before this lands.
• Two placements racing for the same idle runner. The loser queues on the side label behind one job. The idle count already subtracts the jobs the snapshot shows queued there.

No timer, observer or loop added: placement is one bounded read (60 s deadline) per run.

Not touched: nightly workflows (none of these jobs is nightly; test_ci_nightly_off_blacksmith.py passes), signing, and cmux#29033 (the react-apps shards).

Observed / Not run

• Passed locally: tests/test_ci_portable_job_placement.py (18), test_feat_ci_guard_group.py, test_cmux_next_checks_workflow.py (165), test_ci_linux_single_size.py, test_ci_fork_runner_routing.py, test_supplychain_security.py, test_ci_side_lane_placement.py, test_ci_nightly_off_blacksmith.py, test_ci_test_execution_registry.py, test_ci_tui_routes_avoid_nightly_label.py, test_ci_reusable_workflow_permissions.py, test_ci_linux_guard_routing.py.
• Red on the head with the same failures as base 7fc75a9: test_runner_routing_table.py (stable-next.yml and nightly.yml entries), test_ci_blacksmith_linux_failover.py (app-platform.yml, cmux-next-chromium-currency.yml), test_ci_runs_on_fallback.py (nightly.yml), test_ci_signing_keychain_is_deleted_before_bundle_code.py (stable-next.yml). runner_routing_table.py check reports the same 5 problems as base, and none for the new jobs.
• On macOS (run by hand, the way a mini would run them): checks and tooling-tests steps gave 81 pass and 5 fail. Two were macOS differences, fixed here: the web lint pre-push test read nice (BSD nice prints usage), and package_deps_heal.offender() passed \s, \w and \b to git log -G. macOS's regex has none of those, so the heal never found the offending commit there; it now uses POSIX classes. The other three ("No new SwiftUI", "Scrollbars thin", "Package conventions lint") are continue-on-error warnings that report findings in base content.
• Not run: the jobs on a real mini, in this PR's own run. That run uses the base branch's placement, which doesn't exist yet, so it routes the old way. A probe commit forced them onto the side label for one run (then reverted). They waited over 50 minutes with no free mini, because every side runner was busy, and the next push cancelled them. So there is no on-mini result yet. That wait is the case this PR avoids: placement sends a job to a mini only when that mini is idle now.

Routing classifiers

On a mini, a job that portable placement places still runs only its own scripts, none of the app's Mac tiers. scripts/ci/cmux_next_route.py (PORTABLE_RUNNERS) and scripts/ci/detect_ci_change_areas.py (the guard_tests_runner term) therefore read these runners as Linux. Without that, an edit to checks would run every Mac tier, and a change to any guard test the release-notary job names would count as reaching the app. That made test_ci_nightly_push_throttle.py and test_cmux_next_route.py red on this branch; both now pass, and test_ci_change_areas.py fails per test exactly as on base. Main's owned-pool rescue must also watch cmux-next-guards.yml (test_ci_owned_pool_sources.py). That is cmux#29343, into main; this PR lands after it.

Fleet Linux

RUNNER_POOLS.linux and LINUX_RUNNER still resolve to Blacksmith 32 vCPU. No self-hosted Linux Actions runner is registered: the fleet dashboard's runner list is all glaeda Mac labels. So no job can take fleet Linux yet. This PR reads vars.LINUX_RUNNER as step 1 whenever it names a non-Blacksmith label. Once a GCP pool runner label exists and the variable names it, these jobs take it with no further change. Registering those runners is a runner-trust change and is not in this PR.

Changelog

none

🤖 Generated with Claude Code

Agent session: agent session

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 4caaeb3: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr29130 -f ref=ci/linux-jobs-fall-back-to-minis -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4120428201901 (e1849d914e)
• cmux NEXT nightly 1.0.0-nightly.4120480532201 (6b2795d84a)

[teamleaderleo]

Owned by an agent session · session

[teamleaderleo]

SW-15 independent exact-head review — FIX at b47a6add70c5b843b049c69c73ffd1f84bb97edf (owner @teamleaderleo; branch ci/linux-jobs-fall-back-to-minis; base feat-cmux-next). Read the current portable-placement code, all three consuming workflows and the actual tests. Two P1 route-selection gaps undermine the PR's promised fleet Linux → idle owned mini → Blacksmith choice.

[P1] Unconditional vars.CI_SIDE_LANE_RUNNER wins before the new placement output. .github/workflows/cmux-next.yml:1578, cmux-next-guards.yml:230 and :628 all choose ... || vars.CI_SIDE_LANE_RUNNER || needs.portable-placement.outputs...linux_jobs ... || ...mini_jobs ... || blacksmith. GitHub Actions short-circuits ||: whenever CI_SIDE_LANE_RUNNER is configured, its nonempty string wins regardless of which job portable-placement selected, whether its snapshot is stale, whether the mini is occupied/reserved for a nightly, or whether this is an ineligible rerun. The guard group additionally has an unguarded vars.LINUX_RUNNER after the selected outputs. The order test test_ci_portable_job_placement.py:166-184 only checks relative substring indices for linux/mini/Blacksmith and never checks earlier || terms, so it passes while routing bypasses the planner. Fix: remove the unconditional legacy side labels from these eligible jobs; use only attempt-matched linux_jobs/mini_jobs outputs before the cloud route, with a separate intentional emergency override if required. Regression: render the expression (or inspect parsed OR terms) with configured CI_SIDE_LANE_RUNNER and planner returning empty; assert Blacksmith wins, not the side label. With planned fleet Linux assert Linux wins before a side label.

[P1] The Python planner assigns an unverified/offline/busy Linux runner without checking availability. scripts/ci/portable_job_placement.py:81-86,128-147 makes fleet_linux a string-prefix filter, then puts every non-MAC_FIRST job into linux_jobs merely because FLEET_LINUX has a label. runners and the janitor snapshot are consulted only for the remaining mini jobs. In fact, test_ci_portable_job_placement.py:69-73 asserts all jobs route to gcp-linux-32 while the runner fixture contains five Macs and zero Linux hosts. If that configured Linux runner is offline, already busy or all of its slots queued, all those jobs can wait on it indefinitely; the spare mini / Blacksmith overflow are never considered. Fix: validate that the Linux label is an allowlisted owned runner and at least one matching runner is online/idle after already queued work. Allocate at most the available capacity; put remainder through mini and cloud. Treat unreadable runner/snapshot as an unverified Linux choice (fail over), and add explicit offline/missing/busy, partial-capacity, and real idle-Linux tests.

Both defects are visible in current source and tests. I did not trigger CI jobs or claim a live failure. The existing tests exercise the decision function and check for output strings while missing the complete workflow expression and Linux capacity reality. Light fix: gate the workflow's actual runs-on order and verify runner capacity. Heavy rewrite of scheduling/leases is unnecessary here; leave the owner's branch and nightly reservations intact, and rerun workflow/routing tests at its next head before a merge.

[teamleaderleo]

[P1] Placement bypass. A truthy vars.CI_SIDE_LANE_RUNNER is evaluated before any needs.portable-placement output and wins every || chain, so this job takes that mini label even when placement found no free mini, rejected a stale snapshot, or chose fleet Linux. The same bug is in cmux-next-guards.yml:230,628. Remove the unguarded legacy side-route terms; add an expression-order regression that asserts empty placement → Blacksmith with this variable set.

[teamleaderleo]

[P1] Missing/offline/busy fleet Linux always wins. linux_jobs is filled from FLEET_LINUX string alone; the runner list and pool snapshot are considered only for minis. The current test_fleet_linux_comes_before_the_minis passes five Mac runners and zero Linux runners yet asserts all jobs go to Linux. Verify owned Linux runner online/idle capacity and subtract queue before assigning jobs, then carry the rest to mini/cloud; add offline/absent/busy and partial-capacity cases.

[teamleaderleo]

This test uses idle(5) — five Mac runners, zero Linux — but asserts all three jobs route to gcp-linux-32. This fixture currently accepts the broken promise: a configured Linux label is treated as a functioning free runner. Include an actual available Linux fixture for positive priority, then assert fallback when the Linux runner is absent, offline, busy or already queued.

[blacksmith-sh]

Found 17 test failures on Blacksmith runners:

Failures

Test	View Logs
1 runner guard check(s) failed:/1 runner guard check(s) failed:	View Logs
1 runner guard check(s) failed:/1 runner guard check(s) failed:	View Logs
test_big_red (main.FleetRuleLints.test_big_red)/ test_big_red (main.FleetRuleLints.test_big_red)	View Logs
test_every_blacksmith_linux_label_fails_over (main.BlacksmithLinuxFailoverTest.test _every_blacksmith_linux_label_fails_over)/ test_every_blacksmith_linux_label_fails_over (main.BlacksmithLinuxFailoverTest.test _every_blacksmith_linux_label_fails_over)	View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/ test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)	View Logs
test_every_owned_workflow_is_watched (main.OwnedPoolSources.test_every_owned_workfl ow_is_watched) (tree='/home/runner/_work/cmux/cmux')	View Logs
test_every_upload_is_short_lived_and_small_or_allowlisted (main.WorkflowTests.test_ every_upload_is_short_lived_and_small_or_allowlisted)/ test_every_upload_is_short_lived_and_small_or_allowlisted (main.WorkflowTests.test_ every_upload_is_short_lived_and_small_or_allowlisted)	View Logs
test_no_retry_attempt_of_any_workflow_takes_an_owned_label (main.Wiring.test_no_ret ry_attempt_of_any_workflow_takes_an_owned_label) (workflow='cmux-next-guards.yml')/ test_no_retry_attempt_of_any_workflow_takes_an_owned_label (main.Wiring.test_no_ret ry_attempt_of_any_workflow_takes_an_owned_label) (workflow='cmux-next-guards.yml')	View Logs
test_no_retry_attempt_of_any_workflow_takes_an_owned_label (main.Wiring.test_no_ret ry_attempt_of_any_workflow_takes_an_owned_label) (workflow='nightly.yml', job='generate -nightly-deltas', attempt='3')/ test_no_retry_attempt_of_any_workflow_takes_an_owned_label (main.Wiring.test_no_ret ry_attempt_of_any_workflow_takes_an_owned_label) (workflow='nightly.yml', job='generate -nightly-deltas', attempt='3')	View Logs
test_no_retry_attempt_of_any_workflow_takes_an_owned_label (main.Wiring.test_no_ret ry_attempt_of_any_workflow_takes_an_owned_label) (workflow='nightly.yml', job='sign-nig htly-identity-legacy', attempt='3')/ test_no_retry_attempt_of_any_workflow_takes_an_owned_label (main.Wiring.test_no_ret ry_attempt_of_any_workflow_takes_an_owned_label) (workflow='nightly.yml', job='sign-nig htly-identity-legacy', attempt='3')	View Logs

...and 7 more test failures. View all on Blacksmith

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Review (codex, router-review): SHIP at 4caaeb3

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 3. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at 4caaeb3

Visual verdict: no images; Leo tries the build

1. [P2] [follow-up] portable_job_placement.py:158: Pool selection happens before queue subtraction and nightly reservations. Reproduced with three idle light runners, three queued light jobs, and three idle std runners: everything goes to Blacksmith. Evaluate effective availability across compatible pools before choosing one.

2. [P2] [follow-up] cmux-tui-artifacts.yml:302: TUI placement precedes tree-owner-wait, which can run for minutes before handing ownership to this run. The build then consumes the old mini label without rechecking availability. Those runners may now be busy or reserved for nightly work, putting publication back into the queue this change aims to avoid. Refresh placement after ownership resolves.

3. [P2] [follow-up] cmux-next-guards.yml:230: Moving ci-guard-group onto macOS drops existing test coverage. run_ci_guards.skipped_here() skips “Validate CMUX workload profile contract” on macOS; I confirmed this against the generated plan. Keep that test on a Linux job so mini availability does not determine whether the contract is tested.

Relevant local workflow, placement, fork-isolation, and supply-chain checks passed. Three guard-test cases require Git history absent from this snapshot.

Follow-ups:

• Choose compatible pools using queue-adjusted availability and nightly reservations.
• Refresh TUI placement after the publisher ownership wait.
• Preserve Linux workload-profile test coverage.