[teamleaderleo]

The webview test shards (react-apps-check (1..4) and their base replay react-apps-base-check) move from Blacksmith 32 vCPU to Blacksmith 16 vCPU. The overflow failover still goes to the other size (now 32). No other job changes. (Fork PRs resolve to Blacksmith on this job, as they did before this PR; the review follow-up is filed separately.)

Leo, 2026-10-30:

i assume blacksmith prob cut us off cuz we were using like $10k a day so

y'know, we'll have to be more self-reliant and size the stuff better

and later:

i mean the linux things cant our minis just run them

This PR only resizes jobs. The shards install Playwright Chromium with apt (--with-deps), so they count as Linux-only and the mini fallback does not apply here. The table below marks which of the other top jobs could run on owned minis; that work belongs in follow-ups.

Why these jobs

They cost the most on Blacksmith. Over 2026-10-29/30 (the fleet's daily GitHub job export), the shards used about 99k wall minutes a day at 32 vCPU, roughly 3.2M vCPU-minutes. The same days also had 303 shard runs that the overflow failover had already sent to 16 vCPU, which gives a same-suite comparison:

shard runs, 2026-10-28..30	n	p50	p90	max
32 vCPU, pull_request, success	6830	972 s	1377 s	2251 s
16 vCPU, pull_request, success	257	1107 s	1681 s	2139 s
32 vCPU, push, success	1088	886 s	1274 s	1865 s
16 vCPU, push, success	46	1016 s	1499 s	1812 s

On 16 vCPU the shards take 14% longer at p50 and cost half as much per minute. No 16 vCPU run hit the 40 min timeout or ran out of memory. The 4 vCPU OOM in run 37666132324 is why test_webviews_tests_run_on_a_runner_with_room_for_the_suite still forbids 4 vCPU.

Performance

metric	before	after	delta	harness or run link
shard Blacksmith vCPU-minutes / day	~3.17M (99k min × 32)	~1.81M (113k min × 16)	-1.36M (-43%)	gh-jobs export 2026-10-29/30; the after value scales the measured 16 vCPU p50 ratio (1.14)
shard Blacksmith wall minutes / day	~99k	~113k	+14k	same
est. Blacksmith $ / day (list $0.004/min per 2 vCPU)	~$6.3k	~$3.6k	-$2.7k	same
shard p50 wall (PR)	972 s	1107 s	+135 s	6830 vs 257 runs above
shard p90 wall (PR)	1377 s	1681 s	+304 s	same

Top 10 cmux jobs by Blacksmith use (2026-10-29/30, per day)

#	job (workflow)	wall min/day	vCPU-min/day	measured	runner today	smallest fit / owned option	Linux-only?
1	web / react-apps-check (1-4) (ci-web via ci.yml and cmux-next web)	~99k	~3.17M	p50 972 s PR, Test webviews step 765-898 s	32 vCPU	16 vCPU (this PR)	yes (Playwright apt deps)
2	gallery health / health (4 shards)	44.3k	716k	mean 619 s, render 448 s	16 vCPU	8 vCPU to measure; fewer shards on a PR that touches few entries	yes (Playwright apt deps)
3	cmux-tui artifacts / build aarch64-apple-darwin	11.0k	349k	mean 610 s	32 vCPU	owned Mac builds it natively	no (darwin target; a mini builds it natively)
4	cmux-tui artifacts / build x86_64-apple-darwin	10.7k	339k	mean 594 s	32 vCPU	owned Mac	no
5	cmux-next fast tier (cargo)	9.9k	282k	mean 217 s; cargo step 92 s at 32, 194 s at 16	32 vCPU	stays 32 (CPU-bound) or an owned Mac	no (cargo)
6	cmux-next checks	16.4k	264k	mean 346 s, mostly Python script tests	16 vCPU	4 vCPU or any owned mini	no
7	cmux-tui artifacts / build aarch64-unknown-linux-musl	7.7k	240k	mean 417 s	32 vCPU	16 vCPU	yes (musl)
8	cmux-tui artifacts / build x86_64-unknown-linux-musl	7.4k	231k	mean 403 s	32 vCPU	16 vCPU	yes (musl)
9	cmux-next guards / CI guard group	12.4k	221k	145 s at 32, 277 s at 4	32 / 4 vCPU	4 vCPU or any owned mini	no (Python guards)
10	Fast / checks (fast-checks.yml, dispatch on main)	6.2k	199k	mean 188 s	32 vCPU	16 vCPU	mixed

Not in the vCPU top 10 but high in wall minutes:

• cmux-tui artifacts / build-windows (20.2k on 4 vCPU Windows).
• nightly-guards / workflow-guard-tests / release-notary (18.3k on 4 vCPU; Python, so not Linux-only and could move to a mini).
• generated catch-up / catch up (9.5k on 2 vCPU, 279 s of it in checkout).

Risk

• Slower webview verdict. A PR's verdict comes about 2 min later at p50 and 5 min later at p90. The webview verdict gates landing (rulebook), so this is wall time on the landing path.
• Timeout tail. The slowest 16 vCPU run took 2139 s against the 2400 s timeout-minutes: 40. The 1920 s retry deadline still stops retries before the uploads. If timeouts appear, the fix is a fifth shard, not going back to 32.
• Memory. No 16 vCPU shard was SIGKILLed in 303 runs. run-webviews-tests.sh sizes its file and dev-server pools from nproc, so it runs fewer at once on its own.
• Failover. While CI_CLOUD_OVERFLOW_SAVED is set, the shards go to 32 vCPU, the switch's pairing for 16 (test_ci_blacksmith_linux_failover).

Tests

• tests/test_web_validation.py now expects 16 vCPU with the 32 failover. Run locally: 14 OK.
• tests/test_runs_on_labels.py: OK.
• tests/test_ci_blacksmith_linux_failover.py: 1 failure, app-platform.yml:45. It fails the same way on base cecaf56b, so it is a base red and not from this change.
• pr-overlap: no open PR on feat-cmux-next overlaps this change. Prior art was read: ci(webviews): isolate base comparison from react shards #24927 set 32 vCPU when the base replay was split out, and ci: hard-coded Blacksmith Linux runs-on fail over to another Blacksmith size #26439 added the failover. Neither had a cost reason for 32.

🤖 Generated with Claude Code

Agent session: agent session

[teamleaderleo]

Owned by an agent session · session

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 6a76ba2: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr29033 -f ref=ci/react-apps-check-16vcpu -f platform=macos -f backend_mode=local

Passes: CI passes on 5f06c0e4a5.

CI passes on 5f06c0e4a5 (run 41203255886 attempt 1).

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

web and size checks: green

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4120590216903 (810351da56)
• cmux NEXT nightly 1.0.0-nightly.4120932117101 (f6dafe0b13)
• cmux NEXT nightly 1.0.0-nightly.4121152269601 (d3f317b93c)
• cmux NEXT nightly 1.0.0-nightly.4121223757501 (9a6e1c2aa1)

[teamleaderleo]

Review (codex, router-review): SHIP at 8dc4d87

Follow-ups: 1. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at 8dc4d87

Visual verdict: no images; Leo tries the build

1. [P1][follow-up] Fork PRs still resolve to Blacksmith instead of hosted Ubuntu. For a fork PR targeting manaflow-ai/cmux, github.repository_owner is the base owner, so these expressions fall through to blacksmith-16vcpu (or the 32-vCPU failover) because they lack the head-repository fork branch. This contradicts the PR description and can leave fork checks queued on organization-only capacity. See ci-web.yml:302 and ci-web.yml:620.

Focused web validation tests pass; the failover suite retains only the documented base failure.

Follow-ups:

• [P1][follow-up] Add the explicit fork-PR ubuntu-24.04 branch to both selectors.

[blacksmith-sh]

Found 3 test failures on Blacksmith runners:

Failures

Test	View Logs
test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_ a_picked_signing_label_has_a_rerun_alternative)/ test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_ a_picked_signing_label_has_a_rerun_alternative)	View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/ test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)	View Logs
test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes t_no_runs_on_chain_ends_on_the_reserved_side_label)/ test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes t_no_runs_on_chain_ends_on_the_reserved_side_label)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}