ci: split test.yml into four concurrent jobs behind a fail-closed result gate

[flora131]

What this does

.github/workflows/test.yml ran ~14 steps sequentially in one job per platform, so its wall clock was the sum of every step rather than the longest dependent chain. This splits the work into four independent jobs and keeps job id test as a fail-closed result gate.

job	chain	Linux	Windows
suites	build package → unit → integration	121 s	195 s
agent-suite	native bindings → coding-agent vitest	126 s	232 s
release-archive	build package → build-binaries.sh → archive smoke	74 s	149 s
static-checks (Linux only)	typecheck, file length, docs links, Mintlify, CI contracts	30 s	–
test (gate, 2 legs)	assert every work-job result is success	15 s	–

Steps stay together only when one consumes another's build output: unit tests stay behind the package build (test/unit/pi-0.82.1-artifacts.test.ts silently test.skips without dist/), integration stays behind the package build plus Node, vitest stays behind the natives build, and release-archive rebuilds natives itself because scripts/build-binaries.sh reuses or rebuilds packages/natives/native/*.node. Rust is dropped from suites and static-checks; Node is dropped from static-checks. The blanket 10/15-minute caps become measured per-job caps (suites 6/8, agent-suite 6/9, release-archive 5/6, static-checks 6, gate 5), each leaving room for the one bounded flake retry its job owns.

Before / after wall clock

	before	after
PR wall clock (max of both platform jobs)	452 s avg (434–483 s, n=3 healthy runs, GitHub API)	~247 s
saving		~205 s (45 %), 7 m 32 s → 4 m 07 s
Linux job	257 s	126 s
runner-seconds	709 s	~957 s (+35 %)

The critical path today is the Windows job at 452 s. After the split its longest chain is 48 s setup + 42 s native bindings + 142 s coding-agent vitest = 232 s, plus a ~15 s Linux result gate. The runner-second rise is the deliberate price of the wall-clock cut, not a surprise.

⚠️ Measured on the first two split runs — read before merging

Both runs were green, but the estimate above did not hold. Recording it honestly:

job	run 1 (30527771985)	run 2 (30528920082)
static-checks (linux-x64)	32 s	50 s
release-archive Linux / Windows	84 s / 162 s	83 s / 175 s
suites Linux / Windows	230 s / 348 s	147 s / 238 s
agent-suite Linux / Windows	138 s / 349 s	203 s / 380 s
test gate, both legs	3 s / 4 s	4 s / 5 s
whole run	433 s	440 s

Baseline for comparison: 452 s avg (434–483 s, n=3 healthy). Queue delays on run 1 were 9–10 s on Linux and 42–68 s on Windows.

The topology worked exactly as designed. All seven work jobs started within 68 s of run creation, so Blacksmith does not cap concurrency below seven and risk #1 (queueing) did not materialize. Both required contexts appeared with byte-identical names, the gate ran in 3–4 s, and static-checks was green in 32 s — feedback on typecheck and file length that previously arrived only at the end of a 257 s job.

The saving is ~15 s, not ~205 s. The sequential-job sampling the plan was built on understated every Windows step by roughly 1.5×:

step	sampled	run 1	run 2
Windows coding-agent vitest	142 s	221 s	237 s
Windows native binding build	42 s	63 s	72 s
Linux coding-agent vitest	70 s	78 s	126 s
Windows unit step	127 s	267 s (retried)	150 s
Linux unit step	84 s	190 s (retried)	101 s

On both runs the critical path was Windows agent-suite, whose real cost is 349–380 s rather than the 232 s the estimate assumed. That single number is where the plan's 247 s estimate came from, so the estimate cannot be recovered by tuning anything else.

On run 1 the unit step's one bounded flake retry fired on both platforms, from two different pre-existing flakes, each of which passed on the retry (run 2 had no retry at all):

• Linux: isolated default main lists and executes engine-only /workflow and /workflows while the host has no extensions (13.1 s, "Timed out waiting for fixture report") in interactive-engine-default-main.test.ts.
• Windows: raw child stdout and stderr share the bounded telemetry budget — one of the tests the plan already identified as timing-sensitive.

Neither is caused by this PR. test/unit is untouched by the diff and each job owns its own runner, so nothing here runs under added contention. They are reported, not repaired: repairing them is out of scope and would mean weakening assertions. The retry machinery behaved correctly and the duration guard printed its table on both platforms.

The useful structural result still stands: wall clock is now dominated by two steps instead of fourteen, so sharding coding-agent vitest (the plan's phase 2) now has a direct effect rather than a diluted one, and static-checks returns typecheck/file-length feedback in 32–50 s instead of at the end of a 257 s job. This PR deliberately does not shard anything, per the brief. docs/ci.md carries both runs so the next decision starts from measurements rather than from the estimate.

Follow-up commit: two Windows caps widened

suites Windows at 348 s was 72 % of its 8-minute cap on the very first run, with the retry. A cap that cancels a passing retried run is worse than a late hang detection, so the second commit applies the plan's own "roughly 2× measured p100" rule to the newer measurement: suites 6/8 → 8/12, agent-suite 6/9 → 6/12, release-archive unchanged at 5/6, static-checks 6, gate 5. Every cap still sits under the 15-minute Windows blanket it replaced, and the contract test now enforces that ceiling as well as the exact values.

Executed vs distinct test counts

Measured in this worktree, Bun 1.3.14, bun install fresh.

test/ci — the suite this PR changes

run	executed	distinct
before (bun run test:ci-contracts)	25	25
after, run 1	31	31
after, run 2	31	31
after, run 3	31	31
after, --shard=1/2	22	22
after, --shard=2/2	9	9

All three full runs and the union of the two shards produce byte-identical 31-name sets, so the suite is order- and partition-independent. Distinct coverage grows 25 → 31: one test (test workflow preserves its two-platform matrix and deterministic contracts) is replaced by six stronger ones, and every assertion it made is still made. Nothing was removed.

test/unit — unchanged by this PR, re-run to prove it

run	executed records	distinct	result
before, run 1	4427	4426	pass, 123.2 s
before, run 2	4427	4426	pass, 129.1 s
after, via scripts/run-flaky-test-suite.ts -- bun run test:unit	4427	4426	pass, 125.6 s, no retry

(The 4427th record is the same test.skip name printed twice; Bun's own footer reports Ran 4426 tests across 581 files in every run.)

test/integration: 289 executed, 289 distinct, pass, through the flaky-suite runner.

The 950 duplicate executions

They are real re-execution, not a reporting artifact, and this PR avoids them by not using --parallel anywhere.

--parallel implies --isolate. 20 files in test/unit import 108 sibling *.test.ts files (e.g. executor.test.ts imports 20+ siblings), and Bun also collects those 108 files in their own right. Sequentially the module registry is shared so the second collection is a no-op; under --isolate every file gets a fresh registry and each imported file's tests run twice. The arithmetic checks out: the sequential log attributes 607 records to the 20 aggregator headers and 360 to the 108 children = 967, against an observed excess of 981. A clean 8-file/24-test probe under --parallel=4 executed each test exactly once, so --parallel does not duplicate by itself. The subagents-attempt-watchdog.test.ts header appearing 9 times is separate and benign: Bun's parallel reporter reprints the file header on every worker flush.

Consequence for the duration guard: every duplicated test is scored twice, once under contention, so the 70 % fail ratio starts firing on tests that are healthy sequentially. That alone disqualifies --parallel for this repository. A follow-up issue is worth filing for the redundant aggregator files, which already make "which file owns this test" depend on collection order.

The 600 ms watchdog test

Nothing was changed about it. test/unit/subagents-attempt-watchdog.test.ts and test/unit/subagents-attempt-watchdog-helpers.ts are not deleted, skipped, weakened, or excluded, and the diff does not touch them.

Under a saturation proxy (--parallel=12 on 12 cores, the same one-worker-per-core oversubscription --parallel=4 would create on a 4 vCPU runner) the failure is a class, not one test, and it is unstable: three identical invocations failed 4, 3 and 3 tests, and the failing set changed between them. Fixing only the test named in the brief would not have made the first run green. The fixture sets a 1000 ms idle bound and a 4000 ms attempt cap for the whole file and individual tests tighten the wall cap to 600–700 ms; those bounds also apply to the healthy fallback attempt, so under contention the good child cannot produce first activity inside the idle window and the watchdog kills it. The wall-cap test already spends 928 ms of its 600 ms cap sequentially on an idle machine (330 ms of that is pure double child spawn).

The test is not wrong — the environment stops being able to serve a sub-second deadline. So this PR simply never runs the unit suite under contention. If parallel execution is revisited, the correct fix is to derive the fixture's idle bound from a spawn-latency baseline measured once per suite (idleMs = max(1000, k × measured spawn ms)) while leaving every assertion exact. Raising a magic constant would only move the cliff.

Duration-headroom guard

Unchanged, because nothing it reads changed. Each split job still runs its own scripts/run-flaky-test-suite.ts invocation over an unmodified bun run test:unit / bun run test:integration / bun run --cwd packages/coding-agent --bun test, so budget resolution, blind detection and the 40 %/70 % ratios are untouched. Verified end to end in this worktree:

• unit: gate enabled, default budget 30000 ms, 4303 samples of 4426, not blind, 0 warn, 0 fail, explicit per-test budgets resolved (12000/20000/30000 ms).
• integration: gate enabled, 277 samples of 289, not blind, 0 warn, 0 fail, explicit budgets resolved (30000/60000/240000 ms).
• Both wrote their .ci-diagnostics/<suite>-durations.md table on a green run.

A new contract asserts the three retried suites still reach the guard through unmodified bun run commands and that --parallel, --shard, --concurrent and --max-concurrency appear nowhere in the workflow, so a future edit cannot silently reshape the records the guard scores.

.ci-diagnostics artifact names are now job-unique (test-diagnostics-<job>-<binary_platform>); actions/upload-artifact@v4+ fails the whole run on a duplicate name.

Required check names — no maintainer action needed

No ruleset edit is required and no check name changes. Ruleset 9310196 requires exactly:

• test (blacksmith-4vcpu-ubuntu-2404, linux-x64)
• test (blacksmith-4vcpu-windows-2025, windows-x64)

Job id test keeps those two matrix rows and the same name: test (${{ matrix.os }}, ${{ matrix.binary_platform }}) expression, so both strings are byte-identical. It becomes a result gate with if: always() that fails on failure, cancelled and skipped.

The skipped case is the one that matters: a job whose needs failed is skipped, and GitHub counts a skipped required check as satisfied. Without always() plus an explicit result check, a failing suite would turn the required context green — strictly worse than today. Because needs.<job>.result collapses a matrix to one value, each gate leg asserts every platform's work jobs, which is strictly stronger than the per-platform meaning the context had before. The gate does no platform work, so both legs run on the Linux runner rather than burning a Windows runner and its measured 33 s queue.

If maintainers later prefer real per-job required contexts (which would remove the ~15 s gate), that is a separate deliberate change: replace the two contexts with the eight work-job contexts in the same window as the workflow merge. Do not do both at once.

One thing to verify on the first run of this PR: the estimate assumes 7 jobs start about as promptly as 2 do today (measured queue: Linux 9 s, Windows 33 s). Read gh api .../jobs --jq '.jobs[]|{name,created_at,started_at}'; if Windows queueing exceeds ~60 s, merge release-archive back into suites on Windows.

Contract-test changes

test/ci/ci-workflow-contracts.test.ts shrinks: its workflow-text helpers move to a new test/ci/workflow-text.ts shared by all four CI contract suites, and the topology contracts land in a new test/ci/test-workflow-topology.test.ts:

• the gate's two contexts are derived from the gate job's own matrix, plus if: always(), its needs set, and that its single step fails on failure|cancelled|skipped;
• per-job caps replace the 10/15 pair, and timeout_minutes: 10|15 may not come back;
• the vitest/native-binding adjacency is re-anchored inside agent-suite, and the smoke steps inside release-archive, so neither depends on file-order adjacency across jobs;
• the sticky-disk useblacksmith/actions checkout pair is asserted for each of the three cross-platform jobs, and the gate is asserted to check out nothing;
• artifact names must be unique across jobs.

Moving test:ci-contracts into a Linux-only job drops its only CRLF checkout (.gitattributes marks *.yml as text without eol=lf), so a new contract requires every CI contract suite to read workflow text through the newline-normalizing reader. That converts a platform-dependent trap into one Linux can enforce; release-publisher-contracts.test.ts and release-recovery-contracts.test.ts were converted to comply.

docs/ci.md is rewritten for the new job graph, the gate's role, the exact required contexts, and the measured basis for each cap.

No packages/*/CHANGELOG.md entry: this is CI-only infrastructure per AGENTS.md.

Validation

All run in a dedicated worktree with Bun 1.3.14 after a fresh bun install:

• two full green CI runs of the new workflow (30527771985, 30528920082), both required contexts present with identical names
• bun run typecheck ✅ · bun run lint ✅ · bun run check:file-length ✅ (max 500; largest touched file is 398 lines)
• bun run test:ci-contracts ✅ ×3 plus --shard=1/2 and --shard=2/2, identical 31-name sets
• bun run scripts/run-flaky-test-suite.ts ... -- bun run test:unit ✅ 4426 tests, first attempt, guard not blind
• bun run scripts/run-flaky-test-suite.ts ... -- bun run test:integration ✅ 289 tests, guard not blind
• bun run --cwd packages/coding-agent docs:check ✅
• Bun.YAML.parse on the new workflow, and the gate's shell logic exercised against success,failure,cancelled,skipped and empty inputs (empty fails closed)
• the full prek pre-commit hook set, including bun run test:unit, passed on commit

No unrelated failures were observed.

⚠️ Merge-order note

Open PR #2073 (ci: guard Blacksmith runner coverage and trim workflow comments) also edits .github/workflows/test.yml and test/ci/ci-workflow-contracts.test.ts. Whichever of the two merges second will need a rebase.

Greptile Summary

This change separates CI work into concurrent suite, native-agent, release-archive, and static-check jobs while preserving the two existing required test check names through a result gate. The gate was exercised with successful, failed, cancelled, skipped, and empty dependency-result sets: only the all-success case passed, and every incomplete or unsuccessful result failed closed. The repository’s CI workflow contract suite completed with 32 passing tests.

Confidence Score: 5/5

T-Rex Logs

What T-Rex did

• Validated that baseline results produced success across all four work items and that the extracted gate exited with code 0.
• Verified that failure, cancelled, and skipped scenarios cause the extracted gate to emit a rejection message and exit with code 1, and that empty results fail closed.
• Confirmed the contract test suite passed, including the topology test that asserts the exact required contexts, always() behavior, all four dependencies, and blocking result patterns.

_{Ran code and verified through T-Rex}

_{Reviews (4): Last reviewed commit: "docs(ci): record the second split run an..." | Re-trigger Greptile}