Shard Test262 across parallel CI runners + fix cross-OS suite cache#2661
Open
lahma wants to merge 2 commits into
Open
Shard Test262 across parallel CI runners + fix cross-OS suite cache#2661lahma wants to merge 2 commits into
lahma wants to merge 2 commits into
Conversation
…ache
Test262 execution is ~55-60% of every CI job (349s linux / 380s windows) and
free runners are capped at 4 vCPU, which NUnit already saturates — an experiment
sweeping NumberOfTestWorkers from 4 to 16 on a 4-vCPU runner showed no gain (flat
to 12, worse at 16). So the only lever for wall-clock is horizontal sharding.
- Jint.Tests.Test262.csproj: generate/compile/run only the shard named by
TEST262_SHARD_COUNT / TEST262_SHARD_INDEX (defaults to the full suite locally).
- pr.yml: 4-way shard on the slow x64 linux/windows runners; run the full suite
unsharded on the scarcer macOS/ARM runners; a fast `unit` job covers everything
else once per OS.
- generate-test262 job produces each shard once into a per-shard, cross-OS cache,
removing the previous 4-way same-key cache race ("another job may be creating
this cache") and the redundant per-OS regeneration.
- .gitattributes: force the settings file to LF so the cross-OS cache key and the
regeneration gate are byte-identical (it currently checks out CRLF on Windows).
- Bump test262harness.console to 1.1.0 (adds --shard-count / --shard-index) and
refresh actions/checkout (v7) and actions/cache (v6).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lahma
marked this pull request as ready for review
July 13, 2026 10:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Test262 execution is ~55-60% of every CI job (349s on linux, 380s on windows), and free GitHub runners are capped at 4 vCPU — which NUnit already saturates (
NumberOfTestWorkersdefaults toProcessorCount). An A/B sweep on a real 4-vCPU runner confirmed raising the worker count does not help:Flat through 12 (inside the noise band of
workers=4), worse at 16 — textbook CPU-bound saturation. So the only lever for wall-clock is horizontal sharding across runners, which is free for public repos.What
Jint.Tests.Test262.csproj):TEST262_SHARD_COUNT/TEST262_SHARD_INDEXmake the project generate, compile and run only its deterministic slice. Defaults to the full suite, so local builds are unchanged. Shard assignment lives in the harness (stable FNV-1a hash → disjoint shards whose union is the full 99,566-case suite, verified).pr.yml: 4-way shard on the slow x64 linux/windows runners; macOS/ARM run the full suite unsharded (one runner each — they're scarcer and already fast per-core); a fastunitjob runs everything except Test262 once per OS.generate-test262job produces each shard once into a per-shard, cross-OS cache. This removes the previous 4-way same-key race ("another job may be creating this cache") and the redundant per-OS regeneration..gitattributes: forcesTest262Harness.settings.jsontoeol=lf— required so the cross-OS cache key (hashFiles) and the MSBuild regeneration gate are byte-identical (it currently checks out CRLF on Windows).actions/checkout→ v7,actions/cache→ v6.Expected payoff
Per-shard Test262 exec ~349s → ~87s; a run drops from ~10-12 min to roughly ~4-5 min on the sharded runners (macOS/ARM stay ~7 min by design), and the cache is effective on warm runs instead of racing.
Validation
The harness feature and the csproj plumbing were validated locally (4 shards summing exactly to 99,566;
TEST262_SHARD_INDEX=1generated the correct 24,592-case slice with the shard folded into the regen hash). The workflow behavior itself should get one real Actions run once 1.1.0 is published.🤖 Generated with Claude Code