Wpt: derive the corpus inventory table instead of maintaining it by hand by lahma · Pull Request #3276 · sebastienros/jint

lahma · 2026-08-23T08:24:49Z

Closes #3251, which asked for an analysis that starts by trying to kill the idea. It did, and it killed most of it: of the four things that issue floated — a corpus generator, a wpt sync tool, a report mode and a census — only the census survives, and it survives as a test rather than as a tool.

The kill criteria, against `git log` rather than recollection

1. Frequency — fires for sync, and saves the census. The entire history of Jint.Tests/Wpt/ is 27 commits over three days. Ten of them added vendored files (+93, +83, +69, +48, +43, +13, +11, +1, +1, +1) — more than the four the issue assumed — but every one is inside a single campaign that has now ended. Sync is burst-shaped; extrapolating a rate from a burst is unfounded, and a documented recipe beats a tool.

The census is a different quantity, and this is the finding that turns the criterion around: the inventory table has moved 6 times in ~36 hours, and 5 of those came from engine-fix commits, not corpus additions (#3243, #3245, #3247, #3250). Two commits (#3238, #3244) fixed WPT-found defects and left the table stale behind them. Table staleness is driven by ongoing engine work, which is unbounded — so frequency argues for the census and against the sync tool.

2. "A shell script suffices" — fires, and it is now written down instead of built. The byte-verification is 48 gh api directory calls and git hash-object, ~20 lines, under a minute, no downloads (a GitHub blob id is git hash-object), and .gitattributes already pins the tree text eol=lf so a Windows checkout cannot confound it. Ran it: 359 files, zero drift, zero absent upstream. It is documented in Vendor/README.md as a recipe. The criterion does not fire for the census, because assertion counts can only come from the driver's own results and a script would have to reimplement the harness to get them.

3. The census wants to be a test. It does. That is what shipped.

4. Prior art — does not fire; adopting either model would be a downgrade. Checked live, not from memory. Deno's tests/wpt/runner/expectations/ carries no reasoning at all: schema.json sets additionalProperties: false, and wpt.ts update rebuilds every leaf, so a hand-added comment is erased. Node's test/wpt/status/*.{json,cjs} enforces only fail.expected; skip — 146 of ~240 entries — is never checked against reality, and its one free-text field (note) is undocumented and never read by the runner. Neither has a divergence category. Jint's table is two-sided over every entry and mandates a category plus prose; adopting either shape would lose both properties.

One incidental finding: §3's motivating example — the not-vendored row claiming Jint does not expose DedicatedWorkerGlobalScope — has already been corrected on main (it cites #3195 now), which weakens that section further.

What the census is

Vendor/README.md's inventory table stops being prose somebody maintains and becomes a derived artifact. Two tests, split by what each column costs to know:

Standards, suite counts, file counts — checked always, 45 ms. Derived by reading the embedded corpus, no execution. It is also what catches a corpus arriving with no row of its own: the prefixes must partition every vendored .any.js, so a new standard fails here rather than being silently left out.
Assertion and not-passing totals — opt-in. These can only be known by running the corpus.

WptCensus.Record hooks WptHarness.Run, the single funnel every vendored file's outcome returns through — so the census sees the whole corpus without the theories knowing it exists, and the driver's own rules are untouched. Measure() then runs only the files the theories have not already run, so the two chains overlap rather than duplicate.

JINT_WPT_CENSUS=1 checks the table; JINT_WPT_CENSUS=update rewrites it from the run — and that second mode is the maintenance procedure, replacing the hand re-census the README records having needed.

Proof it fails when the table is wrong

Four perturbations, each reverted afterwards:

perturbation	result
Fetch assertions 388→389 and total 40,657→40,658 — internally consistent, so only a measured check can see it	FAIL, exit 1, both tables side by side, fix command named
DOM files 13→14	FAIL, exit 1, in 45 ms (the always-on half)
removed `("DOM","dom/")` from the census rows	FAIL, exit 1, naming all 13 orphaned `dom/` files
Streams 1,170/4 → 1,999/9, then `JINT_WPT_CENSUS=update`	repaired, back to 1,170/4 and the total

update round-trips byte-identically on a correct table (same git hash-object) and preserves LF. And the table on main today is correct: the census independently reproduced 40,657 / 2,889 and all 13 rows.

Cost, and why it is gated


always-on half	45 ms, every platform, every filtered run
measured half, default	0 — skipped
measured half enabled	`Jint.Tests` net10.0 70 s → 79 s; an earlier always-on measurement read 65 s → 85 s

+15–30 % is material, so the measured half is gated on JINT_WPT_CENSUS and wired into the PR workflow's Windows leg only — the table is Windows-measured and assertion counts move per operating system — mirroring the JINT_HOST_CONTRACT_VERIFICATION leg beside it. A non-Windows leg gets an empty variable, which does not match the gate.

Two mechanisms built, measured and discarded

Worth recording so nobody rebuilds them:

xUnit assembly fixture — gates correctly (dotnet test exits 1) but this runner reports [Test Assembly Cleanup Failure] Xunit.Sdk.TestPipelineException with the message dropped, prints Passed!, and swallows Console.Out/Console.Error. Useless as a maintainer-facing check.
ITestCaseOrderer — invoked per test method here (invocations=1 seen=1), so it cannot order a fact after theories at all.

Verification

suite	result
Solution build, Release, all TFMs	0 errors (1 pre-existing `MSB3277`)
`Jint.Tests` net10.0	10,515 passed / 5 skipped / 0 failed — exactly +1 pass, +1 skip against baseline
`Jint.Tests` net472	7,256 / 4 / 0
`Jint.Tests.PublicInterface` net10.0 / net472	2,503 / 9 and 1,909 / 9, 0 failed
`Jint.Tests` net10.0, `JINT_WPT_CENSUS=1` (CI-equivalent)	10,516 / 4 / 0
WPT area, census on, after rebase onto current `main`	424 / 424 (422 corpus rows + 2 census tests)

The corpus is unchanged: git diff over Jint.Tests/Wpt/Vendor touches only README.md, no vendored .js/.json is modified, and the diff contains zero +|/-| table lines. 35 suites, 273 files, 40,657 assertions, 2,889 not passing — identical. Independently, all 359 vendored files still byte-match the pinned upstream SHA.

Against the verdict

The census can run the corpus twice. If the census test happens to run before the theories, Measure() runs nearly the whole corpus itself — which is where the +15–30 % comes from, and it doubles exposure to any timing flake in the worker/atomics lanes. It passed cleanly on repeated runs and the blast radius is one CI leg, but ordering could not be forced: ITestCaseOrderer does not work under this runner, as above.
The measured half is Windows-only, so a Linux-only contributor never sees it fail. That is deliberate — the table says in its own first line that it is Windows-measured, and updating it from a Linux run would write figures the table does not claim — but it is a hole, and the three platform-independent columns are what cover everyone else.
The pinned SHA is still only in prose. Putting it somewhere machine-readable was considered and deliberately not done: nothing automated consumes it today (the recipe greps it out of the README), so a constant nothing reads is scaffolding.

Filed rather than built

#3260 — the wpt server, which this analysis concluded is worth more than every tooling idea in the issue combined: 13 not-vendored rows cite it. Already open and being worked separately.

🤖 Generated with Claude Code

Issue sebastienros#3251 asked whether the WPT driver should get a Test262Harness-style tool, and required Phase 0 to try to kill the proposal first. Three of its four kill criteria fire; only the census survives, so only the census is built here. Frequency (criterion 1) says the whole history of Jint.Tests/Wpt is 27 commits across three days, with ten corpus-sync events all inside one campaign that has ended — so sync is burst-shaped and wants a recipe, not a tool. A shell recipe (criterion 2) is genuinely sufficient for the sync half: the byte-verification is 20 lines, it runs in under a minute over 48 API calls, and running it now reports 359 files with no drift at this pin. Prior art (criterion 4) would be a downgrade rather than a subsumption: Deno's expectation.json carries no reasoning at all — its schema forbids extra keys and `update` overwrites leaves — and Node's status files enforce only `fail.expected`, leaving `skip`, the majority of their entries, never checked against reality. Neither carries a divergence category. Jint's exclusion table is two-sided over every entry and mandates a category plus prose, so it is left exactly as it is. The staleness the census closes is real and is not tied to corpus additions: of the six times the table moved, five came from engine-fix commits, and two commits fixed WPT-found defects without touching it at all. Every figure in it is arithmetic the driver already does. So the table becomes a derived artifact, in two halves split by cost: - The standards, suite counts and file counts are read off the embedded corpus and are checked always, on every platform and in every filtered run, in 45 ms. This is also what refuses a corpus vendored with no row of its own: the census prefixes must partition every vendored .any.js. - The assertion and not-passing totals need the suites to have run. The check reuses every outcome the driver already produced — WptHarness.Run hands each file's results to WptCensus on the way back out — but always-on it still measured Jint.Tests at 65 s -> 85 s, which is material, so it is gated on JINT_WPT_CENSUS. The PR workflow's Windows leg sets it, in the same shape as the host-contract-verification leg beside it; an ordinary run pays nothing. JINT_WPT_CENSUS=update rewrites the table instead of failing against it, which replaces the hand re-census that had to be done four times. WptTestRunner's theories, the shim and the exclusion table's runtime enforcement are untouched. The corpus is unchanged: no vendored file is modified and no row of the table moves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHnbgQGW5QgfatxL6DgGEd

…ally hiding Thirteen rows of `_notVendored` read "needs a wpt server" — roughly a quarter of everything parked, and understated twice over, because a file needing one usually cannot produce a test report at all and so lands in that table rather than in the exclusion table. `WptServer` is that server: an in-process HTTP/1.1 origin on the loopback interface, a raw `TcpListener` on port 0, started once per test run. Not `wptserve` — that would mean a Python dependency in a .NET test suite's CI and would give up the vendored-and-byte-verified model `Vendor/README.md` depends on. It serves the *vendored* corpus, plus a C# port of six `.py` handlers, and `WptServerTests` holds each of those to the upstream source at the pin so a reimplementation cannot quietly answer something else. The census the thirteen rows deserved turned out to matter more than the server: only about a third of what they parked was waiting on HTTP. `cors/` (21 files) needs a CORS request mode `RequestConstructor` documents not having, and a second origin; `credentials/` needs a cookie jar `SocketsHttpHandler` deliberately does not carry; and `policies/` holds no `.any.js` file at all, so that row was parking nothing runnable. Every row is now stated file by file, in terms of the engine feature it needs rather than in terms of a server. Twenty files come back — nine of `basic/`, two of `body/`, four of `redirect/`, two of `headers/` and three of `response/`, in three new theories. Two of them (`body/formdata.any.js`, `body/mime-type.any.js`) never wanted a server at all: they build their own Request and Response, and the old directory glob was simply wrong about them. 326 assertions, 232 of them passing. Three things about the lane are load-bearing, and each is documented where it lives. It is a *list*, not a rule — giving the whole corpus a real `fetch` would turn thirty green rows into network-dependent ones for nothing, and the lane is the only place in the driver that grants `WebApiFeatures.Fetch`, so `TheServerLaneHoldsExactlyTheFilesItNames` holds it in both directions; the engine's `UrlFilter` is the server's own port, re-checked on every redirect hop, so nothing can reach the network even inside it. The shim supplies the API base URL the engine deliberately has not got, exactly as `RequestConstructor` tells a host to (`new URL(relative, base).href`), and pointedly does not wrap `Request`. And the drive loop learned to wait for work with no due time: a request in flight is a thread-pool completion `TimeUntilNextScheduledWork` cannot report, so the server lane polls with an idle timer reset by a test settling. The 94 rows that do not pass are named in the exclusion table under six categories, four of them new. Five are genuine defects the corpus found the moment it could make a real request, and they are deliberately left unfixed here so that the change which first runs a suite is not also the change that moves the engine: no default `Accept: */*` (sebastienros#3279), a body stream on a HEAD response (sebastienros#3280), a mutable header list on a fetched response (sebastienros#3281), three content headers dropped from a bodiless request (sebastienros#3282), and `clone()` teeing without structured-cloning the chunks (sebastienros#3283). `NeedsTriage` is no longer empty, which is that mechanism working. `MatchesPattern` gained one thing: `\*` for a literal asterisk. Without it there is no pattern at all that names `accept-header.any.js`'s "…with value '*/*'" row and not its passing sibling "…with value 'custom/*'". Verified: solution builds Release on every target framework; `Jint.Tests` green on net10.0 (10,567) and net472 (7,256), and again with `JINT_HOST_CONTRACT_VERIFICATION=1` alongside `Jint.Tests.PublicInterface` (2,507 / 1,913). The wpt suite ran five times over and reported the same 458 cases every time; with three runs of `upstream/main` for comparison the medians are 68 s before and 73 s after, so the twenty new files cost about 5 s. The inventory table is sebastienros#3276's derived one, and `JINT_WPT_CENSUS=1` agrees with the figures written here — which is the table checking the prose rather than the other way round. Every vendored file was byte-verified against its upstream blob id at the pin. Closes sebastienros#3260 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHnbgQGW5QgfatxL6DgGEd

…ally hiding (#3287) Thirteen rows of `_notVendored` read "needs a wpt server" — roughly a quarter of everything parked, and understated twice over, because a file needing one usually cannot produce a test report at all and so lands in that table rather than in the exclusion table. `WptServer` is that server: an in-process HTTP/1.1 origin on the loopback interface, a raw `TcpListener` on port 0, started once per test run. Not `wptserve` — that would mean a Python dependency in a .NET test suite's CI and would give up the vendored-and-byte-verified model `Vendor/README.md` depends on. It serves the *vendored* corpus, plus a C# port of six `.py` handlers, and `WptServerTests` holds each of those to the upstream source at the pin so a reimplementation cannot quietly answer something else. The census the thirteen rows deserved turned out to matter more than the server: only about a third of what they parked was waiting on HTTP. `cors/` (21 files) needs a CORS request mode `RequestConstructor` documents not having, and a second origin; `credentials/` needs a cookie jar `SocketsHttpHandler` deliberately does not carry; and `policies/` holds no `.any.js` file at all, so that row was parking nothing runnable. Every row is now stated file by file, in terms of the engine feature it needs rather than in terms of a server. Twenty files come back — nine of `basic/`, two of `body/`, four of `redirect/`, two of `headers/` and three of `response/`, in three new theories. Two of them (`body/formdata.any.js`, `body/mime-type.any.js`) never wanted a server at all: they build their own Request and Response, and the old directory glob was simply wrong about them. 326 assertions, 232 of them passing. Three things about the lane are load-bearing, and each is documented where it lives. It is a *list*, not a rule — giving the whole corpus a real `fetch` would turn thirty green rows into network-dependent ones for nothing, and the lane is the only place in the driver that grants `WebApiFeatures.Fetch`, so `TheServerLaneHoldsExactlyTheFilesItNames` holds it in both directions; the engine's `UrlFilter` is the server's own port, re-checked on every redirect hop, so nothing can reach the network even inside it. The shim supplies the API base URL the engine deliberately has not got, exactly as `RequestConstructor` tells a host to (`new URL(relative, base).href`), and pointedly does not wrap `Request`. And the drive loop learned to wait for work with no due time: a request in flight is a thread-pool completion `TimeUntilNextScheduledWork` cannot report, so the server lane polls with an idle timer reset by a test settling. The 94 rows that do not pass are named in the exclusion table under six categories, four of them new. Five are genuine defects the corpus found the moment it could make a real request, and they are deliberately left unfixed here so that the change which first runs a suite is not also the change that moves the engine: no default `Accept: */*` (#3279), a body stream on a HEAD response (#3280), a mutable header list on a fetched response (#3281), three content headers dropped from a bodiless request (#3282), and `clone()` teeing without structured-cloning the chunks (#3283). `NeedsTriage` is no longer empty, which is that mechanism working. `MatchesPattern` gained one thing: `\*` for a literal asterisk. Without it there is no pattern at all that names `accept-header.any.js`'s "…with value '*/*'" row and not its passing sibling "…with value 'custom/*'". Verified: solution builds Release on every target framework; `Jint.Tests` green on net10.0 (10,567) and net472 (7,256), and again with `JINT_HOST_CONTRACT_VERIFICATION=1` alongside `Jint.Tests.PublicInterface` (2,507 / 1,913). The wpt suite ran five times over and reported the same 458 cases every time; with three runs of `upstream/main` for comparison the medians are 68 s before and 73 s after, so the twenty new files cost about 5 s. The inventory table is #3276's derived one, and `JINT_WPT_CENSUS=1` agrees with the figures written here — which is the table checking the prose rather than the other way round. Every vendored file was byte-verified against its upstream blob id at the pin. Closes #3260 Claude-Session: https://claude.ai/code/session_01FHnbgQGW5QgfatxL6DgGEd Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

lahma merged commit 50d4219 into sebastienros:main Aug 23, 2026
5 checks passed

lahma mentioned this pull request Aug 23, 2026

Wpt: stand a server up, and find out what "needs a wpt server" was really hiding #3287

Merged

Navigation Menu

Repository navigation

Wpt: derive the corpus inventory table instead of maintaining it by hand - #3276

Wpt: derive the corpus inventory table instead of maintaining it by hand#3276
lahma merged 1 commit into
sebastienros:mainfrom
lahma:work/3251

lahma commented Aug 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lahma commented Aug 23, 2026

The kill criteria, against git log rather than recollection

What the census is

Proof it fails when the table is wrong

Cost, and why it is gated

Two mechanisms built, measured and discarded

Verification

Against the verdict

Filed rather than built

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

The kill criteria, against `git log` rather than recollection