Wpt: stand a server up, and find out what "needs a wpt server" was really hiding

[lahma]

Closes #3260.

Thirteen rows of WptTestRunner._notVendored read "needs a wpt server" — roughly a quarter of everything parked, and understated twice over, because a file needing one usually cannot produce a test report at all and so lands in the not-vendored table rather than in the exclusion table, where NeedsWptServer was a single row.

This stands the server up. It also does the census those thirteen rows deserved, and that turned out to matter more than the server: only about a third of what they parked was waiting on HTTP.

Phase A — the inventory, before anything was built

The thirteen rows are seven directory globs and six named files. Behind them: 82 .any.js files, read at the pinned commit 6c7127bd.

Row	.any.js behind it	Unlockable by a server alone	Why not
fetch/api/abort/*	3	0	general.any.js includes /common/get-host-info.sub.js (wptserve substitution) and ../request/request-error.js at file scope; the other two build a Request from a relative url
fetch/api/basic/*	31	9	22 need a browser header (Referer, forbidden-header filtering), a second origin, an HTTP cache, HTTP/2, blob:/data:, garbageCollect(), .sub. substitution, or an /xhr/resources/ handler
fetch/api/body/*	3	3	2 of the 3 need no server at all — they build their own Request/Response. The third wants Response.prototype.textStream, an unmerged Fetch pull
fetch/api/cors/*	21	0	The CORS request mode, a preflight, Access-Control-*, an opaque filtered response, and a second origin. RequestConstructor documents neither reading nor validating mode — there is no origin model to assert against
fetch/api/credentials/*	3	0	A cookie jar (SocketsHttpHandler.UseCookies is deliberately off), HTTP authentication, the credentials init member
fetch/api/policies/*	0	0	The directory holds no .any.js file at all — .html documents, .headers sidecars and the .js they load. This row was parking nothing runnable
fetch/api/redirect/*	15	4	11 need a second origin, Referer, keepalive + a window, HTTP/2, blob:/data:, or are .tentative
headers/header-values.any.js	1	1	—
headers/header-values-normalize.any.js	1	1	—
headers/headers-no-cors.any.js	1	0	The "no-cors" request mode
response/response-cancel-stream.any.js	1	1	—
response/response-clone.any.js	1	1	—
response/response-headers-guard.any.js	1	1	—
total	82	21

The endpoint inventory behind those 21 is small and repetitive: seven handlers (inspect-headers.py, status.py, method.py, redirect.py, redirect-empty-location.py, clean-stash.py, trickle.py), one static file (top.txt), one helper (/common/utils.js) and a root GET /. No .headers file, no charset generation, no cross-origin alias, one trickled body.

Payoff, in tests rather than files. 20 files vendored (the 21st, body/textstream.any.js, wants an unimplemented member), registering 326 assertions, of which 232 pass. Two of the 20 needed nothing but a corrected reason.

Two findings that changed the shape of the work, both of which arrived only when the files actually ran:

1. There is no API base URL, and the whole fetch corpus writes relative URLs. fetch("../resources/status.py?…") is a TypeError in Jint by design — RequestConstructor documents declining to invent a document URL and tells a host to "resolve it yourself with new URL(relative, base).href". Without that, a server unlocks nothing. The shim now does exactly what the doc tells a host to do, against a location.href that is where the server really serves the file. It pointedly does not wrap Request: rows that construct one from a relative url stay excluded under NeedsApiBaseUrl.
2. The drive loop treated "nothing scheduled" as "stalled". A request in flight is a thread-pool completion with no due time — the case Advanced.WaitForScheduledWork documents as findable only by polling. The server lane polls, with an idle timer reset by progress (a test settling) rather than by the wall clock.

Phase B — what was built

WptServer: raw TcpListener, port 0, in-process, per run

Not wptserve (Deno's model): a Python dependency in a .NET test suite's CI, and it gives up the vendored-and-byte-verified model Vendor/README.md depends on. WptServer serves the vendored corpus — the same WptCorpus bytes every other suite reads.

Not HttpListener and not Kestrel, and each reason is a property the suites need:

• Port 0. HttpListener takes a URL prefix and has no ephemeral form, so it would need a port picked in advance and a race to go with it. A collision in CI is a flake, and this cannot have one.
• No URL ACL, no framework reference. HttpListener needs an ACL for anything but localhost on Windows; Kestrel needs Microsoft.AspNetCore.App.
• It owns the bytes on the wire, which is what lets trickle.py dribble a body out and what makes the query's percent-decoding byte-exact (below).

The divergences from upstream's handlers, stated

The .py files cannot run, so six handlers are reimplemented in C#. This is the one part of the corpus that is not vendored bytes, and WptServerTests is what stops a reimplementation quietly answering something else — 13 tests, each written from the upstream source at the pin and named after it.

Upstream handler	What the C# does	What it deliberately does not
inspect-headers.py	echoes the named request headers back as x-request-<name>	the ?cors branch — no CORS model, so every file passing it is parked for that instead
status.py	status code, reason phrase, content type and body out of the query	—
method.py	echoes the method, the four content headers (or NO) and the body	the ?cors branch
redirect.py	redirect_status, location, simple, the query-preserving location rewrite with its &count= hop counter, and the token/max_count stash	Access-Control-*, the OPTIONS preflight branch, redirect_referrerpolicy, delay (no vendored file passes one)
redirect-empty-location.py	a 302 with an empty Location	—
clean-stash.py	drops one token's stash entry	—
trickle.py	count lines of TEST_TRICKLE, one every ms ms, same delay before the headers	—

How each was verified. Every row above has a test asserting the behaviour the vendored file depends on: InspectHeadersEchoesTheHeadersItWasAskedFor, StatusAnswersWithWhatTheQueryAsksFor, MethodEchoesTheRequestBackToTheCaller, RedirectAnswersTheStatusAndRewritesTheLocation, RedirectLeavesASimpleLocationAlone, RedirectCountsHopsInTheStashAndReportsThemAtMaxCount, RedirectEmptyLocationSendsAnEmptyLocationHeader, TrickleWritesTheLinesItWasAskedFor, TrickleSendsNoContentTypeWhenAskedNotTo, StaticFilesComeOutOfTheVendoredCorpus, APathTheCorpusDoesNotHoldIsA404.

One divergence was found by a suite before a test caught it, and is worth stating on its own: the whole server holds wire bytes as Latin-1. wptserve works in bytes — request.GET.first(b"content") is a percent-decoded byte string its handlers write straight back. basic/text-utf8.any.js asks for content=%fe%ff%4e%09, a UTF-16BE body served as text/plain;charset=UTF-8, precisely to see what Response.text() makes of bytes that are not valid UTF-8. A first pass used Uri.UnescapeDataString, which replaced them with U+FFFD before the engine ever saw them, and the file asserted against the server's mistake. TheQueryIsDecodedToBytesRatherThanToText pins it.

The lane is a list, not a rule

Giving the whole fetch/api/ corpus a real fetch would turn thirty already-green rows into network-dependent ones for nothing — they build their own Response and never ask for a URL. WptHarness._serverBackedFiles names 17 files, and TheServerLaneHoldsExactlyTheFilesItNames holds it in both directions, because the lane is the only place in the driver that grants WebApiFeatures.Fetch and is therefore the boundary of the promise that no suite can open a socket. Inside it, Options.WebApi.Fetch.UrlFilter is WptServer.Owns — re-checked on every redirect hop — so nothing can reach the network there either.

Tables brought back in line

• The 13 "needs a wpt server" rows are gone. Every remaining fetch row states the engine feature the file needs, file by file, and where a row still says "server" it names the handler shape this one does not reproduce.
• Three new theories (fetch/api/basic, body, redirect), 20 minimum-test entries, and the driver's own inventory checks extended.
• MatchesPattern gained \* for a literal asterisk. Without it there is no pattern that names accept-header.any.js's failing …with value '*/*' row and not its passing sibling …with value 'custom/*' — every literal segment of the one is a segment of the other, so the two-sided rule could never be satisfied.

The 94 rows that do not pass

Six categories, four of them new.

Five are genuine engine defects the corpus found the moment it could make a real request. Filed separately and deliberately not fixed here, so that the change which first runs a suite is not also the change that moves the engine. NeedsTriage is no longer empty, which is that mechanism working:

#	Issue	Rows	Citation
1	#3279 no default Accept: */*	1	HTTP-network-or-cache fetch step 8
2	#3280 a HEAD response carries a body stream	1	HTTP-network fetch
3	#3281 a fetched response's Headers are mutable	1	response header list, immutable guard
4	#3282 three content headers dropped from a bodiless request	8	Fetch has no content-header category; the BCL does
5	#3283 clone() tees without structured-cloning the chunks	14	clone a body

Each is localised by a passing sibling — the row that sets Accept explicitly passes, the 204/205/304 rows pass, redirect-method's POST rows pass, response-clone's other ten rows pass — which is what says these are the defects rather than the harness.

The other 69 are decisions or environment, and add up exactly: 35 + 21 + 10 + 2 + 1.

• 35 NeedsXmlHttpRequest (new). Both header-values* files run their whole table twice, once through XMLHttpRequest.setRequestHeader. The driver's XHR is a corpus reader that refuses anything but a GET, by name.
• 21 NeedsOpaqueRedirect (new). redirect: "manual" hands the script the redirect response rather than a browser's opaque filtered one — FetchTransport documents this as Node's reading of HTTP fetch step 6. The follow and error rows of both redirect files pass.
• 10 NeedsApiBaseUrl. text-utf8.any.js builds a Request from the empty string; its other ten rows go through fetch and pass.
• 2 NeedsPermissiveHeaderTransport (new). See below.
• 1 NeedsBrowserRequestHeaders (new). Accept-Language is a browser reporting a user's language preferences.

One divergence that is neither the engine nor the server

AHeaderValueAboveAsciiDoesNotSurviveTheHttpStack probes every byte from 0x01 to 0xFF through a real request and a real response, with no engine in the picture, and measures the line exactly: every byte up to 0x7F round-trips — control characters included, which are the ones that look most likely to be refused — and every byte from 0x80 up does not. The Fetch Standard's header value is any byte but NUL, LF and CR, so header-values.any.js and request-headers-nonascii.any.js are asserting something a browser carries and the .NET request/response header encodings (ASCII by default) do not. That is what puts those two rows in their own category rather than in NeedsTriage.

Flake evidence

A network-dependent suite is a new flake surface and this repository has spent real effort removing those (#3213, #3221, #3234). What makes this one deterministic:

• No fixed port. TcpListener(IPAddress.Loopback, 0) and the kernel picks; there is nothing to collide with.
• No external dependency, no DNS, no second process. The server is in the test process, on loopback.
• No wall-clock assertion anywhere. TrickleWritesTheLinesItWasAskedFor asserts the bytes, not the timing. The drive loop's idle timer is reset by a test settling, not by elapsed time.
• A request the server never answers becomes a failing test, not a stalled file. The lane's fetch timeout (10 s) is deliberately inside the drive loop's grace period (20 s).

Measured: five consecutive runs of --filter Jint.Tests.Wpt, 458 cases every time, 71/73/73/81/83 s. Three runs of upstream/main for comparison: 424 cases, 64/68/73 s. Median 68 s → 73 s, so the twenty new files cost about 5 s.

Verification

• dotnet build -c Release for the solution, every target framework: 0 errors. (One pre-existing MSB3277 in Jint.Tests.CommonScripts' net472 leg, untouched by this change.)
• Jint.Tests: 10,567 passed / 5 skipped on net10.0, 7,256 / 4 on net472.
• JINT_HOST_CONTRACT_VERIFICATION=1: same counts, plus Jint.Tests.PublicInterface 2,507 (net10.0) and 1,913 (net472).
• JINT_WPT_CENSUS=1: the inventory table Wpt: derive the corpus inventory table instead of maintaining it by hand #3276 derives agrees with every figure written here — the table checking the prose rather than the other way round.
• Every vendored file byte-verified against its upstream blob id at the pin (22 files, 0 drift).
• Rebased onto upstream/main after Wpt: derive the corpus inventory table instead of maintaining it by hand #3276 landed and re-verified in full.

Against the verdict

"A quarter of everything parked" was not a quarter of anything. The honest reading of the census is that the thirteen rows were badly over-stated: policies/* parked no runnable file at all, and cors/* plus credentials/* — 24 of the 82 — are blocked by engine features a server does nothing about. If this PR had only corrected the reasons and vendored the two body/ files that needed nothing, it would have delivered a real part of the value at none of the risk. Someone who thinks the flake surface is not worth 232 passing assertions has a coherent position, and the reason-corrections could be split out of this change cleanly.

The shim now resolves relative URLs, and that is the harness doing something the engine will not. It is defensible — RequestConstructor prescribes exactly this to a host, and the shim already plays the browsing environment by supplying location and GLOBAL — but it is one more place where a green row depends on the harness. The mitigation is that Request is deliberately not wrapped, so the constructor's own divergence is still asserted by 10 excluded rows and by the whole of fetch/api/request/ staying unvendored.

Six handlers in C# is six things that can drift from upstream. WptServerTests is the answer, but it is a test of the reimplementation, not the byte-verification the rest of the corpus gets. A corpus bump that changes redirect.py will not fail anything until someone reads it.

The 20-second grace period is a number. It is not derived from anything: it has to exceed the lane's 10 s fetch timeout and be well inside the 5-minute harness deadline, and 20 was picked in that window. A genuinely hung server-lane file costs 20 s of a run before it reports, where every other lane reports at once.

NeedsTriage going from 0 to 25 rows is a real cost. AGENTS.md says the emptiness is the signal, and this change spends it. Five issues are filed, but until they are paid the category no longer means "the corpus has found something new".

🤖 Generated with Claude Code