Skip to content

test262: five staging files excluded as non-terminating do terminate, and pass - #3513

Merged
lahma merged 1 commit into
sebastienros:mainfrom
lahma:staging-stale-exclusions
Aug 30, 2026
Merged

lahma merged 1 commit into
sebastienros:mainfrom
lahma:staging-stale-exclusions

Conversation

@lahma

@lahma lahma commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Fixes the third finding in #3505. Six of the twenty staging/ entries under the === STAGING EXCLUSIONS === banner were filed as "does not terminate inside the 30-second TimeoutInterval". All six terminate and all six pass. Five leave the list; the sixth stays, for a reason that is not the one on the banner.

What they actually cost

Per-test duration with the machine to itself (--logger "console;verbosity=detailed", slowest mode shown):

file
staging/sm/regress/regress-619003-1.js 1.0 s removed
staging/sm/RegExp/unicode-braced.js 0.55 s removed
staging/sm/RegExp/unicode-class-braced.js 0.56 s removed
staging/sm/RegExp/unicode-ignoreCase.js 1.0 s removed
staging/sm/Function/has-instance-jitted.js 0.9 s removed
staging/sm/expressions/nullish-coalescing.js 3.0 s stays

TimeoutInterval is wall clock, not work, so a test whose worker is starved by a neighbour holding the whole 30 seconds times out on the neighbour's behalf. This group was filed off one run in which every genuinely non-terminating file was still in the suite — the banner's own second paragraph says as much ("starves the rest of the run badly enough to make unrelated tests elsewhere in the suite time out too"); it just did not follow that the starved test could be one of these. Twenty-four PRs landed between #3016 and now, and #3070 ("Choose the regex engine per subject") is the likeliest reason the three RegExp files are also cheap in absolute terms today.

Four full runs, and what they show

Four consecutive full dotnet test -c Release Jint.Tests.Test262 runs with all six removed, on a workstation that had other work on it:

run result
A 102,526 passed / 9 failed / 153 skipped
B 102,533 passed / 2 failed / 153 skipped
C 102,528 passed / 7 failed / 153 skipped
D 102,530 passed / 5 failed / 153 skipped

Every run: passed + failed = 102,535, and skipped 153 — exactly control's 165 minus the twelve entries removed. The total stays 102,688 because an excluded file is still a generated test that reports as skipped.

Every failure in all four runs was a 30-33 s TimeoutException, and the set was not confined to the six:

file failed / 4 runs excluded today? uncontended cost
staging/sm/expressions/nullish-coalescing.js 4 yes 3.0 s
staging/sm/expressions/short-circuit-compound-assignment.js 3 no 1.0 s
intl402/supportedLocalesOf-unicode-extensions-ignored.js 2 no
staging/sm/expressions/object-literal-__proto__.js 2 no 1.0 s
staging/sm/Function/has-instance-jitted.js 2 yes 0.9 s
the other four of the six 0 yes 0.2-1.0 s

That table is the whole argument. Three files main ships unexcluded and green in CIshort-circuit-compound-assignment.js, object-literal-__proto__.js and the intl402 one — flake on this machine as often as or more often than has-instance-jitted.js does, and has-instance-jitted.js is cheaper than either of the two staging ones. Excluding a 0.9 s file while the suite already runs 1.0 s files next to it would be incoherent, so it goes. Each of them passes in isolation in seconds.

nullish-coalescing.js is a different animal: 3 seconds per mode, three times the next slowest file in the suite, 1e5 iterations of a thirteen-assertion function against a staging/sm median in the single-digit milliseconds. It failed 4 of 4 runs in both modes, including run B, in which nothing else failed at all. Its entry stays — but the comment above it now says what is true: it is a cost, not a loop that never ends, and what removes it is interpreter throughput or a budget that measures work instead of wall clock.

Final configuration

Five entries removed, and two full runs to confirm:

  • Run 1: 102,532 passed / 1 failed / 155 skipped of 102,688 — the failure staging/sm/expressions/short-circuit-compound-assignment.js, which this PR does not touch and which passes in 1 s standalone
  • Run 2: 102,532 passed / 1 failed / 155 skipped of 102,688 — the failure intl402/supportedLocalesOf-unicode-extensions-ignored.js, likewise untouched here and passing in 4 s standalone

Control on main: 102,523 passed / 0 failed / 165 skipped of 102,688. Ten skips (five files x two modes) become ten passes: 102,533 / 0 / 155 once the one contention flake each run is discounted. With nullish-coalescing.js back out of the running set the failure count drops straight back to the machine's one-per-run noise floor, and none of the five removed files appears in it.

The banner

The two sub-groups merge — the distinction between them ("found by running the suite") is exactly what produced the mistake — and the remaining group's comment now says what earns a place in it: a file that does not finish on its own, confirmed by running the file on its own. Eleven entries remain there and are unchanged: extensions/recursion.js, the eight Date/dst-offset-caching-*, JSON/parse-mega-huge-array.js, regress/regress-610026.js and regress/regress-1507322-deep-weakmap.js, plus generators/delegating-yield-9.js until #3506 removes it. Array/to-length.js leaves via #3510.

No engine change: this PR is the settings file only.

Re-measured after rebasing onto 8f0aac567, in a quiet moment: 102,533 passed / 0 failed / 155 skipped of 102,688 — the predicted figure exactly, with no flake at all.

… and pass (sebastienros#3505)

Six of the twenty entries under the STAGING EXCLUSIONS banner were filed as
"does not terminate inside the 30-second TimeoutInterval". All six terminate and
all six pass; measured on their own they take 0.2 to 3.0 seconds. TimeoutInterval
is wall clock, not work, so a test whose worker is starved by a neighbour holding
the whole 30 seconds times out on the neighbour's behalf, and this group was
filed off one run in which every genuinely non-terminating file was still in the
suite.

Five leave the list. Four full runs with all six removed put every failure in a
30-second TimeoutException, and the set was not confined to the six: three files
main already ships unexcluded and green -- short-circuit-compound-assignment.js
and object-literal-__proto__.js at 1.0 s, plus an intl402 one -- failed as often
as or more often than has-instance-jitted.js at 0.9 s, so that one goes too.

nullish-coalescing.js stays, with the reason corrected. At 3.0 s per mode it is
three times the next slowest file in the suite -- 1e5 iterations of a
thirteen-assertion function against a staging/sm median in the single-digit
milliseconds -- and it failed all four runs in both modes, including the run in
which nothing else failed at all. That is a cost, not a loop that never ends.

The two sub-groups merge, since the distinction between them is what produced the
mistake, and the remaining comment says what earns a place in it: a file that
does not finish on its own, confirmed by running the file on its own.

test262, two runs on the final configuration: 102,532 passed / 1 failed /
155 skipped of 102,688, each run's single failure a contention flake on a file
this change does not touch. Control on main is 102,523 / 0 / 165.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014W5mbjGhyvgAS4pivXoc4S
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant