test262: five staging files excluded as non-terminating do terminate, and pass - #3513
Merged
Merged
Conversation
… and pass (sebastienros#3505) Six of the twenty entries under the STAGING EXCLUSIONS banner were filed as "does not terminate inside the 30-second TimeoutInterval". All six terminate and all six pass; measured on their own they take 0.2 to 3.0 seconds. TimeoutInterval is wall clock, not work, so a test whose worker is starved by a neighbour holding the whole 30 seconds times out on the neighbour's behalf, and this group was filed off one run in which every genuinely non-terminating file was still in the suite. Five leave the list. Four full runs with all six removed put every failure in a 30-second TimeoutException, and the set was not confined to the six: three files main already ships unexcluded and green -- short-circuit-compound-assignment.js and object-literal-__proto__.js at 1.0 s, plus an intl402 one -- failed as often as or more often than has-instance-jitted.js at 0.9 s, so that one goes too. nullish-coalescing.js stays, with the reason corrected. At 3.0 s per mode it is three times the next slowest file in the suite -- 1e5 iterations of a thirteen-assertion function against a staging/sm median in the single-digit milliseconds -- and it failed all four runs in both modes, including the run in which nothing else failed at all. That is a cost, not a loop that never ends. The two sub-groups merge, since the distinction between them is what produced the mistake, and the remaining comment says what earns a place in it: a file that does not finish on its own, confirmed by running the file on its own. test262, two runs on the final configuration: 102,532 passed / 1 failed / 155 skipped of 102,688, each run's single failure a contention flake on a file this change does not touch. Control on main is 102,523 / 0 / 165. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014W5mbjGhyvgAS4pivXoc4S
lahma
force-pushed
the
staging-stale-exclusions
branch
from
August 30, 2026 15:49
4608a89 to
d3fee7f
Compare
This was referenced Aug 30, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the third finding in #3505. Six of the twenty
staging/entries under the=== STAGING EXCLUSIONS ===banner were filed as "does not terminate inside the 30-secondTimeoutInterval". All six terminate and all six pass. Five leave the list; the sixth stays, for a reason that is not the one on the banner.What they actually cost
Per-test duration with the machine to itself (
--logger "console;verbosity=detailed", slowest mode shown):staging/sm/regress/regress-619003-1.jsstaging/sm/RegExp/unicode-braced.jsstaging/sm/RegExp/unicode-class-braced.jsstaging/sm/RegExp/unicode-ignoreCase.jsstaging/sm/Function/has-instance-jitted.jsstaging/sm/expressions/nullish-coalescing.jsTimeoutIntervalis wall clock, not work, so a test whose worker is starved by a neighbour holding the whole 30 seconds times out on the neighbour's behalf. This group was filed off one run in which every genuinely non-terminating file was still in the suite — the banner's own second paragraph says as much ("starves the rest of the run badly enough to make unrelated tests elsewhere in the suite time out too"); it just did not follow that the starved test could be one of these. Twenty-four PRs landed between #3016 and now, and #3070 ("Choose the regex engine per subject") is the likeliest reason the threeRegExpfiles are also cheap in absolute terms today.Four full runs, and what they show
Four consecutive full
dotnet test -c Release Jint.Tests.Test262runs with all six removed, on a workstation that had other work on it:Every run: passed + failed = 102,535, and skipped 153 — exactly control's 165 minus the twelve entries removed. The total stays 102,688 because an excluded file is still a generated test that reports as skipped.
Every failure in all four runs was a 30-33 s
TimeoutException, and the set was not confined to the six:staging/sm/expressions/nullish-coalescing.jsstaging/sm/expressions/short-circuit-compound-assignment.jsintl402/supportedLocalesOf-unicode-extensions-ignored.jsstaging/sm/expressions/object-literal-__proto__.jsstaging/sm/Function/has-instance-jitted.jsThat table is the whole argument. Three files
mainships unexcluded and green in CI —short-circuit-compound-assignment.js,object-literal-__proto__.jsand theintl402one — flake on this machine as often as or more often thanhas-instance-jitted.jsdoes, andhas-instance-jitted.jsis cheaper than either of the two staging ones. Excluding a 0.9 s file while the suite already runs 1.0 s files next to it would be incoherent, so it goes. Each of them passes in isolation in seconds.nullish-coalescing.jsis a different animal: 3 seconds per mode, three times the next slowest file in the suite,1e5iterations of a thirteen-assertion function against astaging/smmedian in the single-digit milliseconds. It failed 4 of 4 runs in both modes, including run B, in which nothing else failed at all. Its entry stays — but the comment above it now says what is true: it is a cost, not a loop that never ends, and what removes it is interpreter throughput or a budget that measures work instead of wall clock.Final configuration
Five entries removed, and two full runs to confirm:
staging/sm/expressions/short-circuit-compound-assignment.js, which this PR does not touch and which passes in 1 s standaloneintl402/supportedLocalesOf-unicode-extensions-ignored.js, likewise untouched here and passing in 4 s standaloneControl on
main: 102,523 passed / 0 failed / 165 skipped of 102,688. Ten skips (five files x two modes) become ten passes: 102,533 / 0 / 155 once the one contention flake each run is discounted. Withnullish-coalescing.jsback out of the running set the failure count drops straight back to the machine's one-per-run noise floor, and none of the five removed files appears in it.The banner
The two sub-groups merge — the distinction between them ("found by running the suite") is exactly what produced the mistake — and the remaining group's comment now says what earns a place in it: a file that does not finish on its own, confirmed by running the file on its own. Eleven entries remain there and are unchanged:
extensions/recursion.js, the eightDate/dst-offset-caching-*,JSON/parse-mega-huge-array.js,regress/regress-610026.jsandregress/regress-1507322-deep-weakmap.js, plusgenerators/delegating-yield-9.jsuntil #3506 removes it.Array/to-length.jsleaves via #3510.No engine change: this PR is the settings file only.
Re-measured after rebasing onto
8f0aac567, in a quiet moment: 102,533 passed / 0 failed / 155 skipped of 102,688 — the predicted figure exactly, with no flake at all.