Stop ClassBenchmark rebuilding its engine per iteration - #3053
Merged
Merged
Conversation
AGENTS.md says never to reach for [IterationSetup]: it forces InvocationCount=1 and UnrollFactor=1, which leaks tiered-JIT warm-up into the measured iterations and made identical code report 2.489 ms and 9.811 ms in different runs. ClassBenchmark still used it; every other class in the project already follows the rule, so this one was missed. Nothing needed it. Every row's script is idempotent with respect to engine state -- the two construction rows allocate garbage, GetSet rewrites the same property, and the class-declaration row runs inside an IIFE -- so one engine per row serves all iterations. It now follows the documented IsolatedScript pattern: each row gets its own engine, built in [GlobalSetup] and warmed with that row's own script and nothing else, so engine construction and warm-up stay out of the measurement. The prepared scripts move into [GlobalSetup] too. They were being parsed inside the benchmark method, which put Engine.PrepareScript on the measured path. Numbers from this class are not comparable to any published before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 23, 2026
legrab
added a commit
to legrab/pocok
that referenced
this pull request
Aug 25, 2026
Updated [Jint](https://github.com/sebastienros/jint) from 4.16.0 to 4.16.1. <details> <summary>Release notes</summary> _Sourced from [Jint's releases](https://github.com/sebastienros/jint/releases)._ ## 4.16.1 Jint 4.16.1 is the **first release from the new `4.x` maintenance branch**, and it marks the point where the two lines separate: `main` is now **5.0.0 development**, and `4.x` is where the 4.16.x line continues. **What that means for you.** If you are on 4.16.0, this is a drop-in update — it is correctness and conformance work only, **no API change and no changed default**. Every public signature is the same one 4.16.0 shipped, on all five target frameworks. If you want the 4.x line, take it from `4.x` and expect fixes rather than features. If you want to follow where the engine is going, watch `main` — v5 brings breaking API changes, an opt-in WHATWG web API surface, Web Workers, Node compatibility and a raised .NET Framework floor, and every one of them is recorded as it lands in [`docs/v5-migration.md`](https://github.com/sebastienros/jint/blob/main/docs/v5-migration.md). From this release onward the 4.x public surface is snapshotted per target framework in `Jint.Tests.PublicInterface/Verify/`, so "did the API move?" is a diff rather than a judgement call — on this branch a diff there is a bug, and comparing those files against `main`'s is the v4→v5 delta. ### Highlights **Conformance, from a suite that now runs more of test262.** The `staging/` directory is generated and executed for the first time (#3016), which is roughly 2,800 additional cases — largely SpiderMonkey's own suite contributed upstream, covering behaviour the stable directories never reach. Much of the work below is what it found. **Built-ins do what the spec says, step by step.** The array built-ins perform the internal methods they name rather than equivalents (#3066); `Array.from` honours `IsConstructor` and a typed array's `length` write throws (#3043); an array truncation walks downwards and the generics report the writes they fail (#3072); argument validation and evaluation order are corrected in five built-ins (#3069); `Map` and `Set` get the `[[SetData]]` tombstone their traversals are specified over (#3073); `Date.prototype.setTime` stores the clipped time value (#3042); and `Array.prototype.values`/`keys`/`entries` no longer gate on an array-like receiver (#3236). **Iterators and control flow.** A throw from the iterator step no longer closes the iterator (#3047); the `done` flag is consulted before stepping again (#3048); a rejected `return()` propagates out of an abandoned `for await` loop (#3113); an optional-chain short circuit is distinguished from a genuine `undefined` (#3040); a computed property key is evaluated even when spelled as a literal (#3039) and survives an `await` or `yield` intact (#3144, #3150); and destructuring the rest of an exhausted array yields an empty array rather than 2³² elements (#3263). **Numeric and string accuracy.** `Math.acosh`, `asinh`, `atanh`, `cbrt`, `expm1` and `log1p` are ported from fdlibm for correctly-rounded results across every target framework (#3050); `toFixed` formats from the double's exact value and reads `this` from `[[NumberData]]` (#3071); `String.prototype` case conversion derives from Jint's own Unicode tables rather than the host's culture data (#3068); and the regex engine is chosen per subject, with `RegExp.prototype.replace` no longer rewriting `lastIndex` (#3070). **Bounds that hold.** JavaScript strings have a maximum length instead of a wrapped array rent (#3015); a JSON document too long to become a string is refused while it is being built (#3028); a frame displaced by a proper tail call keeps counting while its trampoline runs, so `MaxRecursionDepth` cannot be evaded by leaving and re-entering the trampoline (#3022); and an `Atomics` waiter is released when nothing can ever notify it again (#3029). **Error messages no longer run user JavaScript** (#3041) — rendering a message for a value with a script-supplied `toString` used to invoke it, from inside the failure path. **Internationalization.** The five Temporal members the proposal removed are dropped (#3014), and `u`-extension options are canonicalized with every date format the spec allows (#3018). Two fixes in this release come from **@svenrog** — a sloppy function answering its own `arguments` (#3061) and the outer link on a parked `Function`-constructor environment (#3063). ## What's Changed * Drop the five Temporal members the proposal removed by @lahma in sebastienros/jint#3014 * Canonicalize u-extension options and format every date the spec allows by @lahma in sebastienros/jint#3018 * Mark a global created by an unresolvable assignment, and stop a waitAsync timeout outliving its engine by @lahma in sebastienros/jint#3019 * Run test262's staging/ directory too by @lahma in sebastienros/jint#3016 * Give JavaScript strings a maximum length instead of a wrapped array rent by @lahma in sebastienros/jint#3015 * Let a for-of frame decline the unwind it can only rethrow by @lahma in sebastienros/jint#3017 * Keep counting a frame a tail call replaced while its trampoline runs by @lahma in sebastienros/jint#3022 * Unpark staging/Temporal/removed-methods.js, which #3014 already fixed by @lahma in sebastienros/jint#3023 * Drop the Islamic date conversions no calendar path reaches by @lahma in sebastienros/jint#3027 * Let an Atomics waiter go when nothing can ever notify it again by @lahma in sebastienros/jint#3029 * Refuse a JSON document too long to be a string while it is being built by @lahma in sebastienros/jint#3028 * Bump the microsoft group with 3 updates by @dependabot[bot] in sebastienros/jint#3033 * Bump the analyzers group with 1 update by @dependabot[bot] in sebastienros/jint#3031 * Add initial threat model for untrusted scripts by @sebastienros in sebastienros/jint#3030 * Stop ClassBenchmark rebuilding its engine per iteration by @lahma in sebastienros/jint#3053 * createRealm installs a full $262 on the new realm and returns it by @lahma in sebastienros/jint#3044 * Give the benchmark suite a measurement environment by @lahma in sebastienros/jint#3055 * Evaluate a computed property key even when it is spelled as a literal by @lahma in sebastienros/jint#3039 * Stop error messages from running user JavaScript by @lahma in sebastienros/jint#3041 * Array.from honours IsConstructor, and a typed array's length write throws by @lahma in sebastienros/jint#3043 * Consult the iterator's done flag before stepping it again by @lahma in sebastienros/jint#3048 * Date.prototype.setTime must store the clipped time value by @lahma in sebastienros/jint#3042 * Answer a sloppy function's own arguments instead of throwing by @svenrog in sebastienros/jint#3061 * Keep the outer link on a parked Function-constructor environment by @svenrog in sebastienros/jint#3063 * A throw from the iterator step must not close the iterator by @lahma in sebastienros/jint#3047 ... (truncated) Commits viewable in [compare view](sebastienros/jint@v4.16.0...v4.16.1). </details> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>
This was referenced Aug 26, 2026
This was referenced Sep 2, 2026
This was referenced Sep 7, 2026
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ClassBenchmarkused[IterationSetup], whichAGENTS.mdexplicitly forbids: it forcesInvocationCount=1andUnrollFactor=1, leaking tiered-JIT warm-up into the measured iterations. The repo's own note records this making identical code report 2.489 ms and 9.811 ms in different runs.Every other benchmark class already follows the rule — this one was missed.
Why nothing needed it
Each row's script is idempotent with respect to engine state, so one engine per row serves all iterations:
ConstructSimple/ConstructDeepInheritanceGetSetReEvaluateClassDeclarationsIt now follows the documented
IsolatedScriptpattern — each row gets its own engine, built in[GlobalSetup]and warmed with that row's own script and nothing else — so engine construction and warm-up stay outside the measurement while the rows keep their warm-dispatch character.The prepared scripts also move into
[GlobalSetup]; they were being parsed inside the benchmark method, puttingEngine.PrepareScripton the measured path.Numbers from this class are not comparable to any published before this change.
🤖 Generated with Claude Code