Add custom JavaScript regex engine (QuickJS libregexp port)

[lahma]

Summary

Port of QuickJS's libregexp engine to C# as a custom regex engine for patterns where .NET's System.Text.RegularExpressions produces incorrect results. Uses a three-tier architecture: direct .NET Regex for simple patterns, Jint's converter for patterns needing rewriting, and custom bytecode engine for ECMAScript-specific semantics.

97,638 test262 passed, 0 failures (+1,624 from 96,014 baseline on main)

Architecture

Parse time:  Acornima validates regex syntax (OnRegExp → Validate)
             NeedCustomEngine check prevents incorrect pre-compilation

Runtime:     JintLiteralExpression caches compiled result on AST node

             RegExpInitialize (three-tier routing):
               1. NeedCustomEngine(pattern, flags)?
                  → yes: compile to QuickJS bytecode (correctness over speed)
               2. JsRegExpConverter.TryConvert?
                  → yes: .NET Regex in standard mode (fast path)
               3. Otherwise:
                  → custom engine fallback

Why the hybrid approach matters

The custom bytecode interpreter is 13-17x slower than .NET's JIT-compiled Regex on heavy workloads. Benchmarked with dromaeo-object-regexp (40 regex tests — split/match/test/replace on 64KB strings), the branch shows no meaningful regression on existing workloads (confidence intervals overlap with main across all scenarios).

The three-tier routing keeps .NET Regex on the fast path for compatible patterns while the custom engine handles the categories of .NET divergence from ECMAScript spec.

What's included

Custom regex engine (Jint/Runtime/RegExp/)

• RegExpCompiler.cs (~2,840 lines) — pattern → bytecode compiler, ported from QuickJS libregexp.c
• RegExpInterpreter.cs (~1,300 lines) — backtracking NFA bytecode interpreter with full-snapshot state
• RegExpOpcode.cs — 58 opcodes (char matching, assertions, lookaround, backreferences, ranges)
• Unicode/UnicodeProperties.cs — Unicode property escapes (\p{}), case folding, set operations for v-flag
• Unicode/UnicodeData.cs — Unicode 17 character data tables (scripts, categories, properties)

Pattern routing (NeedCustomEngine)

Routes to custom engine for: /u and /v flags, named groups, scoped modifiers ((?i:...)), forward backreferences, lookbehind with backrefs, nullable quantifier captures, case-insensitive with non-ASCII content, duplicate named groups.

Converter (JsRegExpConverter)

Lightweight JS-to-.NET converter handling . → [^\n\r\u2028\u2029], \s/\S → JS whitespace sets, \b/\B → lookaround expansions, multiline ^/$ → anchors, and more. Falls back to custom engine on rejection.

Safety

• Backtracking stack depth limit (16M elements / ~64MB memory guard)
• Cancellation token checked every 10K opcodes + inside backreference loops
• Compilation constrained by RegexTimeout
• RegexTimeout propagated from engine options through preparation handlers

API compatibility

• [Obsolete] ParseResult shim on JsRegExp for backward compatibility
• Unused regExpParseResult parameter removed from Construct

Acornima dependency

Requires Acornima 1.4.0 which adds v-flag syntax validation and Unicode 17 script names.

Benchmark results (vs main)

Scenario	Main (ms)	Branch (ms)	Delta
ObjectRegExp Classic NonPrepared	126.3 ± 7.2	123.7 ± 4.4	-2% (noise)
ObjectRegExp Classic Prepared	105.5 ± 11.6	118.3 ± 27.5	+12% (high variance, CIs overlap)
ObjectRegExp Modern NonPrepared	122.8 ± 6.9	126.3 ± 4.9	+3% (noise)
ObjectRegExp Modern Prepared	103.8 ± 7.5	104.0 ± 8.9	~0%
SunSpider regexp-dna	101.6 ± 0.9	102.7 ± 5.4	+1% (noise)

Test plan

• Jint.Tests: 2,818 passed, 0 failed (net10.0) + 2,771 passed (net472)
• Test262: 97,638 passed, 0 failed, 1,270 skipped
• PublicInterface: 79 passed, 0 failed
• No regressions from baseline
• Regex timeout tests pass
• Benchmark: no meaningful regression on regex workloads

🤖 Generated with Claude Code

[lahma]

  Time estimate: ~2.5 hours total (wall clock), including:
  - ~30 min: Research and planning (exploring test262 exclusions, Acornima, QuickJS, alternatives, web searches, acornima PR discussion)
  - ~20 min: Phase 1 — abstraction layer + hybrid wiring (JsRegExp, RegExpPrototype, RegExpConstructor modifications)
  - ~90 min: Phase 2 — porting QuickJS libregexp (3 parallel agents for compiler/interpreter/unicode, plus manual build fixes)
  - ~20 min: Phase 3 — UnicodeProperties implementation (agent) + final testing and PR creation

What was built: 7,129 lines added across 12 files — a complete JavaScript regex engine ported from QuickJS's libregexp to managed C#, integrated as a hybrid fallback alongside the existing .NET Regex fast path. Zero test regressions.

This Claude is quite optimistic about those no regressions..

[Copilot]

Pull request overview

Adds a QuickJS libregexp-ported regex engine to Jint and integrates it as a runtime fallback when Acornima/.NET Regex cannot represent JavaScript RegExp semantics (notably v-flag, unicode property escapes, forward backrefs, and other semantic mismatches). This shifts regex handling toward “parse-time validation, runtime compilation” to preserve correctness while retaining .NET Regex as a fast path when safe.

Changes:

• Introduces a custom regex compiler + bytecode interpreter (QuickJS port) and Unicode property support.
• Routes RegExp initialization/execution to either .NET Regex or the custom engine via detection + fallback.
• Updates parsing defaults/Prepare* flow to validate regex syntax without requiring conversion at parse time; enables additional test262 RegExp coverage.

Reviewed changes

Copilot reviewed 22 out of 24 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
Jint/Runtime/RegExp/Unicode/UnicodeProperties.cs	Adds Unicode property lookup, case folding, and set operations used by the custom engine.
Jint/Runtime/RegExp/RegExpSyntaxException.cs	Adds a dedicated exception type for custom engine compilation failures.
Jint/Runtime/RegExp/RegExpOpcode.cs	Defines custom-engine bytecode opcodes, sizes, and header layout.
Jint/Runtime/RegExp/RegExpMatchResult.cs	Adds match/capture result structs for the custom engine execution path.
Jint/Runtime/RegExp/RegExpInterpreter.cs	Implements the QuickJS-style bytecode interpreter with backtracking and timeout support.
Jint/Runtime/RegExp/RegExpCompiler.cs	Implements the QuickJS-style regex compiler (parsing → bytecode), including /v unicode-sets support.
Jint/Runtime/RegExp/JintRegExpEngine.cs	Wraps compilation/execution of the custom engine and builds match results.
Jint/Runtime/RegExp/DotNetRegExpEngine.cs	Wraps .NET Regex + Acornima parse metadata for the fast path.
Jint/Runtime/Interpreter/Expressions/JintLiteralExpression.cs	Updates RegExp literal evaluation to cache runtime-compiled .NET/custom engines.
Jint/ParsingOptions.cs	Stops overriding RegExp parse mode to AdaptToInterpreted; relies on base Validate mode.
Jint/Native/RegExp/RegExpPrototype.cs	Integrates custom-engine execution into exec and adjusts fast paths for engine type and v semantics.
Jint/Native/RegExp/RegExpConstructor.cs	Adds flag validation, custom-engine routing/fallback, and constructors for cached custom-engine instances.
Jint/Native/JsRegExp.cs	Adds CustomEngine, separates unicode vs FullUnicode semantics, and exposes UsesDotNetEngine.
Jint/Native/Iterator/IteratorInstance.cs	Uses AdvanceStringIndex for Unicode-aware advancement during regexp iteration.
Jint/Jint.csproj	Switches Acornima dependency from package to external project reference (fork).
Jint/Engine.Defaults.cs	Bumps default ECMAScript version and sets RegExpParseMode to Validate by default.
Jint/Engine.Ast.cs	Adds fallback to Validate mode when RegExp conversion fails during PrepareScript/PrepareModule.
Jint.Tests/Jint.Tests.csproj	Switches Acornima.Extras dependency to external project reference (fork).
Jint.Tests.Test262/Test262Harness.settings.json	Re-enables previously excluded regexp features/tests in test262 settings.
Jint.Tests.Test262/Jint.Tests.Test262.csproj	Switches Acornima.Extras dependency to external project reference (fork).
Jint.Tests.PublicInterface/Jint.Tests.PublicInterface.csproj	Switches Acornima.Extras dependency to external project reference (fork).
Jint.Tests.CommonScripts/Jint.Tests.CommonScripts.csproj	Switches Acornima.Extras dependency to external project reference (fork).
Directory.Packages.props	Comments out Acornima/Acornima.Extras central package versions (no longer used).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[adams85]

@lahma FYI, I just released flag v validation and the Unicode 17 upgrade.

Hopefully, this will unblock this PR. But let me know if you need any further changes on the parser side.

Some remarks:

As planned, the regexp conversion-related APIs have been deprecated, so you will get a lot of deprecation warnings. These can just be ignored for now.

However, if you want to get ahead of the wave, you can switch to using the new OnRegExp option, which allows you to run custom logic when a regexp is encountered instead of the built-in one. Built-in validation logic is still available via the context.Validate the method and the conversion logic via Tokenizer.AdaptRegExp. You can check out this test to get an idea of how this will work in Acornima v2, after the planned refactor (of course, AdaptRegExp will reside in another package in the future).

[lahma]

@adams85 congratulations on the release, great leap forward. Big thanks for cleaning up and polishing my AI slop 😅 I'll integrate the new version and then check your feedback, much appreciated!