test(estree/tokens): add tests for tokens via raw transfer

[overlookmotel]

Implement the beginnings of support for tokens in napi/parser.

This PR only adds support for tokens via raw transfer, and behind an undocumented experimentalTokens option.

Add tests checking that tokens received on JS side via raw transfer match snapshots for all Test262, AcornJSX, and TypeScript test cases. They do!

The tests are the main purpose of this PR, making sure it works before we switch over to tokens via raw transfer in Oxlint.

We can add full tokens support to napi/parser (including via JSON transfer) later on.

[overlookmotel]

• refactor(linter/plugins): add regex field to all tokens #19894
• perf(linter/plugins): transfer tokens via raw transfer #19893
• refactor(estree/tokens): move raw transfer-related files into subdirectory #19892
• test(estree/tokens): add tests for tokens via raw transfer #19885 👈 (View in Graphite)
• test(napi/parser): fix tests for range, parent, and lazy on TS fixtures #19884
• refactor(estree/tokens): convert token kinds to ESTreeKind #19875
• feat(parser): add VARIANTS const to Kind via fieldless_enum! macro #19877
• feat(data_structures): add fieldless_enum! macro #19876
• feat(parser): make all Kind::is_* methods const #19874
• docs(estree/tokens): correct comment #19873
• refactor(estree/tokens): move JSON-related code into subdirectory #19869
• refactor(estree/tokens): shorten code #19868
• refactor(estree/tokens): add Context methods from different AST nodes #19867
• refactor(estree/tokens): rename UpdateContext to RawContext #19866
• refactor(estree/tokens): break up into multiple files #19865
• feat(estree/tokens): add function to update tokens in place #19856 : 1 other dependent PR (#19858 )
• refactor(estree/tokens): reduce repeated code #19857
• ci(benchmarks): add benchmark for updating tokens for raw transfer #19854
• feat(parser): add mutate_tokens Cargo feature #19853
• refactor(parser): Token::set_* method visible only within lexer #19852
• perf(estree/tokens): preallocate sufficient space for tokens JSON #19851
• refactor(estree/tokens): break up into multiple files #19850
• perf(estree/tokens): introduce ESTreeTokenConfig trait #19842
• refactor(estree/tokens): move logic for JSX identifiers into JSXState struct #19839
• fix(linter/plugins): decode escapes in identifier tokens #19838
• refactor(estree/tokens): re-order code #19837
• docs(estree/tokens): improve comments #19836
• refactor(estree/tokens): make ESTreeUnsafeToken type private #19835
• refactor(estree/tokens): rename types #19834
• fix(estree/tokens): convert this tokens in TSTypeName #19815
• perf(estree/tokens): do not JSON-encode keyword, punctuator, etc tokens #19814
• perf(estree/tokens): do not JSON-encode this identifiers #19813
• perf(estree/tokens): do not JSON-encode JSX identifiers #19812
• perf(estree/tokens): handle regex tokens separately #19796
• perf(estree/tokens): use strings from AST for identifier tokens #19744 : 1 other dependent PR (#19735 )
• refactor(estree/tokens): add TokenType type #19789
• feat(estree): add IS_COMPACT const to Formatter trait #19787
• feat(estree): expose buffer and formatter of serializers #19773
• perf(estree): use CodeBuffer::print_strs_array to reduce bounds checks #19766
• feat(data_structures): add CodeBuffer::print_strs_array #19760
• perf(estree): use CodeBuffer::print_indent #19727
• perf(estree/tokens): serialize tokens while visiting AST #19726
• perf(estree/tokens): serialize with ESTree not serde #19725
• feat(estree): oxc_estree crate export config and formatter types #19724
• refactor(estree/tokens): convert tokens spans before AST spans #19723
• refactor(estree/tokens): simplify code #19722
• perf(estree/tokens): do not branch on presence of override twice #19721
• refactor(estree/tokens): move logic for excluding legacy identifiers into visitor #19720
• perf(estree/tokens): replace hash map with Vec #19718
• perf(estree/tokens): replace multiple hash sets into a single hash map #19716
• refactor(estree/tokens): replace stored token when re-lexing #19698
• perf(parser): remove branches from finish_next_inner #19695
• perf(parser): remove const generic param from finish_next_inner #19684
• feat(parser): add LexerConfig::TOKENS_METHOD_IS_STATIC const #19683
• docs(parser): improve doc comments for ParserConfig and LexerConfig #19682
• refactor(estree/tokens): use wildcard import for AST types #19681
• test(estree): ESTree conformance check files with hashbangs #19680
• fix(estree/tokens): reverse field order of regex object in tokens #19679
• perf(estree/tokens): serialize tokens to compact JSON #19572
• ci(benchmarks): correct comment #19650
• perf(parser/tokens): pre-allocate capacity for tokens #19543
• fix(estree/tokens): generate tokens for files with BOM #19535
• refactor(estree/tokens): combine collect_token_context into to_estree_tokens_json #19534
• refactor(estree): reorder tokens code #19533
• refactor(coverage): standardize ESTree tokens testing functions #19532
• perf(linter/plugins): remove typescript from bundle #19531
• docs(linter/plugins): correct doc comments for initTokens #19530
• refactor(linter/plugins): combine functions for getting tokens #19529
• ci(benchmarks): benchmark parser with tokens and ESTree tokens conversion #19649
• perf(linter/plugins): use Oxc tokens in plugins #19498
• test(estree/tokens): do not skip tokens conformance tests for files with hashbangs #19643
• fix(estree): fix tokens for JSX #19524
• perf(parser): introduce ParserConfig #19637
• feat(parser,estree,coverage): collect tokens in parser and convert to ESTree format #19497
• feat(coverage): add estree token conformance scaffolding #19496
• fix(tasks/coverage): bring back acorn-jsx estree testing #19516 : 1 other dependent PR (#19515 )
• chore(submodules): update submodule SHAs to latest commits #19518
• main

---

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• 0-merge - adds this PR to the back of the merge queue
• hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codspeed-hq]

Merging this PR will not alter performance

✅ 53 untouched benchmarks
⏩ 3 skipped benchmarks¹

---

_{Comparing om/03-01-test_estree_tokens_add_tests_for_tokens_via_raw_transfer (be741da) with main (449080c)²}

Footnotes

1. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on main (8b89309) during the generation of this report, so 449080c was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[Copilot]

Pull request overview

Adds initial (experimental) token support to napi/parser via raw transfer, and introduces snapshot-based tests to validate that raw-transferred tokens match expected outputs across existing fixture suites.

Changes:

• Introduces an undocumented experimentalTokens option and wires it through the Rust parser configuration and raw-transfer pipeline.
• Adds raw-transfer token deserialization on the JS side and stores token offsets/lengths in raw-transfer metadata.
• Adds/extends Vitest + CI configuration and raw-transfer test runners to execute token snapshot tests for Test262, Acorn JSX, and TypeScript fixtures.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
napi/parser/vitest.config.ts	Enables raw-test execution when RUN_RAW_TOKENS_TESTS is set.
napi/parser/test/parser.ts	Extends test-only option typings with experimentalTokens.
napi/parser/test/parse-raw.test.ts	Adds token test suites gated by RUN_RAW_TOKENS_TESTS.
napi/parser/test/parse-raw-worker.ts	Adds worker-side token snapshot verification and token-fixture loading logic.
napi/parser/test/parse-raw-common.ts	Adds a new test-type bitflag and a tokens fixture directory path constant.
napi/parser/src/types.rs	Exposes experimentalTokens (Rust ParserOptions.tokens) to JS bindings.
napi/parser/src/raw_transfer_types.rs	Extends RawTransferMetadata::new to include token offset/length.
napi/parser/src/raw_transfer.rs	Populates token offset/length in metadata and converts tokens for raw transfer.
napi/parser/src/lib.rs	Enables runtime token collection through RuntimeParserConfig.
napi/parser/src-js/raw-transfer/eager.js	Adds raw-transfer token deserialization and tokens getter when enabled.
napi/parser/Cargo.toml	Adds dependency on oxc_estree_tokens.
Cargo.lock	Locks the new oxc_estree_tokens dependency.
.github/workflows/ci.yml	Enables RUN_RAW_TOKENS_TESTS in the NAPI CI job.

Comments suppressed due to low confidence (2)

napi/parser/src/types.rs:40

• The docs say experimentalTokens is “only available with experimental raw transfer”, but the option is now plumbed into parse_impl for all parsing paths (parseSync/parse too). That means callers can enable experimentalTokens without raw transfer and still pay the tokenization cost, while never receiving tokens in the returned JS object. Consider enforcing this constraint (ignore/throw unless experimentalRawTransfer/experimentalLazy is set) or ensuring tokens are only enabled in the raw-transfer code path.

    /// Controls whether parser should return tokens.
    ///
    /// This option is not stable yet, and only available with experimental raw transfer.
    ///
    /// @default false
    #[napi(skip_typescript, js_name = "experimentalTokens")]
    pub tokens: Option<bool>,

napi/parser/src/raw_transfer_types.rs:54

• RawTransferMetadata::new now accepts tokens_offset/tokens_len, which means these fields are no longer always 0 in the parser. The doc comments on the struct fields above still state they’re parser-invariant and only used in the linter; please update that documentation to match the new behavior (and clarify what these offsets/lengths point to in the parser case).

impl RawTransferMetadata {
    pub fn new(data_offset: u32, is_ts: bool, tokens_offset: u32, tokens_len: u32) -> Self {
        Self { data_offset, is_ts, is_jsx: false, has_bom: false, tokens_offset, tokens_len }
    }

[graphite-app]

Merge activity

• Mar 1, 8:00 PM UTC: overlookmotel added this pull request to the Graphite merge queue.
• Mar 1, 8:09 PM UTC: Merged by the Graphite merge queue.