feat(linter/plugins): serialize rust-parsed tokens

[lilnasy]

No description provided.

[lilnasy]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• refactor(linter/plugins): remove dependency on TypeScript #17026
• feat(linter/plugins): serialize rust-parsed tokens #17025 👈 (View in Graphite)
• perf(oxc_parser): preallocate tokens vector #17095
• feat(parser): add token collection option #17024
• test(benchmarks): add parser_tokens benchmark #17047
• main

---

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• 0-merge - adds this PR to the back of the merge queue
• hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codspeed-hq]

CodSpeed Performance Report

Merging #17025 will degrade performances by 25.15%

_{Comparing 12-17-serialization (7ff857c) with 12-17-feat_oxc_parser_store_tokens_in_lexer_ (e7c19d8)}

Summary

❌ 4 regressions
✅ 38 untouched
⏩ 3 skipped¹

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Mode	Benchmark	BASE	HEAD	Change
❌	Simulation	parser[cal.com.tsx]	32.7 ms	43.7 ms	-25.15%
❌	Simulation	parser[RadixUIAdoptionSection.jsx]	96.9 µs	121.4 µs	-20.16%
❌	Simulation	parser[react.development.js]	1.8 ms	2.3 ms	-22.42%
❌	Simulation	parser[binder.ts]	4.3 ms	5.5 ms	-22.64%

Footnotes

1. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[overlookmotel]

I imagine perf regression is coming from the loop which calls to_tseslint_type. Hopefully my suggestions below solve that.

[overlookmotel]

Can we avoid adding this field to Program? Tokens can be returned in ParserReturn instead.

[overlookmotel]

We shouldn't need this. We want JS side code to read the original Vec<Token> directly, without converting it.

Instead:

• Add #[ast] attribute to the existing Token struct (and remove #[repr(transparent)] attr - #[ast] macro adds it automatically).
• Add #[ast] #[generate_derive(ESTree)] #[estree(rename = "TokenKind")] attributes to the existing Kind (token kind) enum.
• Implement ESTree trait on Token.
• Add crates/oxc_parser/src/lexer/token.rs and crates/oxc_parser/src/lexer/kind.rs to the list of files that codegen processes at in ast_tools.
• Add another type #[ast] struct Tokens<'a>(Vec<'a, Tokens>); in same file. We don't need to use that type, but adding it should make ast_tools generate a deserializeVecToken function.
• Add a TOKEN flag to FLAG_NAMES in raw_transfer.rs generator.
• Export deserializeVecToken from deserializer when TOKEN flag is enabled.

ESTree impl on Token will need to be manually defined (including the #[estree(raw_deser)] attr). Shout if you have any difficulty with this. Writing raw_deser implementations is pretty dreadful - my fault, it's terribly designed - and you'll need to write the byte offsets manually (since Token doesn't have real fields).

But you should be able to use deserializeTokenKind which codegen will generate, and also use the generated ESTree impl for Kind.

The conversion from Kind (token kind) to ESTree token type is the tricky part. Should be able to get codegen to do most of the work by adding #[estree(rename = "Keyword")] (etc) to all the variants of Kind.

No doubt, you didn't need me to explain every one of the steps above. I've just included them for clarity, to (hopefully) save you a little time. Please excuse me if I'm "teaching grandpa to suck eggs".

I've laid a bit of groundwork for this in #17050 and #17052.

[overlookmotel]

Oh actually maybe we can get codegen to generate everything for us by:

• Don't put #[ast] attr on Token.
• Instead, add this:

/// Dummy type to communicate the content of `Token` to `oxc_ast_tools`.
#[ast(foreign = Token)]
#[generate_derive(ESTree)]
#[expect(dead_code)]
struct TokenAlias {
  pub span: Span,
  pub kind: Kind,
  #[estree(skip)]
  pub _align: U128Align,
}

/// Zero-sized type which has alignment of `u128`
#[repr(transparent)]
struct U128Align([u128; 0]);

This tells the codegen "treat Token as if it's defined like this".

You'd need to add a "special case" to codegen for U128Align (search ast_tools for PointerAlign).

EDIT: No this is a bad idea. Need to implement ESTree and raw_deser manually to handle the special case of extra fields on regexp tokens.

[overlookmotel]

This is probably where the perf regression is coming from. Should be able to remove it once my other comments are actioned.