perf(linter/plugins): transfer tokens via raw transfer by overlookmotel · Pull Request #19893 · oxc-project/oxc

overlookmotel · 2026-03-01T19:25:56Z

Transfer tokens via raw transfer in Oxlint JS plugins.

Deserializer is written by hand. Token is not a normal struct, so deserializer cannot be generated.

overlookmotel · 2026-03-01T19:26:37Z

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

0-merge - adds this PR to the back of the merge queue
hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

codspeed-hq · 2026-03-01T19:32:29Z

Merging this PR will not alter performance

✅ 4 untouched benchmarks
⏩ 52 skipped benchmarks¹

_{Comparing om/02-23-perf_linter_plugins_transfer_tokens_via_raw_transfer (eb9c0ad) with main (c2a42f6)}

52 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Copilot

Pull request overview

This PR updates Oxlint’s JS plugin raw-transfer pipeline to send lexer Tokens directly (via offsets into the raw-transfer buffer) instead of serializing ESTree tokens to JSON, aiming to reduce overhead and improve performance.

Changes:

Switch Rust-side token preparation from to_estree_tokens_json to in-place update_tokens, and write raw token pointer/length metadata into the transfer buffer.
Implement a JS-side deserializer that reconstructs ESTree-compatible token objects from the raw Rust Token representation.
Update raw-transfer metadata documentation/comments to reflect the new token transfer format.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tasks/ast_tools/src/generators/raw_transfer.rs	Updates comments describing token offset/length metadata semantics.
napi/parser/src/raw_transfer_types.rs	Updates metadata docs to reflect raw lexer-token offsets/counts.
crates/oxc_linter/src/lib.rs	Switches linter plugin path from token JSON generation to in-place token updates + raw pointer/len metadata.
crates/oxc_linter/src/context/host.rs	Exposes mutable access to stored parser tokens and makes current sub-host mutable accessor usable in crate.
apps/oxlint/src/js_plugins/parse.rs	Switches N-API parser raw-transfer path from token JSON to raw token pointer/len metadata.
apps/oxlint/src-js/utils/globals.ts	Adds `String.fromCodePoint` export for TSDown/global usage.
apps/oxlint/src-js/plugins/tokens.ts	Replaces JSON token parsing with raw token deserialization and adds identifier unescaping logic.
apps/oxlint/src-js/plugins/source_code.ts	Stops exporting `textDecoder` and exports `sourceText` for token deserialization.

crates/oxc_linter/src/lib.rs

apps/oxlint/src-js/plugins/tokens.ts

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

napi/parser/src/raw_transfer_types.rs

apps/oxlint/src-js/plugins/tokens.ts

graphite-app · 2026-03-01T21:46:42Z

Merge activity

Mar 1, 9:46 PM UTC: overlookmotel added this pull request to the Graphite merge queue.
Mar 1, 9:54 PM UTC: Merged by the Graphite merge queue.

Transfer tokens via raw transfer in Oxlint JS plugins. Deserializer is written by hand. `Token` is not a normal struct, so deserializer cannot be generated.

### 🚀 Features - 733d6dc parser: Report error on `infer` outside conditional type (#19879) (camc314) - c2a42f6 allocator: Add `Vec::into_bump_slice_mut` (#19895) (overlookmotel) - ee4982b parser: Add `VARIANTS` const to `Kind` via `fieldless_enum!` macro (#19877) (overlookmotel) - b3dceae data_structures: Add `fieldless_enum!` macro (#19876) (overlookmotel) - 12b841e parser: Make all `Kind::is_*` methods `const` (#19874) (overlookmotel) - 25c2e25 estree/tokens: Add function to update tokens in place (#19856) (overlookmotel) - f78e6df parser: Add `mutate_tokens` Cargo feature (#19853) (overlookmotel) - 5036bb6 parser: Report error on `for await` in static blocks (#19844) (camc314) - 42bd431 parser: Report error for missing initializer in using decl (#19824) (camc314) - a2f58e5 parser: Report error for `implements` clause in non-ts files (#19820) (Cameron) - b25228a estree: Add `IS_COMPACT` const to `Formatter` trait (#19787) (overlookmotel) - e2a1b79 estree: Expose buffer and formatter of serializers (#19773) (overlookmotel) - 4699498 data_structures: Add `CodeBuffer::print_strs_array` (#19760) (overlookmotel) - 233f947 estree: `oxc_estree` crate export config and formatter types (#19724) (overlookmotel) - 5937a32 semantic: Introduce `symbol_declarations` method (#19609) (camc314) - ea6b796 parser: Add `LexerConfig::TOKENS_METHOD_IS_STATIC` const (#19683) (overlookmotel) - 655c38f semantic: Add "did you mean?" suggestions to undefined name errors (#19102) (copilot-swe-agent) - 9e11dc6 parser,estree,coverage: Collect tokens in parser and convert to ESTree format (#19497) (camc314) - c4a3677 parser: Report error for initializer in ambient context (#19187) (camc314) ### 🐛 Bug Fixes - abc7e19 codegen: Improve parenthesised checks when printing types (#19880) (camc314) - 017de5d parser: Update error code for type annotation in `for...in` statement (#19882) (camc314) - 7682e5a linter/plugins: Decode escapes in identifier tokens (#19838) (overlookmotel) - 06767ed estree/tokens: Convert `this` tokens in `TSTypeName` (#19815) (overlookmotel) - ef798af parser: Use TS8037 for satisfies expression in JS files diagnostic (#19819) (camc314) - 98ea5c5 parser: Use TS8016 for type assertions in JS files diagnostic (#19818) (camc314) - 1710f56 codegen: Remove double indentation for enum inside namespace (#19775) (Dunqing) - 9e4995c codegen: Print type annotation on `CatchParameter` (#19790) (camc314) - 297b2bb codegen: Wrap `TSConditionalType` in parens when necessary (#19788) (camc314) - cec7878 codegen: Print `definite` property on AccessorProperty (#19786) (camc314) - 6f395cf codegen: Print `definite` property on PropertyDefinition (#19785) (camc314) - b749373 codegen: Correctly parenthesise TSArrayType (#19784) (camc314) - 876dc1b codegen: Print object property `this` param (#19783) (camc314) - 93bb861 formatter: Trim trailing whitespace before breaking line (#19740) (leaysgur) - ed17bbf codegen: Print `override` keyword for method and property definitions (#19753) (Dunqing) - 6a59a76 parser: Improve error recovery for private identifiers in property names (#19710) (Boshen) - 3b96f41 codegen: Print comments in JSX expression containers and spread attributes (#19701) (Boshen) - f5694ce estree/tokens: Reverse field order of `regex` object in tokens (#19679) (overlookmotel) - b2b7a55 estree/tokens: Generate tokens for files with BOM (#19535) (overlookmotel) - 50a7514 estree: Fix tokens for JSX (#19524) (overlookmotel) - a35063e minifier: Preserve side effects for meta property url reads (#19668) (Boshen) - 8ad3430 semantic/jsdoc: Handle even-numbered backtick sequences in JSDoc parsing (#19664) (Boshen) ### ⚡ Performance - 05ccf9f linter/plugins: Transfer tokens via raw transfer (#19893) (overlookmotel) - c1bfdcf estree/tokens: Preallocate sufficient space for tokens JSON (#19851) (overlookmotel) - 4b0611a estree/tokens: Introduce `ESTreeTokenConfig` trait (#19842) (overlookmotel) - 81bab90 estree/tokens: Do not JSON-encode keyword, punctuator, etc tokens (#19814) (overlookmotel) - 6260ddd estree/tokens: Do not JSON-encode `this` identifiers (#19813) (overlookmotel) - b378f4a estree/tokens: Do not JSON-encode JSX identifiers (#19812) (overlookmotel) - 5016d92 estree/tokens: Handle regex tokens separately (#19796) (overlookmotel) - 780a68e estree/tokens: Use strings from AST for identifier tokens (#19744) (overlookmotel) - dc9c2e3 estree: Use `CodeBuffer::print_strs_array` to reduce bounds checks (#19766) (overlookmotel) - 845da35 estree: Use `CodeBuffer::print_indent` (#19727) (overlookmotel) - ec88f6a estree/tokens: Serialize tokens while visiting AST (#19726) (overlookmotel) - bc6507f estree/tokens: Serialize with `ESTree` not `serde` (#19725) (overlookmotel) - ec24859 estree/tokens: Do not branch on presence of override twice (#19721) (overlookmotel) - dac14be estree/tokens: Replace hash map with `Vec` (#19718) (overlookmotel) - b9d2443 estree/tokens: Replace multiple hash sets into a single hash map (#19716) (overlookmotel) - 7233548 parser: Remove branches from `finish_next_inner` (#19695) (overlookmotel) - b5d9845 parser: Remove const generic param from `finish_next_inner` (#19684) (overlookmotel) - 8940f66 estree/tokens: Serialize tokens to compact JSON (#19572) (overlookmotel) - 136e39b parser/tokens: Pre-allocate capacity for tokens (#19543) (overlookmotel) - 6a6513c linter/plugins: Use Oxc tokens in plugins (#19498) (camc314) - b3b2d30 parser: Introduce `ParserConfig` (#19637) (overlookmotel) ### 📚 Documentation - b2b7a64 estree/tokens: Correct comment (#19873) (overlookmotel) - 0399311 estree/tokens: Improve comments (#19836) (overlookmotel) - 1b392de minifier: Add `Function.prototype.toString` assumption (#19758) (sapphi-red) - 75c9cd8 parser: Improve doc comments for `ParserConfig` and `LexerConfig` (#19682) (overlookmotel) - 2fa936f README.md: Map npm package links to npmx.dev (#19666) (Boshen) Co-authored-by: Boshen <1430279+Boshen@users.noreply.github.com>

# Oxlint ### 🚀 Features - 2e0e1d0 linter/no-unused-vars: Add experimental fix mode controls (off|suggestion|fix) (#19774) (camc314) - f34f6fa linter: Introduce typeCheck config option (#19764) (camc314) - 694be7d linter: Introduce typeAware as config options (#19614) (camc314) - 655c38f semantic: Add "did you mean?" suggestions to undefined name errors (#19102) (copilot-swe-agent) - e97a57e linter/id-length: Use serde to deserialize rule options (#19636) (camc314) - c4a3677 parser: Report error for initializer in ambient context (#19187) (camc314) - 346045a linter/id-length: Add `checkGeneric` option (#19634) (camc314) ### 🐛 Bug Fixes - 1b7a937 linter: Correct double-comparisons fix with swapped operands (#19846) (camc314) - c308857 linter/consistent_type_imports: Add missing help and notes to diagnostics (#19827) (Daniel Osmond) - 7682e5a linter/plugins: Decode escapes in identifier tokens (#19838) (overlookmotel) - f368fcd linter/consistent_type_assertions: Add missing with_help and with_note to diagnostics (#19826) (Daniel Osmond) - 04e6223 npm: Add `preferUnplugged` for Yarn PnP compatibility (#19829) (Boshen) - 86d5037 linter: Add help text to no-extend-native, no-useless-backreference (#19733) (Anthony Amaro) - 50e8eff linter: Add .with_help() to operator-assignment, no-nonoctal-decimal-escape (#19732) (Anthony Amaro) - 1417bdc linter/no-wrapper-object-types: Add help messages to missing diagnostics (#19771) (Daniel Osmond) - 0838477 linter/ban_ts_comment: Add help and notes to missing diagnostics (#19781) (Daniel Osmond) - e8c77cf linter/adjacent_overload_signatures: Add missing diagnostics (#19780) (Daniel Osmond) - 28834ac linter/ban_types: Add missing help and note to diagnostics (#19782) (Daniel Osmond) - fd938d3 linter/prefer-enum-initializers: Add help messages to missing diagnostics (#19772) (Daniel Osmond) - eb928ee linter/no-dynamic-delete: Add help messages to missing diagnostics (#19768) (Daniel Osmond) - a985666 linter/no-empty-interface: Add help messages to missing diagnostics (#19769) (Daniel Osmond) - 2dc0ceb linter/no-extra-non-null-assertion: Add help messages to missing diagnostics (#19770) (Daniel Osmond) - 95d5d66 linter/no-dupe-keys: Handle `__proto__` proto setters in (#19762) (camc314) - 24ff0db linter/exhaustive-deps: False positive for member expressions in IIFEs (#19751) (Dennis Chen) - 7243a58 linter/no-use-before-define: Honor `ignoreTypeReferences` when value and type name collisions (#19747) (Dimava) - eefd818 linter/explicit-module-boundary-types: Add help messages to missing diagnostics (#19736) (Daniel Osmond) - 0440e9a linter: Add help text to no_control_regex, no_fallthrough, no_param_reassign (#19655) (Anthony Amaro) - e84cb2f react/display-name: Handle merged type+value context symbols (#19608) (camc314) - ce7e253 linter/prefer-object-from-entries: Require exact path match in unicorn helper (#19687) (camc314) - f5694ce estree/tokens: Reverse field order of `regex` object in tokens (#19679) (overlookmotel) - b2b7a55 estree/tokens: Generate tokens for files with BOM (#19535) (overlookmotel) - 0722721 linter/jsx-curly-brace-presence: False positive with prop & mixed quotes (#19674) (camc314) - 3496acd linter: Enhance diagnostic help messages for eslint rules (#19653) (Anthony Amaro) - e384e94 linter: Enhance help diagnostic messages for more eslint rules (#19658) (Anthony Amaro) - a4d5b34 linter: Avoid non-promise catch false positives (#19574) (camc314) - 5706f38 linter: `unicorn/no-array-callback-reference` skip `Effect.*` array-like methods name. (#19633) (Said Atrahouch) ### ⚡ Performance - 05ccf9f linter/plugins: Transfer tokens via raw transfer (#19893) (overlookmotel) - 4b0611a estree/tokens: Introduce `ESTreeTokenConfig` trait (#19842) (overlookmotel) - ec88f6a estree/tokens: Serialize tokens while visiting AST (#19726) (overlookmotel) - d4dcf26 linter/plugins: Remove `typescript` from bundle (#19531) (overlookmotel) - 6a6513c linter/plugins: Use Oxc tokens in plugins (#19498) (camc314) ### 📚 Documentation - d86f59e linter: Improve docs for no-useless-concat, mark as pending fixer. (#19859) (connorshea) - caa091d linter/plugins: Correct doc comments for `initTokens` (#19530) (overlookmotel) - 2fa936f README.md: Map npm package links to npmx.dev (#19666) (Boshen) - dc0ff73 linter/no-useless-constructor: Warn for parameter properties as well (#19638) (Ole Asteo) # Oxfmt ### 🚀 Features - 5141bc2 formatter: Support trailing ignore comments (#19304) (Andreas Lubbe) - 4888a99 oxfmt/lsp: Support other schemes beside `file://` and `untitled://` (#19872) (Sysix) - 14a0181 oxfmt: Support `graphql()` variant for gql-in-js (#19703) (leaysgur) - ca68ea6 oxfmt: Support gql-in-js substitution (#19670) (leaysgur) - 035933c formatter,oxfmt: Support js-in-vue (partially) (#19514) (leaysgur) - 9e11dc6 parser,estree,coverage: Collect tokens in parser and convert to ESTree format (#19497) (camc314) ### 🐛 Bug Fixes - 8e3842d oxfmt: Avoid embedded TSFN crash by returning errors as data (take2) (#19806) (Yuji Sugiura) - 04e6223 npm: Add `preferUnplugged` for Yarn PnP compatibility (#19829) (Boshen) - e540585 oxfmt: Support tailwind sort for CSS/LESS/SCSS (#19803) (leaysgur) - 93bb861 formatter: Trim trailing whitespace before breaking line (#19740) (leaysgur) - b85f97b formatter: Drop blank line between terminal call and first chain member (#19659) (Dunqing) ### ⚡ Performance - b3b2d30 parser: Introduce `ParserConfig` (#19637) (overlookmotel) ### 📚 Documentation - 2fa936f README.md: Map npm package links to npmx.dev (#19666) (Boshen) Co-authored-by: Boshen <1430279+Boshen@users.noreply.github.com>

overlookmotel mentioned this pull request Mar 1, 2026

test(estree/tokens): add tests for tokens via raw transfer #19885

Merged

github-actions bot added A-linter Area - Linter A-parser Area - Parser A-cli Area - CLI A-ast-tools Area - AST tools A-linter-plugins Area - Linter JS plugins C-performance Category - Solution not expected to change functional behavior, only performance labels Mar 1, 2026

overlookmotel mentioned this pull request Mar 1, 2026

refactor(estree/tokens): move raw transfer-related files into subdirectory #19892

Merged

overlookmotel marked this pull request as ready for review March 1, 2026 19:28

overlookmotel requested a review from camc314 as a code owner March 1, 2026 19:28

Copilot AI review requested due to automatic review settings March 1, 2026 19:28

overlookmotel self-assigned this Mar 1, 2026

Copilot started reviewing on behalf of overlookmotel March 1, 2026 19:28 View session

Copilot AI reviewed Mar 1, 2026

View reviewed changes

crates/oxc_linter/src/lib.rs Outdated Show resolved Hide resolved

apps/oxlint/src-js/plugins/tokens.ts Show resolved Hide resolved

overlookmotel force-pushed the om/02-23-perf_linter_plugins_transfer_tokens_via_raw_transfer branch from 9d65223 to 886d5cb Compare March 1, 2026 19:39

overlookmotel force-pushed the om/03-01-refactor_estree_tokens_move_raw_transfer-related_files_into_subdirectory branch from c17ff02 to b0e8d04 Compare March 1, 2026 19:39

overlookmotel mentioned this pull request Mar 1, 2026

refactor(linter/plugins): add regex field to all tokens #19894

Merged

graphite-app bot changed the base branch from om/03-01-refactor_estree_tokens_move_raw_transfer-related_files_into_subdirectory to graphite-base/19893 March 1, 2026 20:01

graphite-app bot force-pushed the graphite-base/19893 branch from b0e8d04 to 68152f3 Compare March 1, 2026 20:10

graphite-app bot force-pushed the om/02-23-perf_linter_plugins_transfer_tokens_via_raw_transfer branch from 886d5cb to 3f5a898 Compare March 1, 2026 20:10

graphite-app bot changed the base branch from graphite-base/19893 to main March 1, 2026 20:11

graphite-app bot force-pushed the om/02-23-perf_linter_plugins_transfer_tokens_via_raw_transfer branch from 3f5a898 to 9518469 Compare March 1, 2026 20:11

overlookmotel force-pushed the om/02-23-perf_linter_plugins_transfer_tokens_via_raw_transfer branch from 9518469 to f7633c1 Compare March 1, 2026 21:20

overlookmotel requested a review from Copilot March 1, 2026 21:26

Copilot started reviewing on behalf of overlookmotel March 1, 2026 21:26 View session

Copilot AI reviewed Mar 1, 2026

View reviewed changes

napi/parser/src/raw_transfer_types.rs Outdated Show resolved Hide resolved

apps/oxlint/src-js/plugins/tokens.ts Show resolved Hide resolved

overlookmotel force-pushed the om/02-23-perf_linter_plugins_transfer_tokens_via_raw_transfer branch from f7633c1 to eb9c0ad Compare March 1, 2026 21:38

graphite-app bot added the 0-merge Merge with Graphite Merge Queue label Mar 1, 2026

perf(linter/plugins): transfer tokens via raw transfer (#19893)

05ccf9f

Transfer tokens via raw transfer in Oxlint JS plugins. Deserializer is written by hand. `Token` is not a normal struct, so deserializer cannot be generated.

graphite-app bot force-pushed the om/02-23-perf_linter_plugins_transfer_tokens_via_raw_transfer branch from eb9c0ad to 05ccf9f Compare March 1, 2026 21:47

graphite-app bot merged commit 05ccf9f into main Mar 1, 2026
21 checks passed

graphite-app bot deleted the om/02-23-perf_linter_plugins_transfer_tokens_via_raw_transfer branch March 1, 2026 21:54

graphite-app bot removed the 0-merge Merge with Graphite Merge Queue label Mar 1, 2026

This was referenced Mar 2, 2026

Linter plugins: Rustify token methods #16207

Closed

☂️ Oxlint JS Plugins Milestone 2 #14826

Closed

This was referenced Mar 2, 2026

release(crates): oxc v0.116.0 #19911

Merged

release(apps): oxlint v1.51.0 && oxfmt v0.36.0 #19912

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf(linter/plugins): transfer tokens via raw transfer#19893

perf(linter/plugins): transfer tokens via raw transfer#19893
graphite-app[bot] merged 1 commit intomainfrom
om/02-23-perf_linter_plugins_transfer_tokens_via_raw_transfer

overlookmotel commented Mar 1, 2026 •

edited

Loading

Uh oh!

overlookmotel commented Mar 1, 2026 •

edited

Loading

Uh oh!

codspeed-hq bot commented Mar 1, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

graphite-app bot commented Mar 1, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

overlookmotel commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

overlookmotel commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to use the Graphite Merge Queue

Uh oh!

codspeed-hq bot commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

graphite-app bot commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

overlookmotel commented Mar 1, 2026 •

edited

Loading

overlookmotel commented Mar 1, 2026 •

edited

Loading

codspeed-hq bot commented Mar 1, 2026 •

edited

Loading

graphite-app bot commented Mar 1, 2026 •

edited

Loading