perf(linter/plugins): optimize merging of tokens and comments#16071
Merged
overlookmotel merged 4 commits intooxc-project:mainfrom Nov 25, 2025
Merged
Conversation
github-actions
bot
added
A-linter
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes the merging of tokens and comments arrays by replacing a sort-based approach with a two-pointer merge algorithm. Since both input arrays are already sorted by their start positions, the O(n log n) sort operation is replaced with an O(n) linear merge, which is significantly more efficient for performance-critical linting operations.
Key Changes
- Implemented two-pointer merge algorithm to combine sorted
tokensandcommentsarrays - Added debug assertion to validate merge correctness in development builds
- Removed TODO comment about potential optimization (now implemented)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
overlookmotel
approved these changes
Nov 24, 2025
Member
overlookmotel
left a comment
overlookmotel
left a comment
There was a problem hiding this comment.
Since we're optimizing, could try and speed it up even more - suggestion below. But this PR is likely faster than what we had before, so we could merge it as is if you don't have the time/inclination to try to shave off a few more cycles.
lilnasy
force-pushed
the
perf/linter/plugins/initTokensWithComments
branch
2 times, most recently
from
bad004b to
7699cda
Compare
overlookmotel
force-pushed
the
perf/linter/plugins/initTokensWithComments
branch
from
7699cda to
d513973
Compare
This was referenced Nov 25, 2025
graphite-app bot
pushed a commit
that referenced
this pull request
Nov 25, 2025
Follow-on after #16071. Seeing that more performant implementation made me see another couple of optimizations. 1. Fast path for when file has no comments (probably not so uncommon). 2. Extend the "push in runs" approach. Ensure that in the main loop, there's always at least 1 token and 1 comment remaining, and you know which one it is, which reduces branches. 3. Avoid ever reading out of bounds of the arrays (expensive).
leaysgur
pushed a commit
that referenced
this pull request
Nov 26, 2025
Follow-on after #16071. Seeing that more performant implementation made me see another couple of optimizations. 1. Fast path for when file has no comments (probably not so uncommon). 2. Extend the "push in runs" approach. Ensure that in the main loop, there's always at least 1 token and 1 comment remaining, and you know which one it is, which reduces branches. 3. Avoid ever reading out of bounds of the arrays (expensive).
overlookmotel
pushed a commit
that referenced
this pull request
Dec 1, 2025
# Oxlint ### 🚀 Features - 5da1a63 linter/plugins: Introduce `RuleTester` (#16206) (overlookmotel) - 41129ab linter/plugins: Implement `languageOptions.parser` (#16292) (overlookmotel) - 4b24825 linter: Implement no-restricted-types rule (#16276) (camc314) - 7150209 linter/plugins: Implement `SourceCode#getNodeByRangeIndex` (#16256) (overlookmotel) - 3226864 linter/plugins: Implement options merging (#16217) (overlookmotel) - cbb108a linter/plugins: Support default options (#16170) (overlookmotel) - 04a3a66 linter/plugins: Implement `SourceCode#getTokenOrCommentAfter()` (#16045) (Arsh) - 68b63d9 linter/plugins: Implement `SourceCode#getTokenOrCommentBefore()` (#16044) (Arsh) - 04d9454 linter/plugins: Implement `SourceCode#getTokenByRangeStart()` (#16043) (Arsh) - 7b8d578 linter/plugins: Implement `SourceCode#getTokensBetween()` (#16034) (Arsh) - 79c242f linter/plugins: Implement `SourceCode#getLastTokensBetween()` (#16033) (Arsh) - 1772078 linter/plugins: Implement `SourceCode#getFirstTokenBetween()` (#16032) (Arsh) - 21bb86d linter/plugins: Implement `SourceCode#getFirstTokensBetween()` (#16019) (Arsh) - 78f74b1 linter/plugins: Implement `SourceCode#getLastTokenBetween()` (#16008) (Arsh) - df0b948 linter/plugins: Implement `SourceCode#getLastToken()` (#16003) (Arsh) ### 🐛 Bug Fixes - cf249f5 linter/plugins: Fix message interpolation (#16300) (overlookmotel) - 9149a26 linter/plugins, napi/parser: Deep freeze visitor keys (#16293) (overlookmotel) - 71271ee linter: Fix `typescript/no-empty-interface` config option casing. (#16282) (connorshea) - 653fa6c oxlint/oxfmt/lsp: Tell client the real tool name & version (#16212) (Sysix) - 385e13e linter: Support both prefer-inline and preferInline for import/no-duplicates (#16275) (copilot-swe-agent) - efc08d3 linter: Fix config docs for no-return-assign and unicode-bom rules. (#16260) (connorshea) - dc96d3f linter: Correct docs to use string option for import/first rule. (#16264) (connorshea) - 15ce491 linter: Correct the default config options for the switch-exhaustiveness-check rule. (#16268) (connorshea) - 13f985c linter: Fix casing for unicorn/explicit-length-check config option. (#16269) (connorshea) - 0df1901 linter/plugins: Reset state after error during AST visitation (#16246) (overlookmotel) - 42e9dcc linter: Fix docs for `consistent-type-specifier-style` and `consistent-type-definitions` rules. (#16253) (connorshea) - 505ceb1 linter: Fix the radix rule docs to correctly render as a string-only config option (#16248) (connorshea) - 135cee6 oxlint/lsp: Run diagnostics after delayed worker initialization (#16244) (Sysix) - 78aa294 linter/plugins: Deep freeze options (#16218) (overlookmotel) - 123bffe linter/plugins: Handle zero-token files in `SourceCode#getLastToken()` (#16184) (Arsh) - 55fcfba linter: Add considerDefaultExhaustiveForUnions option to switch-exhaustiveness-check (#16204) (camc314) - 9cc20a1 minifier: Avoid merging side effectful expressions to next assignment statement if the side effect may change the left hand side reference (#16165) (sapphi-red) - 75249e0 linter/plugins: Handle non-UTF8 file paths (#16157) (overlookmotel) - 86fa667 linter/plugins: Improve type def for `RuleMeta` `defaultOptions` property (#16159) (overlookmotel) - d2b7dcb tsgolint: Improve lsp diagnostic when linting fails (#16115) (camc314) - 8810bd5 linter: Prevent conflicting fixes between prefer_number_properties and prefer_numeric_literals (#16113) (camc314) - 91eb3f2 ast/estree: Convert `TSImportType` `argument` field to `Literal` (#16109) (overlookmotel) - 2da2fc1 linter: Fix prefer-string-raw fixer producing invalid JS for non-ASCII strings (#16102) (camc314) - f5cb601 linter/plugins: Perform length checks before continuing loops (#16025) (Arsh) - e1c21ca linter/jsx-key: Implement missing options (#15731) (camc314) ### ⚡ Performance - 02bdf90 linter/plugins, napi/parser: Reuse arrays in visitor keys (#16294) (overlookmotel) - 714a91a linter/no-restricted-types: Skip running if config is empty (#16278) (camc314) - d3a34f8 linter/plugins: Optimize `getTokens()` and other methods (#16188) (Arsh) - c05db06 linter/plugins: Speed up `initTokensWithComments` (#16117) (overlookmotel) - 4846886 linter/plugins: Optimize merging of `tokens` and `comments` (#16071) (Arsh) - e232d35 linter/plugins: Recycle objects in token methods (#16068) (overlookmotel) ### 📚 Documentation - e928732 linter/plugins: Fix JSDoc comment (#16295) (overlookmotel) - d4a1581 linter: Add a link to MDN for the no-proto rule. (#16213) (connorshea) - be36e36 linter/plugins: Fix JSDoc comment for `loadPluginImpl` (#16211) (overlookmotel) - 6d6e9c9 linter: Improve the docs for oxc/no-async-await rule. (#16181) (connorshea) - 0e1d38a linter/plugins: Clarify JSDoc comment for `getTokensBetween` (#16070) (overlookmotel) - 3ee22b2 linter/plugins: Fix JSDoc comments for tokens methods (#16063) (overlookmotel) - f257b5c linter/plugins: Clarify JSDoc comments for tokens methods (#16062) (overlookmotel) # Oxfmt ### 🚀 Features - 116e0d1 website: Auto generate oxfmt docs (#15985) (Boshen) - 862bdf7 oxfmt: Detect unsupported experimental options (take2) (#16088) (leaysgur) ### 🐛 Bug Fixes - 75ac90c formatter: Comments in call arguments should be printed as-is (#16327) (Dunqing) - fd77568 formatter: Don't wrap parenthesis for yield expression if there is no leading comment (#16326) (Dunqing) - 8ccfb06 formatter: Should indent class extends and interface heritage when it is a member expression without type arguments (#16323) (Dunqing) - 0faa978 oxfmt: JsFormatEmbeddedCb types (#16324) (Brooooooklyn) - 2b8f982 formatter: JSX text wrapping incorrect (#16318) (Dunqing) - f3ffebe formatter: Should indent variable declarator if there is a trailing comment (#16243) (Dunqing) - 31d3186 formatter: Incorrect handling of directives with comments (#16235) (Dunqing) - ac8fcaf formatter: Add parens for new: private field expr (#16312) (leaysgur) - 653fa6c oxlint/oxfmt/lsp: Tell client the real tool name & version (#16212) (Sysix) - 380a0af formatter: Incorrect printing of class binding trailing comments (#16234) (Dunqing) - 0ca8154 formatter: Incorrect printing of trailing comments of callee when the call arguments are empty (#16232) (Dunqing) - ac3a92e formatter: Print comment in ternary jsx (#16224) (leaysgur) - e3a7388 formatter: Fix parens for static member chain (#16229) (leaysgur) - 55334c3 formatter: Incorrect printing of dangling comments in the if statement (#16228) (Dunqing) - 9096a63 formatter: Correct printing of trailing comments after the semicolon for class properties (#16225) (Dunqing) - 75fd568 formatter: Inconsistent union type output between two runs (#16222) (Dunqing) - cd70484 formatter: Should not add a hard space before function body (#16221) (Dunqing) - 9097167 formatter: Incorrect printing of union types with comments (#16205) (Dunqing) - 79b78b3 formatter: Template literal element should not be indented (#16189) (Dunqing) - 48d6ed2 formatter: Nested assignment pattern should not expand outer object pattern (#16160) (Dunqing) - 8f4137d formatter: Output is incorrect when using comments inside JSX which is the right hand-side of `LogicalExpression` (#16156) (Dunqing) - 85c3a10 formatter/sort_imports: Handle internal prefixes correctly (#16128) (leaysgur) - 38b7bc4 oxfmt: Make no-napi build work (#16134) (leaysgur) - 889d2e7 formatter: Handle poor layout for grouped call arguments (#16093) (Dunqing) - 14b0a6a oxfmt: Fix JS-ish file detection (#16092) (leaysgur) - 9706a1a oxfmt: Ignore unsupported options (#16085) (leaysgur)
taearls
pushed a commit
to taearls/oxc
that referenced
this pull request
Dec 11, 2025
…c-project#16071) - Follow up to oxc-project#15861. - Implements the performance optimization pointed out in oxc-project#15952.
taearls
pushed a commit
to taearls/oxc
that referenced
this pull request
Dec 11, 2025
…ments` (oxc-project#16116) Follow-on after oxc-project#16071. Add more debug assertions to ensure `comments` and `tokens` are merged correctly. This code is removed in release build - just for tests.
taearls
pushed a commit
to taearls/oxc
that referenced
this pull request
Dec 11, 2025
…16117) Follow-on after oxc-project#16071. Seeing that more performant implementation made me see another couple of optimizations. 1. Fast path for when file has no comments (probably not so uncommon). 2. Extend the "push in runs" approach. Ensure that in the main loop, there's always at least 1 token and 1 comment remaining, and you know which one it is, which reduces branches. 3. Avoid ever reading out of bounds of the arrays (expensive).
taearls
pushed a commit
to taearls/oxc
that referenced
this pull request
Dec 11, 2025
# Oxlint ### 🚀 Features - 5da1a63 linter/plugins: Introduce `RuleTester` (oxc-project#16206) (overlookmotel) - 41129ab linter/plugins: Implement `languageOptions.parser` (oxc-project#16292) (overlookmotel) - 4b24825 linter: Implement no-restricted-types rule (oxc-project#16276) (camc314) - 7150209 linter/plugins: Implement `SourceCode#getNodeByRangeIndex` (oxc-project#16256) (overlookmotel) - 3226864 linter/plugins: Implement options merging (oxc-project#16217) (overlookmotel) - cbb108a linter/plugins: Support default options (oxc-project#16170) (overlookmotel) - 04a3a66 linter/plugins: Implement `SourceCode#getTokenOrCommentAfter()` (oxc-project#16045) (Arsh) - 68b63d9 linter/plugins: Implement `SourceCode#getTokenOrCommentBefore()` (oxc-project#16044) (Arsh) - 04d9454 linter/plugins: Implement `SourceCode#getTokenByRangeStart()` (oxc-project#16043) (Arsh) - 7b8d578 linter/plugins: Implement `SourceCode#getTokensBetween()` (oxc-project#16034) (Arsh) - 79c242f linter/plugins: Implement `SourceCode#getLastTokensBetween()` (oxc-project#16033) (Arsh) - 1772078 linter/plugins: Implement `SourceCode#getFirstTokenBetween()` (oxc-project#16032) (Arsh) - 21bb86d linter/plugins: Implement `SourceCode#getFirstTokensBetween()` (oxc-project#16019) (Arsh) - 78f74b1 linter/plugins: Implement `SourceCode#getLastTokenBetween()` (oxc-project#16008) (Arsh) - df0b948 linter/plugins: Implement `SourceCode#getLastToken()` (oxc-project#16003) (Arsh) ### 🐛 Bug Fixes - cf249f5 linter/plugins: Fix message interpolation (oxc-project#16300) (overlookmotel) - 9149a26 linter/plugins, napi/parser: Deep freeze visitor keys (oxc-project#16293) (overlookmotel) - 71271ee linter: Fix `typescript/no-empty-interface` config option casing. (oxc-project#16282) (connorshea) - 653fa6c oxlint/oxfmt/lsp: Tell client the real tool name & version (oxc-project#16212) (Sysix) - 385e13e linter: Support both prefer-inline and preferInline for import/no-duplicates (oxc-project#16275) (copilot-swe-agent) - efc08d3 linter: Fix config docs for no-return-assign and unicode-bom rules. (oxc-project#16260) (connorshea) - dc96d3f linter: Correct docs to use string option for import/first rule. (oxc-project#16264) (connorshea) - 15ce491 linter: Correct the default config options for the switch-exhaustiveness-check rule. (oxc-project#16268) (connorshea) - 13f985c linter: Fix casing for unicorn/explicit-length-check config option. (oxc-project#16269) (connorshea) - 0df1901 linter/plugins: Reset state after error during AST visitation (oxc-project#16246) (overlookmotel) - 42e9dcc linter: Fix docs for `consistent-type-specifier-style` and `consistent-type-definitions` rules. (oxc-project#16253) (connorshea) - 505ceb1 linter: Fix the radix rule docs to correctly render as a string-only config option (oxc-project#16248) (connorshea) - 135cee6 oxlint/lsp: Run diagnostics after delayed worker initialization (oxc-project#16244) (Sysix) - 78aa294 linter/plugins: Deep freeze options (oxc-project#16218) (overlookmotel) - 123bffe linter/plugins: Handle zero-token files in `SourceCode#getLastToken()` (oxc-project#16184) (Arsh) - 55fcfba linter: Add considerDefaultExhaustiveForUnions option to switch-exhaustiveness-check (oxc-project#16204) (camc314) - 9cc20a1 minifier: Avoid merging side effectful expressions to next assignment statement if the side effect may change the left hand side reference (oxc-project#16165) (sapphi-red) - 75249e0 linter/plugins: Handle non-UTF8 file paths (oxc-project#16157) (overlookmotel) - 86fa667 linter/plugins: Improve type def for `RuleMeta` `defaultOptions` property (oxc-project#16159) (overlookmotel) - d2b7dcb tsgolint: Improve lsp diagnostic when linting fails (oxc-project#16115) (camc314) - 8810bd5 linter: Prevent conflicting fixes between prefer_number_properties and prefer_numeric_literals (oxc-project#16113) (camc314) - 91eb3f2 ast/estree: Convert `TSImportType` `argument` field to `Literal` (oxc-project#16109) (overlookmotel) - 2da2fc1 linter: Fix prefer-string-raw fixer producing invalid JS for non-ASCII strings (oxc-project#16102) (camc314) - f5cb601 linter/plugins: Perform length checks before continuing loops (oxc-project#16025) (Arsh) - e1c21ca linter/jsx-key: Implement missing options (oxc-project#15731) (camc314) ### ⚡ Performance - 02bdf90 linter/plugins, napi/parser: Reuse arrays in visitor keys (oxc-project#16294) (overlookmotel) - 714a91a linter/no-restricted-types: Skip running if config is empty (oxc-project#16278) (camc314) - d3a34f8 linter/plugins: Optimize `getTokens()` and other methods (oxc-project#16188) (Arsh) - c05db06 linter/plugins: Speed up `initTokensWithComments` (oxc-project#16117) (overlookmotel) - 4846886 linter/plugins: Optimize merging of `tokens` and `comments` (oxc-project#16071) (Arsh) - e232d35 linter/plugins: Recycle objects in token methods (oxc-project#16068) (overlookmotel) ### 📚 Documentation - e928732 linter/plugins: Fix JSDoc comment (oxc-project#16295) (overlookmotel) - d4a1581 linter: Add a link to MDN for the no-proto rule. (oxc-project#16213) (connorshea) - be36e36 linter/plugins: Fix JSDoc comment for `loadPluginImpl` (oxc-project#16211) (overlookmotel) - 6d6e9c9 linter: Improve the docs for oxc/no-async-await rule. (oxc-project#16181) (connorshea) - 0e1d38a linter/plugins: Clarify JSDoc comment for `getTokensBetween` (oxc-project#16070) (overlookmotel) - 3ee22b2 linter/plugins: Fix JSDoc comments for tokens methods (oxc-project#16063) (overlookmotel) - f257b5c linter/plugins: Clarify JSDoc comments for tokens methods (oxc-project#16062) (overlookmotel) # Oxfmt ### 🚀 Features - 116e0d1 website: Auto generate oxfmt docs (oxc-project#15985) (Boshen) - 862bdf7 oxfmt: Detect unsupported experimental options (take2) (oxc-project#16088) (leaysgur) ### 🐛 Bug Fixes - 75ac90c formatter: Comments in call arguments should be printed as-is (oxc-project#16327) (Dunqing) - fd77568 formatter: Don't wrap parenthesis for yield expression if there is no leading comment (oxc-project#16326) (Dunqing) - 8ccfb06 formatter: Should indent class extends and interface heritage when it is a member expression without type arguments (oxc-project#16323) (Dunqing) - 0faa978 oxfmt: JsFormatEmbeddedCb types (oxc-project#16324) (Brooooooklyn) - 2b8f982 formatter: JSX text wrapping incorrect (oxc-project#16318) (Dunqing) - f3ffebe formatter: Should indent variable declarator if there is a trailing comment (oxc-project#16243) (Dunqing) - 31d3186 formatter: Incorrect handling of directives with comments (oxc-project#16235) (Dunqing) - ac8fcaf formatter: Add parens for new: private field expr (oxc-project#16312) (leaysgur) - 653fa6c oxlint/oxfmt/lsp: Tell client the real tool name & version (oxc-project#16212) (Sysix) - 380a0af formatter: Incorrect printing of class binding trailing comments (oxc-project#16234) (Dunqing) - 0ca8154 formatter: Incorrect printing of trailing comments of callee when the call arguments are empty (oxc-project#16232) (Dunqing) - ac3a92e formatter: Print comment in ternary jsx (oxc-project#16224) (leaysgur) - e3a7388 formatter: Fix parens for static member chain (oxc-project#16229) (leaysgur) - 55334c3 formatter: Incorrect printing of dangling comments in the if statement (oxc-project#16228) (Dunqing) - 9096a63 formatter: Correct printing of trailing comments after the semicolon for class properties (oxc-project#16225) (Dunqing) - 75fd568 formatter: Inconsistent union type output between two runs (oxc-project#16222) (Dunqing) - cd70484 formatter: Should not add a hard space before function body (oxc-project#16221) (Dunqing) - 9097167 formatter: Incorrect printing of union types with comments (oxc-project#16205) (Dunqing) - 79b78b3 formatter: Template literal element should not be indented (oxc-project#16189) (Dunqing) - 48d6ed2 formatter: Nested assignment pattern should not expand outer object pattern (oxc-project#16160) (Dunqing) - 8f4137d formatter: Output is incorrect when using comments inside JSX which is the right hand-side of `LogicalExpression` (oxc-project#16156) (Dunqing) - 85c3a10 formatter/sort_imports: Handle internal prefixes correctly (oxc-project#16128) (leaysgur) - 38b7bc4 oxfmt: Make no-napi build work (oxc-project#16134) (leaysgur) - 889d2e7 formatter: Handle poor layout for grouped call arguments (oxc-project#16093) (Dunqing) - 14b0a6a oxfmt: Fix JS-ish file detection (oxc-project#16092) (leaysgur) - 9706a1a oxfmt: Ignore unsupported options (oxc-project#16085) (leaysgur)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SourceCodeAPIs (TS ESLint implementation) #15861.