perf(parser): remove lexer lookahead

[Boshen]

No description provided.

[Boshen]

• perf(parser): remove lexer lookahead #11349 👈 (View in Graphite)
• main

---

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• 0-merge - adds this PR to the back of the merge queue
• hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Copilot]

Pull Request Overview

This PR removes the lexer lookahead functionality to improve parsing performance. Key changes include:

• Removing explicit clearing of the lookahead buffer in various lexer implementations.
• Eliminating the lookahead field and associated methods from the lexer module and dependent cursor functions.

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
crates/oxc_parser/src/lexer/typescript.rs	Removed lookahead clearing in LAngle and RAngle token processing.
crates/oxc_parser/src/lexer/template.rs	Removed lookahead clearing in template literal substitution.
crates/oxc_parser/src/lexer/regex.rs	Removed lookahead clearing after reading regex.
crates/oxc_parser/src/lexer/punctuation.rs	Removed conditional lookahead clearing in re-lexing right angle tokens.
crates/oxc_parser/src/lexer/mod.rs	Removed lookahead field, related struct, and associated methods.
crates/oxc_parser/src/lexer/jsx.rs	Removed lookahead clearing for JSX token processing.
crates/oxc_parser/src/cursor.rs	Removed peek methods that depended on the lookahead API.

Comments suppressed due to low confidence (8)

crates/oxc_parser/src/lexer/typescript.rs:13

• Removing the lookahead clearing here improves performance; please confirm that no token state reset is needed for proper LAngle token handling.

self.lookahead.clear();

crates/oxc_parser/src/lexer/typescript.rs:26

• Ensure that clearing the lookahead for RAngle tokens is safely removed without affecting the correct token transitions.

self.lookahead.clear();

crates/oxc_parser/src/lexer/template.rs:391

• Verify that the removal of lookahead clearing in template literal substitutions does not compromise token consistency.

self.lookahead.clear();

crates/oxc_parser/src/lexer/regex.rs:23

• Confirm that the removal of lookahead clearing after reading regex patterns maintains the correct parsing behavior.

self.lookahead.clear();

crates/oxc_parser/src/lexer/punctuation.rs:62

• Review the removal of conditional lookahead clearing to ensure that edge cases for punctuation tokens are properly handled.

if kind != Kind::RAngle { self.lookahead.clear(); }

crates/oxc_parser/src/lexer/mod.rs:59

• With the removal of the lookahead field and its methods, please verify that all parser logic referencing lookahead has been updated accordingly.

struct Lookahead<'a> { position: SourcePosition<'a>, token: Token, }

crates/oxc_parser/src/lexer/jsx.rs:116

• Ensure that clearing the lookahead for JSX tokens is no longer required after the overall lookahead removal.

self.lookahead.clear();

crates/oxc_parser/src/cursor.rs:61

• Since peeking methods that depend on lookahead have been removed, check that the parser now uses an alternative approach for token lookahead.

pub(crate) fn peek_token(&mut self) -> Token { self.lexer.lookahead(1) }

[codspeed-hq]

CodSpeed Instrumentation Performance Report

Merging #11349 will degrade performances by 4.61%

_{Comparing 05-27-perf_parser_remove_lexer_lookahead (1bdeed2) with main (c91697e)}

Summary

⚡ 4 improvements
❌ 1 regressions
✅ 33 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
⚡	lexer[RadixUIAdoptionSection.jsx]	21.8 µs	21.1 µs	+3.49%
⚡	lexer[binder.ts]	961.2 µs	929.4 µs	+3.41%
⚡	lexer[cal.com.tsx]	6 ms	5.8 ms	+4.23%
⚡	lexer[react.development.js]	397.2 µs	383.7 µs	+3.52%
❌	parser[RadixUIAdoptionSection.jsx]	86.2 µs	90.4 µs	-4.61%

[camchenry]

Third times the charm maybe? 😄

[Boshen]

Performance regression after we remove code? 👎

[Boshen]

Merge activity

• May 28, 2:05 PM UTC: The merge label '0-merge' was detected. This PR will be added to the Graphite merge queue once it meets the requirements.
• May 28, 2:05 PM UTC: Boshen added this pull request to the Graphite merge queue.
• May 28, 2:12 PM UTC: Merged by the Graphite merge queue.