perf(parser): remove lexer lookahead

[camchenry]

• Closes parser: use checkpoints instead of token lookahead #11194

Removes the lookahead VecDeque in the lexer and all of the bookkeeping code depending on it.

[camchenry]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• perf(parser): remove lexer lookahead #11332 👈 (View in Graphite)
• refactor(parser): remove lexer lookahead in module parsing #11330 : 2 other dependent PRs (#11336 , #11337 )
• main

---

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• 0-merge - adds this PR to the back of the merge queue
• hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codspeed-hq]

CodSpeed Instrumentation Performance Report

Merging #11332 will degrade performances by 4.92%

_{Comparing 05-27-perf_parser_remove_lexer_lookahead (f0a9349) with 05-27-refactor_parser_remove_lexer_lookahead_in_module_parsing (f85a6a4)}

Summary

⚡ 4 improvements
❌ 2 regressions
✅ 32 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
⚡	lexer[RadixUIAdoptionSection.jsx]	21.9 µs	21 µs	+4.33%
⚡	lexer[binder.ts]	961.2 µs	929.3 µs	+3.43%
⚡	lexer[cal.com.tsx]	6 ms	5.8 ms	+4.23%
⚡	lexer[react.development.js]	397.3 µs	383.8 µs	+3.5%
❌	parser[RadixUIAdoptionSection.jsx]	87.1 µs	91.7 µs	-4.92%
❌	parser[react.development.js]	1.4 ms	1.4 ms	-3.16%

[camchenry]

Similar to before, this speeds up the lexer (~3-4%) but slows down the parser (~3-4%) which nets out to being slower than before. However, once we work on #11334 and other small perf improvements, we should be able to recover this performance and do even better in the long run.

[overlookmotel]

I suggest splitting this change out into a separate PR. Presumably these lines is where the parser perf regression is coming from. Removing the VecDeque from the lexer should be a gain in both lexer and parser benchmarks.

Then in this block of code, I think we can refactor to avoid using lookahead at all. I think that should be possible since we're immediately bumping on to next token anyway in the next line self.expect(separator). If we can do that, hopefully it'll remove the perf regression.

But it'd be useful to benchmark that in isolation without the change of removing the VecDeque also in the mix - hence why I think it'd be better in a separate PR.

[camchenry]

@overlookmotel got it, I can an intermediate PR to benchmark this specifically. I was planning to write a follow-up PR to improve the performance in this function and hopefully remove the lookahead.

[camchenry]

Okay Graphite seems to think this PR doesn't exist. I'm just going to close and re-create.