ci(benchmarks/lexer): fix lexer benchmark

[overlookmotel]

Fix a bug in lexer benchmark. SourceCleaner checks that lexer doesn't produce any errors on the cleaned source. But it wasn't using exactly the same code for lexing as the benchmark itself - it wasn't using Lexer#first_token to get the first token. Fix that.

[overlookmotel]

• perf(lexer): skip single space in read_next_token #15513
• perf(lexer): inline handle_byte into read_next_token #15520
• ci(benchmarks/lexer): ensure next_token is inlined into lexer benchmark #15519
• ci(benchmarks/lexer): fix lexer benchmark #15517 👈 (View in Graphite)
• main

---

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• 0-merge - adds this PR to the back of the merge queue
• hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Copilot]

Pull Request Overview

This PR refactors the lexer benchmark code by extracting the common lexing logic into a reusable lex_whole_file helper function. The refactoring aims to reduce code duplication and improve maintainability.

Key Changes:

• Extracted common lexer iteration logic into a new lex_whole_file function with #[inline(always)] attribute
• Updated both the benchmark loop and SourceCleaner::clean method to use the new helper function

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codspeed-hq]

CodSpeed Performance Report

Merging #15517 will not alter performance

_{Comparing 11-09-ci_benchmarks_lexer_fix_lexer_benchmark (c8de3e4) with main (2f0518d)¹}

Summary

✅ 37 untouched

Footnotes

1. No successful run was found on main (bc2e35d) during the generation of this report, so 2f0518d was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[graphite-app]

Merge activity

• Nov 9, 4:45 PM UTC: Boshen added this pull request to the Graphite merge queue.
• Nov 9, 4:51 PM UTC: Merged by the Graphite merge queue.