perf(lexer): remove branches from unicode handling

[overlookmotel]

#12768 split next_char, next_2_chars, and peek_char into separate functions for the hot and cold paths.

That was a good change, but had one side-effect - because the unicode branch is now in a separate function which isn't inlined, the compiler loses knowledge of the context - that Source isn't at EOF, and that (in 2 of the 3 methods) the next character is known not to be ASCII.

Add unchecked assertions to inform compiler of the known facts, so it can remove 2 branches when calling chars.next().unwrap().

This code is on a cold path, so will likely not make a noticeable difference in files which don't contain many Unicode chars (like our benchmark fixtures), but why not?

[overlookmotel]

• perf(lexer): remove branches from unicode handling #15328 👈 (View in Graphite)
• test(transformer): update transformer conformance snapshots #15330
• main

---

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• 0-merge - adds this PR to the back of the merge queue
• hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codspeed-hq]

CodSpeed Performance Report

Merging #15328 will not alter performance

_{Comparing 11-01-perf_lexer_remove_branches_from_unicode_handling (f39e645) with main (70bf817)¹}

Summary

✅ 37 untouched

Footnotes

1. No successful run was found on main (6c09c1f) during the generation of this report, so 70bf817 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[Copilot]

Pull Request Overview

This PR refactors Unicode character handling in the lexer's Source module to improve performance by providing better hints to the compiler through assert_unchecked!. The changes replace unwrap_unchecked() calls with unwrap() paired with assert_unchecked! to communicate invariants more explicitly.

• Replaced unsafe unwrap_unchecked() with safe unwrap() after informing the compiler of invariants via assert_unchecked!
• Refactored Unicode character handling functions to remove the byte parameter and make them fully responsible for handling Unicode
• Restructured control flow to use if-else expressions instead of early returns

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[overlookmotel]

Merge activity

• Nov 5, 3:25 PM UTC: The merge label '0-merge' was detected. This PR will be added to the Graphite merge queue once it meets the requirements.
• Nov 5, 3:25 PM UTC: overlookmotel added this pull request to the Graphite merge queue.
• Nov 5, 3:32 PM UTC: Merged by the Graphite merge queue.