Avoid per-token source index visitor calls - #26506
Merged
Merged
Conversation
Merging this PR will improve performance by 4.63%
Performance Changes
Tip Curious why this is faster? Use the CodSpeed MCP and ask your agent. Comparing 
Footnotes
|
charliermarsh
changed the title
charliermarsh
force-pushed
the
charlie/codex-parser-throughput
branch
from
July 1, 2026 14:17
0bcbd8b to
f673f29
Compare
charliermarsh
changed the title
charliermarsh
marked this pull request as ready for review
July 1, 2026 15:15
charliermarsh
added a commit
that referenced
this pull request
Jul 1, 2026
## Summary `Indexer::from_tokens` scans the trivia between consecutive tokens to find continuation newlines. Most tokens are adjacent, but we still construct an empty slice and run `match_indices` for those pairs. Skip that scan when the previous token ends where the current token starts. Non-empty gaps continue through the existing newline logic, preserving the indexed continuation lines. This is split from #26506 so CodSpeed can measure the empty-trivia fast path independently. ## Performance The [full CodSpeed report](https://app.codspeed.io/astral-sh/ruff/branches/charlie%2Fskip-empty-indexer-trivia) compares `779b0c2` with `dc42d1d`. All 15 linter CPU-simulation measurements improve: | Configuration | Gain across the five workloads | | --- | ---: | | Default rules | 1.06-2.72% | | All rules | 0.30-0.64% | | All rules with preview | 0.25-0.53% | Summing the five default-rules head and base measurements gives a 2.38% aggregate improvement. No individual benchmark crosses CodSpeed's significance threshold, which is why the GitHub bot does not list these gains even though they appear in the full report.
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Indexer::from_tokenspreviously called both range builders and checked string flags for every token, even though only string tokens can affect those indexes or span multiple lines.Dispatch the existing builders from the token-kind match instead. Interpolated-string boundaries visit the interpolated-range builder, regular strings and f-string middles visit the multiline-range builder, and t-string middles only update the current line start. This preserves the indexed ranges while removing visitor calls and flag lookup from the common non-string path.
The empty-trivia fast path was split into #26507 so CodSpeed can measure the two optimizations independently.
Performance
The full CodSpeed report compares
f673f29withdc42d1d. All 15 linter CPU-simulation measurements improve:Summing the five default-rules head and base measurements gives a 4.00% aggregate improvement. CodSpeed classifies
linter/default-rules[large/dataset.py]at 4.63% as a significant improvement; the other measured gains are below its reporting threshold.