refactor(estree/tokens): convert tokens spans before AST spans#19723
Merged
graphite-app[bot] merged 1 commit intomainfrom Feb 25, 2026
Merged
Conversation
This was referenced Feb 25, 2026
This was referenced Feb 25, 2026
github-actions
bot
added
the
C-cleanup
This was referenced Feb 25, 2026
graphite-app
bot
force-pushed
the
om/02-22-refactor_estree_tokens_convert_tokens_spans_before_ast_spans
branch
from
e15f399 to
8b030d9
Compare
Base automatically changed from
om/02-22-refactor_estree_tokens_simplify_code
to
main
February 25, 2026 20:42
graphite-app
bot
deleted the
om/02-22-refactor_estree_tokens_convert_tokens_spans_before_ast_spans
branch
This was referenced Feb 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refactor.
Previously we converted spans from UTF-8 to UTF-16 in AST first, then serialized tokens. This meant that when serializing tokens, we had to convert spans to UTF-16 before comparing offsets of overrides to the token's start offset (because the override offsets are taken from the AST nodes during the AST walk pass).
Instead, convert token spans first, then the AST. This removes span conversion from the critical path. On a CPU with instruction-level parallelism, other work can start without being blocked on waiting for the result of span conversion, and the two can happen in parallel.
This should be a slight perf improvement, though it likely won't show up on CodSpeed benchmarks as CodSpeed's CPU simulation to basic to emulate instruction-level parallelism.
But mainly, the motivation is that this makes the process simpler and easier to reason about.