feat: scan compact ASCII lexer range classes - #127
Conversation
|
Warning Review limit reached
Next review available in: 49 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe compiled lexer now represents eligible ASCII self-loop sets as bounded range descriptors, scans them through a scalar range-aware path, serializes and validates the descriptors, records performance metrics, and adds lexer and benchmark coverage for identifier, number, and whitespace ranges. ChangesASCII range lexer acceleration
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Lexer
participant CompiledLexerDfa
participant AsciiRun
participant PerfCounters
Lexer->>CompiledLexerDfa: scan_ascii_run(state, input)
CompiledLexerDfa->>AsciiRun: scan(input)
AsciiRun-->>CompiledLexerDfa: scan result and matched range
CompiledLexerDfa-->>Lexer: scan result
Lexer->>PerfCounters: record range-scan metrics
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Copy/Paste DetectionFound 10 duplication(s) across 5 changed Rust file(s) (threshold: 100 tokens). Show duplicationsFound a 54 line (320 tokens) duplication in the following files:
atn: &LexerAtn,
hooks: &mut H,
mut generated_action: A,
mut generated_predicate: P,
unknown_policy: UnknownSemanticPolicy,
mut accept_adjuster: E,
) -> Result<TokenId, TokenStoreError>
where
I: CharStream,
H: SemanticHooks,
A: FnMut(&mut BaseLexer<I>, LexerCustomAction) -> bool,
P: FnMut(&BaseLexer<I>, LexerPredicate) -> Option<bool>,
E: FnMut(&mut BaseLexer<I>, i32, usize),
{
let hooks = RefCell::new(hooks);
let token = next_token_with_hooks_impl(
lexer,
sink,
atn,
&mut |lexer, action| {
if !generated_action(lexer, action)
&& !dispatch_lexer_action_hook(&hooks, lexer, action)
&& unknown_policy == UnknownSemanticPolicy::Error
&& let (Ok(rule), Ok(index)) = (
usize::try_from(action.rule_index()),
usize::try_from(action.action_index()),
)
{
lexer.record_semantic_error(true, rule, index);
}
},
&mut |lexer, predicate| {
generated_predicate(lexer, predicate)
.or_else(|| dispatch_lexer_predicate_hook(&hooks, lexer, predicate))
.unwrap_or_else(|| match unknown_policy {
UnknownSemanticPolicy::AssumeTrue => true,
UnknownSemanticPolicy::AssumeFalse => false,
UnknownSemanticPolicy::Error => {
lexer.record_semantic_error(
false,
predicate.rule_index(),
predicate.pred_index(),
);
false
}
})
},
&mut |lexer| dispatch_lexer_before_token_hook(&hooks, lexer),
&mut accept_adjuster,
&mut |lexer, accept_position| {
dispatch_lexer_after_accept_hook(&hooks, lexer, accept_position);
},
LexerMatchStrategy {
compiled: None,Found a 26 line (153 tokens) duplication in the following files:
pub fn next_token_with_hooks<I, A, P, E>(
lexer: &mut BaseLexer<I>,
sink: &mut TokenSink<'_>,
atn: &LexerAtn,
mut custom_action: A,
mut semantic_predicate: P,
mut accept_adjuster: E,
) -> Result<TokenId, TokenStoreError>
where
I: CharStream,
A: FnMut(&mut BaseLexer<I>, LexerCustomAction),
P: FnMut(&BaseLexer<I>, LexerPredicate) -> bool,
E: FnMut(&mut BaseLexer<I>, i32, usize),
{
next_token_with_hooks_impl(
lexer,
sink,
atn,
&mut custom_action,
&mut semantic_predicate,
&mut |_| {},
&mut accept_adjuster,
&mut |_, _| {},
LexerMatchStrategy {
compiled: None,
use_cache: false,Found a 25 line (142 tokens) duplication in the following files:
atn: &LexerAtn,
hooks: &mut H,
) -> Result<TokenId, TokenStoreError>
where
I: CharStream,
H: SemanticHooks,
{
let hooks = RefCell::new(hooks);
let token = next_token_with_hooks_impl(
lexer,
sink,
atn,
&mut |lexer, action| {
let _ = dispatch_lexer_action_hook(&hooks, lexer, action);
},
&mut |lexer, predicate| {
dispatch_lexer_predicate_hook(&hooks, lexer, predicate).unwrap_or(true)
},
&mut |lexer| dispatch_lexer_before_token_hook(&hooks, lexer),
&mut |_, _, _| {},
&mut |lexer, accept_position| {
dispatch_lexer_after_accept_hook(&hooks, lexer, accept_position);
},
LexerMatchStrategy {
compiled: None,Found a 23 line (138 tokens) duplication in the following files:
let mut next = Vec::new();
for config in active {
let Some(state) = atn.state(config.state) else {
continue;
};
for transition in &state.transitions {
if !transition.matches(symbol, MIN_CHAR_VALUE, MAX_CHAR_VALUE) {
continue;
}
let mut advanced = config.clone();
set_config_state(atn, &mut advanced, transition.target());
if symbol == EOF {
advanced.consumed_eof = true;
} else {
advanced.position += 1;
}
next.push(advanced);
}
}
let closure = epsilon_closure(atn, next, &mut |predicate| {
semantic_predicate(lexer, predicate)
});Found a 24 line (136 tokens) duplication in the following files:
let source_has_semantic_context = dfa_state_has_semantic_context;
for config in active {
let Some(state) = atn.state(config.state) else {
continue;
};
for transition in &state.transitions {
if !transition.matches(symbol, MIN_CHAR_VALUE, MAX_CHAR_VALUE) {
continue;
}
let mut advanced = config.clone();
set_config_state(atn, &mut advanced, transition.target());
if symbol == EOF {
advanced.consumed_eof = true;
} else {
advanced.position += 1;
}
next.push(advanced);
}
}
let closure = epsilon_closure(atn, next, &mut |predicate| {
semantic_predicate(lexer, predicate)
});
let target_has_semantic_context = closure.has_semantic_context;Found a 24 line (134 tokens) duplication in the following files:
let atn = AtnDeserializer::new(&SerializedAtn::from_i32(&[
4, 0, 2, // version, lexer, max token type
9, // states
6, -1, // 0 token start
2, 0, // 1 rule 0 start
1, 0, // 2
1, 0, // 3
7, 0, // 4 rule 0 stop
2, 1, // 5 rule 1 start
1, 1, // 6
1, 1, // 7
7, 1, // 8 rule 1 stop
0, // non-greedy
0, // precedence
2, // rules
1, 1, // rule 0 starts at 1, token type 1
5, 2, // rule 1 starts at 5, token type 2
1, // modes
0, // default mode starts at 0
0, // sets
8, // edges
0, 1, 1, 0, 0, 0, // start -> rule 0
0, 5, 1, 0, 0, 0, // start -> rule 1
1, 2, 5, 'a' as i32, 0, 0, 2, 3, 5, 'b' as i32, 0, 0, 3, 4, 1, 0, 0, 0, 5, 6, 5,Found a 23 line (129 tokens) duplication in the following files:
let source_has_semantic_context = dfa_state_has_semantic_context;
for config in active {
let Some(state) = atn.state(config.state) else {
continue;
};
for transition in &state.transitions {
if !transition.matches(symbol, MIN_CHAR_VALUE, MAX_CHAR_VALUE) {
continue;
}
let mut advanced = config.clone();
set_config_state(atn, &mut advanced, transition.target());
if symbol == EOF {
advanced.consumed_eof = true;
} else {
advanced.position += 1;
}
next.push(advanced);
}
}
let closure = epsilon_closure(atn, next, &mut |predicate| {
semantic_predicate(lexer, predicate)
});Found a 22 line (117 tokens) duplication in the following files:
atn: &LexerAtn,
mut custom_action: A,
mut semantic_predicate: P,
mut accept_adjuster: E,
) -> Result<TokenId, TokenStoreError>
where
I: CharStream,
A: FnMut(&mut BaseLexer<I>, LexerCustomAction),
P: FnMut(&BaseLexer<I>, LexerPredicate) -> bool,
E: FnMut(&mut BaseLexer<I>, i32, usize),
{
next_token_with_hooks_impl(
lexer,
sink,
atn,
&mut custom_action,
&mut semantic_predicate,
&mut |_| {},
&mut accept_adjuster,
&mut |_, _| {},
LexerMatchStrategy {
compiled: None,Found a 13 line (107 tokens) duplication in the following files:
let mut hooks = LifecycleRecordingHooks::default();
let mut store = TokenStore::new(lexer.source_text(), lexer.source_name());
let mut sink = TokenSink::new(&mut store);
let mut ids = Vec::new();
for _ in 0..3 {
let id = if compiled {
next_token_compiled_with_semantic_hooks(
&mut lexer, &mut sink, &atn, &dfa, &mut hooks,
)
} else {
next_token_with_semantic_hooks(&mut lexer, &mut sink, &atn, &mut hooks)
}
.expect("lifecycle token should fit");Found a 14 line (102 tokens) duplication in the following files:
let id = next_token_compiled(lexer, &mut sink, atn, dfa).expect("test token should fit");
let token = sink.view(id).expect("emitted token should exist");
TokenSnapshot {
token_type: token.token_type(),
text: token.text().to_owned(),
channel: token.channel(),
start: token.start(),
stop: token.stop(),
start_byte: token.start_byte(),
stop_byte: token.stop_byte(),
line: token.line(),
column: token.column(),
}
} |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements exact ASCII range descriptors and range-prefix scanners to optimize lexer performance for ASCII self-loop sets. It introduces the AsciiRange and AsciiRanges structures, integrates them into the compiled DFA representation (AsciiRun::Ranges), updates serialization formats, and adds comprehensive tests and benchmark fixtures. Feedback on the changes suggests hoisting the active ranges slice out of the loop in scan_scalar to avoid repeated dynamic slicing and bounds checks on the hot path.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
tinovyatkin
changed the title
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
|
@codex review |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/atn/ascii_range.rs (2)
103-111: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSimplify
from_packedusing bitwise shifts.You can use bitwise shifts to decode the ranges symmetrically to how
packed_wordsencodes them. This completely avoids the intermediateto_le_bytesarrays and simplifies the nested array indexing logic.♻️ Proposed refactor
- pub(super) fn from_packed(count: u8, words: [u32; 2]) -> Option<Self> { - let bytes = [words[0].to_le_bytes(), words[1].to_le_bytes()]; - let mut ranges = [AsciiRange::default(); MAX_RANGES]; - for (index, range) in ranges.iter_mut().enumerate() { - range.low = bytes[index / 2][(index % 2) * 2]; - range.high = bytes[index / 2][(index % 2) * 2 + 1]; - } - Self::new(count, ranges) - } + pub(super) fn from_packed(count: u8, words: [u32; 2]) -> Option<Self> { + let mut ranges = [AsciiRange::default(); MAX_RANGES]; + for (index, range) in ranges.iter_mut().enumerate() { + let shift = (index % 2) * 16; + range.low = (words[index / 2] >> shift) as u8; + range.high = (words[index / 2] >> (shift + 8)) as u8; + } + Self::new(count, ranges) + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/atn/ascii_range.rs` around lines 103 - 111, Update AsciiRange::from_packed to decode each low/high pair directly from words using bitwise shifts, matching the encoding performed by packed_words. Remove the intermediate to_le_bytes array and nested byte indexing while preserving range ordering and the existing Self::new(count, ranges) validation.
65-72: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRemove unnecessary
checked_add.Since
range.highis already validated to be an ASCII character (<= 127) in the preceding loop,pair[0].high + 1will never overflow au8(max255). You can safely replacechecked_addwith direct addition for better readability.♻️ Proposed refactor
- if ranges[..count].windows(2).any(|pair| { - pair[0] - .high - .checked_add(1) - .is_some_and(|next| next >= pair[1].low) - }) { + if ranges[..count] + .windows(2) + .any(|pair| pair[0].high + 1 >= pair[1].low) + {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/atn/ascii_range.rs` around lines 65 - 72, In the range-overlap check using windows(2), replace pair[0].high.checked_add(1) with direct u8 addition before comparing against pair[1].low. Keep the existing overlap condition and return behavior unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/atn/ascii_range.rs`:
- Around line 103-111: Update AsciiRange::from_packed to decode each low/high
pair directly from words using bitwise shifts, matching the encoding performed
by packed_words. Remove the intermediate to_le_bytes array and nested byte
indexing while preserving range ordering and the existing Self::new(count,
ranges) validation.
- Around line 65-72: In the range-overlap check using windows(2), replace
pair[0].high.checked_add(1) with direct u8 addition before comparing against
pair[1].low. Keep the existing overlap condition and return behavior unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 5c89896c-026e-40c2-8d0f-f51283432fa1
📒 Files selected for processing (2)
src/atn/ascii_range.rssrc/atn/lexer_dfa.rs
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Summary
docs/issue-80-lexer-range-benchmark.mdWhy
The existing
Until1/Until2/Until3acceleration handles self-loops with only a few exit bytes, but it cannot represent common identifier, numeric, or whitespace classes. These descriptors cover that remaining shape without embedding grammar-specific knowledge in the runtime.AVX2 and NEON candidates were implemented and tested during development. Native NEON measurements at 32- and 64-byte thresholds were neutral or slower on both stress workloads and representative Bazel/Trino fixtures, and this host could not provide native AVX2 benchmark evidence. The candidates were therefore removed under issue #80's decision gate; the final implementation contains no architecture-specific or unsafe code.
Impact
0.9796xgeometric time versusmain, with a1.0275xworst case0.8972xand0.8565xValidation
cargo test --locked --all-targets --all-featurescargo clippy --locked --all-targets --all-features -- -D warningscargo +1.95.0 check --locked --all-targets --all-featurescargo test --locked --target x86_64-apple-darwin --all-targets --all-features357 passed, 0 failed, 0 skipped1.15xregression guardCloses #80
Summary by CodeRabbit
Performance
Benchmarking
Documentation