Fix exponential block-comment lexing - #137
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThe lexer prediction engine now represents recursive caller paths with shared context graphs instead of per-configuration stacks. Interpreted, cached, and compiled matching paths, DFA serialization, validation, and regression tests were updated for the new representation. ChangesLexer caller-context refactor
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant BaseLexer
participant LexerContextArena
participant epsilon_closure_with_lexer
participant LexerDfaCache
participant SemanticPredicate
BaseLexer->>LexerContextArena: Reset prediction workspace
BaseLexer->>epsilon_closure_with_lexer: Expand active configurations
epsilon_closure_with_lexer->>LexerContextArena: Merge caller contexts
epsilon_closure_with_lexer->>SemanticPredicate: Evaluate predicate transitions
epsilon_closure_with_lexer->>LexerDfaCache: Cache context-aware DFA state
LexerDfaCache-->>BaseLexer: Resume cached or interpreted matching
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Copy/Paste DetectionFound 9 duplication(s) across 3 changed Rust file(s) (threshold: 100 tokens). Show duplicationsFound a 54 line (320 tokens) duplication in the following files:
atn: &LexerAtn,
hooks: &mut H,
mut generated_action: A,
mut generated_predicate: P,
unknown_policy: UnknownSemanticPolicy,
mut accept_adjuster: E,
) -> Result<TokenId, TokenStoreError>
where
I: CharStream,
H: SemanticHooks,
A: FnMut(&mut BaseLexer<I>, LexerCustomAction) -> bool,
P: FnMut(&BaseLexer<I>, LexerPredicate) -> Option<bool>,
E: FnMut(&mut BaseLexer<I>, i32, usize),
{
let hooks = RefCell::new(hooks);
let token = next_token_with_hooks_impl(
lexer,
sink,
atn,
&mut |lexer, action| {
if !generated_action(lexer, action)
&& !dispatch_lexer_action_hook(&hooks, lexer, action)
&& unknown_policy == UnknownSemanticPolicy::Error
&& let (Ok(rule), Ok(index)) = (
usize::try_from(action.rule_index()),
usize::try_from(action.action_index()),
)
{
lexer.record_semantic_error(true, rule, index);
}
},
&mut |lexer, predicate| {
generated_predicate(lexer, predicate)
.or_else(|| dispatch_lexer_predicate_hook(&hooks, lexer, predicate))
.unwrap_or_else(|| match unknown_policy {
UnknownSemanticPolicy::AssumeTrue => true,
UnknownSemanticPolicy::AssumeFalse => false,
UnknownSemanticPolicy::Error => {
lexer.record_semantic_error(
false,
predicate.rule_index(),
predicate.pred_index(),
);
false
}
})
},
&mut |lexer| dispatch_lexer_before_token_hook(&hooks, lexer),
&mut accept_adjuster,
&mut |lexer, accept_position| {
dispatch_lexer_after_accept_hook(&hooks, lexer, accept_position);
},
LexerMatchStrategy {
compiled: None,Found a 26 line (153 tokens) duplication in the following files:
pub fn next_token_with_hooks<I, A, P, E>(
lexer: &mut BaseLexer<I>,
sink: &mut TokenSink<'_>,
atn: &LexerAtn,
mut custom_action: A,
mut semantic_predicate: P,
mut accept_adjuster: E,
) -> Result<TokenId, TokenStoreError>
where
I: CharStream,
A: FnMut(&mut BaseLexer<I>, LexerCustomAction),
P: FnMut(&BaseLexer<I>, LexerPredicate) -> bool,
E: FnMut(&mut BaseLexer<I>, i32, usize),
{
next_token_with_hooks_impl(
lexer,
sink,
atn,
&mut custom_action,
&mut semantic_predicate,
&mut |_| {},
&mut accept_adjuster,
&mut |_, _| {},
LexerMatchStrategy {
compiled: None,
use_cache: false,Found a 25 line (142 tokens) duplication in the following files:
atn: &LexerAtn,
hooks: &mut H,
) -> Result<TokenId, TokenStoreError>
where
I: CharStream,
H: SemanticHooks,
{
let hooks = RefCell::new(hooks);
let token = next_token_with_hooks_impl(
lexer,
sink,
atn,
&mut |lexer, action| {
let _ = dispatch_lexer_action_hook(&hooks, lexer, action);
},
&mut |lexer, predicate| {
dispatch_lexer_predicate_hook(&hooks, lexer, predicate).unwrap_or(true)
},
&mut |lexer| dispatch_lexer_before_token_hook(&hooks, lexer),
&mut |_, _, _| {},
&mut |lexer, accept_position| {
dispatch_lexer_after_accept_hook(&hooks, lexer, accept_position);
},
LexerMatchStrategy {
compiled: None,Found a 24 line (134 tokens) duplication in the following files:
let atn = AtnDeserializer::new(&SerializedAtn::from_i32(&[
4, 0, 2, // version, lexer, max token type
9, // states
6, -1, // 0 token start
2, 0, // 1 rule 0 start
1, 0, // 2
1, 0, // 3
7, 0, // 4 rule 0 stop
2, 1, // 5 rule 1 start
1, 1, // 6
1, 1, // 7
7, 1, // 8 rule 1 stop
0, // non-greedy
0, // precedence
2, // rules
1, 1, // rule 0 starts at 1, token type 1
5, 2, // rule 1 starts at 5, token type 2
1, // modes
0, // default mode starts at 0
0, // sets
8, // edges
0, 1, 1, 0, 0, 0, // start -> rule 0
0, 5, 1, 0, 0, 0, // start -> rule 1
1, 2, 5, 'a' as i32, 0, 0, 2, 3, 5, 'b' as i32, 0, 0, 3, 4, 1, 0, 0, 0, 5, 6, 5,Found a 23 line (133 tokens) duplication in the following files:
let source_has_semantic_context = dfa_state_has_semantic_context;
for config in active {
let Some(state) = atn.state(config.state) else {
continue;
};
for transition in &state.transitions {
if !transition.matches(symbol, MIN_CHAR_VALUE, MAX_CHAR_VALUE) {
continue;
}
let mut advanced = config.clone();
set_config_state(atn, &mut advanced, transition.target());
if symbol == EOF {
advanced.consumed_eof = true;
} else {
advanced.position += 1;
}
next.push(advanced);
}
}
let closure = epsilon_closure_with_lexer(lexer, atn, next, semantic_predicate);
let target_has_semantic_context = closure.has_semantic_context;
let suppress_edge = source_has_semantic_context || target_has_semantic_context;Found a 28 line (119 tokens) duplication in the following files:
impl IntStream for FallbackInput {
fn consume(&mut self) {
self.0.consume();
}
fn la(&mut self, offset: isize) -> i32 {
self.0.la(offset)
}
fn index(&self) -> usize {
self.0.index()
}
fn seek(&mut self, index: usize) {
self.0.seek(index);
}
fn size(&self) -> usize {
self.0.size()
}
fn source_name(&self) -> &str {
self.0.source_name()
}
}
// Deliberately implements none of the optional fast paths.
impl CharStream for FallbackInput {Found a 22 line (117 tokens) duplication in the following files:
atn: &LexerAtn,
mut custom_action: A,
mut semantic_predicate: P,
mut accept_adjuster: E,
) -> Result<TokenId, TokenStoreError>
where
I: CharStream,
A: FnMut(&mut BaseLexer<I>, LexerCustomAction),
P: FnMut(&BaseLexer<I>, LexerPredicate) -> bool,
E: FnMut(&mut BaseLexer<I>, i32, usize),
{
next_token_with_hooks_impl(
lexer,
sink,
atn,
&mut custom_action,
&mut semantic_predicate,
&mut |_| {},
&mut accept_adjuster,
&mut |_, _| {},
LexerMatchStrategy {
compiled: None,Found a 13 line (107 tokens) duplication in the following files:
let mut hooks = LifecycleRecordingHooks::default();
let mut store = TokenStore::new(lexer.source_text(), lexer.source_name());
let mut sink = TokenSink::new(&mut store);
let mut ids = Vec::new();
for _ in 0..3 {
let id = if compiled {
next_token_compiled_with_semantic_hooks(
&mut lexer, &mut sink, &atn, &dfa, &mut hooks,
)
} else {
next_token_with_semantic_hooks(&mut lexer, &mut sink, &atn, &mut hooks)
}
.expect("lifecycle token should fit");Found a 14 line (102 tokens) duplication in the following files:
let id = next_token_compiled(lexer, &mut sink, atn, dfa).expect("test token should fit");
let token = sink.view(id).expect("emitted token should exist");
TokenSnapshot {
token_type: token.token_type(),
text: token.text().to_owned(),
channel: token.channel(),
start: token.start(),
stop: token.stop(),
start_byte: token.start_byte(),
stop_byte: token.stop_byte(),
line: token.line(),
column: token.column(),
}
} |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request replaces the explicit call stack (Vec<usize>) in the lexer configuration with graph-structured caller contexts managed by a new LexerContextArena. This optimization allows equivalent configurations with different caller paths to merge their contexts, preventing unbounded state growth in recursive lexer rules such as nested comments. The review feedback highlights a potential stack-overflow risk in the recursive visit helper within has_recursive_context and suggests refactoring it to use an iterative approach with an explicit stack.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
tinovyatkin
changed the title
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9d5369e249
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Summary
Root cause
The lexer ATN simulator included a concrete cloned return-state stack in each configuration. Recursive/non-greedy comment rules can reach the same lexer state, input position, rule, and action history through many equivalent caller paths. Keeping every concrete path separate caused the closure to materialize and revisit a combinatorial number of configurations; successive block comments exposed that growth as approximately
O(2^n).The fix stores caller paths as canonical graph-structured contexts and unions them when the rest of a configuration is equivalent. Rule-stop traversal tracks visited context nodes, so shared DAG branches are not expanded repeatedly. Ordered unions retain the traversal priority needed for greedy/non-greedy behavior. The same representation is now used by learned DFA states and compiled-DFA continuations.
This is lexer prediction work, before a matched token's channel action is applied. Hidden-channel tokens exposed the problem in the Kotlin grammar but were not incorrectly entering parser
ALL(*)lookahead.Benchmarks
Mehen's vendored Kotlin-spec lexer grammar, same Apple M3 Pro host:
0.13.0The fixed path scales approximately linearly through 80 comments.
The documented grammars-v4 Rust-vs-Go matrix was also rerun with 10 iterations and 2 warmups. Geometric-mean
go avg_ns / rust avg_nsratios are now:Validation
cargo test --lockedcargo test --locked --all-features --lib(241 passed)cargo clippy --locked --all-targets --all-features -- -D warningscargo fmt --checkgit diff --check357 passed, 0 failed, 0 skippedCloses #135
Summary by CodeRabbit
Improvements
Bug Fixes
Documentation