fix(parser): make speculative recognition stack-safe by tinovyatkin · Pull Request #147 · ophi-dev/antlr-rust-runtime

tinovyatkin · 2026-07-20T21:32:12Z

Root cause

The v0.14.0 adaptive token-set change increased the native frame used by
recognize_state_fast. v0.14.1 reduced that frame enough for the first Mehen
reproducer, but it did not fix the underlying invariant: valid speculative ATN
depth was still mapped directly onto native call depth. A deeper Kotlin rule
chain therefore crossed the same unbounded limit. The empty-path cycle walk
also used native recursion for branching rule graphs.

Fix

preserve the optimized recursive recognizer, but check stack capacity before
each bounded group of recursive entries and grow a segmented stack when the
red zone is reached
track native nesting independently from the existing semantic ATN depth limit
make empty-path cycle reachability iterative
cover 4,096 nested calls, branching paths, consuming follows, and recovery on
a 256 KiB thread stack

Validation

reopened Mehen reproducer:
kotlin_nested_if_inside_else_if_chain_counts passes on the default stack
full Mehen snapshot gate: 938 passed, 0 skipped
cargo test --locked --all-targets --all-features
cargo clippy --locked --all-targets --all-features -- -D warnings
ANTLR runtime testsuite: 357 passed, 0 failed, 0 skipped
Kotlin parity: all 9 parse-tree fixtures match Python byte-for-byte
same-machine Kotlin A-B-B-A minimum-time ratios: 1.04x, 1.03x, 1.04x, and
1.02x (all below the 1.15x watchdog)

Fixes #142

Summary by CodeRabbit

Bug Fixes
- Improved parser stability when processing deeply nested or complex inputs.
- Prevented native stack overflow scenarios during parsing and recovery.
- Preserved existing recognition and reachability behavior while improving handling of deep parse paths.
Tests
- Expanded stress coverage for deep rule chains, branching structures, follow paths, and nested recovery cases.

tinovyatkin · 2026-07-20T21:32:18Z

@codex review

gemini-code-assist · 2026-07-20T21:32:24Z

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

coderabbitai · 2026-07-20T21:32:27Z

📝 Walkthrough

Walkthrough

The fast recognizer now tracks native call depth, conditionally grows the stack with stacker, and uses iterative empty-path reachability. New configurable nested ATN builders and small-stack tests cover deep recognition, branching, rule follows, and recovery.

Changes

Fast recognizer stack safety

Layer / File(s)	Summary
Fast recognizer stack guard `Cargo.toml`, `src/parser.rs`	Adds `stacker`, tracks `native_depth` across fast-recognition paths, and routes sufficiently deep calls through `stacker::maybe_grow`.
Iterative empty-path reachability `src/parser.rs`	Replaces recursive empty-path traversal with an explicit work stack.
Nested graph stress coverage `src/parser.rs`	Adds configurable branching and consuming-follow ATN shapes, deep small-stack parsing tests, and `native_depth` initialization in direct recognizer tests.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant FastRecognizer
  participant StackGuard
  participant RecognizerInner
  FastRecognizer->>FastRecognizer: Track native_depth
  FastRecognizer->>StackGuard: Check stack capacity at interval
  StackGuard->>RecognizerInner: Execute with maybe_grow
  RecognizerInner->>FastRecognizer: Continue nested recognition

Possibly related PRs

ophi-dev/antlr-rust-runtime#144: Targets native-stack overflow in the parser with related deep-nesting coverage and stack-usage changes.
ophi-dev/antlr-rust-runtime#66: Modifies fast-recognizer recovery and empty-parse cycle handling.
ophi-dev/antlr-rust-runtime#117: Updates empty-cycle reachability handling in src/parser.rs.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes add stack checks, iterative reachability, and deep-stack tests, addressing the stack-overflow issue in `#142`.
Out of Scope Changes check	✅ Passed	The shown changes stay focused on stack-safety fixes and test coverage, with no clear unrelated modifications.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: making speculative parser recognition stack-safe.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/iterative-fast-recognizer

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-07-20T21:32:33Z

Copy/Paste Detection

Found 16 duplication(s) across 1 changed Rust file(s) (threshold: 100 tokens).

Show duplications

Found a 27 line (145 tokens) duplication in the following files:

Starting at line 14994 of src/parser.rs
Starting at line 15127 of src/parser.rs

    fn generated_match_token_recovers_missing_token_from_context_follow() {
        let atn = generated_match_recovery_atn();
        let data = RecognizerData::new(
            "Mini.g4",
            Vocabulary::new(
                [None, Some("'X'"), Some("'Y'")],
                [None, Some("X"), Some("Y")],
                [None::<&str>, None, None],
            ),
        );
        let mut parser = BaseParser::new(
            CommonTokenStream::new(Source {
                tokens: vec![TestToken::eof("parser-test", 3, 1, 3)],
                index: 0,
            }),
            data,
        );
        parser.rule_context_stack = vec![
            RuleContextFrame {
                rule_index: 0,
                invoking_state: 0,
            },
            RuleContextFrame {
                rule_index: 1,
                invoking_state: 1,
            },
        ];

Found a 34 line (132 tokens) duplication in the following files:

Starting at line 8518 of src/parser.rs
Starting at line 8557 of src/parser.rs

                    if self.fast_parser_predicate_matches(predicate_context, transition, index) {
                        let boundary = left_recursive_boundary(atn, state, target);
                        outcomes.extend(
                            self.recognize_state_fast(
                                atn,
                                FastRecognizeRequest {
                                    state_number: target,
                                    stop_state,
                                    index,
                                    rule_start_index,
                                    decision_start_index: next_decision_start_index,
                                    precedence,
                                    depth: depth + 1,
                                    recovery_symbols: Rc::clone(&epsilon_recovery_symbols),
                                    recovery_state: epsilon_recovery_state,
                                },
                                FastRecognizeScratch {
                                    predicate_context,
                                    visiting,
                                    memo,
                                    expected,
                                    native_depth: native_depth + 1,
                                },
                            )
                            .into_iter()
                            .map(|mut outcome| {
                                if let Some(rule_index) = boundary {
                                    let boundary = self.arena_boundary_node(rule_index);
                                    self.defer_fast_outcome_node(&mut outcome, boundary);
                                }
                                outcome
                            }),
                        );
                    } else {

Found a 33 line (131 tokens) duplication in the following files:

Starting at line 8482 of src/parser.rs
Starting at line 8519 of src/parser.rs
Starting at line 8558 of src/parser.rs

                    let boundary = left_recursive_boundary(atn, state, target);
                    outcomes.extend(
                        self.recognize_state_fast(
                            atn,
                            FastRecognizeRequest {
                                state_number: target,
                                stop_state,
                                index,
                                rule_start_index,
                                decision_start_index: next_decision_start_index,
                                precedence,
                                depth: depth + 1,
                                recovery_symbols: Rc::clone(&epsilon_recovery_symbols),
                                recovery_state: epsilon_recovery_state,
                            },
                            FastRecognizeScratch {
                                predicate_context,
                                visiting,
                                memo,
                                expected,
                                native_depth: native_depth + 1,
                            },
                        )
                        .into_iter()
                        .map(|mut outcome| {
                            if let Some(rule_index) = boundary {
                                let boundary = self.arena_boundary_node(rule_index);
                                self.defer_fast_outcome_node(&mut outcome, boundary);
                            }
                            outcome
                        }),
                    );
                }

Found a 27 line (127 tokens) duplication in the following files:

Starting at line 13952 of src/parser.rs
Starting at line 14024 of src/parser.rs

        let mut atn = ParserAtnBuilder::new(2);
        assert_eq!(
            atn.add_state(AtnStateKind::RuleStart, Some(0))
                .expect("state")
                .index(),
            0
        );
        assert_eq!(
            atn.add_state(AtnStateKind::BlockStart, Some(0))
                .expect("state")
                .index(),
            1
        );
        assert_eq!(
            atn.add_state(AtnStateKind::Basic, Some(0))
                .expect("state")
                .index(),
            2
        );
        assert_eq!(
            atn.add_state(AtnStateKind::Basic, Some(0))
                .expect("state")
                .index(),
            3
        );
        assert_eq!(
            atn.add_state(AtnStateKind::BlockEnd, Some(0))

Found a 22 line (125 tokens) duplication in the following files:

Starting at line 13978 of src/parser.rs
Starting at line 14050 of src/parser.rs

            atn.add_state(AtnStateKind::BlockEnd, Some(0))
                .expect("state")
                .index(),
            4
        );
        assert_eq!(
            atn.add_state(AtnStateKind::RuleStop, Some(0))
                .expect("state")
                .index(),
            5
        );
        atn.set_rule_to_start_state(vec![0])
            .expect("rule start states");
        atn.set_rule_to_stop_state(vec![5])
            .expect("rule stop states");
        atn.add_decision_state(1).expect("decision state");
        atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
            .expect("transition");
        atn.add_transition(
            1,
            ParserTransitionSpec::Atom {
                target: 2,

Found a 33 line (115 tokens) duplication in the following files:

Starting at line 9442 of src/parser.rs
Starting at line 9515 of src/parser.rs

                        outcomes.extend(
                            self.recognize_state(
                                atn,
                                RecognizeRequest {
                                    state_number: *target,
                                    stop_state,
                                    index,
                                    rule_start_index,
                                    decision_start_index: next_decision_start_index,
                                    init_action_rules,
                                    predicates,
                                    semantics,
                                    rule_args,
                                    member_actions,
                                    return_actions,
                                    local_int_arg,
                                    member_values: member_values.clone(),
                                    return_values: return_values.clone(),
                                    rule_alt_number: next_alt_number,
                                    track_alt_numbers,
                                    consumed_eof,
                                    precedence,
                                    depth: depth + 1,
                                    recovery_symbols: epsilon_recovery_symbols.clone(),
                                    recovery_state: epsilon_recovery_state,
                                },
                                visiting,
                                memo,
                                expected,
                            )
                            .into_iter()
                            .map(|mut outcome| {
                                prepend_decision(&mut outcome, decision);

Found a 15 line (113 tokens) duplication in the following files:

Starting at line 15052 of src/parser.rs
Starting at line 15342 of src/parser.rs

    fn generated_match_token_counts_single_token_deletion_recovery() {
        let atn = generated_match_recovery_atn();
        let data = RecognizerData::new(
            "Mini.g4",
            Vocabulary::new(
                [None, Some("'X'"), Some("'Y'"), Some("'Z'")],
                [None, Some("X"), Some("Y"), Some("Z")],
                [None::<&str>, None, None, None],
            ),
        );
        let mut parser = BaseParser::new(
            CommonTokenStream::new(Source {
                tokens: vec![
                    TestToken::new(3).with_text("z"),
                    TestToken::new(2).with_text("y"),

Found a 12 line (112 tokens) duplication in the following files:

Starting at line 12933 of src/parser.rs
Starting at line 13016 of src/parser.rs

        let mut atn = ParserAtnBuilder::new(1);
        for (state, kind, rule) in [
            (0, AtnStateKind::RuleStart, 0),
            (1, AtnStateKind::StarLoopEntry, 0),
            (2, AtnStateKind::Basic, 0), // ops hub
            (3, AtnStateKind::Basic, 0), // shift prec
            (4, AtnStateKind::Basic, 0), // shift first >
            (5, AtnStateKind::Basic, 0), // shift second >
            (6, AtnStateKind::Basic, 0), // rel prec
            (7, AtnStateKind::Basic, 0), // rel >
            (8, AtnStateKind::LoopEnd, 0),
            (9, AtnStateKind::RuleStop, 0),

Found a 22 line (112 tokens) duplication in the following files:

Starting at line 14389 of src/parser.rs
Starting at line 14590 of src/parser.rs

    fn predicate_after_token_atn() -> Atn {
        let mut atn = ParserAtnBuilder::new(2);
        assert_eq!(
            atn.add_state(AtnStateKind::RuleStart, Some(0))
                .expect("state")
                .index(),
            0
        );
        assert_eq!(
            atn.add_state(AtnStateKind::Basic, Some(0))
                .expect("state")
                .index(),
            1
        );
        assert_eq!(
            atn.add_state(AtnStateKind::Basic, Some(0))
                .expect("state")
                .index(),
            2
        );
        assert_eq!(
            atn.add_state(AtnStateKind::Basic, Some(0))

Found a 22 line (111 tokens) duplication in the following files:

Starting at line 12798 of src/parser.rs
Starting at line 14389 of src/parser.rs
Starting at line 14590 of src/parser.rs

    fn left_recursive_loop_with_caller_follow_atn(caller_symbol: i32) -> Atn {
        let mut atn = ParserAtnBuilder::new(2);
        assert_eq!(
            atn.add_state(AtnStateKind::RuleStart, Some(0))
                .expect("state")
                .index(),
            0
        );
        assert_eq!(
            atn.add_state(AtnStateKind::Basic, Some(0))
                .expect("state")
                .index(),
            1
        );
        assert_eq!(
            atn.add_state(AtnStateKind::Basic, Some(0))
                .expect("state")
                .index(),
            2
        );
        assert_eq!(
            atn.add_state(AtnStateKind::RuleStart, Some(1))

Found a 14 line (110 tokens) duplication in the following files:

Starting at line 14723 of src/parser.rs
Starting at line 17279 of src/parser.rs

    fn parser_matches_token_and_reports_mismatch() {
        let source = Source {
            tokens: vec![
                TestToken::new(1).with_text("x"),
                TestToken::eof("parser-test", 1, 1, 1),
            ],
            index: 0,
        };
        let data = RecognizerData::new(
            "Mini.g4",
            Vocabulary::new([None, Some("'x'")], [None, Some("X")], [None::<&str>, None]),
        );
        let mut parser = BaseParser::new(CommonTokenStream::new(source), data);
        let matched = parser.match_token(1).expect("token 1 should match");

Found a 13 line (109 tokens) duplication in the following files:

Starting at line 14723 of src/parser.rs
Starting at line 17304 of src/parser.rs

    fn parser_matches_token_and_reports_mismatch() {
        let source = Source {
            tokens: vec![
                TestToken::new(1).with_text("x"),
                TestToken::eof("parser-test", 1, 1, 1),
            ],
            index: 0,
        };
        let data = RecognizerData::new(
            "Mini.g4",
            Vocabulary::new([None, Some("'x'")], [None, Some("X")], [None::<&str>, None]),
        );
        let mut parser = BaseParser::new(CommonTokenStream::new(source), data);

Found a 22 line (108 tokens) duplication in the following files:

Starting at line 7351 of src/parser.rs
Starting at line 7742 of src/parser.rs

    ) -> Option<RecognizeOutcome> {
        let (error_index, message) = self.expected_error_message(rule_index, start_index, expected);
        let diagnostic = diagnostic_for_token(self.token_at(error_index), message);
        let mut next_index = error_index;
        loop {
            let symbol = self.token_type_at(next_index);
            if sync_symbols.contains(&symbol) {
                if next_index == error_index {
                    return None;
                }
                break;
            }
            if symbol == TOKEN_EOF {
                break;
            }
            let after = self.consume_index(next_index, symbol);
            if after == next_index {
                break;
            }
            next_index = after;
        }
        let mut nodes = NodeSeqId::EMPTY;

Found a 15 line (108 tokens) duplication in the following files:

Starting at line 17558 of src/parser.rs
Starting at line 17582 of src/parser.rs

    fn outcome_ties_keep_later_non_recursive_alternative() {
        let arena = RecognitionArena::default();
        let first = RecognizeOutcome {
            index: 1,
            consumed_eof: false,
            alt_number: 0,
            member_values: BTreeMap::new(),
            return_values: BTreeMap::new(),
            diagnostics: DiagnosticSeqId::EMPTY,
            decisions: Vec::new(),
            actions: vec![ParserAction::new(1, 0, 0, None)],
            nodes: NodeSeqId::EMPTY,
        };
        let second = RecognizeOutcome {
            actions: vec![ParserAction::new(2, 0, 0, None)],

Found a 16 line (105 tokens) duplication in the following files:

Starting at line 6547 of src/parser.rs
Starting at line 7118 of src/parser.rs

        let start_state = atn.rule_to_start_state().get(rule_index).ok_or_else(|| {
            AntlrError::Unsupported(format!("rule {rule_index} has no start state"))
        })?;
        let stop_state = atn
            .rule_to_stop_state()
            .get(rule_index)
            .filter(|state| *state != usize::MAX)
            .ok_or_else(|| {
                AntlrError::Unsupported(format!("rule {rule_index} has no stop state"))
            })?;

        let start_index = self.current_visible_index();
        self.clear_prediction_diagnostics();
        self.reset_per_parse_caches();
        self.reset_recognition_arena();
        let caller_follow_state = self.pending_invoking_follow_state(atn);

Found a 13 line (100 tokens) duplication in the following files:

Starting at line 6282 of src/parser.rs
Starting at line 6306 of src/parser.rs

        let mut expected = BTreeSet::new();
        for index in (1..self.rule_context_stack.len()).rev() {
            let invoking_state = self.rule_context_stack[index].invoking_state;
            let Ok(state_number) = usize::try_from(invoking_state) else {
                continue;
            };
            let Some(Transition::Rule { follow_state, .. }) = atn
                .state(state_number)
                .and_then(|state| state.transitions().first())
                .map(ParserTransition::data)
            else {
                continue;
            };

tinovyatkin · 2026-07-20T21:34:23Z

Consolidated the duplicated epsilon-like continuation setup and rebased onto current main in d415f65. cargo test --locked --all-targets --all-features and strict Clippy remain green.

@codex review

chatgpt-codex-connector · 2026-07-20T21:40:36Z

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: d415f656f0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Keep the optimized recognizer recursive while checking native stack capacity at bounded intervals and growing a segmented stack when needed. Make empty-path cycle analysis iterative and cover deep calls, branches, follows, and recovery on a 256 KiB thread.

tinovyatkin · 2026-07-20T23:38:40Z

@codex review

chatgpt-codex-connector · 2026-07-20T23:43:22Z

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: e988bae3a7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…flow Generated recursive-descent rule methods mapped grammar-rule nesting directly onto native call depth: the CEL grammar walks ~9 rules per `[`, so ~850 nesting levels aborted the process on an 8 MiB stack while the interpreted path (recognize_state_fast, #147) and the tree walker were already segmented-stack safe. Emit a capacity probe at the shared parse_generated_rule_N_dispatch boundary, sampled once per 8 rule-context frames, growing onto a segmented stack via the same red-zone constants recognize_state_fast uses. Deeply nested input now parses (or reports a syntax error) instead of aborting. Fixes #193

tinovyatkin force-pushed the fix/iterative-fast-recognizer branch from 55a08bf to d415f65 Compare July 20, 2026 21:34

tinovyatkin force-pushed the fix/iterative-fast-recognizer branch from d415f65 to e988bae Compare July 20, 2026 23:38

coderabbitai Bot approved these changes Jul 20, 2026

View reviewed changes

tinovyatkin merged commit 84f365e into main Jul 21, 2026
12 of 13 checks passed

tinovyatkin deleted the fix/iterative-fast-recognizer branch July 21, 2026 00:08

ophiarch Bot mentioned this pull request Jul 21, 2026

chore(main): release 0.14.2 #148

Merged

This was referenced Jul 25, 2026

bug(codegen): generated recursive-descent rules overflow the native stack on deeply nested input #193

Closed

fix(codegen): guard generated rule dispatch against native stack overflow #194

Merged

Navigation Menu

Uh oh!

Uh oh!

fix(parser): make speculative recognition stack-safe - #147

fix(parser): make speculative recognition stack-safe#147
tinovyatkin merged 1 commit into
mainfrom
fix/iterative-fast-recognizer

tinovyatkin commented Jul 20, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

tinovyatkin commented Jul 20, 2026

Uh oh!

gemini-code-assist Bot commented Jul 20, 2026

Uh oh!

coderabbitai Bot commented Jul 20, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

github-actions Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

tinovyatkin commented Jul 20, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 20, 2026

Uh oh!

tinovyatkin commented Jul 20, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

tinovyatkin commented Jul 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Root cause

Fix

Validation

Summary by CodeRabbit

Uh oh!

tinovyatkin commented Jul 20, 2026

Uh oh!

gemini-code-assist Bot commented Jul 20, 2026

Uh oh!

coderabbitai Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

github-actions Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Copy/Paste Detection

Uh oh!

tinovyatkin commented Jul 20, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 20, 2026

Uh oh!

tinovyatkin commented Jul 20, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

tinovyatkin commented Jul 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 20, 2026 •

edited

Loading

github-actions Bot commented Jul 20, 2026 •

edited

Loading