perf: compile the lexer DFA to static tables at codegen time - #54
Conversation
ANTLR lexers discover their DFA lazily: the ATN simulator computes epsilon closures per character and caches the resulting config sets per process (and per thread). This adds an ahead-of-time subset construction that runs inside antlr4-rust-gen, embeds the finished tables in the generated lexer as a serialized u32 stream, and matches tokens with one array lookup per character instead of closure computation, hashing, or config allocation. Construction reuses the interpreter's own closure, pruning, and accept-selection code, so a compiled walk reproduces interpreter behavior exactly. Constructs a finite DFA cannot represent compile as *escape edges*: reaching one re-matches that single token through the ATN interpreter. This covers semantic predicates, recursive lexer rules (Kotlin's nested DelimitedComment), position-dependent custom-action traces, and the state budget, so one dynamic rule never disqualifies the rest of its mode. Two state-explosion sources had to be normalized away for real grammars to determinize: action traces from referenced token rules that the accept-time dispatcher would suppress anyway (they made state identity input-offset dependent), and recursive rule-call stacks (detected via duplicated follow states). With both fixes the Kotlin lexer converges to 1,325 states across all five modes; Java 489, Trino 1,226, C# 1,068 (its predicate-guarded interpolation mode correctly stays interpreted). Generated lexers deserialize the embedded stream once per process (sub-ms, version-tagged with fallback to runtime compilation, then interpretation). showDFA conformance cases opt out through the new set_force_interpreted, since the learned-DFA trace only exists under interpretation. Kotlin fixture results (Apple Silicon, parse-bench min): 2-7% faster parses, and first-parse latency drops 1.54ms -> 1.08ms because cache warmup is gone. Conformance: full runtime testsuite 357 passed / 0 failed; kotlin-parity byte-for-byte green. Also fixes bench_tree_sitter.py for tree-sitter-language-pack wheels that ship the Rust-native binding (str input, method accessors) so parse-bench runs on macOS arm64 again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 1 minute Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
WalkthroughChangesThis PR adds an ahead-of-time compiled DFA matching path for lexers alongside the existing ATN interpreter. A new Sequence Diagram(s)See diagrams embedded in the hidden review stack artifact for token matching dispatch and compiled DFA construction flows. Related Issues: None specified. Related PRs: None specified. Suggested labels: enhancement, performance, lexer Suggested reviewers: ophi-dev Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Copy/Paste DetectionFound 12 duplication(s) across 6 changed Rust file(s) (threshold: 100 tokens). Show duplicationsFound a 26 line (168 tokens) duplication in the following files:
atn.add_state(AtnState::new(5, AtnStateKind::RuleStop).with_rule_index(0));
atn.state_mut(0)
.expect("state 0")
.add_transition(Transition::Epsilon { target: 1 });
atn.state_mut(1)
.expect("state 1")
.add_transition(Transition::Epsilon { target: 2 });
atn.state_mut(1)
.expect("state 1")
.add_transition(Transition::Epsilon { target: 3 });
atn.state_mut(2)
.expect("state 2")
.add_transition(Transition::Atom {
target: 4,
label: 1,
});
atn.state_mut(3)
.expect("state 3")
.add_transition(Transition::Atom {
target: 4,
label: 2,
});
atn.state_mut(4)
.expect("state 4")
.add_transition(Transition::Epsilon { target: 5 });
atn.add_decision_state(1);Found a 43 line (157 tokens) duplication in the following files:
)
.expect("writing to a string cannot fail");
// Capture the rule start AFTER `enter_rule`, which advances the cursor past any
// leading hidden-channel tokens to the first visible token. Capturing before
// would make `$start`/`$text` in generated actions include a leading hidden
// prefix (e.g. whitespace), diverging from ANTLR and the rule context start.
writeln!(
out,
" let __rule_start = antlr4_runtime::IntStream::index(self.base.input());"
)
.expect("writing to a string cannot fail");
// Member-setting `@init` runs on rule entry (before the body) so same-rule
// predicates and actions observe the state it sets.
render_generated_init_action_entry(
out,
index,
step_render_context.init_entry_action_statements,
2,
);
// Queue the `@init` action event before the body steps so the buffered replay
// (`run_generated_action`) runs it ahead of body actions, matching ANTLR's
// "init before body" order. It sits after `__generated_action_marker`, so a
// fatal-sync abort that truncates back to the marker discards it too.
render_generated_init_action(out, index, entry_state, init_action_statements, 2);
writeln!(out, " let mut __consumed_eof = false;")
.expect("writing to a string cannot fail");
writeln!(
out,
" let mut __sync_error: Option<antlr4_runtime::AntlrError> = None;"
)
.expect("writing to a string cannot fail");
writeln!(
out,
" let __result = (|| -> Result<(), antlr4_runtime::AntlrError> {{"
)
.expect("writing to a string cannot fail");
render_generated_steps(out, &rule.steps, 3, step_render_context);
writeln!(out, " Ok(())").expect("writing to a string cannot fail");
writeln!(out, " }})();").expect("writing to a string cannot fail");
writeln!(out, " match __result {{").expect("writing to a string cannot fail");
writeln!(out, " Ok(()) => {{").expect("writing to a string cannot fail");
writeln!(
out,Found a 24 line (134 tokens) duplication in the following files:
let atn = AtnDeserializer::new(&SerializedAtn::from_i32(&[
4, 0, 2, // version, lexer, max token type
9, // states
6, -1, // 0 token start
2, 0, // 1 rule 0 start
1, 0, // 2
1, 0, // 3
7, 0, // 4 rule 0 stop
2, 1, // 5 rule 1 start
1, 1, // 6
1, 1, // 7
7, 1, // 8 rule 1 stop
0, // non-greedy
0, // precedence
2, // rules
1, 1, // rule 0 starts at 1, token type 1
5, 2, // rule 1 starts at 5, token type 2
1, // modes
0, // default mode starts at 0
0, // sets
8, // edges
0, 1, 1, 0, 0, 0, // start -> rule 0
0, 5, 1, 0, 0, 0, // start -> rule 1
1, 2, 5, 'a' as i32, 0, 0, 2, 3, 5, 'b' as i32, 0, 0, 3, 4, 1, 0, 0, 0, 5, 6, 5,Found a 18 line (134 tokens) duplication in the following files:
atn.add_state(AtnState::new(4, AtnStateKind::BlockEnd).with_rule_index(0));
atn.add_state(AtnState::new(5, AtnStateKind::RuleStop).with_rule_index(0));
atn.state_mut(0)
.expect("state 0")
.add_transition(Transition::Epsilon { target: 1 });
atn.state_mut(1)
.expect("state 1")
.add_transition(Transition::Epsilon { target: 2 });
atn.state_mut(1)
.expect("state 1")
.add_transition(Transition::Epsilon { target: 3 });
atn.state_mut(2)
.expect("state 2")
.add_transition(Transition::Atom {
target: 4,
label: 1,
});
atn.state_mut(3)Found a 23 line (133 tokens) duplication in the following files:
pub fn next_token_with_hooks<I, F, A, P, E>(
lexer: &mut BaseLexer<I, F>,
atn: &Atn,
mut custom_action: A,
mut semantic_predicate: P,
mut accept_adjuster: E,
) -> CommonToken
where
I: CharStream,
F: TokenFactory,
A: FnMut(&mut BaseLexer<I, F>, LexerCustomAction),
P: FnMut(&BaseLexer<I, F>, LexerPredicate) -> bool,
E: FnMut(&mut BaseLexer<I, F>, i32, usize),
{
next_token_with_hooks_impl(
lexer,
atn,
&mut custom_action,
&mut semantic_predicate,
&mut accept_adjuster,
LexerMatchStrategy {
compiled: None,
use_cache: false,Found a 22 line (130 tokens) duplication in the following files:
let Some(state) = atn.state(config.state) else {
continue;
};
for transition in &state.transitions {
if !transition.matches(symbol, MIN_CHAR_VALUE, MAX_CHAR_VALUE) {
continue;
}
let mut advanced = config.clone();
set_config_state(atn, &mut advanced, transition.target());
if symbol == EOF {
advanced.consumed_eof = true;
} else {
advanced.position += 1;
}
next.push(advanced);
}
}
let closure = epsilon_closure(atn, next, &mut |predicate| {
semantic_predicate(lexer, predicate)
});
let target_has_semantic_context = closure.has_semantic_context;Found a 17 line (117 tokens) duplication in the following files:
atn.add_state(AtnState::new(5, AtnStateKind::RuleStop).with_rule_index(0));
atn.state_mut(0)
.expect("state 0")
.add_transition(Transition::Epsilon { target: 1 });
atn.state_mut(1)
.expect("state 1")
.add_transition(Transition::Epsilon { target: 2 });
atn.state_mut(1)
.expect("state 1")
.add_transition(Transition::Epsilon { target: 3 });
atn.state_mut(2)
.expect("state 2")
.add_transition(Transition::Atom {
target: 4,
label: 1,
});
atn.state_mut(4)Found a 20 line (109 tokens) duplication in the following files:
[GeneratedParserStep::Decision {
state: 1,
decision: 0,
track_alt_number: true,
allow_semantic_context: false,
force_context: false,
fast_path: Some(GeneratedDecisionFastPath {
arms: vec![
GeneratedDecisionFastArm {
alt: 1,
intervals: vec![(1, 1)],
},
GeneratedDecisionFastArm {
alt: 2,
intervals: vec![(2, 2)],
},
],
}),
alts: vec![vec![mt(1, 4)], vec![mt(2, 4)]],
}]Found a 13 line (105 tokens) duplication in the following files:
fn compile_generated_parser_star_loop(
context: &GeneratedParserCompileContext<'_>,
state: &antlr4_runtime::atn::AtnState,
decision: usize,
stop_state: usize,
visited: &mut BTreeSet<usize>,
) -> Option<Vec<GeneratedParserStep>> {
let mut enter = None;
let mut exit = None;
for (index, transition) in state.transitions.iter().enumerate() {
let alt = index + 1;
let target = transition.target();
let target_state = context.atn.state(target)?;Found a 17 line (105 tokens) duplication in the following files:
}
ActionTemplate::Noop
| ActionTemplate::Text { .. }
| ActionTemplate::TextWithPrefix { .. }
| ActionTemplate::RuleTextWithPrefix { .. }
| ActionTemplate::StringTree { .. }
| ActionTemplate::RuleInvocationStack { .. }
| ActionTemplate::ListenerWalk { .. }
| ActionTemplate::RuleValue { .. }
| ActionTemplate::RuleReturnValue { .. }
| ActionTemplate::SetIntReturn { .. }
| ActionTemplate::TokenText { .. }
| ActionTemplate::TokenTextWithPrefix { .. }
| ActionTemplate::TokenDisplay { .. }
| ActionTemplate::ExpectedTokenNames { .. }
| ActionTemplate::Literal { .. }
| ActionTemplate::MemberValue { .. }Found a 35 line (103 tokens) duplication in the following files:
writeln!(out, " self.base.exit_rule();")
.expect("writing to a string cannot fail");
writeln!(
out,
" self.generated_actions.truncate(__generated_action_marker);"
)
.expect("writing to a string cannot fail");
writeln!(
out,
" self.base.restore_int_members(__generated_member_checkpoint);"
)
.expect("writing to a string cannot fail");
writeln!(
out,
" self.base.restore_generated_diagnostics(__generated_diagnostic_marker);"
)
.expect("writing to a string cannot fail");
writeln!(
out,
" self.base.record_generated_syntax_error();"
)
.expect("writing to a string cannot fail");
writeln!(
out,
" return Err(GeneratedRuleError::Fatal(__error));"
)
.expect("writing to a string cannot fail");
writeln!(out, " }}").expect("writing to a string cannot fail");
writeln!(
out,
" self.base.recover_generated_rule(&mut __ctx, atn(), __error);"
)
.expect("writing to a string cannot fail");
writeln!(
out,Found a 20 line (102 tokens) duplication in the following files:
atn: &Atn,
mut custom_action: A,
mut semantic_predicate: P,
mut accept_adjuster: E,
) -> CommonToken
where
I: CharStream,
F: TokenFactory,
A: FnMut(&mut BaseLexer<I, F>, LexerCustomAction),
P: FnMut(&BaseLexer<I, F>, LexerPredicate) -> bool,
E: FnMut(&mut BaseLexer<I, F>, i32, usize),
{
next_token_with_hooks_impl(
lexer,
atn,
&mut custom_action,
&mut semantic_predicate,
&mut accept_adjuster,
LexerMatchStrategy {
compiled: None, |
Confidence Score: 5/5This looks safe to merge.
Reviews (4): Last reviewed commit: "fix: reject unsorted or inverted wide ro..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces ahead-of-time (AOT) lexer DFA compilation to optimize token matching by replacing runtime closure computations with static table lookups. It adds the CompiledLexerDfa module, updates the code generator to embed compiled DFA tables, and refactors the lexer to support both compiled and interpreted matching strategies. Feedback on these changes highlights opportunities to optimize vector allocation during serialization and recommends replacing unstable let_chains syntax with nested if let blocks to ensure compatibility with older Rust editions.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Breaking out of the compiled walk kept the best accept seen so far, which could return a shorter token (or a spurious recognition error) than the interpreter for a grammar with a legal EOF chain longer than the bound. Escaping instead re-matches the token through the interpreter, which owns the semantics of longer chains. Also pre-allocates the exact serialized-stream capacity, asserted in debug builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tools/parse-bench/run.py (1)
616-637: 🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick winUse a throwaway parser for source probing.
probe_source()does one untimed parse on the same parser used for the benchmark, so--warmups=0still gets an implicit warmup. Probe with a separate parser, then create the parser used by the timed loop.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tools/parse-bench/run.py` around lines 616 - 637, probe_source() is using the same parser instance that later gets benchmarked, which adds an implicit untimed warmup even when --warmups is 0. Update main() to probe input with a throwaway parser created via get_parser(SPECS[args.language]), then instantiate a fresh parser for the timed loop and use that one for parse_once() iterations.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/atn/lexer_dfa.rs`:
- Around line 149-159: The mode_state_counts() implementation in LexerDfa drops
None entries by flattening mode_starts, so uncompiled lexer modes disappear from
the per-mode diagnostics. Update this method to preserve one count per mode by
iterating over mode_starts directly, using zero for None entries and computing
each compiled mode’s count from its start offset and the next available
boundary, so the returned vector stays aligned with compiled_mode_flags() and
includes zeros for uncompiled modes.
- Around line 303-323: `table_indexes_are_valid` only verifies wide-row targets,
so malformed serialized DFA data with unsorted, overlapping, or inverted ranges
can still be accepted. Update the validation path in
`LexerDFA::from_serialized`/`table_indexes_are_valid` to also inspect each wide
row’s range ordering and bounds before trusting the data, and reject the table
if `char_target` could binary-search ambiguous or invalid ranges; keep the
existing target checks and add range-shape validation for `wide_rows`.
---
Outside diff comments:
In `@tools/parse-bench/run.py`:
- Around line 616-637: probe_source() is using the same parser instance that
later gets benchmarked, which adds an implicit untimed warmup even when
--warmups is 0. Update main() to probe input with a throwaway parser created via
get_parser(SPECS[args.language]), then instantiate a fresh parser for the timed
loop and use that one for parse_once() iterations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: e7726beb-ef17-4f05-bbca-ca4b7ffee536
📒 Files selected for processing (7)
src/atn/lexer.rssrc/atn/lexer_dfa.rssrc/atn/mod.rssrc/bin/antlr4-runtime-testsuite.rssrc/bin/antlr4-rust-gen.rssrc/lexer.rstools/parse-bench/run.py
find_action_open_brace and last_rule_header_colon carried two verbatim copies of the comment/string/char-set skipping state machine (CPD's largest finding on this file). Both now drive a single GrammarSourceCursor that yields structurally significant characters, so the skip rules cannot drift apart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The largest CPD finding (the 70-line duplication at antlr4-rust-gen.rs:4913/5089) is resolved in 98a0cfd: |
table_indexes_are_valid promised that corrupted embedded streams degrade to runtime compilation, but only checked transition targets: a wide row with unsorted, overlapping, or inverted ranges would pass validation and make char_target's binary search silently misroute transitions. Validation now also requires each wide row to be well-formed, sorted, and disjoint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/bin/antlr4-rust-gen.rs (1)
5133-5154: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winSkip comments in
matching_action_brace. It ignores strings and char literals, but not//or/* */, so braces inside a comment can misplace the closing-brace match and throw offlast_rule_header_colon.src/bin_support/templates.rs:163🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/bin/antlr4-rust-gen.rs` around lines 5133 - 5154, The brace matcher used by last_rule_header_colon is still sensitive to comment text, so braces inside // or /* */ can break matching_action_brace and shift the colon scan. Update matching_action_brace to skip over line and block comments the same way it already ignores strings and char literals, then verify last_rule_header_colon still uses the returned close position correctly when advancing the GrammarSourceCursor.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/bin/antlr4-rust-gen.rs`:
- Around line 5133-5154: The brace matcher used by last_rule_header_colon is
still sensitive to comment text, so braces inside // or /* */ can break
matching_action_brace and shift the colon scan. Update matching_action_brace to
skip over line and block comments the same way it already ignores strings and
char literals, then verify last_rule_header_colon still uses the returned close
position correctly when advancing the GrammarSourceCursor.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 848b20fe-23b6-4a4d-9ee3-0f8c36dec4a5
📒 Files selected for processing (1)
src/bin/antlr4-rust-gen.rs
The performance section claimed lexer DFAs are learned and shared across recognizer instances; they are now compiled at generation time and embedded in the generated lexer, so tokenization has no warmup. Also documents the new codegen step, lists the compiled tables among runtime capabilities, and bumps the dependency snippet to the current 0.6 release line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
tinovyatkin has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
1 participant
Summary
ANTLR lexers discover their DFA lazily: the ATN simulator computes epsilon closures per character and caches the resulting config sets per process (and per thread). This PR runs that subset construction ahead of time inside
antlr4-rust-gen, embeds the finished tables in the generated lexer as a serializedu32stream, and matches tokens with one array lookup per character — no closure computation, hashing, or config allocation on the hot path, and no warmup on first parse.src/atn/lexer_dfa.rs(new): the compiler. Subset construction per mode over the lexer ATN, dense ASCII rows + binary-searched Unicode ranges + EOF edges, with row pooling. It reuses the interpreter's own closure/pruning/accept-selection code, so a compiled walk reproduces interpreter behavior exactly.src/atn/lexer.rs: a compiled-table token matcher that plugs into the existingnext_tokenouter loop (emission, skip/more, mode handling all shared). New entry pointsnext_token_compiled/next_token_compiled_with_hooks; every existing entry point is unchanged.src/bin/antlr4-rust-gen.rs: compiles + serializes the DFA at generation time; the generated lexer deserializes it once per process (sub-ms). The stream is version-tagged and falls back to runtime compilation, then to interpretation.Escape edges instead of per-mode bailouts
Constructs a finite DFA cannot represent compile as escape edges: a walk that reaches one re-matches that single token through the ATN interpreter. This covers semantic predicates, recursive lexer rules (Kotlin's nested
DelimitedComment), position-dependent custom-action traces, and the state budget — so one dynamic rule never disqualifies the rest of its mode. C#'s predicate-guarded interpolation mode stays interpreted while its other four modes compile.Two state-explosion sources had to be normalized for real grammars to determinize at all:
lexer_action_belongs_to_accept). They made DFA-state identity input-offset-dependent — one fresh state per character. Pruning them during construction is behavior-preserving and collapsed the Kotlin default mode from >65k states (non-converging) to 581.Resulting sizes: Kotlin 1,325 states across all 5 modes, Java 489, Trino 1,226, C# 1,068.
Correctness
antlr4-python3-runtime, all casesshowDFAdescriptors print the learned-DFA trace, which only exists under interpretation — the harness now opts those lexers out via the newBaseLexer::set_force_interpretedPerformance (Apple Silicon, parse-bench Kotlin fixtures, min of 20 iters)
The 2–7% deltas are on parse-bench's deliberately parser-prediction-heavy stress fixtures, where the lexer is a small slice of total time. On the lexer-heavier kotlin-parity snippets the improvement reaches ~20%. Two structural wins don't show in steady-state numbers at all:
thread_local— multi-threaded consumers previously re-warmed the lexer DFA per thread.Size cost, measured
The eye-catching 128KB → 1.4MB jump is the generated source text for the Kotlin lexer module (decimal literals are a verbose encoding). What ships is the table payload. Measured on the kotlin-parity dumper, a complete release binary (lexer + parser + runtime + driver):
__TEXT,__constThe payload scales with DFA states × distinct edge rows — Unicode identifier classes are the multiplier, so Kotlin is the worst case, not the typical one:
For calibration against the benchmark's reference point: the official
tree-sitter-kotlingrammar lib is 3.3 MB, of which 3.27 MB is__const— its LR tables (plus ~314 KB core runtime). Precompiled tables are how table-driven parsers buy their speed; this PR joins that trade at ~4× smaller table size than tree-sitter pays for the same language, because only the lexer is table-compiled.If a size-sensitive consumer appears, the fallback path makes an opt-out nearly free: a
--no-embedded-lexer-dfacodegen flag would skip the embedded stream and compile the same DFA at first use (~130 ms one-time for Kotlin). Not included here to keep the change focused.Also in this PR
bench_tree_sitter.pynow handlestree-sitter-language-packwheels that ship the Rust-native binding (strinput, method accessors instead of attributes), so parse-bench's tree-sitter runner works on macOS arm64 as well as the Linux CI wheels.🤖 Generated with Claude Code