[codex] Optimize lexer input and position hot paths

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## Unreleased
 
+### Performance
+
+- Compiled lexers read in-memory ASCII directly from their static DFA tables
+  and commit accepted spans in bulk. Optional `CharStream` fast paths preserve
+  scalar fallback behavior for custom streams and Unicode input.
+
 ### Breaking
 
 - Buffered tokens now live once in a compact `TokenStore` and are addressed by

docs/issue-78-lexer-benchmark.md

@@ -0,0 +1,98 @@
+# Issue 78 lexer fast-path benchmark record
+
+Measurements were taken on 2026-07-17 on an Apple M3 Pro with Rust 1.96.0.
+The baseline was `origin/main` at `73f33a407`; generated lexers and parsers
+were rebuilt with the matching generator/runtime for each revision. The
+grammars-v4 checkout was
+`284602b3f23ca54dc30778204ab7ae9e969145e9`.
+
+## Current-main finding
+
+Ahead-of-time DFA compilation already removes ATN closure, hashing, and config
+allocation from ordinary lexer matching. Issue #78 still applied after that
+work in three places:
+
+- each compiled-DFA symbol read still changed the shared cursor with
+  `seek(position)` followed by `la(1)`;
+- each accepted or recovered span was replayed through `consume_char()` to
+  rebuild line and column;
+- position queries and accept rewinds still used higher-level text or stream
+  operations.
+
+The scalar change therefore keeps the compiled DFA and lexer lifecycle model
+intact. It adds optional immutable-access and position-summary methods to
+`CharStream`, specializes the compiled ASCII walk, and centralizes accepted
+position commits. Streams that do not implement the optional methods retain
+the original scalar fallback.
+
+## Lex-only results
+
+The four configurations were built with the lex-only benchmark runner:
+
+1. `main`;
+2. the scalar fast paths with the ordinary release profile;
+3. the scalar fast paths plus `-C target-cpu=native`;
+4. the scalar fast paths plus ThinLTO and one codegen unit.
+
+After all builds completed, the already-built runners were measured in eight
+rotating, alternating process rounds per fixture. Each process used 20 warmups
+and 100 timed lexes. The table reports the median process average across all 19
+fixtures; ratios below one are faster.
+
+| Configuration | Geometric ratio vs main | Aggregate ratio vs main |
+|---|---:|---:|
+| scalar release | 0.8853x | 0.8642x |
+| scalar + native CPU | 0.8772x | 0.8671x |
+| scalar + ThinLTO / one codegen unit | 0.7725x | 0.7450x |
+
+Native CPU tuning was effectively neutral relative to the ordinary scalar
+build (`0.9908x` geometric, `1.0034x` aggregate). ThinLTO and one codegen unit
+improved on the ordinary scalar build by a further `0.8725x` geometric and
+`0.8621x` aggregate.
+
+The ordinary scalar build produced these per-language geometric ratios:
+
+| Fixtures | Count | Scalar vs main |
+|---|---:|---:|
+| Kotlin, including two lexer stress fixtures | 6 | 0.9173x |
+| C# | 4 | 0.8505x |
+| Java | 4 | 0.8598x |
+| Trino SQL | 5 | 0.8970x |
+
+Every source-derived fixture improved. The short Unicode stress fixture had
+overlapping 35-41 microsecond samples in the broad run, so it was repeated in
+15 alternating process pairs with 100 warmups and 5,000 timed lexes. Its
+median process average was 34.1 microseconds for the scalar build and 34.8
+microseconds for main (`0.9805x`). The ASCII stress fixture measured `0.8872x`
+in the broad run.
+
+## End-to-end parse results
+
+The same ordinary baseline and scalar binaries were measured over the 17
+source-derived fixtures in six alternating process rounds, each with 5
+warmups and 20 timed parses.
+
+| Fixtures | Scalar vs main |
+|---|---:|
+| Kotlin | 0.9977x |
+| C# | 0.9946x |
+| Java | 1.0025x |
+| Trino SQL | 0.9900x |
+| **Geometric mean** | **0.9958x** |
+| **Aggregate time** | **0.9961x** |
+
+The largest fixture ratio was `1.0182x` on the Java Trino filter fixture, so
+all 17 fixtures remained within the 2% regression threshold.
+
+## Fast-path counters
+
+A three-iteration instrumentation run demonstrated that the two synthetic
+fixtures use the intended paths:
+
+| Fixture | Direct ASCII reads | Generic reads | Scalar replay | Bulk committed |
+|---|---:|---:|---:|---:|
+| ASCII stress | 6,057 | 0 | 0 | 5,193 |
+| Unicode fallback | 0 | 2,181 | 0 | 1,845 |
+
+The Unicode stream remains indexed by scalar value. The generic count records
+immutable scalar lookups; it does not indicate cursor mutation or replay.

src/atn/lexer.rs

@@ -717,24 +717,19 @@ where
         let accept = match token_match {
             MatchResult::Accept(accept) => accept,
             MatchResult::NoViableAlt { stop } => {
-                lexer.input_mut().seek(start);
+                lexer.commit_position(start, start);
                 if lexer.input_mut().la(1) == EOF {
                     lexer.set_hit_eof(true);
                     return lexer.emit_eof_or_pending(sink);
                 }
                 record_token_recognition_error(lexer, start, stop);
-                while lexer.input().index() < stop {
-                    lexer.consume_char();
-                }
+                lexer.commit_position(start, stop);
                 continuing_more = false;
                 continue;
             }
         };
 
-        lexer.input_mut().seek(start);
-        while lexer.input().index() < accept.position {
-            lexer.consume_char();
-        }
+        lexer.commit_position(start, accept.position);
 
         let token_type = atn
             .rule_to_token_type()
@@ -1076,6 +1071,72 @@ fn match_token_compiled<I>(
     start_state: u16,
     start: usize,
 ) -> Option<MatchResult>
+where
+    I: CharStream,
+{
+    if let Some(input) = lexer.input().contiguous_ascii() {
+        return match_token_compiled_ascii(input, dfa, start_state, start);
+    }
+    match_token_compiled_generic(lexer, dfa, start_state, start)
+}
+
+fn match_token_compiled_ascii(
+    input: &[u8],
+    dfa: &CompiledLexerDfa,
+    start_state: u16,
+    start: usize,
+) -> Option<MatchResult> {
+    let mut state = start_state;
+    let mut position = start;
+    let mut best: Option<AcceptState> = None;
+    let mut error_stop = start;
+    let mut eof_edges = 0_u32;
+    #[cfg(feature = "perf-counters")]
+    let mut direct_chars = 0;
+    let result = loop {
+        if let Some(accept) = dfa.accept(state) {
+            record_compiled_accept(accept, position, &mut best);
+        }
+        let (target, at_eof) = if position < input.len() {
+            let symbol = input[position];
+            #[cfg(feature = "perf-counters")]
+            {
+                direct_chars += 1;
+            }
+            error_stop = error_stop.max(position.saturating_add(1));
+            (dfa.ascii_target(state, symbol), false)
+        } else {
+            eof_edges += 1;
+            if eof_edges > MAX_COMPILED_EOF_EDGES {
+                break None;
+            }
+            (dfa.eof_target(state), true)
+        };
+        if target == DEAD_STATE {
+            break Some(best.map_or(
+                MatchResult::NoViableAlt { stop: error_stop },
+                MatchResult::Accept,
+            ));
+        }
+        if target == ESCAPE_STATE {
+            break None;
+        }
+        if !at_eof {
+            position += 1;
+        }
+        state = target;
+    };
+    #[cfg(feature = "perf-counters")]
+    crate::perf::record_lexer_direct_ascii(direct_chars);
+    result
+}
+
+fn match_token_compiled_generic<I>(
+    lexer: &mut BaseLexer<I>,
+    dfa: &CompiledLexerDfa,
+    start_state: u16,
+    start: usize,
+) -> Option<MatchResult>
 where
     I: CharStream,
 {
@@ -1570,14 +1631,21 @@ fn display_error_text(text: &str) -> String {
 
 /// Reads the Unicode scalar value at an absolute character-stream index.
 ///
-/// The interpreter explores many paths at different input offsets, so it seeks
-/// the shared input stream before each lookahead instead of cloning the stream.
+/// Streams with immutable random access avoid touching their committed cursor;
+/// custom streams retain the compatible seek-and-lookahead path.
 fn symbol_at<I>(lexer: &mut BaseLexer<I>, position: usize) -> i32
 where
     I: CharStream,
 {
-    lexer.input_mut().seek(position);
-    lexer.input_mut().la(1)
+    let symbol = lexer.input().symbol_at(position).unwrap_or_else(|| {
+        lexer.input_mut().seek(position);
+        lexer.input_mut().la(1)
+    });
+    #[cfg(feature = "perf-counters")]
+    if symbol != EOF {
+        crate::perf::record_lexer_generic_char();
+    }
+    symbol
 }
 
 #[cfg(test)]

src/atn/lexer_dfa.rs

@@ -170,6 +170,13 @@ impl CompiledLexerDfa {
             .get(self.states[usize::from(state)].accept as usize)
     }
 
+    /// Target for one byte from a stream known to contain only ASCII.
+    pub(super) fn ascii_target(&self, state: u16, symbol: u8) -> u16 {
+        debug_assert!(symbol.is_ascii());
+        let compiled = &self.states[usize::from(state)];
+        self.ascii_rows[compiled.ascii_row as usize][usize::from(symbol)]
+    }
+
     /// `LexerTransition` target for a non-EOF symbol, or [`DEAD_STATE`].
     pub(super) fn char_target(&self, state: u16, symbol: i32) -> u16 {
         let compiled = &self.states[usize::from(state)];
@@ -921,7 +928,8 @@ mod tests {
     use super::*;
     use crate::atn::lexer::{next_token, next_token_compiled, next_token_compiled_with_hooks};
     use crate::atn::serialized::{AtnDeserializer, SerializedAtn};
-    use crate::char_stream::InputStream;
+    use crate::char_stream::{CharStream, InputStream, TextInterval};
+    use crate::int_stream::IntStream;
     use crate::lexer::BaseLexer;
     use crate::recognizer::RecognizerData;
     use crate::token::{TOKEN_EOF, Token, TokenSink, TokenStore};
@@ -931,20 +939,27 @@ mod tests {
     struct TokenSnapshot {
         token_type: i32,
         text: String,
+        line: usize,
+        column: usize,
     }
 
-    fn compiled_token(
-        lexer: &mut BaseLexer<InputStream>,
+    fn compiled_token<I>(
+        lexer: &mut BaseLexer<I>,
         atn: &LexerAtn,
         dfa: &CompiledLexerDfa,
-    ) -> TokenSnapshot {
+    ) -> TokenSnapshot
+    where
+        I: CharStream,
+    {
         let mut store = TokenStore::new(lexer.source_text(), lexer.source_name());
         let mut sink = TokenSink::new(&mut store);
         let id = next_token_compiled(lexer, &mut sink, atn, dfa).expect("test token should fit");
         let token = sink.view(id).expect("emitted token should exist");
         TokenSnapshot {
             token_type: token.token_type(),
             text: token.text().to_owned(),
+            line: token.line(),
+            column: token.column(),
         }
     }
 
@@ -956,6 +971,44 @@ mod tests {
         TokenSnapshot {
             token_type: token.token_type(),
             text: token.text().to_owned(),
+            line: token.line(),
+            column: token.column(),
+        }
+    }
+
+    #[derive(Clone, Debug)]
+    struct FallbackInput(InputStream);
+
+    impl IntStream for FallbackInput {
+        fn consume(&mut self) {
+            self.0.consume();
+        }
+
+        fn la(&mut self, offset: isize) -> i32 {
+            self.0.la(offset)
+        }
+
+        fn index(&self) -> usize {
+            self.0.index()
+        }
+
+        fn seek(&mut self, index: usize) {
+            self.0.seek(index);
+        }
+
+        fn size(&self) -> usize {
+            self.0.size()
+        }
+
+        fn source_name(&self) -> &str {
+            self.0.source_name()
+        }
+    }
+
+    // Deliberately implements none of the optional fast paths.
+    impl CharStream for FallbackInput {
+        fn text(&self, interval: TextInterval) -> String {
+            self.0.text(interval)
         }
     }
 
@@ -1103,6 +1156,58 @@ mod tests {
         assert_eq!(compiled_token(&mut lexer, &atn, &dfa).token_type, TOKEN_EOF);
     }
 
+    #[test]
+    fn compiled_dfa_keeps_custom_streams_on_the_compatible_fallback() {
+        let atn = two_rule_atn(false);
+        let dfa = CompiledLexerDfa::compile(&atn);
+        let mut lexer = BaseLexer::new(FallbackInput(InputStream::new(" ab")), recognizer_data());
+
+        let token = compiled_token(&mut lexer, &atn, &dfa);
+        assert_eq!(token.token_type, 1);
+        assert_eq!(token.text, "ab");
+        assert_eq!((token.line, token.column), (1, 1));
+        assert_eq!(lexer.input().index(), 3);
+    }
+
+    #[cfg(feature = "perf-counters")]
+    #[test]
+    fn lexer_counters_distinguish_ascii_unicode_and_replay_paths() {
+        let ascii_atn = two_rule_atn(false);
+        let ascii_dfa = CompiledLexerDfa::compile(&ascii_atn);
+        crate::perf::reset();
+        let mut ascii = BaseLexer::new(InputStream::new(" ab"), recognizer_data());
+        let token = compiled_token(&mut ascii, &ascii_atn, &ascii_dfa);
+        assert_eq!(token.text, "ab");
+        let [direct, generic, replay, bulk] = crate::perf::lexer_snapshot();
+        assert!(direct >= 3, "{direct}");
+        assert_eq!(generic, 0);
+        assert_eq!(replay, 0);
+        assert_eq!(bulk, 3);
+
+        let unicode_atn = wide_range_atn();
+        let unicode_dfa = CompiledLexerDfa::compile(&unicode_atn);
+        crate::perf::reset();
+        let mut unicode = BaseLexer::new(InputStream::new("ĀĂ"), recognizer_data());
+        let token = compiled_token(&mut unicode, &unicode_atn, &unicode_dfa);
+        assert_eq!(token.text, "ĀĂ");
+        let [direct, generic, replay, bulk] = crate::perf::lexer_snapshot();
+        assert_eq!(direct, 0);
+        assert!(generic >= 2, "{generic}");
+        assert_eq!(replay, 0);
+        assert_eq!(bulk, 2);
+
+        crate::perf::reset();
+        let mut fallback =
+            BaseLexer::new(FallbackInput(InputStream::new(" ab")), recognizer_data());
+        let token = compiled_token(&mut fallback, &ascii_atn, &ascii_dfa);
+        assert_eq!(token.text, "ab");
+        let [direct, generic, replay, bulk] = crate::perf::lexer_snapshot();
+        assert_eq!(direct, 0);
+        assert!(generic >= 3, "{generic}");
+        assert_eq!(replay, 3);
+        assert_eq!(bulk, 0);
+    }
+
     #[test]
     fn compiled_dfa_reports_recognition_errors_like_the_interpreter() {
         let atn = wide_range_atn();