From 8a5422b0faf031179c0822dd1646d4bdd14cc9fd Mon Sep 17 00:00:00 2001 From: Konstantin Vyatkin Date: Fri, 24 Jul 2026 17:15:17 +0200 Subject: [PATCH 1/2] feat(runtime): add ByteStream for binary parsing + MIDI example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce `ByteStream`, a byte-oriented `CharStream` for parsing binary formats. Each byte is one symbol in `0..=255` and the stream index is the byte offset, so grammars can use the Latin-1 convention (`BYTE: ' '..'ÿ'`) that ANTLR's reference runtimes use for binary input, without the transcoding and per-byte allocation `InputStream` incurs on non-UTF-8 bytes. `ByteStream>` is generic over `AsRef<[u8]>` to map onto Rust IO primitives: `new(vec)` owns, `new(&buf[..])` borrows a network buffer zero-copy, and `from_reader(r)?` drains any `std::io::Read` (files, sockets, stdin, `Cursor`). Because the bytes are not text, `text()` renders a matched span as lowercase hex. The ASCII fast-path (`contiguous_ascii`) is deliberately not implemented: it feeds a 128-wide DFA row, so bytes >= 0x80 route through the generic path instead. Add a worked binary-parsing example under tests/fixtures/antlr4-rust-gen/midi-binary/: a Standard MIDI File grammar (MThd/MTrk chunks, VLQ delta-times, note and meta events) plus a small `SemanticHooks` chunk-framing "superClass" that counts down each chunk's declared byte length and synthesizes END_OF_CHUNK -- the data-dependent "read N, then N bytes" pattern ANTLR's bencoding grammar solves the same way. The bare `{beginChunk();}` lexer action lowers to a typed hook via a --sem-patterns helper. An integration test generates the recognizer from the committed grammar and parses a real .mid fixture over a ByteStream with zero syntax errors. The grammar is adapted (simplified) from milnet2/midi-grammar by Tobias Blaschke (BSD-3-Clause). README gains a "Binary and Byte-Oriented Parsing" section documenting the approach. --- README.md | 33 ++ src/byte_stream.rs | 282 ++++++++++++++++++ src/lib.rs | 2 + tests/antlr4_rust_gen_cli.rs | 133 +++++++++ .../antlr4-rust-gen/midi-binary/MidiLexer.g4 | 65 ++++ .../antlr4-rust-gen/midi-binary/MidiParser.g4 | 32 ++ .../midi-binary/make_fixture.py | 56 ++++ .../antlr4-rust-gen/midi-binary/patterns.toml | 14 + .../antlr4-rust-gen/midi-binary/twinkle.mid | Bin 0 -> 41 bytes 9 files changed, 617 insertions(+) create mode 100644 src/byte_stream.rs create mode 100644 tests/fixtures/antlr4-rust-gen/midi-binary/MidiLexer.g4 create mode 100644 tests/fixtures/antlr4-rust-gen/midi-binary/MidiParser.g4 create mode 100644 tests/fixtures/antlr4-rust-gen/midi-binary/make_fixture.py create mode 100644 tests/fixtures/antlr4-rust-gen/midi-binary/patterns.toml create mode 100644 tests/fixtures/antlr4-rust-gen/midi-binary/twinkle.mid diff --git a/README.md b/README.md index 23c71ec1..9fe5c168 100644 --- a/README.md +++ b/README.md @@ -456,6 +456,39 @@ inline at their ATN action/predicate coordinates. This is the mode the conformance harness uses after rendering descriptor grammars through `Rust.test.stg` (see below). +### Binary and Byte-Oriented Parsing + +ANTLR grammars can parse binary formats, not just text. The convention the +reference runtimes use is to treat each byte as a codepoint in +`U+0000..=U+00FF` and write lexer rules over that range +(`BYTE : '\u0000' .. '\u00FF';`). This runtime ships +[`ByteStream`](src/byte_stream.rs) for exactly that: a `CharStream` backed by +raw bytes where the stream index is the byte offset and lookahead returns the +byte value (`0..=255`). It is generic over the +backing store — `ByteStream::new(vec)` owns, `ByteStream::new(&buf[..])` borrows +a network buffer zero-copy, and `ByteStream::from_reader(file)?` drains any +`std::io::Read`. Because the bytes are not text, `text()` renders a matched span +as lowercase hex. + +Length-prefixed formats ("read N, then consume N bytes") are data-dependent, so +a pure grammar cannot frame them alone — the same constraint ANTLR's `bencoding` +grammar solves with a lexer `superClass`. Here that role is filled by a +[`SemanticHooks`](src/parser.rs) implementation: `LexerSemCtx`/`LexerLifecycleCtx` +expose `push_mode`/`pop_mode`, `enqueue_token` (to synthesize framing tokens), +and raw `la()` lookbehind, so a small hook struct can count down a declared +chunk length and emit an end-of-chunk token. A bare `{helper();}` lexer action +lowers to a typed hook method via a `--sem-patterns` `[[helper]]` entry with +`kind = "lexer-action"`, `lower = "hook"`. + +A complete worked example — a Standard MIDI File grammar (MThd/MTrk chunks, +variable-length delta-times, note and meta events) with a chunk-framing hook, +parsed over a `ByteStream` from a real `.mid` fixture — lives in +[`tests/fixtures/antlr4-rust-gen/midi-binary/`](tests/fixtures/antlr4-rust-gen/midi-binary/) +and its integration test (`midi_binary_grammar_parses_standard_midi_file_over_byte_stream` +in [tests/antlr4_rust_gen_cli.rs](tests/antlr4_rust_gen_cli.rs)). The grammar is +adapted from [milnet2/midi-grammar](https://github.com/milnet2/midi-grammar) +(Tobias Blaschke, BSD-3-Clause). + ## Runtime Testsuite On the maintainer checkout, where the ANTLR jar and upstream runtime-testsuite diff --git a/src/byte_stream.rs b/src/byte_stream.rs new file mode 100644 index 00000000..c86b1125 --- /dev/null +++ b/src/byte_stream.rs @@ -0,0 +1,282 @@ +//! A byte-oriented [`CharStream`] for parsing binary formats. +//! +//! ANTLR grammars normally consume Unicode text, but many real-world formats +//! are raw bytes: chunk containers (RIFF/WAV), fixed-width records (tar), and +//! self-describing tag streams (CBOR, Standard MIDI). The reference runtimes +//! parse these by treating each byte as a codepoint in `U+0000..=U+00FF` (a +//! "Latin-1" view) and writing lexer rules over `' '..'ÿ'`. +//! +//! [`InputStream`](crate::InputStream) can do this too, but only after decoding +//! the bytes into a `String`: any byte `>= 0x80` is not valid UTF-8 on its own, +//! so the whole input takes the non-ASCII path and is materialized into a +//! `Vec` plus a byte-offset table — roughly 12 bytes of heap per input +//! byte, and the compiled-DFA ASCII scanner is disabled. +//! +//! `ByteStream` avoids all of that. It is generic over any `AsRef<[u8]>` +//! backing store, so stream index equals byte offset, lookahead is a single +//! array read, and there is no transcoding or auxiliary allocation. +//! +//! # Mapping to Rust IO primitives +//! +//! ANTLR parsing needs random access — the lexer and parser `seek`, look +//! behind with `la(-1)`, and `mark`/`release` for prediction — so the bytes +//! must live fully in memory; `ByteStream` cannot lazily pull from a socket +//! mid-parse. The design instead meets the two IO shapes that matter: +//! +//! - **Bytes you already hold** (a network read buffer, an `mmap`, a slice of a +//! larger frame): borrow them zero-copy with `ByteStream::new(&buf[..])`. +//! Nothing is copied; the stream lives as long as the borrow. +//! - **A reader** (`File`, `TcpStream`, `Stdin`, `Cursor`): drain it into an +//! owned buffer with [`ByteStream::from_reader`], which is just a thin +//! wrapper over [`std::io::Read::read_to_end`]. +//! - **An owned `Vec`**: hand it over with `ByteStream::new(vec)` and the +//! stream takes ownership without copying. +//! +//! ```ignore +//! // From a file: +//! let stream = ByteStream::from_reader(std::fs::File::open(path)?)?; +//! // Zero-copy from an in-memory buffer (e.g. bytes read off a socket): +//! let stream = ByteStream::new(&packet[..]); +//! +//! let lexer = MidiLexer::new(stream); +//! let tokens = CommonTokenStream::new(lexer); +//! let mut parser = MidiParser::new(tokens); +//! let tree = parser.file()?; +//! ``` +//! +//! Write lexer rules against the byte range, e.g. `BYTE : ' ' .. 'ÿ';`. +//! +//! # Token text is hex +//! +//! Because the bytes are not text, [`CharStream::text`] renders the matched +//! span as a lowercase hex string with no separators (`[0xDE, 0xAD]` becomes +//! `"dead"`). Token *positions* are still exact byte offsets; use +//! [`IntStream::index`](crate::IntStream::index) or a token's byte span when you +//! need to slice the original bytes. + +use std::io; + +use crate::char_stream::{CharStream, TextInterval}; +use crate::int_stream::{EOF, IntStream, UNKNOWN_SOURCE_NAME}; + +/// A [`CharStream`] backed by raw bytes, where each byte is one symbol in +/// `0..=255` and the stream index is the byte offset. +/// +/// Generic over the backing store `B: AsRef<[u8]>`: use `Vec` for owned +/// bytes or `&[u8]` to borrow an existing buffer zero-copy. See the +/// [module documentation](self) for how this maps onto Rust IO primitives. +#[derive(Clone, Debug)] +pub struct ByteStream> { + bytes: B, + cursor: usize, + source_name: String, +} + +impl ByteStream> { + /// Creates a byte stream by draining a [`std::io::Read`] into an owned + /// buffer — the bridge for files, sockets, stdin, and [`std::io::Cursor`]. + /// + /// # Errors + /// + /// Returns any error produced while reading `reader` to end. + pub fn from_reader(mut reader: impl io::Read) -> io::Result { + let mut bytes = Vec::new(); + reader.read_to_end(&mut bytes)?; + Ok(Self::new(bytes)) + } +} + +impl> ByteStream { + /// Creates a byte stream over `bytes`, using ANTLR's unknown source-name + /// placeholder. + /// + /// `bytes` may be an owned `Vec` or a borrowed `&[u8]` (zero-copy). + pub fn new(bytes: B) -> Self { + Self::with_source_name(bytes, UNKNOWN_SOURCE_NAME) + } + + /// Creates a byte stream with an explicit source name for tokens and + /// diagnostics. + pub fn with_source_name(bytes: B, source_name: impl Into) -> Self { + Self { + bytes, + cursor: 0, + source_name: source_name.into(), + } + } + + /// Returns the backing bytes. + #[must_use] + pub fn bytes(&self) -> &[u8] { + self.bytes.as_ref() + } + + /// Returns true when the cursor has reached or passed the end of input. + #[must_use] + pub fn is_eof(&self) -> bool { + self.cursor >= self.bytes.as_ref().len() + } +} + +impl> IntStream for ByteStream { + fn consume(&mut self) { + if !self.is_eof() { + self.cursor += 1; + } + } + + fn la(&mut self, offset: isize) -> i32 { + if offset == 0 { + return 0; + } + + // Mirror `InputStream::la`: `+1` is the symbol under the cursor, and + // negative offsets look behind. `checked_*` keeps `isize::MIN` and + // out-of-range lookahead on the EOF path instead of panicking. + let absolute = if offset > 0 { + self.cursor.checked_add((offset - 1).cast_unsigned()) + } else { + offset + .checked_neg() + .and_then(|distance| usize::try_from(distance).ok()) + .and_then(|distance| self.cursor.checked_sub(distance)) + }; + + absolute.map_or(EOF, |index| self.symbol_at(index).unwrap_or(EOF)) + } + + fn index(&self) -> usize { + self.cursor + } + + fn seek(&mut self, index: usize) { + self.cursor = index.min(self.bytes.as_ref().len()); + } + + fn size(&self) -> usize { + self.bytes.as_ref().len() + } + + fn source_name(&self) -> &str { + &self.source_name + } +} + +impl> CharStream for ByteStream { + /// Renders the inclusive byte interval as a lowercase, separator-free hex + /// string. See the [module documentation](self) for the rationale. + fn text(&self, interval: TextInterval) -> String { + let bytes = self.bytes.as_ref(); + if interval.is_empty() { + return String::new(); + } + let stop = (interval.stop + 1).min(bytes.len()); + let start = interval.start.min(stop); + use std::fmt::Write as _; + bytes[start..stop].iter().fold( + String::with_capacity((stop - start) * 2), + |mut acc, byte| { + // Writing to a String is infallible. + let _ = write!(acc, "{byte:02x}"); + acc + }, + ) + } + + fn symbol_at(&self, index: usize) -> Option { + Some( + self.bytes + .as_ref() + .get(index) + .map_or(EOF, |&byte| i32::from(byte)), + ) + } + + // NOTE: `contiguous_ascii` is deliberately NOT implemented. That fast path + // feeds bytes into a 128-wide ASCII DFA row (`ascii_target`), which is only + // valid for 7-bit input; bytes `>= 0x80` route correctly through the + // generic path's `wide_rows` instead. + + fn byte_interval(&self, interval: TextInterval) -> Option<(usize, usize)> { + // Index == byte offset, so the byte span is exact. + let len = self.bytes.as_ref().len(); + if interval.is_empty() { + let at = self.cursor.min(len); + return Some((at, at)); + } + let stop = (interval.stop + 1).min(len); + let start = interval.start.min(stop); + Some((start, stop)) + } +} + +#[cfg(test)] +#[allow(clippy::disallowed_methods)] // insta assertion macros unwrap internal I/O. +mod tests { + use super::*; + + #[test] + fn lookahead_reads_bytes_including_high_bytes() { + let mut stream = ByteStream::new(vec![0x00, 0x7F, 0x80, 0xFF]); + assert_eq!(stream.la(0), 0, "la(0) is the ANTLR sentinel, not EOF"); + assert_eq!(stream.la(1), 0x00); + assert_eq!(stream.la(2), 0x7F); + assert_eq!(stream.la(3), 0x80, "high byte is 128, not sign-extended"); + assert_eq!(stream.la(4), 0xFF); + assert_eq!(stream.la(5), EOF); + stream.consume(); + assert_eq!(stream.index(), 1); + assert_eq!(stream.la(-1), 0x00); + assert_eq!(stream.la(isize::MIN), EOF, "no panic on extreme offset"); + } + + #[test] + fn consume_stops_at_eof_and_seek_clamps() { + let mut stream = ByteStream::new(vec![0x01, 0x02]); + assert_eq!(stream.size(), 2); + stream.consume(); + stream.consume(); + stream.consume(); // past EOF is a no-op + assert_eq!(stream.index(), 2); + assert!(stream.is_eof()); + stream.seek(99); + assert_eq!(stream.index(), 2, "seek clamps to size"); + stream.seek(1); + assert_eq!(stream.la(1), 0x02); + } + + #[test] + fn text_is_lowercase_hex_and_byte_interval_is_exact() { + let stream = ByteStream::new(vec![0xDE, 0xAD, 0xBE, 0xEF]); + assert_eq!(stream.text(TextInterval::new(0, 3)), "deadbeef"); + assert_eq!(stream.text(TextInterval::new(1, 2)), "adbe"); + assert_eq!(stream.text(TextInterval::empty()), ""); + // Inclusive char interval [1, 2] -> half-open byte span [1, 3). + assert_eq!(stream.byte_interval(TextInterval::new(1, 2)), Some((1, 3))); + assert_eq!(stream.symbol_at(0), Some(0xDE)); + assert_eq!(stream.symbol_at(4), Some(EOF)); + } + + #[test] + fn borrows_bytes_zero_copy() { + // The network-buffer case: parse a slice we already hold without + // handing ownership to the stream. + let buffer: [u8; 4] = [0xCA, 0xFE, 0xBA, 0xBE]; + let mut stream = ByteStream::new(&buffer[..]); + assert_eq!(stream.la(1), 0xCA); + assert_eq!(stream.size(), 4); + // `buffer` is still ours afterwards. + assert_eq!(buffer[0], 0xCA); + } + + #[test] + fn from_reader_drains_any_read() { + // The file/socket case, exercised here with an in-memory Cursor that + // implements the same `io::Read` contract as `File`/`TcpStream`. + let source = io::Cursor::new(vec![0x4D, 0x54, 0x68, 0x64]); // "MThd" + let mut stream = ByteStream::from_reader(source).expect("cursor read is infallible"); + assert_eq!(stream.size(), 4); + assert_eq!(stream.la(1), 0x4D); + assert_eq!(stream.text(TextInterval::new(0, 3)), "4d546864"); + } +} diff --git a/src/lib.rs b/src/lib.rs index e770514c..cbbce5d0 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -3,6 +3,7 @@ extern crate self as antlr4_runtime; pub mod atn; +pub mod byte_stream; pub mod char_stream; pub mod dfa; pub mod errors; @@ -22,6 +23,7 @@ pub mod vocabulary; pub mod xpath; pub use atn::parser::{ParserAtnPrediction, ParserAtnSimulator, ParserAtnSimulatorError}; +pub use byte_stream::ByteStream; pub use char_stream::{CharStream, InputStream, PositionSummary, TextInterval}; pub use dfa::{DfaStateId, DfaTransition, ParserDfa, ParserDfaStateView, ParserDfaStats}; pub use errors::{AntlrError, ConsoleErrorListener, ErrorListener}; diff --git a/tests/antlr4_rust_gen_cli.rs b/tests/antlr4_rust_gen_cli.rs index 22cea405..f14d287f 100644 --- a/tests/antlr4_rust_gen_cli.rs +++ b/tests/antlr4_rust_gen_cli.rs @@ -1281,3 +1281,136 @@ fn unsupported_grammar_options_warn_and_exact_hooks_acknowledge_them() { ); assert!(acknowledged_out.join("options_lexer.rs").is_file()); } + +/// End-to-end binary-parsing example: generate the MIDI recognizer from the +/// committed byte-oriented grammar, then parse a real Standard MIDI File +/// through a `ByteStream`. Exercises the whole binary path — raw high bytes as +/// codepoints, a `SemanticHooks` chunk-length superClass emitting synthesized +/// `END_OF_CHUNK` tokens (the `bencoding` pattern), and the generated parser. +#[test] +fn midi_binary_grammar_parses_standard_midi_file_over_byte_stream() { + let temp = temporary_directory("midi-binary"); + let dir = + Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/antlr4-rust-gen/midi-binary"); + let out = temp.path().join("generated"); + + let output = run_antlr4_rust_gen(&[ + dir.join("MidiLexer.g4").as_os_str(), + dir.join("MidiParser.g4").as_os_str(), + OsStr::new("--sem-patterns"), + dir.join("patterns.toml").as_os_str(), + OsStr::new("--out-dir"), + out.as_os_str(), + ]); + assert!( + output.status.success(), + "stdout: {}\nstderr: {}", + utf8(&output.stdout), + utf8(&output.stderr) + ); + + // The bare `{beginChunk();}` lexer action lowers to a typed hook method. + let lexer = fs::read_to_string(out.join("midi_lexer.rs")).expect("lexer should be emitted"); + assert!(lexer.contains("pub trait MidiLexerHooks"), "{lexer}"); + assert!(lexer.contains("fn begin_chunk"), "{lexer}"); + + let fixture = dir.join("twinkle.mid"); + let fixture = fixture.to_str().expect("fixture path should be UTF-8"); + let test_source = format!( + r####" +#[cfg(test)] +mod midi_tests {{ + use super::midi_lexer::{{MidiLexer, MidiLexerHooks, END_OF_CHUNK}}; + use super::midi_parser::MidiParser; + use antlr4_runtime::{{ + ByteStream, CommonTokenStream, LexerLifecycleCtx, LexerSemCtx, Parser as _, Token as _, + }}; + + /// A minimal chunk-framing superClass: reads each MThd/MTrk chunk's declared + /// byte length and synthesizes END_OF_CHUNK once the body is consumed — the + /// "read N, then frame N bytes" pattern, in Rust, on plain `ByteStream`. + #[derive(Default)] + struct MidiHooks {{ + end_of_chunk: Option, + }} + + impl MidiLexerHooks for MidiHooks {{ + fn begin_chunk(&mut self, ctx: &mut LexerSemCtx<'_, I>) + where + I: antlr4_runtime::CharStream, + {{ + // The header token just matched magic(4) + big-endian length(4). + // Read the four length bytes RAW via lookbehind — `text_so_far()` + // would return `ByteStream`'s hex rendering, not the bytes. + let b3 = ctx.la(-4) as u32; + let b2 = ctx.la(-3) as u32; + let b1 = ctx.la(-2) as u32; + let b0 = ctx.la(-1) as u32; + let len = ((b3 << 24) | (b2 << 16) | (b1 << 8) | b0) as usize; + self.end_of_chunk = Some(ctx.position() + len); + }} + + fn lexer_before_token(&mut self, ctx: &mut LexerLifecycleCtx<'_, I>) + where + I: antlr4_runtime::CharStream, + {{ + // Fires after the previous body token was emitted and before the + // next match — the clean point to close the chunk so END_OF_CHUNK + // lands AFTER the last body token rather than inverting with it. + if let Some(end) = self.end_of_chunk {{ + let pos = ctx.input_position(); + if pos >= end {{ + self.end_of_chunk = None; + ctx.pop_mode(); + ctx.enqueue_token(END_OF_CHUNK, pos.saturating_sub(1)); + }} + }} + }} + }} + + fn parse(bytes: Vec) -> (Vec, usize) {{ + let lexer = MidiLexer::with_typed_hooks(ByteStream::new(bytes.clone()), MidiHooks::default()); + let mut stream = CommonTokenStream::new(lexer); + stream.fill(); + let types: Vec = stream.tokens().map(|t| t.token_type()).collect(); + + let lexer = MidiLexer::with_typed_hooks(ByteStream::new(bytes), MidiHooks::default()); + let mut parser = MidiParser::new(CommonTokenStream::new(lexer)); + parser.file().expect("well-formed MIDI parses"); + (types, parser.number_of_syntax_errors()) + }} + + #[test] + fn parses_a_real_standard_midi_file() {{ + let bytes = include_bytes!({fixture:?}).to_vec(); + let (types, errors) = parse(bytes); + + // BEGIN_HEADER, six HDR_BYTE, END_OF_CHUNK; BEGIN_TRACK, four + // (DELTA_TIME, event) pairs, END_OF_CHUNK; EOF (-1). + assert_eq!(errors, 0, "no syntax errors on a well-formed file"); + assert_eq!( + types, + vec![ + 2, // BEGIN_HEADER + 4, 4, 4, 4, 4, 4, // six HDR_BYTE (format, ntracks, division) + 1, // END_OF_CHUNK (MThd body framed by its length = 6) + 3, // BEGIN_TRACK + 5, 7, // delta, NOTE_ON + 5, 6, // delta, NOTE_OFF + 5, 9, // delta, META_SET_TEMPO + 5, 8, // delta, META_END_OF_TRACK + 1, // END_OF_CHUNK (MTrk body framed by its length = 19) + -1, // EOF + ], + ); + }} +}} +"#### + ); + + assert_generated_project( + temp.path(), + &["midi_lexer.rs", "midi_parser.rs"], + &test_source, + ); +} diff --git a/tests/fixtures/antlr4-rust-gen/midi-binary/MidiLexer.g4 b/tests/fixtures/antlr4-rust-gen/midi-binary/MidiLexer.g4 new file mode 100644 index 00000000..5b494bb3 --- /dev/null +++ b/tests/fixtures/antlr4-rust-gen/midi-binary/MidiLexer.g4 @@ -0,0 +1,65 @@ +// Standard MIDI File (SMF) lexer -- a worked example of byte-oriented parsing. +// +// Each byte is one symbol in U+0000..=U+00FF (the "Latin-1" view used by +// ANTLR's binary grammars); feed it through `ByteStream`, which maps raw bytes +// to exactly those codepoints. Character ranges use backslash-u escapes so the +// grammar stays plain ASCII on disk. +// +// The lexer leans on a small `SemanticHooks` implementation (`MidiHooks`, in +// the test) to frame each chunk by its declared length -- the "read N, then N +// bytes" pattern that ANTLR's `bencoding` grammar solves with a lexer +// superClass. Everything else is plain grammar: a track body cycles between a +// DELTA mode (which matches one variable-length-quantity delta-time) and an +// EVENT mode (which matches one whole event as a single fixed-width token). +// Matching each event whole avoids the classic MIDI ambiguity where a status +// byte and a VLQ continuation byte share the range 0x80..0xFF -- they never +// share a lexer mode here. +// +// Adapted (simplified) from milnet2/midi-grammar by Tobias Blaschke, BSD-3: +// https://github.com/milnet2/midi-grammar. Scope is deliberately small: MThd + +// MTrk chunks, VLQ delta-times, note-on/off, and set-tempo / end-of-track meta +// events. Running status, sysex, and most meta events are out of scope. + +lexer grammar MidiLexer; + +tokens { + // Synthesized by MidiHooks when a chunk's declared byte length is reached. + END_OF_CHUNK +} + +// Chunk headers: 4-byte magic + 4-byte big-endian length. `{beginChunk();}` +// tells MidiHooks to start counting down the chunk body from the length bytes. +BEGIN_HEADER : 'MThd' BYTE BYTE BYTE BYTE {beginChunk();} -> pushMode(HEADER_BODY); +BEGIN_TRACK : 'MTrk' BYTE BYTE BYTE BYTE {beginChunk();} -> pushMode(DELTA); + +fragment BYTE : '\u0000' .. '\u00FF'; + +// --------------------------------------------------------------------------- +// MThd body: format (u16), ntracks (u16), division (u16). MidiHooks emits +// END_OF_CHUNK after the sixth byte, which pops back to the default mode. +mode HEADER_BODY; + +HDR_BYTE : '\u0000' .. '\u00FF'; + +// --------------------------------------------------------------------------- +// One delta-time before each event: a variable-length quantity whose +// continuation bytes set the high bit and whose final byte clears it. +mode DELTA; + +DELTA_TIME : '\u0080' .. '\u00FF'* '\u0000' .. '\u007F' -> mode(EVENT); + +// --------------------------------------------------------------------------- +// Exactly one event, matched whole so its data bytes never look like a status +// byte or a VLQ group. Each rule returns to DELTA for the next event. +mode EVENT; + +// Channel-voice messages: high nibble = command, low nibble = channel, then +// two data bytes (note/velocity, etc.). +NOTE_OFF : '\u0080' .. '\u008F' EVENT_BYTE EVENT_BYTE -> mode(DELTA); +NOTE_ON : '\u0090' .. '\u009F' EVENT_BYTE EVENT_BYTE -> mode(DELTA); + +// Meta events: 0xFF, type, then a (here fixed) length and payload. +META_END_OF_TRACK : '\u00FF' '\u002F' '\u0000' -> mode(DELTA); +META_SET_TEMPO : '\u00FF' '\u0051' '\u0003' EVENT_BYTE EVENT_BYTE EVENT_BYTE -> mode(DELTA); + +fragment EVENT_BYTE : '\u0000' .. '\u00FF'; diff --git a/tests/fixtures/antlr4-rust-gen/midi-binary/MidiParser.g4 b/tests/fixtures/antlr4-rust-gen/midi-binary/MidiParser.g4 new file mode 100644 index 00000000..0e3525ae --- /dev/null +++ b/tests/fixtures/antlr4-rust-gen/midi-binary/MidiParser.g4 @@ -0,0 +1,32 @@ +// Standard MIDI File (SMF) parser. Pairs with MidiLexer.g4 and the MidiHooks +// SemanticHooks implementation, driven over a `ByteStream`. See MidiLexer.g4 +// for the byte-oriented lexing strategy and attribution. + +parser grammar MidiParser; + +options { + tokenVocab = MidiLexer; +} + +// A file is one header chunk followed by one or more track chunks. +file : header track+ EOF; + +// MThd: the BEGIN_HEADER token already consumed the magic + length; its six +// body bytes arrive as HDR_BYTE, then MidiHooks closes the chunk. +header : BEGIN_HEADER HDR_BYTE HDR_BYTE HDR_BYTE HDR_BYTE HDR_BYTE HDR_BYTE END_OF_CHUNK; + +// MTrk: a run of timed events, closed by MidiHooks when the declared byte +// length is reached. A well-formed track ends with an End-of-Track meta event +// as its final event. +track : BEGIN_TRACK event+ END_OF_CHUNK; + +// Each event is a variable-length delta-time followed by the event body. The +// lexer matches each body whole, so the parser just names the alternatives. +event : DELTA_TIME body; + +body + : NOTE_ON # noteOn + | NOTE_OFF # noteOff + | META_SET_TEMPO # setTempo + | META_END_OF_TRACK # endOfTrack + ; diff --git a/tests/fixtures/antlr4-rust-gen/midi-binary/make_fixture.py b/tests/fixtures/antlr4-rust-gen/midi-binary/make_fixture.py new file mode 100644 index 00000000..2bda2590 --- /dev/null +++ b/tests/fixtures/antlr4-rust-gen/midi-binary/make_fixture.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +"""Regenerate twinkle.mid, the byte fixture for the MIDI binary-parsing test. + +The bytes are committed so the test needs no toolchain, but this script keeps +them auditable. Run it from anywhere; it writes next to itself. + + python3 make_fixture.py + +Produces a spec-correct Standard MIDI File (format 0, one track): + MThd len=6 format=0 ntracks=1 division=96 + MTrk: + delta 0 NoteOn ch0 note60 vel64 (90 3C 40) + delta 96 NoteOff ch0 note60 vel64 (80 3C 40) + delta 0 Meta SetTempo 500000 us (FF 51 03 07 A1 20) + delta 0 Meta EndOfTrack (FF 2F 00) + +`file(1)` reports: "Standard MIDI data (format 0) using 1 track at 1/96". +""" +import os +import struct + + +def vlq(value: int) -> bytes: + """Encode an int as a MIDI variable-length quantity (MSB group first).""" + out = [value & 0x7F] + value >>= 7 + while value: + out.insert(0, (value & 0x7F) | 0x80) + value >>= 7 + return bytes(out) + + +def build() -> bytes: + track = b"" + track += vlq(0) + bytes([0x90, 60, 64]) # NoteOn ch0 + track += vlq(96) + bytes([0x80, 60, 64]) # NoteOff ch0 + # Set Tempo 500000 us/quarter-note (the 3-byte payload after FF 51 03). + track += vlq(0) + bytes([0xFF, 0x51, 0x03]) + struct.pack(">I", 500000)[1:] + track += vlq(0) + bytes([0xFF, 0x2F, 0x00]) # EndOfTrack + + mthd = b"MThd" + struct.pack(">IHHH", 6, 0, 1, 96) + mtrk = b"MTrk" + struct.pack(">I", len(track)) + track + return mthd + mtrk + + +def main() -> None: + data = build() + path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "twinkle.mid") + with open(path, "wb") as handle: + handle.write(data) + print(f"wrote {len(data)} bytes to {path}") + print("hex:", data.hex()) + + +if __name__ == "__main__": + main() diff --git a/tests/fixtures/antlr4-rust-gen/midi-binary/patterns.toml b/tests/fixtures/antlr4-rust-gen/midi-binary/patterns.toml new file mode 100644 index 00000000..ff6045b5 --- /dev/null +++ b/tests/fixtures/antlr4-rust-gen/midi-binary/patterns.toml @@ -0,0 +1,14 @@ +# Semantic-helper patterns for the MIDI binary-parsing example. +# +# The lexer's `{beginChunk();}` action is a bare helper call with no portable +# lowering, so we route it to a generated hook method. `antlr4-rust-gen` then +# emits a `MidiLexerHooks` trait with `fn begin_chunk(&mut self, ctx)`, which +# `MidiHooks` implements to start the chunk-length countdown. +version = 1 + +[[helper]] +kind = "lexer-action" +name = "beginChunk" +arguments = "" +returns = "unit" +lower = "hook" diff --git a/tests/fixtures/antlr4-rust-gen/midi-binary/twinkle.mid b/tests/fixtures/antlr4-rust-gen/midi-binary/twinkle.mid new file mode 100644 index 0000000000000000000000000000000000000000..f3e6121d67d053403cb921c0f74207747b915402 GIT binary patch literal 41 tcmeYb$w*;fU|<7cMur66kfLlLOPFDTjYC3%jRV8~KxX!Z3Jm}C833fF2^RnW literal 0 HcmV?d00001 From b7898072be8b5964804f6b5e52d05578de43a8ae Mon Sep 17 00:00:00 2001 From: Konstantin Vyatkin Date: Fri, 24 Jul 2026 18:01:14 +0200 Subject: [PATCH 2/2] fix(runtime): address Codex review on ByteStream binary parsing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - byte_stream: clamp interval `stop` before the `+1` in `text` and `byte_interval`, mirroring `InputStream`, so a `usize::MAX` span (e.g. an EOF-token interval) can no longer overflow. Empty/out-of-range now returns `None`/`""` consistently. - byte_stream: implement `position_summary` by scanning the raw bytes. Without it, `BaseLexer::position_at` fell back to iterating the hex `text()`, so N bytes counted as 2N columns and `0x0A` newlines were invisible — corrupting line/column for split or synthesized tokens. - byte_stream: keep the module-doc example grammar-agnostic (`FooLexer`, per AGENTS.md), pointing to the MIDI fixture for the worked example. - midi fixture: carry the upstream milnet2/midi-grammar BSD-3-Clause notice verbatim (LICENSE-midi-grammar) and reference it from both grammar headers, satisfying the source-redistribution condition. Adds unit tests for the usize::MAX clamp and raw-byte position summaries. --- src/byte_stream.rs | 110 +++++++++++++++--- .../midi-binary/LICENSE-midi-grammar | 37 ++++++ .../antlr4-rust-gen/midi-binary/MidiLexer.g4 | 11 +- .../antlr4-rust-gen/midi-binary/MidiParser.g4 | 3 + 4 files changed, 140 insertions(+), 21 deletions(-) create mode 100644 tests/fixtures/antlr4-rust-gen/midi-binary/LICENSE-midi-grammar diff --git a/src/byte_stream.rs b/src/byte_stream.rs index c86b1125..759eb2c4 100644 --- a/src/byte_stream.rs +++ b/src/byte_stream.rs @@ -38,13 +38,16 @@ //! // Zero-copy from an in-memory buffer (e.g. bytes read off a socket): //! let stream = ByteStream::new(&packet[..]); //! -//! let lexer = MidiLexer::new(stream); +//! // Feed it to any generated lexer built from a byte-oriented grammar. +//! let lexer = FooLexer::new(stream); //! let tokens = CommonTokenStream::new(lexer); -//! let mut parser = MidiParser::new(tokens); -//! let tree = parser.file()?; +//! let mut parser = FooParser::new(tokens); +//! let tree = parser.entry_rule()?; //! ``` //! -//! Write lexer rules against the byte range, e.g. `BYTE : ' ' .. 'ÿ';`. +//! Write lexer rules against the byte range, e.g. `BYTE : ' ' .. 'ÿ';`. A +//! complete worked example — a Standard MIDI File grammar parsed over a +//! `ByteStream` — lives under `tests/fixtures/antlr4-rust-gen/midi-binary/`. //! //! # Token text is hex //! @@ -56,7 +59,7 @@ use std::io; -use crate::char_stream::{CharStream, TextInterval}; +use crate::char_stream::{CharStream, PositionSummary, TextInterval}; use crate::int_stream::{EOF, IntStream, UNKNOWN_SOURCE_NAME}; /// A [`CharStream`] backed by raw bytes, where each byte is one symbol in @@ -166,15 +169,22 @@ impl> CharStream for ByteStream { /// Renders the inclusive byte interval as a lowercase, separator-free hex /// string. See the [module documentation](self) for the rationale. fn text(&self, interval: TextInterval) -> String { + // Clamp `stop` before any `+1`, mirroring `InputStream`: a caller + // passing `TextInterval::new(_, usize::MAX)` (e.g. an EOF token span) + // must not overflow. let bytes = self.bytes.as_ref(); - if interval.is_empty() { + let len = bytes.len(); + if interval.is_empty() || len == 0 { + return String::new(); + } + let start = interval.start.min(len); + let stop = interval.stop.min(len - 1); + if start > stop { return String::new(); } - let stop = (interval.stop + 1).min(bytes.len()); - let start = interval.start.min(stop); use std::fmt::Write as _; - bytes[start..stop].iter().fold( - String::with_capacity((stop - start) * 2), + bytes[start..=stop].iter().fold( + String::with_capacity((stop - start + 1) * 2), |mut acc, byte| { // Writing to a String is infallible. let _ = write!(acc, "{byte:02x}"); @@ -197,16 +207,43 @@ impl> CharStream for ByteStream { // valid for 7-bit input; bytes `>= 0x80` route correctly through the // generic path's `wide_rows` instead. + /// Summarizes line/column movement over `[start, end)` by scanning the raw + /// bytes. + /// + /// Without this, [`BaseLexer`](crate::lexer::BaseLexer) would fall back to + /// iterating [`Self::text`], which is hex — so a span of N bytes would count + /// as 2N columns and `0x0A` newline bytes would be invisible, corrupting the + /// line/column of split or synthesized tokens. + fn position_summary(&self, start: usize, end: usize) -> Option { + let bytes = self.bytes.as_ref(); + let len = bytes.len(); + if start > end { + return None; + } + let start = start.min(len); + let end = end.min(len); + let mut summary = PositionSummary::default(); + for &byte in &bytes[start..end] { + if byte == b'\n' { + summary.line_breaks += 1; + summary.trailing_columns = 0; + } else { + summary.trailing_columns += 1; + } + } + Some(summary) + } + fn byte_interval(&self, interval: TextInterval) -> Option<(usize, usize)> { - // Index == byte offset, so the byte span is exact. + // Index == byte offset, so the byte span is exact. Clamp `stop` before + // the `+1` for the same overflow reason as `text`. let len = self.bytes.as_ref().len(); - if interval.is_empty() { - let at = self.cursor.min(len); - return Some((at, at)); + if interval.is_empty() || len == 0 { + return None; } - let stop = (interval.stop + 1).min(len); - let start = interval.start.min(stop); - Some((start, stop)) + let start = interval.start.min(len); + let stop = interval.stop.min(len - 1); + (start <= stop).then_some((start, stop + 1)) } } @@ -253,10 +290,49 @@ mod tests { assert_eq!(stream.text(TextInterval::empty()), ""); // Inclusive char interval [1, 2] -> half-open byte span [1, 3). assert_eq!(stream.byte_interval(TextInterval::new(1, 2)), Some((1, 3))); + assert_eq!(stream.byte_interval(TextInterval::empty()), None); assert_eq!(stream.symbol_at(0), Some(0xDE)); assert_eq!(stream.symbol_at(4), Some(EOF)); } + #[test] + fn text_and_byte_interval_clamp_usize_max_without_overflow() { + // An EOF-token span can carry `stop == usize::MAX`; clamping before the + // `+1` must not overflow (debug panic / release wrap). + let stream = ByteStream::new(vec![0xDE, 0xAD]); + assert_eq!(stream.text(TextInterval::new(0, usize::MAX)), "dead"); + assert_eq!( + stream.byte_interval(TextInterval::new(0, usize::MAX)), + Some((0, 2)), + ); + // Out-of-range start clamps to empty rather than panicking. + assert_eq!(stream.text(TextInterval::new(5, usize::MAX)), ""); + } + + #[test] + fn position_summary_scans_raw_bytes_not_hex() { + // Bytes, not hex: a newline byte (0x0A) is one line break, and each + // other byte is one column — never doubled as the hex rendering would. + let stream = ByteStream::new(vec![0x41, 0x0A, 0x42, 0x43]); + assert_eq!( + stream.position_summary(0, 4), + Some(PositionSummary { + line_breaks: 1, + trailing_columns: 2, + }), + ); + // No newline in [2, 4): two raw bytes are two columns (hex would say + // four). + assert_eq!( + stream.position_summary(2, 4), + Some(PositionSummary { + line_breaks: 0, + trailing_columns: 2, + }), + ); + assert_eq!(stream.position_summary(4, 2), None); + } + #[test] fn borrows_bytes_zero_copy() { // The network-buffer case: parse a slice we already hold without diff --git a/tests/fixtures/antlr4-rust-gen/midi-binary/LICENSE-midi-grammar b/tests/fixtures/antlr4-rust-gen/midi-binary/LICENSE-midi-grammar new file mode 100644 index 00000000..656f4ad0 --- /dev/null +++ b/tests/fixtures/antlr4-rust-gen/midi-binary/LICENSE-midi-grammar @@ -0,0 +1,37 @@ +The MIDI grammar in this directory (MidiLexer.g4, MidiParser.g4) is adapted +(simplified) from the milnet2/midi-grammar project: +https://github.com/milnet2/midi-grammar + +Its upstream license and copyright notice, retained per BSD-3-Clause condition 1, +follow verbatim. + +-------------------------------------------------------------------------------- + +BSD 3-Clause License + +Copyright (c) 2024, Tobias Blaschke + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/tests/fixtures/antlr4-rust-gen/midi-binary/MidiLexer.g4 b/tests/fixtures/antlr4-rust-gen/midi-binary/MidiLexer.g4 index 5b494bb3..19024ae4 100644 --- a/tests/fixtures/antlr4-rust-gen/midi-binary/MidiLexer.g4 +++ b/tests/fixtures/antlr4-rust-gen/midi-binary/MidiLexer.g4 @@ -15,10 +15,13 @@ // byte and a VLQ continuation byte share the range 0x80..0xFF -- they never // share a lexer mode here. // -// Adapted (simplified) from milnet2/midi-grammar by Tobias Blaschke, BSD-3: -// https://github.com/milnet2/midi-grammar. Scope is deliberately small: MThd + -// MTrk chunks, VLQ delta-times, note-on/off, and set-tempo / end-of-track meta -// events. Running status, sysex, and most meta events are out of scope. +// Adapted (simplified) from milnet2/midi-grammar by Tobias Blaschke: +// https://github.com/milnet2/midi-grammar. Copyright (c) 2024, Tobias Blaschke; +// licensed BSD-3-Clause. The upstream copyright notice, conditions, and +// disclaimer are retained verbatim in LICENSE-midi-grammar in this directory. +// Scope is deliberately small: MThd + MTrk chunks, VLQ delta-times, +// note-on/off, and set-tempo / end-of-track meta events. Running status, +// sysex, and most meta events are out of scope. lexer grammar MidiLexer; diff --git a/tests/fixtures/antlr4-rust-gen/midi-binary/MidiParser.g4 b/tests/fixtures/antlr4-rust-gen/midi-binary/MidiParser.g4 index 0e3525ae..35d40553 100644 --- a/tests/fixtures/antlr4-rust-gen/midi-binary/MidiParser.g4 +++ b/tests/fixtures/antlr4-rust-gen/midi-binary/MidiParser.g4 @@ -1,6 +1,9 @@ // Standard MIDI File (SMF) parser. Pairs with MidiLexer.g4 and the MidiHooks // SemanticHooks implementation, driven over a `ByteStream`. See MidiLexer.g4 // for the byte-oriented lexing strategy and attribution. +// +// Adapted from milnet2/midi-grammar; Copyright (c) 2024, Tobias Blaschke; +// BSD-3-Clause. Full notice in LICENSE-midi-grammar in this directory. parser grammar MidiParser;