feat(runtime): add ByteStream for binary parsing + MIDI example - #188
Conversation
Introduce `ByteStream`, a byte-oriented `CharStream` for parsing binary
formats. Each byte is one symbol in `0..=255` and the stream index is the
byte offset, so grammars can use the Latin-1 convention (`BYTE: ' '..'ÿ'`)
that ANTLR's reference runtimes use for binary input, without the transcoding
and per-byte allocation `InputStream` incurs on non-UTF-8 bytes.
`ByteStream<B = Vec<u8>>` is generic over `AsRef<[u8]>` to map onto Rust IO
primitives: `new(vec)` owns, `new(&buf[..])` borrows a network buffer
zero-copy, and `from_reader(r)?` drains any `std::io::Read` (files, sockets,
stdin, `Cursor`). Because the bytes are not text, `text()` renders a matched
span as lowercase hex. The ASCII fast-path (`contiguous_ascii`) is
deliberately not implemented: it feeds a 128-wide DFA row, so bytes >= 0x80
route through the generic path instead.
Add a worked binary-parsing example under
tests/fixtures/antlr4-rust-gen/midi-binary/: a Standard MIDI File grammar
(MThd/MTrk chunks, VLQ delta-times, note and meta events) plus a small
`SemanticHooks` chunk-framing "superClass" that counts down each chunk's
declared byte length and synthesizes END_OF_CHUNK -- the data-dependent
"read N, then N bytes" pattern ANTLR's bencoding grammar solves the same way.
The bare `{beginChunk();}` lexer action lowers to a typed hook via a
--sem-patterns helper. An integration test generates the recognizer from the
committed grammar and parses a real .mid fixture over a ByteStream with zero
syntax errors.
The grammar is adapted (simplified) from milnet2/midi-grammar by Tobias
Blaschke (BSD-3-Clause). README gains a "Binary and Byte-Oriented Parsing"
section documenting the approach.
Copy/Paste DetectionFound 1 duplication(s) across 3 changed Rust file(s) (threshold: 100 tokens). Show duplicationsFound a 22 line (132 tokens) duplication in the following files:
"parser grammar Delegate;\ndelegated: {isTypeName()}? ID;\n",
)
.expect("delegate grammar should be writable");
fs::write(&tokens, "lexer grammar Tokens;\nID: [a-z]+;\n")
.expect("token grammar should be writable");
let output = run_antlr4_rust_gen(&[
root.as_os_str(),
tokens.as_os_str(),
OsStr::new("-I"),
temp.path().as_os_str(),
OsStr::new("--out-dir"),
out.as_os_str(),
]);
assert!(
output.status.success(),
"stdout: {}\nstderr: {}",
utf8(&output.stdout),
utf8(&output.stderr)
);
let parser = fs::read_to_string(out.join("root.rs")).expect("parser should be emitted");
assert!(parser.contains("pub trait RootHooks"), "{parser}");
```rust
</details> |
|
Warning Review limit reached
Next review available in: 12 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughAdds a public ChangesBinary parsing support
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant ByteStream
participant GeneratedMidiLexer
participant MidiLexerHooks
participant CommonTokenStream
participant GeneratedMidiParser
ByteStream->>GeneratedMidiLexer: provide byte lookahead
GeneratedMidiLexer->>MidiLexerHooks: invoke begin_chunk and lexer_before_token
MidiLexerHooks->>GeneratedMidiLexer: emit END_OF_CHUNK
GeneratedMidiLexer->>CommonTokenStream: enqueue tokens
CommonTokenStream->>GeneratedMidiParser: provide MIDI tokens
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Claude Code is working… I'll analyze this and get back to you. |
📊 Source Code Metrics (this PR vs
|
| File | Cyclomatic | Cognitive | Functions | LLOC | MI |
|---|---|---|---|---|---|
| src/byte_stream.rs | 44 🆕 | 15 🆕 | 22 🆕 | 98 🆕 | 11.27 🆕 |
| src/lib.rs | 4 ⚪ | 3 ⚪ | 1 ⚪ | 7 ⚪ | 33.59 (main: 33.91) 🔴 |
Generated by mehen v1.7.0 — the code quality watcher.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8a5422b0fa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- byte_stream: clamp interval `stop` before the `+1` in `text` and `byte_interval`, mirroring `InputStream`, so a `usize::MAX` span (e.g. an EOF-token interval) can no longer overflow. Empty/out-of-range now returns `None`/`""` consistently. - byte_stream: implement `position_summary` by scanning the raw bytes. Without it, `BaseLexer::position_at` fell back to iterating the hex `text()`, so N bytes counted as 2N columns and `0x0A` newlines were invisible — corrupting line/column for split or synthesized tokens. - byte_stream: keep the module-doc example grammar-agnostic (`FooLexer`, per AGENTS.md), pointing to the MIDI fixture for the worked example. - midi fixture: carry the upstream milnet2/midi-grammar BSD-3-Clause notice verbatim (LICENSE-midi-grammar) and reference it from both grammar headers, satisfying the source-redistribution condition. Adds unit tests for the usize::MAX clamp and raw-byte position summaries.
|
@codex review All four findings from the previous review are addressed in b789807:
|
|
Claude finished @tinovyatkin's task in 12m 16s —— View job Code Review — PR #188: ByteStream for binary parsing + MIDI exampleVerdict: ✅ No blocking issues found. Checked for bugs and CLAUDE.md compliance.
I reviewed with four independent passes (two CLAUDE.md-compliance, two bug/correctness), cross-checking What I verified
MIDI example — correct
CLAUDE.md compliance — clean
Optional (non-blocking) nit
Nice work on the design writeup and the auditable fixture generator. |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Summary
Adds first-class support for binary / byte-oriented parsing, closing #181.
ByteStream— a new byte-orientedCharStream(src/byte_stream.rs, exported fromlib.rs). Each byte is one symbol in0..=255, the stream index is the byte offset, and lookahead returns the byte value. Grammars use the Latin-1 convention (BYTE : ' ' .. 'ÿ';) that ANTLR's reference runtimes use for binary input — but without theVec<char>+ offset-table transcodingInputStreamincurs on non-UTF-8 bytes.SemanticHooks"superClass" + an integration test that parses a real.midover aByteStream.ByteStreamdesignGeneric over
B: AsRef<[u8]>(defaultVec<u8>) so one type maps onto every Rust IO shape:ByteStream::new(vec)ByteStream::new(&buf[..])CursorByteStream::from_reader(r)?io::ReadBecause the bytes aren't text,
text()renders a matched span as lowercase hex. The ASCII fast path (contiguous_ascii) is deliberately not implemented — it indexes a 128-wide DFA row, so bytes>= 0x80route correctly through the genericwide_rowspath instead. (Verified againstsrc/atn/lexer_dfa.rs.)MIDI example (
tests/fixtures/antlr4-rust-gen/midi-binary/)Length-prefixed formats ("read N, then consume N bytes") are data-dependent, so a pure CFG can't frame them — the same constraint ANTLR's
bencodinggrammar solves with a lexersuperClass. Here that role is a ~20-lineMidiHooksimplementingSemanticHooks:begin_chunkreads the MThd/MTrk declared length (raw bytes viala()lookbehind) and arms a countdown.lexer_before_tokenemits a synthesizedEND_OF_CHUNKonce the body is consumed, viaenqueue_token+pop_mode.The bare
{beginChunk();}lexer action lowers to a typed hook method through a--sem-patterns[[helper]]entry (kind = "lexer-action",lower = "hook"). Scope is deliberately small: MThd/MTrk chunks, VLQ delta-times, note-on/off, set-tempo, end-of-track. Grammar adapted (simplified) from milnet2/midi-grammar by Tobias Blaschke (BSD-3-Clause, attributed in the grammar header and README).The
twinkle.midfixture is validated byfile(1)as "Standard MIDI data (format 0) using 1 track at 1/96";make_fixture.pykeeps its bytes auditable.Test
midi_binary_grammar_parses_standard_midi_file_over_byte_stream(intests/antlr4_rust_gen_cli.rs) generates the recognizer at test time viaantlr4-rust-gen(no ANTLR jar, no network), compiles a crate definingMidiHooks, and parses the fixture over aByteStream— asserting 0 syntax errors and the exact token-type sequence.Verification
cargo test --locked --features codegen— all pass (278 + 3 + 732 + 26).cargo clippy --locked --all-targets --all-features -- -D warnings— clean.cargo fmt-clean; no unrelated formatting churn.Closes #181.
Summary by CodeRabbit
New Features
ByteStreamsupport for parsing binary data from byte arrays or readers.Documentation