feat(codegen): add TestRig command - #302
Merged
Merged
Conversation
Add antlr4-rust-testrig to generate and compile a temporary grammar-specific runner for combined, split, and lexer-only grammars. Rust cannot dispatch parser rules reflectively, so keep runner generation inside codegen and select rule methods from grammar metadata. Forward the standard token, tree, trace, diagnostics, and SLL modes. Process every named input and return a non-zero status for generation, compilation, I/O, lexer, or parser errors so the command is suitable for test automation. Document installation and split-grammar usage, cover success and failure paths end to end, and verify the new binary is included in the published codegen package.
Copy/Paste DetectionFound 2 duplication(s) across 17 changed non-generated Rust file(s) (threshold: 100 tokens). Show duplicationsFound a 23 line (139 tokens) duplication in the following files:
ID: [a-z]+;\n",
)
.expect("delegate grammar should be writable");
let output = run_antlr4_rust_gen(&[
root.as_os_str(),
OsStr::new("-I"),
temp.path().as_os_str(),
OsStr::new("--actions"),
OsStr::new("embedded"),
OsStr::new("--sem-unknown"),
OsStr::new("error"),
OsStr::new("--require-full-semantics"),
OsStr::new("--out-dir"),
out.as_os_str(),
]);
assert!(
output.status.success(),
"stdout: {}\nstderr: {}",
utf8(&output.stdout),
utf8(&output.stderr)
);
let parser = fs::read_to_string(out.join("root_parser.rs")).expect("parser should be emitted");
```rust
---
Found a 20 line (107 tokens) duplication in the following files:
* Starting at line 1159 of crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/compatibility.rs
* Starting at line 1205 of crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/compatibility.rs
```rust
.expect("type-position grammar should be writable");
let output = run_antlr4_rust_gen(&[
grammar.as_os_str(),
OsStr::new("--actions"),
OsStr::new("embedded"),
OsStr::new("--sem-unknown"),
OsStr::new("error"),
OsStr::new("--require-full-semantics"),
OsStr::new("--out-dir"),
out.as_os_str(),
]);
assert!(
output.status.success(),
"stdout: {}\nstderr: {}",
utf8(&output.stdout),
utf8(&output.stderr)
);
let parser =
fs::read_to_string(out.join("type_position_parser.rs")).expect("parser should be emitted"); |
|
Warning Review limit reached
Next review available in: 50 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis change adds the ChangesTestRig CLI
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant TestrigCLI
participant CodeGenerator
participant TemporaryCargoProject
participant GeneratedRunner
User->>TestrigCLI: provide grammar, inputs, and options
TestrigCLI->>CodeGenerator: generate recognizer and test-rig entry point
CodeGenerator->>TemporaryCargoProject: write source and Cargo manifest
TestrigCLI->>GeneratedRunner: run generated binary
GeneratedRunner-->>TestrigCLI: report lexer or parser result
TestrigCLI-->>User: return exit code
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Claude Code is working… I'll analyze this and get back to you. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/antlr-rust-codegen/src/test_rig.rs`:
- Around line 439-448: Add unit tests around select_lexer covering zero lexers,
multiple lexers with matching Parser/Lexer stems, and multiple lexers without a
stem match. Add a tokens start-rule case that exercises rendering of the
lexer-only template, reusing the existing lexer and parser helpers and asserting
the expected selection or generated output.
- Around line 306-311: Remove the discard tuple in the lexer runner and stop
using it to silence unused-field analysis. Prefer moving tree, trace,
diagnostics, and sll out of the shared Options definition into the
parser-specific template so lexer options no longer accept parser-only flags;
otherwise apply per-field dead-code attributes to the shared Options fields.
- Around line 450-488: Replace the hand-written contains assertions in
renders_rule_dispatch_with_generated_method_name with a named external
insta::assert_snapshot! covering the complete rendered runner. In
rejects_unknown_and_ambiguous_rules, snapshot each diagnostic string inline
using insta::assert_snapshot! or an approved debug variant. Add
#[allow(clippy::disallowed_methods)] to the containing test module, and place
the external snapshot in the sibling snapshots directory.
In `@crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/testrig.rs`:
- Line 128: Replace the full parse-tree equality assertion around the
`output.stdout` check with a named `insta::assert_snapshot!` assertion, using
the UTF-8 output as the snapshot content and preserving the existing expected
parse-tree dump through the external snapshot file.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: cf1bd133-7106-4c4b-a5fa-12fe6b95b394
⛔ Files ignored due to path filters (3)
crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__testrig__antlr4_rust_testrig_help.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__testrig__parser_runner_stdout.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__testrig__parser_runner_trace.snapis excluded by!**/*.snap
📒 Files selected for processing (15)
README.mdcrates/antlr-rust-codegen/Cargo.tomlcrates/antlr-rust-codegen/README.mdcrates/antlr-rust-codegen/src/bin/antlr4-rust-testrig.rscrates/antlr-rust-codegen/src/builder.rscrates/antlr-rust-codegen/src/cli.rscrates/antlr-rust-codegen/src/config.rscrates/antlr-rust-codegen/src/driver.rscrates/antlr-rust-codegen/src/lib.rscrates/antlr-rust-codegen/src/test_rig.rscrates/antlr-rust-codegen/src/testrig_cli.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/support.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/testrig.rstools/release/verify-package-contents.sh
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Snapshot complete lexer and parser runners plus split-grammar tree output, and cover missing, ambiguous, and stem-matched lexer selection paths. Replace the lexer runner's discard tuple with an explicit allowance that documents why parser-only TestRig flags remain accepted in lexer-only mode.
📊 Source Code Metrics (this PR vs
|
| File | Cyclomatic | Cognitive | Functions | LLOC | MI |
|---|---|---|---|---|---|
| crates/antlr-rust-codegen/src/builder.rs | 46 ⚪ | 10 ⚪ | 24 ⚪ | 55 ⚪ | 13.71 (main: 13.76) 🔴 |
| crates/antlr-rust-codegen/src/testrig_cli.rs | 75 🆕 | 34 🆕 | 21 🆕 | 88 🆕 | 2.48 🆕 |
| crates/antlr-rust-codegen/src/error.rs | 26 (main: 30) 🟢 | 2 (main: 6) 🟢 | 16 (main: 17) 🟢 | 20 (main: 25) 🟢 | 22.97 (main: 21.39) 🟢 |
| crates/antlr-rust-codegen/src/test_rig.rs | 43 🆕 | 14 🆕 | 16 🆕 | 61 🆕 | 3.80 🆕 |
| crates/antlr-rust-codegen/src/cli_report.rs | 43 🆕 | 8 🆕 | 13 🆕 | 40 🆕 | 18.39 🆕 |
| crates/antlr-rust-codegen/src/driver.rs | 79 (main: 74) 🔴 | 61 (main: 52) 🔴 | 7 ⚪ | 115 (main: 107) 🔴 | 5.20 (main: 6.68) 🔴 |
| crates/antlr-rust-codegen/src/cli.rs | 14 ⚪ | 6 (main: 5) 🔴 | 4 ⚪ | 17 ⚪ | 21.34 (main: 21.55) 🔴 |
| crates/antlr-rust-codegen/src/lib.rs | 3 (main: 2) 🔴 | 0 ⚪ | 2 (main: 1) 🔴 | 4 (main: 1) 🔴 | 38.77 (main: 41.40) 🔴 |
| crates/antlr-rust-codegen/src/bin/antlr4-rust-gen.rs | 2 ⚪ | 0 ⚪ | 1 ⚪ | 1 ⚪ | 77.21 (main: 75.51) 🟢 |
| crates/antlr-rust-codegen/src/bin/antlr4-rust-testrig.rs | 2 🆕 | 0 🆕 | 1 🆕 | 1 🆕 | 76.23 🆕 |
| crates/antlr-rust-codegen/src/config.rs | 1 ⚪ | 0 ⚪ | 0 ⚪ | 0 ⚪ | 43.23 (main: 45.49) 🔴 |
Generated by mehen v1.8.0 — the code quality watcher.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3aca3903d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Render code-generation failures with Miette source diagnostics and enable Clap colors for the public CLI binaries. Generated TestRig runners now report input, lexer, and parser failures through Miette while preserving multi-input aggregation and non-zero exits. Snapshot subprocesses force colors off for deterministic output.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/diagnostics.rs`:
- Around line 99-102: Update the grammar-path normalization used by the Miette
snapshots in
crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/diagnostics.rs:99-102 and
crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/transforms.rs:353-356 to
call replace_miette_path before snapshot normalization. Replace the direct path
substitutions at both sites while preserving the existing snapshot assertions.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: a184aa91-3614-4f7c-a31e-388c82a89262
⛔ Files ignored due to path filters (13)
Cargo.lockis excluded by!**/*.lockcrates/antlr-rust-codegen/src/snapshots/antlr_rust_codegen__test_rig__tests__lexer_runner_rendered.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/src/snapshots/antlr_rust_codegen__test_rig__tests__parser_runner_rendered.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__compatibility__unsupported_antlr4rust_surface_diagnostics.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__diagnostics__lexer_left_recursion_diagnostics.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__semantics__parser_action_hook_signature_conflict_diagnostic.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__testrig__lexer_only_syntax_error.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__testrig__lexer_syntax_error.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__testrig__parser_syntax_error.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__testrig__split_parser_tree.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__testrig__unknown_start_rule_diagnostic.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__transforms__configured_entry_rule_not_found.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__transforms__precedence_ladder_authored_semantic_errors.snapis excluded by!**/*.snap
📒 Files selected for processing (16)
README.mdcrates/antlr-rust-codegen/Cargo.tomlcrates/antlr-rust-codegen/README.mdcrates/antlr-rust-codegen/src/bin/antlr4-rust-gen.rscrates/antlr-rust-codegen/src/bin/antlr4-rust-testrig.rscrates/antlr-rust-codegen/src/cli.rscrates/antlr-rust-codegen/src/cli_report.rscrates/antlr-rust-codegen/src/error.rscrates/antlr-rust-codegen/src/lib.rscrates/antlr-rust-codegen/src/test_rig.rscrates/antlr-rust-codegen/src/testrig_cli.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/compatibility.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/diagnostics.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/support.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/testrig.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/transforms.rs
💤 Files with no reviewable changes (1)
- crates/antlr-rust-codegen/src/error.rs
Disable parse-tree construction unless requested and reject conflicting exact-diagnostics and SLL modes. Alias generated recognizer imports so grammar names cannot collide with runner helpers, and keep the shared Cargo target under a user-specific cache with an explicit override and private fallback.
Route every Miette snapshot path replacement through the wrapper-aware normalizer so line-wrapped paths remain deterministic.
Honor an exported COLUMNS value in the codegen CLIs and generated TestRig runners. Pin the CLI snapshot harness to a wide terminal so checkout path length cannot change Miette wrapping before dynamic paths are normalized.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 97ecca0fbc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Create generated runner projects with private Unix permissions so grammar sources and embedded actions are not exposed through shared temporary directories. Retain the runtime's Rc-backed input and buffer raw diagnostics, converting source text to Miette's Arc-backed representation only when an error is rendered.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/antlr-rust-codegen/README.md`:
- Around line 63-68: Update the command-mode documentation in the package README
to include the supported --trace parser mode alongside --diagnostics and --sll,
preserving the existing statement that --diagnostics and --sll are mutually
exclusive.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 70a8f89a-6d8f-49e7-9a31-81c522eafb09
⛔ Files ignored due to path filters (7)
crates/antlr-rust-codegen/src/snapshots/antlr_rust_codegen__test_rig__tests__lexer_runner_rendered.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/src/snapshots/antlr_rust_codegen__test_rig__tests__parser_runner_rendered.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__compatibility__unsupported_antlr4rust_surface_diagnostics.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__semantics__parser_action_hook_signature_conflict_diagnostic.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__testrig__colliding_recognizer_names_tree.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__testrig__conflicting_prediction_modes.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__testrig__unknown_start_rule_diagnostic.snapis excluded by!**/*.snap
📒 Files selected for processing (10)
README.mdcrates/antlr-rust-codegen/README.mdcrates/antlr-rust-codegen/src/cli_report.rscrates/antlr-rust-codegen/src/lib.rscrates/antlr-rust-codegen/src/test_rig.rscrates/antlr-rust-codegen/src/testrig_cli.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/diagnostics.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/support.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/testrig.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/transforms.rs
Keep the package-level TestRig usage guide aligned with the root README by listing every supported parser mode and the diagnostics/SLL conflict.
Contributor
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
antlr4-rust-testrigas a second binary inantlr-rust-codegenfancydiagnostics and enable Clap colorsDesign
TestRig stays in
antlr-rust-codegenrather than a dedicated crate. Rust cannotreflectively load a generated recognizer or invoke a parser rule by name, so the
command needs codegen's grammar metadata and rule-name normalization to emit
static dispatch. A separate crate would expose or duplicate those internals and
would only move the existing Clap dependency.
The generated runner lives below
src/__antlr4_rust_testrig/so grammars namedMaincannot collide with its entry point. Named inputs are all attempted andtheir results are aggregated, making recovered lexer and parser syntax errors
fail test automation even when a tree can still be produced.
Generated recognizers are imported through private aliases so valid grammar
names cannot collide with runner helpers. Parse trees are built only for
--tree, and--diagnosticsconflicts with--sllrather than silentlychanging prediction mode. Cargo artifacts are shared through the current
user's cache, with
ANTLR4_RUST_TESTRIG_TARGET_DIRas an override and aper-run private fallback when no user cache is available. The generated
project root is created atomically with mode
0700on Unix so grammar sourcesand embedded actions are not exposed through a shared temporary directory.
Codegen diagnostics retain their structured codes and source spans through
Miette. The temporary runner uses the same reporting style for input I/O,
lexer, and parser failures. It retains the runtime's
Rc-backed input and onlycopies the full source into Miette's
Arc-backed representation when adiagnostic is rendered. Snapshot subprocesses force color off through the
standard color-control environment variables, while normal terminal use keeps
the
fancyrenderer and Clap styling.This does not change the generated recognizer/runtime interface, so the
generated-code API revision remains unchanged.
Closes #289.
Validation
cargo test --locked --workspace --all-featurescargo clippy --locked --workspace --all-targets --all-features -- -D warningscargo fmt --all -- --checkgit diff --checkANTLR_RUST_RELEASE_ALLOW_DIRTY=1 tools/release/verify-package-contents.sh antlr-rust-codegencargo package --locked -p antlr-rust-codegen --allow-dirtySummary by CodeRabbit
New Features
antlr4-rust-testrigcommand for running generated lexer and parser grammars.Documentation