feat(codegen): enforce generated runtime compatibility

AGENTS.md

@@ -8,6 +8,23 @@ workarounds to `src/bin/antlr4-rust-gen.rs` or runtime modules; model the
 behavior from ANTLR metadata/ATN structure instead, and keep language-specific
 guidance in docs or tests for that language.
 
+## Generated-code API revision
+
+`src/lib.rs::__ANTLR4_RUST_CODEGEN_API` is the single current revision emitted
+by `antlr4-rust-gen`. The accepted revisions are the literal match arms in
+`src/lib.rs::__antlr4_rust_require_codegen_api!`.
+
+Any change that makes newly generated source require an incompatible runtime
+interface, or makes existing generated source incompatible with the runtime,
+MUST increment `__ANTLR4_RUST_CODEGEN_API`. When incrementing it, update the
+macro's accepted revision arms and supported-revision diagnostic, the
+`generated_modules_enforce_codegen_api_compatibility` integration test and
+snapshots, all checked-in generated recognizers, and the compatibility
+documentation. Retain an older accepted arm only while the runtime still
+provides every API surface that revision's generated source needs. Package
+releases that preserve the generated-source/runtime contract MUST NOT increment
+the revision automatically.
+
 ## Kotlin parser parity perf benchmark
 
 Reproduces the timings against the Kotlin grammar from `antlr/grammars-v4`.

CLAUDE.md

@@ -1,5 +1,22 @@
 # Development notes
 
+## Generated-code API revision
+
+`src/lib.rs::__ANTLR4_RUST_CODEGEN_API` is the single current revision emitted
+by `antlr4-rust-gen`. The accepted revisions are the literal match arms in
+`src/lib.rs::__antlr4_rust_require_codegen_api!`.
+
+Any change that makes newly generated source require an incompatible runtime
+interface, or makes existing generated source incompatible with the runtime,
+MUST increment `__ANTLR4_RUST_CODEGEN_API`. When incrementing it, update the
+macro's accepted revision arms and supported-revision diagnostic, the
+`generated_modules_enforce_codegen_api_compatibility` integration test and
+snapshots, all checked-in generated recognizers, and the compatibility
+documentation. Retain an older accepted arm only while the runtime still
+provides every API surface that revision's generated source needs. Package
+releases that preserve the generated-source/runtime contract MUST NOT increment
+the revision automatically.
+
 ## Inner loop
 
 ```bash

README.md

@@ -75,6 +75,28 @@ antlr4-rust-gen \
 Use multiple roots when a build should emit several independent recognizers in
 one deterministic source-set compilation.
 
+### Generated-source/runtime compatibility
+
+Every newly generated lexer and parser contains a compile-time generated-code
+API check. The check records both the API revision and the
+`antlr4-rust-gen` package version that produced the module. A mismatch reports
+the generated file directly and asks you to either regenerate it with a
+compatible generator or select a compatible `antlr-rust-runtime` dependency.
+
+The generated-code API revision tracks the Rust source interface between
+generated recognizers and the runtime. It is independent of package SemVer and
+the serialized ATN/DFA format versions: compatible package releases may share
+one revision, while an incompatible source-contract change increments it.
+Using the same package release for `antlr4-rust-gen` and
+`antlr-rust-runtime` remains the recommended workflow, but matching the
+generated-code API is the compile-time requirement.
+
+Generated modules created before this check was introduced cannot be detected
+retroactively. When first upgrading to a release that includes the check,
+regenerate every committed lexer and parser once. Thereafter, normal
+`cargo check` builds enforce compatibility for every compiled generated
+module.
+
 ## Complete Example
 
 Suppose you are using `JSON.g4` from `antlr/grammars-v4/json`. Generate both

docs/migration.md

@@ -1,10 +1,19 @@
 # Migration Notes
 
 `antlr-rust-runtime` is pre-1.0. Minor releases may include breaking runtime and
-generator changes. Generated lexers and parsers must use the same release of
-`antlr4-rust-gen` as the runtime. Before regenerating, run
-`antlr4-rust-gen --version` and compare the reported release with the
-`antlr-rust-runtime` dependency version.
+generator changes. Using the same release of `antlr4-rust-gen` and
+`antlr-rust-runtime` remains recommended. Newly generated modules also carry a
+generated-code API revision that is checked against the selected runtime at
+compile time, so releases that deliberately preserve the source contract can
+remain compatible without exact SemVer equality.
+
+Generated modules created before the compatibility check was introduced carry
+no enforceable revision. Regenerate every committed lexer and parser once when
+first adopting a release with this mechanism. If a later build reports a
+generated-code API mismatch, either regenerate the named module with a
+compatible generator or select a runtime that accepts its requested revision.
+`antlr4-rust-gen --version` reports the generator package version for auditing
+and reproducible regeneration.
 
 ## Structured Syntax Error Events and Byte Spans
 

src/bin/antlr4-rust-gen.rs

@@ -52,20 +52,27 @@ use templates::{
     template_sequence_bodies,
 };
 
-const GENERATED_MODULE_HEADER: &str = concat!(
-    "// @generated by ",
-    env!("CARGO_PKG_NAME"),
-    " v",
-    env!("CARGO_PKG_VERSION"),
-    " - do not edit\n",
-    "// project: ",
-    env!("CARGO_PKG_REPOSITORY"),
-    "\n",
-    "#[allow(warnings, missing_docs, clippy::all, clippy::pedantic, clippy::nursery)]\n",
-    "#[rustfmt::skip]\n",
-    "mod __antlr4_rust_generated {\n",
-    "\n",
-);
+fn generated_module_header() -> String {
+    format!(
+        concat!(
+            "// @generated by ",
+            env!("CARGO_PKG_NAME"),
+            " v",
+            env!("CARGO_PKG_VERSION"),
+            " - do not edit\n",
+            "// project: ",
+            env!("CARGO_PKG_REPOSITORY"),
+            "\n",
+            "antlr4_runtime::__antlr4_rust_require_codegen_api!({}, \"{}\");\n",
+            "#[allow(warnings, missing_docs, clippy::all, clippy::pedantic, clippy::nursery)]\n",
+            "#[rustfmt::skip]\n",
+            "mod __antlr4_rust_generated {{\n",
+            "\n",
+        ),
+        antlr4_runtime::__ANTLR4_RUST_CODEGEN_API,
+        env!("CARGO_PKG_VERSION")
+    )
+}
 
 const GENERATED_MODULE_FOOTER: &str = "\
 }
@@ -3885,7 +3892,7 @@ fn render_lexer(
             "if H::ENABLES_LEXER_LIFECYCLE {{\n            {lifecycle_next_token_call}\n        }} else {{\n            {next_token_call}\n        }}"
         )
     };
-    let generated_header = GENERATED_MODULE_HEADER;
+    let generated_header = generated_module_header();
     let generated_footer = GENERATED_MODULE_FOOTER;
 
     Ok(format!(
@@ -12955,7 +12962,7 @@ fn render_parser_with_decision_report(
         embedded_field_inits,
         embedded_impl_items,
     ) = embedded_render_slots(embedded_data.as_ref().or(structural_surface.as_ref()));
-    let generated_header = GENERATED_MODULE_HEADER;
+    let generated_header = generated_module_header();
     let generated_footer = GENERATED_MODULE_FOOTER;
     let AdaptiveAtnParserRenderSlots {
         struct_field: adaptive_atn_preference_struct_field,
@@ -15815,17 +15822,18 @@ mod tests {
         let parser =
             render_parser("TParser", &minimal_parser_data()).expect("parser should render");
 
-        assert!(GENERATED_MODULE_HEADER.contains(concat!(
+        let generated_module_header = generated_module_header();
+        assert!(generated_module_header.contains(concat!(
             "@generated by ",
             env!("CARGO_PKG_NAME"),
             " v",
             env!("CARGO_PKG_VERSION")
         )));
-        assert!(GENERATED_MODULE_HEADER.contains(env!("CARGO_PKG_REPOSITORY")));
+        assert!(generated_module_header.contains(env!("CARGO_PKG_REPOSITORY")));
 
         for rendered in [&lexer, &parser] {
-            assert!(rendered.starts_with(GENERATED_MODULE_HEADER));
-            assert!(rendered[GENERATED_MODULE_HEADER.len()..].starts_with("use antlr4_runtime::"));
+            assert!(rendered.starts_with(&generated_module_header));
+            assert!(rendered[generated_module_header.len()..].starts_with("use antlr4_runtime::"));
             assert!(!rendered.contains("#!["));
             assert!(rendered.contains(GENERATED_MODULE_FOOTER));
             assert!(

src/lib.rs

@@ -2,6 +2,28 @@
 
 extern crate self as antlr4_runtime;
 
+/// Current generated-source/runtime contract revision emitted by the bundled generator.
+#[doc(hidden)]
+pub const __ANTLR4_RUST_CODEGEN_API: u32 = 1;
+
+/// Verifies that generated source is compatible with the selected runtime.
+#[doc(hidden)]
+#[macro_export]
+macro_rules! __antlr4_rust_require_codegen_api {
+    (1, $generator_version:literal) => {};
+    ($requested:literal, $generator_version:literal) => {
+        compile_error!(concat!(
+            "antlr4-rust generated-code API mismatch: antlr4-rust-gen v",
+            $generator_version,
+            " emitted generated-code API revision ",
+            stringify!($requested),
+            ", but the selected antlr-rust-runtime supports revision 1; regenerate this \
+             recognizer with a compatible antlr4-rust-gen or select a compatible \
+             antlr-rust-runtime dependency"
+        ));
+    };
+}
+
 pub mod atn;
 pub mod byte_stream;
 pub mod char_stream;

tests/antlr4_rust_gen_cli.rs

@@ -17,6 +17,16 @@ fn assert_generated_modules_compile(temp_dir: &Path, modules: &[&str]) {
 }
 
 fn assert_generated_project(temp_dir: &Path, modules: &[&str], test_source: &str) {
+    let output = run_generated_project(temp_dir, modules, test_source);
+    assert!(
+        output.status.success(),
+        "generated project failed\nstdout: {}\nstderr: {}",
+        utf8(&output.stdout),
+        utf8(&output.stderr)
+    );
+}
+
+fn run_generated_project(temp_dir: &Path, modules: &[&str], test_source: &str) -> Output {
     let project = temp_dir.join("compile-generated");
     let source = project.join("src");
     let uses_insta = test_source.contains("insta");
@@ -61,7 +71,7 @@ fn assert_generated_project(temp_dir: &Path, modules: &[&str], test_source: &str
             .expect("generated module should be copied into the check crate");
     }
 
-    let output = Command::new(env!("CARGO"))
+    Command::new(env!("CARGO"))
         .args([
             if test_source.is_empty() {
                 "check"
@@ -78,19 +88,26 @@ fn assert_generated_project(temp_dir: &Path, modules: &[&str], test_source: &str
         ])
         .env("CARGO_TARGET_DIR", project.join("target"))
         .output()
-        .expect("cargo check should run");
-    assert!(
-        output.status.success(),
-        "generated project failed\nstdout: {}\nstderr: {}",
-        utf8(&output.stdout),
-        utf8(&output.stderr)
-    );
+        .expect("cargo check should run")
 }
 
 fn utf8(bytes: &[u8]) -> &str {
     std::str::from_utf8(bytes).expect("process output should be UTF-8")
 }
 
+fn normalize_current_package_version(value: &str) -> String {
+    const PLACEHOLDER: &str = "<generator-version>";
+    let version = env!("CARGO_PKG_VERSION");
+    let mut normalized = value.to_owned();
+    let mut search_from = 0;
+    while let Some(relative_start) = normalized[search_from..].find(version) {
+        let start = search_from + relative_start;
+        normalized.replace_range(start..start + version.len(), PLACEHOLDER);
+        search_from = start + PLACEHOLDER.len();
+    }
+    normalized
+}
+
 fn generated_parser_api(source: &str) -> Vec<String> {
     let mut api = BTreeSet::new();
     for line in source.lines().map(str::trim) {
@@ -232,6 +249,78 @@ fn long_and_short_version_exit_successfully_on_stdout() {
     }
 }
 
+#[allow(clippy::disallowed_methods)] // `insta` assertion macros unwrap internal I/O.
+#[test]
+fn generated_modules_enforce_codegen_api_compatibility() {
+    let temp = temporary_directory("codegen-api-compatibility");
+    let grammar = temp.path().join("CodegenApi.g4");
+    let out = temp.path().join("generated");
+    fs::write(
+        &grammar,
+        "grammar CodegenApi;\n\
+         root : ID EOF ;\n\
+         ID : [a-z]+ ;\n\
+         WS : [ \\t\\r\\n]+ -> skip ;\n",
+    )
+    .expect("grammar should be writable");
+
+    let output = run_antlr4_rust_gen(&[
+        grammar.as_os_str(),
+        OsStr::new("--out-dir"),
+        out.as_os_str(),
+    ]);
+    assert!(
+        output.status.success(),
+        "stdout: {}\nstderr: {}",
+        utf8(&output.stdout),
+        utf8(&output.stderr)
+    );
+
+    let lexer_path = out.join("codegen_api_lexer.rs");
+    let parser_path = out.join("codegen_api_parser.rs");
+    let lexer = fs::read_to_string(&lexer_path).expect("lexer should be emitted");
+    let parser = fs::read_to_string(&parser_path).expect("parser should be emitted");
+    let lexer_check = lexer
+        .lines()
+        .find(|line| line.contains("__antlr4_rust_require_codegen_api"))
+        .expect("lexer should require a generated-code API");
+    let parser_check = parser
+        .lines()
+        .find(|line| line.contains("__antlr4_rust_require_codegen_api"))
+        .expect("parser should require a generated-code API");
+    let checks = format!("lexer: {lexer_check}\nparser: {parser_check}");
+    insta::assert_snapshot!(
+        "generated_codegen_api_checks",
+        normalize_current_package_version(&checks)
+    );
+
+    let modules = ["codegen_api_lexer.rs", "codegen_api_parser.rs"];
+    assert_generated_modules_compile(temp.path(), &modules);
+
+    let current = format!(
+        "__antlr4_rust_require_codegen_api!({},",
+        antlr4_runtime::__ANTLR4_RUST_CODEGEN_API
+    );
+    let unsupported = "__antlr4_rust_require_codegen_api!(999,";
+    let mut incompatible_parser = parser;
+    let check_start = incompatible_parser
+        .find(&current)
+        .expect("parser check should contain the current API revision");
+    incompatible_parser.replace_range(check_start..check_start + current.len(), unsupported);
+    fs::write(parser_path, incompatible_parser).expect("parser should be writable");
+
+    let output = run_generated_project(temp.path(), &modules, "");
+    assert!(
+        !output.status.success(),
+        "unsupported generated-code API unexpectedly compiled"
+    );
+    assert_eq!(utf8(&output.stdout), "");
+    insta::assert_snapshot!(
+        "generated_codegen_api_mismatch_diagnostic",
+        normalize_current_package_version(utf8(&output.stderr))
+    );
+}
+
 #[test]
 fn help_flag_as_option_value_is_not_intercepted() {
     let output = run_antlr4_rust_gen(&["--option-hook", "--help"]);

tests/snapshots/antlr4_rust_gen_cli__generated_codegen_api_checks.snap

@@ -0,0 +1,6 @@
+---
+source: tests/antlr4_rust_gen_cli.rs
+expression: normalize_current_package_version(&checks)
+---
+lexer: antlr4_runtime::__antlr4_rust_require_codegen_api!(1, "<generator-version>");
+parser: antlr4_runtime::__antlr4_rust_require_codegen_api!(1, "<generator-version>");

tests/snapshots/antlr4_rust_gen_cli__generated_codegen_api_mismatch_diagnostic.snap

@@ -0,0 +1,13 @@
+---
+source: tests/antlr4_rust_gen_cli.rs
+expression: normalize_current_package_version(utf8(&output.stderr))
+---
+error: antlr4-rust generated-code API mismatch: antlr4-rust-gen v<generator-version> emitted generated-code API revision 999, but the selected antlr-rust-runtime supports revision 1; regenerate this recognizer with a compatible antlr4-rust-gen or select a compatible antlr-rust-runtime dependency
+ --> src/codegen_api_parser.rs:3:1
+  |
+3 | antlr4_runtime::__antlr4_rust_require_codegen_api!(999, "<generator-version>");
+  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  |
+  = note: this error originates in the macro `antlr4_runtime::__antlr4_rust_require_codegen_api` (in Nightly builds, run with -Z macro-backtrace for more info)
+
+error: could not compile `compile-generated` (lib) due to 1 previous error