refactor(codegen): serialize JSON artifacts with serde

[tinovyatkin]

Summary

• replace handwritten serializers for semantics.json, decisions.json,
  optimizations.json, and rust-support.json with typed Serde models
• persist Rust support trust decisions as versioned
  trusted-support.json while preserving locked merges and atomic replacement
• replace the codegen-only toml parser stack with serde_json and use the
  same serializer for the G4 frontend corpus snapshots
• regenerate the checked-in Rust syntax manifests and remove the obsolete
  whole-test-module hash from the six affected Phase A evidence ledgers

Why

Codegen emitted several JSON artifacts through three independent escaping and
layout implementations. Meanwhile, the toml dependency was used only for the
small persistent trust store. A single serializer reduces format-handling code
owned by the project and removes the larger TOML parser stack.

The existing generated manifest schemas and values are unchanged. Every
updated snapshot was compared as parsed, sorted JSON before acceptance; only
pretty-print layout changed. Generated Rust and the generated-code/runtime API
contract are unaffected.

The default persistent trust path changes from trusted-support.toml to
trusted-support.json. This trust-store feature has not appeared in a published
release, so there is no released TOML store to migrate.

The port-evidence validator still checks all 523 active ledgers, declared
evidence files, and remaining locked sections. The removed lock hashed the
entire frontend test module, coupling evidence validity to implementation-only
test helpers such as JSON string escaping.

Validation

• cargo test --locked --workspace --all-features
• cargo test --locked -p antlr-rust-g4-parser --all-features
• cargo test --locked -p antlr-rust-codegen --test antlr4_rust_gen_cli rust_support::
• cargo clippy --locked --workspace --all-targets --all-features -- -D warnings
• cargo fmt --all -- --check
• tools/rust-syntax/update-generated.sh --check
• node tools/grammar-frontend/validate-port-evidence.mjs
• node tools/grammar-frontend/validate-interp-fixtures.mjs
• git diff --check
• parsed old/new manifest snapshot comparison with jq -S
• dependency audit confirming no toml package remains and the runtime graph
  contains neither toml nor serde_json

[github-actions]

Copy/Paste Detection

No duplications found in 10 changed non-generated Rust file(s) (threshold: 100 tokens).

[coderabbitai]

📝 Walkthrough

Walkthrough

The code generator replaces manual JSON and TOML serialization with typed serde and serde_json serialization. Trust stores now use JSON files. Manifest tests parse structured JSON values.

Changes

JSON Serialization Migration

Layer / File(s)	Summary
Serialization foundation crates/antlr-rust-codegen/Cargo.toml, crates/antlr-rust-codegen/src/json.rs, crates/antlr-rust-codegen/src/lib.rs, crates/antlr-rust-g4-parser/Cargo.toml, crates/antlr-rust-g4-parser/src/frontend.rs	Adds the shared to_pretty_json helper and uses serde_json for JSON string escaping.
Typed manifest generation crates/antlr-rust-codegen/src/grammar/transform/artifact.rs, crates/antlr-rust-codegen/src/rust_support/mod.rs, crates/antlr-rust-codegen/src/semantics/manifest.rs	Defines serializable manifest structures and renders optimization, Rust support, semantics, and decisions manifests through typed JSON serialization.
JSON trust-store persistence crates/antlr-rust-codegen/src/rust_support/identity.rs, crates/antlr-rust-codegen/README.md, crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/rust_support.rs	Changes trust-store persistence, default paths, documentation, and integration tests from TOML to JSON.
Structured manifest validation crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/semantics.rs	Updates tests to deserialize manifests and validate structured option and parser-predicate fields.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• ophi-dev/antlr-rust-runtime#304: Directly updates trust-store persistence and Rust-support artifact manifest JSON serialization.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: replacing manual JSON artifact serialization with Serde in code generation.

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/codegen-serde-json

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[claude]

Claude finished @tinovyatkin's task in 45m 1s —— View job

---

Code review

Reviewed origin/main...HEAD (both commits). The core migration is solid — I independently verified the "schemas and values are unchanged" claim rather than taking it on trust, and it holds. Two issues are worth addressing before merge, one of which is scope-related rather than a serialization defect.

---

🔴 1. The port-evidence oracle lock was deleted instead of re-pinned

8aef0a9b sets lockedSections: [] in tools/grammar-frontend/generate-port-evidence.mjs and regenerates six Phase A ledgers to "locked_oracle_sections": []. The causal chain is:

1. crates/antlr-rust-g4-parser/src/frontend.rs:1323-1326 rewrites push_json_string — a #[cfg(test)] helper, in a different crate, that is not one of the JSON artifacts this PR is about.
2. That helper lives inside the #[cfg(test)] section pinned by locked_oracle_sections, so its sha256 moved:

   recorded / base : c07602789cd12a9618a6c1049260d2d140677992319c1b8d1cfe1066f4500f4e
   HEAD            : 41bc0b54f57a1b1021129f9d6e1dff8b93458fef2a5dc02d8b616d8293131c85
   

3. validate-port-evidence.mjs would have failed with locked oracle section hash differs. The response was to remove the lock rather than re-record it.

Also deleted along the way: REFACTOR_BASE_COMMIT, normalizeRelocatedFrontendTests, and two independent guards that had nothing to do with the manifest digest — throw new Error("refactor changed the grammar frontend tests") and throw new Error("implementation commit changed the locked frontend test module").

Net effect (verified):

	base	HEAD
manifests pinning frontend.rs	6	0
CI result	would fail	passes vacuously

validate-port-evidence.mjs:351 is for (const section of manifest.locked_oracle_sections ?? []), so an empty array makes the body dead. The step is .github/workflows/ci.yml:45-46 ("Validate port evidence", job Clippy and Unit Tests) and it exits 0.

Why this trade is bad: the change it accommodates has provably zero output impact. serde_json's escape table escapes exactly 0x00–0x1F, ", \ with the same \b \t \n \f \r shortcuts and lowercase \u00xx, and leaves 0x7F unescaped — byte-identical to the hand-rolled escaper it replaces (the hashed corpus snapshots are unchanged, which confirms it). So the cost is permanently unpinning the ported frontend test oracle for six ledgers, in exchange for deduplicating four lines of test-only code.

Suggested fix: revert frontend.rs:1323-1326 to the hand-rolled escaper. That restores all six locks and the two refactor-equivalence guards at zero functional cost, and drops the now-unneeded serde_json dev-dependency from antlr-rust-g4-parser. If the whole-module lock genuinely is obsolete (as the commit message argues), that's a defensible position — but it deserves its own PR with that rationale, not a drive-by inside a serializer refactor, and it isn't mentioned in the PR description.

Fix this →

---

🟠 2. Trust-store format switch: silent reset by default, hard error via the documented env var

crates/antlr-rust-codegen/src/rust_support/identity.rs:288-311, 320-360. The PR body acknowledges "an existing approval must be recorded again", but the two concrete failure modes are worse than a re-prompt:

(i) Default path — silent reset. The .json path doesn't exist, read_trust_file maps NotFound → empty store, and every persisted decision is discarded with no message. For an interactive user that's a re-prompt; for a non-interactive run with a warmed store, ensure_trusted (rust_support/mod.rs:385-392) now hard-fails:

Rust target support is not trusted

source: <repo>/Rust
fingerprint: sha256:<...>

rerun with --trust-rust-support sha256:<...>

(ii) ANTLR4_RUST_TRUST_STORE at an existing .toml — hard error. crates/antlr-rust-codegen/README.md:73 explicitly documents pointing it at an arbitrary file, and it's read at cli.rs:204. serde_json::from_str fails, wrapped at identity.rs:295-300 and propagated fatally through mod.rs:376-377:

invalid Rust support trust store: expected value at line 1 column 1

Nothing hints that the file is simply in the legacy TOML format — the build just stops until the user finds and deletes it. Only --trust-rust-support <fp> masks either case (it short-circuits at mod.rs:372-374 before TrustStore::load).

This is the same concern Codex raised. A minimal fix: when the parse fails, check whether the content isn't JSON (or the path ends in .toml) and emit a diagnostic naming the format change — or treat a legacy file as an empty store with a warning. A full read-and-migrate would be nicer but isn't required.

Fix this →

---

🟡 3. Nit: stale decisions.json sample in the root README

README.md:826 still shows a compact one-line row:

{"decision": 4, "rule": "namespace_", "state": 113, "canDefer": true, "tier": "fixed", "lookahead": 2}

The generator now pretty-prints — each row is seven indented lines. Keys, values, and ordering are still correct, so this is layout drift only. crates/antlr-rust-codegen/README.md was updated in this PR (trust-store path), so this one just got missed.

---

✅ Verified clean — so reviewers don't re-litigate these

• Schema/value equivalence holds. All seven pre-existing manifest snapshots are jq -S-identical old vs new. crates/antlr-rust-rs-parser/src/generated/{decisions,semantics}.json are identical including key order (order-preserving parse at both revisions). No key renamed, reordered, or type-widened.
• snake_case survived where it had to. SemanticsCoordinateManifest and friends carry no rename_all, so rule_index / atn_state stay snake_case; probedLookahead carries the explicit #[serde(rename)] it needs under rename_all = "lowercase" (which renames variants only); #[serde(flatten)] preserves the old canDefer, tier, lookahead order.
• CodeRabbit's skip_serializing_if request is a false positive. The old writers emitted explicit null for boundaryRule, projected, path, start, end — json_optional_string returned the literal "null" and write_line_column pushed "null". Adding skip_serializing_if = "Option::is_none" would change the artifact shape, not preserve it. Current code is correct.
• CodeRabbit's insta-allow finding is already satisfied. identity.rs:363 carries #[allow(clippy::disallowed_methods)] on mod tests, and both tests/antlr4_rust_gen_cli/{rust_support,semantics}.rs have crate-level #![allow(clippy::disallowed_methods)] at line 1.
• Codex's "regenerate checked-in Rust syntax manifests" was addressed in 8aef0a9b.
• Trailing newlines preserved — all four replaced writers ended with exactly one \n at base; to_pretty_json appends exactly one.
• to_pretty_json's .expect() is unreachable for these models: no floats, no non-string map keys, no i128/u128, no fallible custom Serialize.
• TRANSFORM_PATH centralization is behavior-preserving, including on Windows (Path::join treats / as a separator, so is_file() and file_name() are unchanged; the only consumer is a cargo::rerun-if-changed print).
• Dependency hygiene is correct. serde_json is dev-only for antlr-rust-g4-parser (normal graph is just antlr-rust-runtime); toml is gone from Cargo.lock entirely; no toml:: references remain in the codegen crate.
• No __ANTLR4_RUST_CODEGEN_API bump needed — no generated .rs shape change anywhere in the diff.
• CLAUDE.md audit found no violations. The antlr-rust-g4-parser/Cargo.toml realignment matches the repo's own .taplo.toml (align_entries = true), so it's conformant churn rather than a defect — though it is unrelated noise in this diff.

---

_{Note: cargo test -p antlr-rust-codegen --all-features fails multi_recognizer::three_build_dependency_consumers_share_codegen_and_track_only_resolved_inputs in this sandbox because its nested unlocked build can't compile rustpython-vm (a nix i16→i32 mismatch). That file is untouched by this PR and the --locked workspace build plus all 891 lib tests / 92 CLI tests pass — environmental, not a PR defect.}
• branch codex/codegen-serde-json

[coderabbitai]

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/antlr-rust-codegen/src/grammar/transform/artifact.rs`:
- Around line 15-125: Preserve omitted optional fields in the optimization
manifest serialization. In
crates/antlr-rust-codegen/src/grammar/transform/artifact.rs (lines 15-125), add
serde skip-if-none handling to boundary_rule, projected, and SourceManifest
fields path, start, and end. In
crates/antlr-rust-codegen/src/rust_support/mod.rs (lines 73-89), apply the same
handling to any corresponding optional serialized fields so the legacy artifact
shape remains unchanged.

In `@crates/antlr-rust-codegen/src/rust_support/identity.rs`:
- Around line 413-416: Add #[allow(clippy::disallowed_methods)] to the mod tests
declaration in identity.rs, covering all insta snapshot macros within that test
module.

In `@crates/antlr-rust-codegen/src/rust_support/mod.rs`:
- Line 214: Update the manifest construction in the module’s transform
configuration to derive the transform value from the existing TRANSFORM_FILE
symbol instead of the hardcoded "Rust/transformGrammar.py" literal. Define a
shared module-level manifest-path constant next to TRANSFORM_FILE if needed,
ensuring the path used by the generator and the manifest remain synchronized
when the filename changes.

In `@crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/rust_support.rs`:
- Around line 226-230: The tests currently assert JSON content via
formatting-sensitive substring checks. Replace the raw-text policy assertions in
the shown manifest validation and in
current_java_transform_runs_from_the_staged_tree with structured lookups on the
parsed serde_json::Value, asserting the policy value equals "assume-true";
retain the existing JSON parsing and other test behavior.
- Around line 237-246: The complete JSON structure assertion in the enclosing
test should use a named external insta snapshot instead of the hand-written
serde_json::json! assert_eq!. Replace the assertion around option with an
available snapshot macro, place the named snapshot under the sibling snapshots
directory, and confirm the test has #[allow(clippy::disallowed_methods)].

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: bb4e90f5-5ae0-4640-a070-c17e5a8defca

📥 Commits

Reviewing files that changed from the base of the PR and between cdb3a17 and 330f783.

⛔ Files ignored due to path filters (10)

• Cargo.lock is excluded by !**/*.lock
• crates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__fixed_lookahead_decisions_manifest.snap is excluded by !**/*.snap
• crates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__render_forced_adaptive_decisions_manifest.snap is excluded by !**/*.snap
• crates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__semantics_manifest_with_untranslated_predicate.snap is excluded by !**/*.snap
• crates/antlr-rust-codegen/src/rust_support/snapshots/antlr_rust_codegen__rust_support__identity__tests__persisted_trust_store.snap is excluded by !**/*.snap
• crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__compatibility__antlr4rust_compat_semantics_manifest.snap is excluded by !**/*.snap
• crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__optimizations__ll1_no_fallback_decisions_manifest.snap is excluded by !**/*.snap
• crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__semantics__named_parser_actions_semantics_manifest.snap is excluded by !**/*.snap
• crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__semantics__recog_receiver_semantics_manifest.snap is excluded by !**/*.snap
• crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__transforms__precedence_ladder_optimization_manifest.snap is excluded by !**/*.snap

📒 Files selected for processing (12)

• crates/antlr-rust-codegen/Cargo.toml
• crates/antlr-rust-codegen/README.md
• crates/antlr-rust-codegen/src/grammar/transform/artifact.rs
• crates/antlr-rust-codegen/src/json.rs
• crates/antlr-rust-codegen/src/lib.rs
• crates/antlr-rust-codegen/src/rust_support/identity.rs
• crates/antlr-rust-codegen/src/rust_support/mod.rs
• crates/antlr-rust-codegen/src/semantics/manifest.rs
• crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/rust_support.rs
• crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/semantics.rs
• crates/antlr-rust-g4-parser/Cargo.toml
• crates/antlr-rust-g4-parser/src/frontend.rs

[codecov]

Codecov Report

❌ Patch coverage is 93.50181% with 18 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...tlr-rust-codegen/src/grammar/transform/artifact.rs	92.53%	10 Missing ⚠️
crates/antlr-rust-codegen/src/rust_support/mod.rs	83.33%	6 Missing ⚠️
...es/antlr-rust-codegen/src/rust_support/identity.rs	77.77%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 330f7832de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[tinovyatkin]

Addressed the Claude review in 8aef0a9b:

• regenerated and committed the Rust syntax semantics.json and decisions.json; tools/rust-syntax/update-generated.sh --check passes and parsed values are unchanged
• changed recog_receiver_semantics_manifest to snapshot the raw emitted manifest so production key order remains covered
• replaced the five-field option equality with a named external insta snapshot
• fixed the CI port-evidence failure by removing the obsolete whole-frontend-test-module lock from the six affected ledgers while retaining validation of all 523 active ledgers and remaining locked sections

Also reran workspace clippy, all Rust-support CLI tests, both grammar-frontend validators, and the Rust syntax drift check.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8aef0a9be0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[tinovyatkin]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 8aef0a9be0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

📊 Source Code Metrics (this PR vs main)

File	Cyclomatic	Cognitive	Functions	LLOC	MI
crates/antlr-rust-g4-parser/src/frontend.rs	217 (main: 227) 🟢	88 (main: 91) 🟢	68 ⚪	307 (main: 311) 🟢	0 ⚪
crates/antlr-rust-codegen/src/rust_support/identity.rs	96 (main: 97) 🟢	53 (main: 54) 🟢	23 ⚪	162 (main: 161) 🔴	0 ⚪
crates/antlr-rust-codegen/src/rust_support/mod.rs	125 (main: 136) 🟢	84 (main: 94) 🟢	22 (main: 23) 🟢	133 (main: 160) 🟢	0 ⚪
crates/antlr-rust-codegen/src/grammar/transform/artifact.rs	26 (main: 54) 🟢	2 (main: 41) 🟢	12 (main: 14) 🟢	14 (main: 122) 🟢	14.11 (main: 10.04) 🟢
crates/antlr-rust-codegen/src/semantics/manifest.rs	21 (main: 43) 🟢	4 (main: 40) 🟢	6 (main: 8) 🟢	13 (main: 98) 🟢	18.64 (main: 15.22) 🟢
crates/antlr-rust-codegen/src/lib.rs	3 ⚪	0 ⚪	2 ⚪	4 ⚪	38.36 (main: 38.57) 🔴
crates/antlr-rust-codegen/src/json.rs	2 🆕	0 🆕	1 🆕	3 🆕	63.69 🆕
crates/antlr-rust-codegen/src/rust_support/python.rs	11 ⚪	7 ⚪	1 ⚪	27 ⚪	36.57 (main: 36.52) 🟢

Generated by mehen v1.8.1 — the code quality watcher.