Skip to content
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
b8ccf2e
Implement semantic predicate accountability hooks
tinovyatkin Jul 4, 2026
568a824
Complete semantic plan delivery
tinovyatkin Jul 5, 2026
c695193
Fix grammar-source scanning and close review gaps in semantic delivery
tinovyatkin Jul 5, 2026
3bcc2f2
Close review gaps: fail-loud semantic boundary + manifest provenance
tinovyatkin Jul 6, 2026
88e9fbd
Extend fail-loud boundary to generated-direct + lexer hook policy
tinovyatkin Jul 6, 2026
bfee243
Filter parser/lexer predicate scans to their own rules
tinovyatkin Jul 6, 2026
ec7e495
Make predicate rule-name filter conservative on unresolvable headers
tinovyatkin Jul 6, 2026
d8f3fb9
Close remaining fail-loud + combined-grammar gaps in semantics codegen
tinovyatkin Jul 6, 2026
e2e4120
Surface fail-loud semantic error before replaying buffered actions
tinovyatkin Jul 6, 2026
900f9f1
Distinguish native '<' predicates from templates; filter predicate pr…
tinovyatkin Jul 6, 2026
4f3a052
Honor per-coordinate lexer hook + grammarless overrides
tinovyatkin Jul 6, 2026
960b2fc
Honor per-coordinate lexer assume-false; filter parser actions first
tinovyatkin Jul 6, 2026
1f6482d
Preserve unknown predicate hits across nested interpreted parses
tinovyatkin Jul 6, 2026
1231dad
Make per-coordinate overrides win over built-in translations
tinovyatkin Jul 6, 2026
10ceefa
Escalate to Error policy when a predicate is hook-disposed
tinovyatkin Jul 6, 2026
6c3b3c7
Keep parser actions with unresolvable headers; keep composite overrid…
tinovyatkin Jul 6, 2026
0634873
Never report lexer actions as hooked
tinovyatkin Jul 7, 2026
945f590
Honor lexer action hook override at render; fail loud on unhandled ac…
tinovyatkin Jul 7, 2026
29931f1
Scope hook fail-loud per coordinate; clear hits on fail-loud return
tinovyatkin Jul 7, 2026
bc4b976
Let typed action hooks report handled; surface replay-time action misses
tinovyatkin Jul 7, 2026
29f9874
Propagate fail-loud semantic coordinates through generated parents
tinovyatkin Jul 7, 2026
817074a
Allow mixed translated + untranslated lexer predicates
tinovyatkin Jul 7, 2026
b69e952
Close 4 remaining fail-loud semantics-codegen gaps
tinovyatkin Jul 7, 2026
f65922e
Guard semantic-decision predicate hooks behind lookahead
tinovyatkin Jul 8, 2026
053d7e1
Keep assume-* action overrides out of the hook fallback
tinovyatkin Jul 8, 2026
36e5e83
Apply @init member writes before same-rule predicates
tinovyatkin Jul 8, 2026
dfe9cab
Skip unresolved-FIRST alts in semantic-alt fallback search
tinovyatkin Jul 8, 2026
e717e01
Keep unresolved semantic alts as last-resort fallbacks
tinovyatkin Jul 8, 2026
c8c92e0
Preserve <fail=...> messages for all predicate templates
tinovyatkin Jul 8, 2026
19d1f85
Respect quoted strings when stripping --sem-patterns comments
tinovyatkin Jul 8, 2026
0bc64e8
Give lexer action hooks mutable lexer state
tinovyatkin Jul 8, 2026
07a9cbf
Disambiguate typed-hook predicate method from custom_action
tinovyatkin Jul 8, 2026
f72c6d9
Honor lexer coordinate overrides and single-quoted TOML in sem patterns
tinovyatkin Jul 8, 2026
5a59429
Resolve rule name past a block comment in action attribution
tinovyatkin Jul 9, 2026
f9b401e
Classify ANTLR-synthetic parser actions; fix rule-header attribution
tinovyatkin Jul 9, 2026
69f1bf8
Skip grammar-level declaration blocks in rule-name resolution
tinovyatkin Jul 9, 2026
189f74f
Route synthetic action states to a run_action no-op arm
tinovyatkin Jul 9, 2026
8680035
Document ANTLR's cross-target embedded-action gap
tinovyatkin Jul 9, 2026
ca98b75
Classify synthetic actions by source-span attribution, not position
tinovyatkin Jul 9, 2026
84099e4
Remove conformance-faking RuleValue evaluator; honest 341/357 baseline
tinovyatkin Jul 9, 2026
0b6e728
Add honest Rust.test.stg reference + runtime-gap analysis
tinovyatkin Jul 9, 2026
95163e4
Validate Rust.test.stg with real ST4; correct reference defects
tinovyatkin Jul 9, 2026
2d7a456
Add runtime surface for embedded test actions
tinovyatkin Jul 9, 2026
4f4e515
Embedded-actions pipeline: render Rust.test.stg, splice verbatim Rust
tinovyatkin Jul 10, 2026
a960546
Embedded mode: token consts, rule args, list/literal labels, per-rule…
tinovyatkin Jul 10, 2026
8b86eef
Embedded lexer actions/predicates; manifest walks skip rendered sources
tinovyatkin Jul 10, 2026
57de09a
Typed context views, listener traits, walker bridge; ideal token text…
tinovyatkin Jul 10, 2026
1567ab9
Composites, honest full-context diagnostics, Java-style DFA dump
tinovyatkin Jul 10, 2026
e8de0f2
Clippy pedantic pass: shared replace_all, extracted render helpers
tinovyatkin Jul 10, 2026
4b4c40e
Clippy -D warnings cleanup for the embedded pipeline
tinovyatkin Jul 10, 2026
59b44be
README: document --embedded conformance pipeline and --actions embedded
tinovyatkin Jul 10, 2026
aad15bf
Skip context-view rule accessors that shadow built-in methods
tinovyatkin Jul 10, 2026
bc4b1ef
Retire the legacy template-recognition conformance pipeline
tinovyatkin Jul 10, 2026
537afe5
Snake-case listener callback names derived from alt labels
tinovyatkin Jul 10, 2026
f079506
Fix LR operator-alt classification and prevctx attrs sealing
tinovyatkin Jul 10, 2026
a4b7b87
Fix five conformance clusters; skip-list honest residuals
tinovyatkin Jul 10, 2026
60144bc
Gap doc: note the legacy pipeline's retirement
tinovyatkin Jul 10, 2026
9539006
Drop the --embedded flag entirely
tinovyatkin Jul 10, 2026
b4edad1
Generalize built-in view-accessor yielding to child_count
tinovyatkin Jul 10, 2026
25de401
Merge parallel remote fixes, superseded by this line
tinovyatkin Jul 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,16 @@ cargo run --release --bin antlr4-rust-gen -- \
The output crate must depend on this runtime (`antlr-rust-runtime = { path = ... }`).
Both the kotlin-parity dumper and the parse-bench runner are examples.

Every run also writes a `semantics.json` manifest into `--out-dir` listing each
semantic predicate/action coordinate and its disposition. `--sem-unknown
error|hook|assume-true|assume-false`, `--sem-patterns`, and
`--require-full-semantics` control untranslated coordinates (default
`assume-true`, deprecated; see the README "Semantic Predicates and Actions"
section and issue #9).
Generated parsers emit SemIR tables, `with_hooks(tokens, hooks)`, and typed
hook adapters for bare helper predicates; lexer callers can route closure hooks
through `LexerSemCtx` and the shared `SemanticHooks` trait.

## Kotlin parser parity perf benchmark

Reproduces the timings against the Kotlin grammar from `antlr/grammars-v4`.
Expand Down
69 changes: 69 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,75 @@ See [docs/kotlin-build.md](docs/kotlin-build.md) for the Kotlin smoke workflow.
See [docs/runtime-testsuite.md](docs/runtime-testsuite.md) for the upstream
runtime-testsuite harness.

### Semantic Predicates and Actions: the Compatibility Boundary

ANTLR grammars may embed **target-language** semantic predicates and actions
(`{isTypeName()}?`, `{this.count++;}`). The serialized ATN records *where*
they occur, but not executable code, so a metadata-first runtime cannot run
arbitrary grammar-embedded snippets. The boundary is:

- **Target-agnostic grammars** — no embedded code, or only built-in lexer
commands (`skip`, `channel(...)`, `mode(...)`, `type(...)`) — are fully
supported.
- **Recognized predicate/action shapes** — a library of common idioms
(constant predicates, lookahead text/type checks, integer member counters,
column predicates, and the upstream testsuite's action templates) — are
translated into SemIR by `antlr4-rust-gen` when the grammar source is
passed via `--grammar`.
- **User pattern files** — `--sem-patterns file.toml` can add exact predicate
rewrites, helper-call rewrites, and per-coordinate `hook` /
`assume-true` / `assume-false` / `error` dispositions without changing the
generator.
- **Everything else is not silently guessed.** Each generator run writes a
`semantics.json` manifest next to the generated modules listing every
predicate/action coordinate with its grammar source span, body, and
disposition (`translated`, `hooked`, `assume-true`, `assume-false`,
`ignored`, or `error`).

Unknown coordinates are governed by `--sem-unknown`:

```bash
antlr4-rust-gen --lexer L.interp --parser P.interp --grammar G.g4 \
--out-dir src/generated --sem-unknown error
```

- `assume-true` (current default, deprecated): unknown predicates pass,
unknown actions are no-ops — the historical behavior. A future minor
release changes the default to `error`.
- `hook`: unknown parser predicates are routed to `SemanticHooks` and fail if
the hook does not handle them.
- `assume-false`: unknown predicates fail, removing the guarded alternatives.
- `error`: generation fails, naming each coordinate:

```text
unsupported semantic predicate: rule=s(0) pred_index=0 at 2:4: {isTypeName()}
```

At runtime the same policy exists as
`ParserRuntimeOptions::unknown_predicate_policy`
(`UnknownSemanticPolicy::{AssumeTrue, AssumeFalse, Error}`); under `Error`,
evaluating an unknown predicate coordinate fails the parse with
`AntlrError::Unsupported` instead of producing a tree whose shape silently
depended on a guess.

Generated parsers also expose a parser-side hook escape hatch:
`MyParser::with_hooks(tokens, hooks)`, where `hooks` implements
`SemanticHooks`. Unknown parser predicates are offered to
`SemanticHooks::sempred` before the fallback policy is applied, and unhandled
parser action events are offered to `SemanticHooks::action` after the committed
parse path is selected. Predicate hooks may run speculatively during
prediction, so they must be replay-safe.

For bare helper-call predicates, generated parsers also emit a typed hook
adapter (`MyParserHooks` plus `MyParserTypedHooks<T>`) that maps stable
manifest coordinates to named Rust methods. Lexer callers can use
`LexerSemCtx` with `atn::lexer::next_token_with_semantic_hooks` or the
compiled-DFA variant to route lexer predicates/actions through the same
`SemanticHooks` trait.

Use `--require-full-semantics` in CI when every coordinate must be either
translated or explicitly hooked; policy fallbacks fail generation.

## Runtime Testsuite

On the maintainer checkout, where the ANTLR jar and upstream runtime-testsuite
Expand Down
Loading
Loading