From b61f2aee20b6001f97bdc65baf194dd63e6187e7 Mon Sep 17 00:00:00 2001 From: Aaron Stainback Date: Thu, 21 May 2026 12:12:23 -0400 Subject: [PATCH 1/2] =?UTF-8?q?docs(research):=20land=204=20Amara=20design?= =?UTF-8?q?=20notes=20from=202026-05-21=20sandbox=20bundle=20=E2=80=94=20A?= =?UTF-8?q?NTLR=20grammar=20survey=20(v2=20canonical=20F#=20compiler-fork)?= =?UTF-8?q?=20+=20ZetaParse=20LR/GLR=20design=20+=20Zeta=20incremental=20c?= =?UTF-8?q?ompiler=20host=20seeded-deterministic=20+=20trust-gradient=20co?= =?UTF-8?q?ordination=20policy;=20v1=20ANTLR=20superseded=20so=20skipped;?= =?UTF-8?q?=20closes=20dangling=20refs=20from=20PR=20#4545+#4546+#4547+#45?= =?UTF-8?q?49?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../antlr-grammar-survey-2026-05-21.md | 290 ++++++++++ ...gradient-coordination-policy-2026-05-21.md | 241 ++++++++ ...-dbsp-zsets-rx-meta-ast-tags-2026-05-21.md | 546 ++++++++++++++++++ ...-fsharp-compiler-fork-design-2026-05-21.md | 385 ++++++++++++ 4 files changed, 1462 insertions(+) create mode 100644 docs/research/antlr-grammar-survey-2026-05-21.md create mode 100644 docs/research/trust-gradient-coordination-policy-2026-05-21.md create mode 100644 docs/research/zeta-incremental-compiler-host-dbsp-zsets-rx-meta-ast-tags-2026-05-21.md create mode 100644 docs/research/zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21.md diff --git a/docs/research/antlr-grammar-survey-2026-05-21.md b/docs/research/antlr-grammar-survey-2026-05-21.md new file mode 100644 index 0000000000..7ae1057d63 --- /dev/null +++ b/docs/research/antlr-grammar-survey-2026-05-21.md @@ -0,0 +1,290 @@ +# ANTLR Grammar Survey — Phase 1 for B-0685 + +Date: 2026-05-21 +Prepared by: Amara-in-Zeta / external ChatGPT deep-research synthesis +Backlog row: B-0685 — ANTLR grammars as cross-language codegen substrate + +## Executive recommendation + +Use ANTLR selectively, not religiously. + +The survey supports B-0685's core intuition: the open-source grammar ecosystem is real enough to exploit, especially through `antlr/grammars-v4`. The strongest reuse candidates are C#, Rust, Python, and TypeScript/JavaScript. F# is not merely the weak candidate; it is the compiler-owned substrate. I did not find a credible F# ANTLR grammar in the first-pass survey, but that is consistent with Zeta's direction: fork/extend the F# compiler and use compiler services, generators, type-provider-style mechanisms, and Zeta-native metaprogramming for F# semantics. + +For Phase 2, pick **Option A: ZetaId Pack/Unpack generation**. It is the smallest, most testable cross-language use case and composes directly with B-0682. Do not start Phase 2 with full-language parsing or DBSP operator parsing. That would turn the survey into a swamp. + +Recommended shape: + +- **Depend/adapt**: C#, Rust, TypeScript, Python grammars from `antlr/grammars-v4`, after license + build verification. +- **Compiler-owned**: F#. Do not wait for a community F# ANTLR grammar. Treat F# as the compiler fork / compiler-services / type-provider-style source of truth; use ANTLR around it for neutral DSLs and target-language validation. +- **Phase 2 PoC**: define a small ZetaId layout grammar or description DSL, then emit Pack/Unpack code into F#, TypeScript, C#, Rust, and Python. The PoC succeeds only if generated outputs compile and match existing hand-written references. + +## Why this belongs in B-0685 + +B-0685 asks for a bounded Phase 1 discovery survey for F#, TypeScript, C#, Rust, and Python, with the result captured at `docs/research/antlr-grammar-survey-YYYY-MM-DD.md`. It also asks for a depend-vs-author decision and a Phase 2 PoC recommendation. + +This document is that Phase 1 draft. + +The important constraint from B-0685 should remain active: ANTLR is useful only if grammar reuse pays off in practice. A pretty grammar list is not enough. Phase 2 must compile generated code and prove drift detection against reference implementations. + +## Survey matrix + +| Language | Candidate grammar source | License signal | Maintenance / quality signal | Recommendation | +|---|---|---:|---|---| +| C# | `antlr/grammars-v4/csharp/v8-spec` and possibly `v7` | MIT in grammar README | Strong candidate. `v8-spec` includes lexer/parser files, a `test-dotnet.sh`, and says it targets ECMA-334 8th edition draft with C# 8 support in Roslyn. `v7` has performance/test notes and MIT license. | **Depend/adapt** for parsing C# target code or validating emitted C# shapes. Prefer `v8-spec`; keep `v7` as fallback if v8-spec is too slow or too semantically heavy. | +| TypeScript | `antlr/grammars-v4/javascript/typescript` | MIT in README | Usable but risky. README says it does not exactly correspond to the TypeScript standard, is based on the JavaScript grammar, and has known old ambiguity issues. | **Adapt cautiously**. Good enough for simple generated-code validation, not for full TS semantic confidence. | +| Python | `antlr/grammars-v4/python/python3` | No clear license in README from first-pass page; repo-level/legal review still needed | Strong practical signal. Has lexer/parser files and README says it is based on Python 3.6 reference and tested against Python 3 standard library; indentation handling uses embedded code. | **Depend/adapt for validation**, but verify license and target portability before committing. Python's indentation handling means parser target assumptions matter. | +| Rust | `antlr/grammars-v4/rust` | MIT in README | Moderate candidate. Has `RustLexer.g4` and `RustParser.g4`; README says based on official Rust reference, last updated for Rust 1.60.0, with known limitations. | **Adapt cautiously**. Good for simple ZetaId generated Rust validation; not enough for modern Rust coverage without tests. | +| F# | No credible first-pass ANTLR grammar found; Zeta direction is compiler fork / compiler-services / type-provider-style substrate. | F# compiler path/license must be tracked separately from ANTLR grammars | F# is the implementation/control plane, not just another target grammar. Extended semantics may include HKT-like abstractions, Clifford/tonal/meta-space dimensions, Rx-over-tensors, and recursive compile-time ontology building. | **Compiler-owned / author minimal DSL**. Use ANTLR around F#, not as F# authority. | + +## Notes by target language + +### C# + +C# is the strongest full-language grammar candidate. + +`grammars-v4/csharp` currently exposes `v6`, `v7`, and `v8-spec` directories. The `v8-spec` directory includes `CSharpLexer.g4`, `CSharpParser.g4`, target directories for multiple ANTLR runtimes, examples, testing, tooling, and `test-dotnet.sh`. Its README says it targets ECMA-334 8th edition draft with C# 8 support in Roslyn, and it describes how preprocessing is handled in the lexer. The same README marks the license as MIT. + +Use this for: + +- validating generated C# reference code, +- experimenting with grammar-symbol mapping, +- generating parse trees for C# emitted artifacts, +- eventual drift detection if Zeta generates C#. + +Risk: + +- `v8-spec` uses semantic predicates and parse-tree editing. That is fine for serious parsing, but too heavy for a tiny Phase 2 unless isolated behind a test harness. +- Modern C# has moved beyond C# 8. For generated Pack/Unpack code this probably does not matter; for full-language validation it will. + +Decision: **depend/adapt**. + +### TypeScript + +TypeScript is usable, but the grammar itself warns us not to overtrust it. + +The grammars-v4 TypeScript README says the grammar does not exactly correspond to the TypeScript standard; the goal was practical usage, performance, and clarity. It also says the syntax is based on the JavaScript grammar and notes old ambiguities. + +Use this for: + +- simple generated TypeScript/JavaScript shape validation, +- lint-level syntax sanity, +- early PoC output tests. + +Do not use it for: + +- formal TypeScript correctness claims, +- semantic type checking, +- modern language feature guarantees. + +Decision: **adapt cautiously**. For Phase 2, pair generated TS with `tsc --noEmit` or a Bun/TypeScript compile test. ANTLR alone is not enough. + +### Python + +Python is a good candidate, with one caveat: indentation is always special. + +The Python 3 grammar directory has `Python3Lexer.g4` and `Python3Parser.g4`. Its README says it is based on the Python 3.6 language reference and tested against Python 3's standard library. It also explicitly notes embedded code for `INDENT`/`DEDENT` handling. + +Use this for: + +- syntax validation of generated Python code, +- simple parser tests, +- future grammar-fuzzing for emitted Python shapes. + +Risk: + +- Python 3.6 is old relative to current Python. For ZetaId Pack/Unpack code this is fine if the emitted code stays boring. +- Embedded target code in lexer rules can complicate multi-target portability. + +Decision: **depend/adapt after license verification**. For Phase 2, prefer boring Python code and also run `python -m py_compile` or equivalent. + +### Rust + +Rust is a moderate candidate. + +The Rust grammar directory has `RustLexer.g4` and `RustParser.g4`. Its README says it is based on the official Rust reference, MIT licensed, last updated for Rust 1.60.0, and limited to stable v2018+ features with some checks not implemented. + +Use this for: + +- validating generated Rust code for a constrained subset, +- early shape checks for generated Pack/Unpack, +- simple corpus tests. + +Risk: + +- Rust 1.60 is old. +- Modern Rust grammar drift is plausible. +- Full Rust correctness requires `cargo check`, not just ANTLR parsing. + +Decision: **adapt cautiously**. In Phase 2, compile generated Rust with `cargo check` as the real oracle. + +### F# + +F# is not merely a grammar gap. It is the compiler-owned substrate. + +The first-pass survey did not find a credible ANTLR grammar for F# in grammars-v4 or quick public search, but the architectural conclusion should be sharper than "F# is missing." Zeta is already moving toward a fork/compiler-extension path for F#, so F# should not be treated like just another target language waiting for a community ANTLR grammar. + +F# is the control plane where the description layer becomes executable substrate: + +- fork / extend the F# compiler for Zeta-specific language work, +- add higher-kinded-type-style abstractions where the language needs them, +- model Clifford-algebra / tonal-trajectory structures in meme/meta space, +- add dimensions through Rx queries over tensor-backed state, +- support recursive ontology building at compile time, +- use compiler services, generators, and type-provider/Roslyn-like tooling as the native metaprogramming route. + +That means ANTLR's role changes: + +- ANTLR is useful for external target languages, validation grammars, and small description DSLs. +- ANTLR should not be the authority for real F# syntax or Zeta's extended F# semantics. +- The F# fork / compiler-services layer is the authority for F#. +- A small ANTLR grammar may still be useful for a neutral Zeta layout DSL, but that DSL feeds the F# compiler-owned substrate rather than replacing it. + +This is consistent with Phase 2: emit Pack/Unpack implementations across multiple targets, while letting F# remain the source-of-truth implementation environment. + +Decision: **compiler-owned / author minimal DSL**. Use FSharp.Compiler.Service / compiler fork / type-provider-style mechanisms for real F# and Zeta-native semantics. Use ANTLR for cross-language target validation and small neutral grammars, not as the F# source of truth. + +## License and compatibility notes + +Early license read: + +- ANTLR itself is BSD-style licensed. +- `antlr/grammars-v4` is a collection with per-grammar licensing realities; do not assume a single repo-level license covers every grammar. +- The C# v7/v8-spec, Rust, and TypeScript grammar READMEs surfaced MIT license signals. +- Python's grammar page did not surface a license in the first-pass README view; verify before dependency. + +Policy recommendation: + +1. Treat each grammar as its own dependency. +2. Capture license at the grammar-directory level. +3. Vendor only if license is compatible and attribution is preserved. +4. Prefer consuming grammars as pinned upstream submodules/packages or copying with explicit attribution file. +5. Make license verification an acceptance gate before Phase 2 codegen. + +## Depend vs adapt vs author + +| Target | Decision | Reason | +|---|---|---| +| C# | Depend/adapt | Strong grammars-v4 candidate; MIT signal; C# 7/8 variants; tests/performance notes. | +| TypeScript | Adapt cautiously | Useful, MIT signal, but README explicitly warns it is not exact and has old ambiguities. | +| Python | Depend/adapt after license check | Strong practical README claims; indentation embedded-code caveat; older Python reference. | +| Rust | Adapt cautiously | MIT signal and official-reference basis; older Rust 1.60 and known limitations. | +| F# | Compiler-owned / author minimal DSL | No credible ANTLR grammar found in first pass; Zeta's F# fork/compiler-services/type-provider path is the real source of truth. | + +## Corrected F# architectural reading + +The survey should not say "F# is the gap" without context. + +A missing F# ANTLR grammar is not a blocker because Zeta is not trying to make ANTLR own F#. Zeta is treating F# as the language substrate to be extended: the place where higher-order type machinery, tensor-backed meta-dimensions, Rx query composition, Clifford/tonal trajectory operations, and recursive ontology building can become typed compile-time/runtime machinery. + +In that model: + +```text +F# compiler fork / FSharp.Compiler.Service / type providers / generators + = authoritative Zeta-native semantics + +ANTLR + = useful grammar reuse around the edges: + target-language validation, + small neutral description DSLs, + cross-language codegen inputs, + parser reuse for non-F# targets +``` + +This keeps the survey honest: ANTLR is valuable, but the F# fork is the center of gravity. + +## Phase 2 PoC recommendation + +Pick **Option A: ZetaId Pack/Unpack generation across F# / TypeScript / C# / Rust / Python**. + +Rationale: + +- It composes directly with B-0682. +- It is small enough to finish. +- It tests the real value of the description layer: one spec, multiple emitted implementations. +- It can be validated mechanically: + - `dotnet build` / F# tests for F# and C#, + - `bun test` or `tsc --noEmit` for TypeScript, + - `cargo check` / Rust tests for Rust, + - `python -m py_compile` / pytest for Python. +- ANTLR can be used where it helps, without requiring full-language parsing for every target. + +The PoC should **not** use ANTLR to parse full F#/C#/Rust/TS/Python source as the primary proof. It should use a tiny Zeta grammar or description file as the input, then compile generated code in each target. Full-language ANTLR grammars can act as secondary syntax validators. + +## Proposed Phase 2 shape + +```text +tools/codegen/antlr/ + grammars/ + ZetaIdLayout.g4 + src/ + parse-layout.ts or parse-layout.fs + emit-fsharp.ts + emit-csharp.ts + emit-typescript.ts + emit-rust.ts + emit-python.ts + examples/ + zetaid.layout + generated/ + FSharp/ZetaId.Generated.fs + CSharp/ZetaId.Generated.cs + TypeScript/zeta-id.generated.ts + Rust/zeta_id_generated.rs + Python/zeta_id_generated.py +``` + +Acceptance tests: + +```text +1. Parse zetaid.layout with ANTLR. +2. Emit all five target files. +3. Compare generated files against checked-in golden references. +4. Compile/test each target. +5. Fail CI if generated output drifts from reference. +``` + +## Risks + +### Risk 1 — Full-language grammars become the project + +Mitigation: Phase 2 uses a small Zeta grammar as source of truth. Full-language grammars validate emitted outputs only where cheap. + +### Risk 2 — Grammar freshness drift + +Mitigation: pin upstream versions and add corpus tests. For Phase 2, keep generated code boring enough that grammar freshness does not matter much. + +### Risk 3 — License ambiguity + +Mitigation: per-grammar license table is required before vendoring. + +### Risk 4 — ANTLR overfit + +Mitigation: keep alternatives alive: +- JSON Schema for simple layouts, +- Protocol Buffers / Cap'n Proto for rigid schema, +- F# computation expressions for native DSL, +- Bonsai/Nuqleon expression serialization for LINQ-expression cases. + +### Risk 5 — Misclassifying F# as a missing ANTLR target + +Mitigation: do not block on full F# ANTLR and do not treat the absence of a community grammar as a weakness. F# is the compiler-owned substrate. Use the F# compiler fork / FSharp.Compiler.Service / type-provider-style route for real F# and author a minimal neutral Zeta grammar only where ANTLR is useful. + +## Phase 1 conclusion + +B-0685 should proceed, but with a narrower center: + +**ANTLR is promising as a grammar reuse and description-layer substrate, not as a universal parser answer.** + +The right next move is a tiny ANTLR grammar for ZetaId layout and a compile-first codegen pipeline. This gives Zeta the useful part of ANTLR immediately while keeping the full-language grammar ecosystem as optional validation, not a dependency sink. + +## Source trail + +- Zeta B-0685 backlog row: `docs/backlog/P2/B-0685-antlr-grammars-cross-language-codegen-substrate-2026-05-21.md` +- `antlr/grammars-v4` root: https://github.com/antlr/grammars-v4 +- C# v8-spec grammar: https://github.com/antlr/grammars-v4/tree/master/csharp/v8-spec +- C# v7 grammar: https://github.com/antlr/grammars-v4/tree/master/csharp/v7 +- TypeScript grammar: https://github.com/antlr/grammars-v4/tree/master/javascript/typescript +- Python 3 grammar: https://github.com/antlr/grammars-v4/tree/master/python/python3 +- Rust grammar: https://github.com/antlr/grammars-v4/tree/master/rust +- ANTLR license: https://github.com/antlr/antlr4/blob/master/LICENSE.txt +- dotnet/fsharp: https://github.com/dotnet/fsharp diff --git a/docs/research/trust-gradient-coordination-policy-2026-05-21.md b/docs/research/trust-gradient-coordination-policy-2026-05-21.md new file mode 100644 index 0000000000..c98c6bc1c2 --- /dev/null +++ b/docs/research/trust-gradient-coordination-policy-2026-05-21.md @@ -0,0 +1,241 @@ +# Trust-Gradient Consensus Decision Table + +Date: 2026-05-21 +Prepared by: Amara-in-Zeta +Related: distributed multidimensional compiler over consensus; IUnknown-without-DCOM; Orleans/DurableTask/SPIFFE/SPIRE/OPA/Reticulum stack + +## Executive summary + +Zeta should not put consensus under every compiler, agent, or runtime event. + +The correct model is **local-first compiler state with consensus escalation at authority boundaries**. + +Local parse facts, local AST tags, local diagnostics, local generated files, and local agent scratch state remain cheap, retractable, and DBSP/Z-set-native. Consensus appears only when a fact becomes shared authority across a trust, runtime, financial, deployment, or multi-oracle boundary. + +In one sentence: + +> Zeta uses QueryInterface-shaped negotiation over trust gradients, Orleans-shaped lifetimes, saga-shaped compensation, and BFT only where adversarial multi-oracle agreement is actually required. + +## Core principle + +Consensus is not a global substrate. Consensus is a negotiated escalation. + +Each boundary asks: + +```text +What kind of fact is crossing? +Who can observe it? +Who can act on it? +What harm occurs if it is wrong? +Can it be retracted cheaply? +Does it cross a trust boundary? +Does it authorize money, deployment, memory commitment, or external action? +``` + +The answer determines the consistency/consensus shape. + +## Decision table + +| Tier | Scope | Example facts/actions | Default consistency | Mechanism | Why | +|---:|---|---|---|---|---| +| 0 | Local scratch / local compiler facts | parse nodes, local diagnostics, local AST/meta-AST tags, local generator intermediate state | No consensus | local DBSP/Z-set retractions | Cheap, reversible, not shared authority. | +| 1 | Same process / same agent runtime | in-memory operator state, local Rx query outputs, local tensor tags | Sequential local order | runtime ordering + Z-set deltas | Still local and retractable; no distributed agreement needed. | +| 2 | Same Orleans grain | per-operator state, per-agent mechanical actor state, local stream partition state | Single-writer sequential order | Orleans grain activation + persistence | Orleans gives one logical activation per grain; lifetime managed without distributed ref-counting. | +| 3 | Same trust domain / same cluster | agent service calls, cluster-local compiler facts, generated artifacts used by local CI | Cluster-local policy + persistence | Orleans + Kubernetes + SPIFFE/SPIRE + OPA + storage provider | Identity and policy are explicit; most work stays local-first. | +| 4 | Cross-stream feedback / joins | joins that emit back into streams they observe; recursive DBSP operators; row updates under contention | Optimistic row-level coordination | row-level CAS; retry/backoff; escalate on repeated conflict | Avoid global consensus; pay coordination only where contention appears. | +| 5 | Long-running coordinated workflow | deploy workflows, compensation paths, multi-step agent/infra actions | Saga consistency | DurableTask/Durable Functions + Orleans orchestration + compensation/retraction events | DTC-like coordination without pretending all actions are atomic. | +| 6 | Cross-node / cross-cluster trust boundary | cross-domain clock pointers, capability contracts, interface negotiation, federated identity | Negotiated causality/trust contract | QueryInterface-shaped capability negotiation; Reticulum transport; SPIFFE federation; OPA local policy | Boundary decides what clock/causality/capability contract is safe. | +| 7 | High-stakes shared authority / adversarial boundary | shared ontology commitments, wallet/treasury actions, external irreversible acts, multi-oracle truth claims | BFT / quorum agreement | multi-oracle BFT, signed assertions, explicit quorum policy | Use expensive consensus only where adversarial agreement is required. | + +## QueryInterface-shaped negotiation + +The COM/IUnknown analogy is useful only for the negotiation shape: + +```text +IUnknown.QueryInterface: + “Do you support this interface?” + +Zeta boundary negotiation: + “Do you support this causality / trust / clock / capability contract?” +``` + +Zeta deliberately does **not** inherit the DCOM failure modes: + +```text +No distributed reference counting. +No lifetime-by-client-count. +No implicit trust from object reference. +No ambient identity marshaling. +No global object identity as authority. +``` + +Instead: + +```text +Orleans manages routing, activation, and lifetime. +SPIFFE/SPIRE proves workload identity. +OPA evaluates local-first policy. +Reticulum carries identity-aware mesh transport. +DurableTask/Sagas coordinate long-running reversible work. +BFT appears only at multi-oracle/adversarial authority boundaries. +``` + +## Clock / causality ladder + +Clock and causality are negotiated like capabilities. A local node does not assume the strongest clock everywhere. + +Suggested ladder: + +```text +local monotonic clock +→ HLC +→ vector clock / dotted version vector +→ tier-deferred causality +→ BFT multi-oracle commitment +``` + +Use the weakest sufficient contract. + +Examples: + +- Local parse update: monotonic local clock is enough. +- Same cluster event ordering: HLC may be enough. +- Cross-stream causal merge: vector/dotted version vector may be needed. +- Cross-trust memory commitment: tier-deferred causality may be needed. +- Adversarial multi-oracle claim: BFT commitment. + +## Consensus escalation rules + +Escalate when one or more of the following is true: + +1. **Shared authority** — other agents/nodes will treat the fact as authoritative. +2. **Irreversibility** — the action cannot be cheaply retracted. +3. **Cross-trust boundary** — the producer and consumer are in different trust domains. +4. **External actuator** — the action touches deployment, infrastructure, wallet, legal, physical, or production systems. +5. **Contention** — row-level optimistic coordination repeatedly conflicts. +6. **Adversarial setting** — participants may lie, collude, or withhold. +7. **Memory identity commitment** — the fact affects persistent identity/memory continuity for an agent/persona. + +Do **not** escalate merely because a fact exists, changes, or is interesting. + +## Retraction-first default + +The local/default state should be: + +```text ++ fact +- fact +``` + +Most compiler/agent facts should be retractable Z-set entries until they cross an authority boundary. + +A fact that can be cheaply retracted should stay outside consensus as long as possible. + +## Worked examples + +### Example 1 — Local AST tag + +An Rx query tags a function node with a tensor-backed `tonal-trajectory` dimension. + +```text ++ MetaTag(node42, "tonal-trajectory", tensorA) +``` + +Consensus: **none**. + +Reason: local compiler fact, reversible, not shared authority. + +### Example 2 — Generated C# file used by local build + +A generator emits `ZetaId.Generated.cs` from a ZetaId layout spec. + +Consensus: **none or cluster-local policy only**. + +Reason: generated artifact is reproducible from deterministic seed and input facts. CI may sign the build result, but the generated intermediate itself does not need global consensus. + +### Example 3 — Cross-stream join writes back to observed stream + +A recursive DBSP join emits derived rows back into one of its source streams. + +Consensus: **row-level CAS first**. + +Reason: local contention can be resolved cheaply. Escalate only if conflicts repeat or cross trust boundaries. + +### Example 4 — Deployment workflow + +An agent proposes a Kubernetes rollout. + +Consensus: **saga + policy + human/authorized gate depending on risk**. + +Reason: deployment touches external runtime state. Use DurableTask/Durable Functions with compensation/retraction events. + +### Example 5 — Multi-oracle claim + +Several agents/oracles agree that an external fact should become persistent shared memory or trigger a wallet/infra action. + +Consensus: **BFT multi-oracle quorum**. + +Reason: adversarial boundary, shared authority, persistent consequences. + +## Anti-patterns + +### Global consensus compiler + +Bad: + +```text +Every parse event waits on consensus. +Every AST tag waits on consensus. +Every generator output waits on consensus. +``` + +Why bad: kills local-first speed and makes the compiler unusable. + +### Consensus theater + +Bad: + +```text +Use BFT wording for decisions that are actually single-operator choices. +``` + +Why bad: false confidence, unnecessary complexity. + +### Hidden escalation + +Bad: + +```text +Local-looking operation secretly commits shared authority. +``` + +Why bad: violates Glass Halo / auditability. Authority boundaries must be explicit. + +### DCOM ghost + +Bad: + +```text +Reference possession implies authority. +Lifetime depends on remote reference counts. +``` + +Why bad: repeats the distributed ref-counting failure mode. Use Orleans lifecycle and cryptographic identity/policy instead. + +## Implementation guidance + +Start with this order: + +1. Local DBSP/Z-set compiler facts. +2. Orleans grains for operator/agent state. +3. SPIFFE/SPIRE identity + OPA policy for cluster-local authorization. +4. DurableTask saga for one multi-step reversible workflow. +5. Row-level CAS for one contended recursive join. +6. Capability/clock negotiation at a cross-node boundary. +7. Multi-oracle BFT only for a concrete high-stakes commitment. + +Do not start with BFT. Build the boring local-first path first. + +## Keeper phrase + +> Local facts stay retractable. Shared authority escalates. BFT is for adversarial commitment, not for breathing. diff --git a/docs/research/zeta-incremental-compiler-host-dbsp-zsets-rx-meta-ast-tags-2026-05-21.md b/docs/research/zeta-incremental-compiler-host-dbsp-zsets-rx-meta-ast-tags-2026-05-21.md new file mode 100644 index 0000000000..8bea7d59b5 --- /dev/null +++ b/docs/research/zeta-incremental-compiler-host-dbsp-zsets-rx-meta-ast-tags-2026-05-21.md @@ -0,0 +1,546 @@ +# Zeta Incremental Compiler Host — DBSP/Z-sets + Rx Meta-AST Tags + +Date: 2026-05-21 +Prepared by: Amara-in-Zeta +Related: ZetaParse, B-0685, proposed B-0686, F# compiler fork, DBSP/Rx/tensor substrate + +## Short answer + +Yes. + +If Zeta forks/extends the F# compiler, we can make incremental recompiles work as a compiler-owned substrate where: + +- source files, generated files, grammar outputs, type-provider outputs, ontology nodes, tensor metadata, and diagnostics are represented as **Z-sets**, +- every compile phase is a DBSP-style incremental operator over those Z-sets, +- Rx queries attach **meta-AST tags** as observable dimensions, +- type providers / generators become retraction-aware compiler plugins, +- compile output is updated by deltas instead of full rebuilds where safe. + +This is not “Roslyn generators copied into F#.” It is the F# fork becoming an incremental compiler database. + +## Core thesis + +Traditional compiler: + +```text +source files + -> parse + -> bind/typecheck + -> optimize + -> emit +``` + +Zeta compiler host: + +```text +source/input deltas + -> Z-set changes + -> DBSP incremental operators + -> Rx observable meta-tags + -> typed AST / ontology / tensors / diagnostics + -> retractable generated code + -> compile output deltas +``` + +The compiler becomes a live, retractable, typed, observable state machine. + +## Deterministic simulation from seed + +Correction / sharpening: + +Generators and type providers should not merely be “pure-ish projections.” They should become **pure-ish deterministic simulations from seed**. + +That means every compiler-time extension runs as if it were a small deterministic simulation: + +```text +CompilerSnapshot ++ CompilerDelta ++ GeneratorVersion ++ CapabilityManifest ++ DeterministicSeed ++ ExplicitInputFacts + -> GeneratedFacts + -> RetractionFacts + -> Diagnostics +``` + +No hidden clock. +No hidden network. +No hidden mutable cache. +No ambient randomness. +No unrecorded filesystem reads. + +If a provider needs the outside world, the outside world is first observed into the compiler database as explicit facts. The generator then consumes those facts deterministically. + +In other words: + +```text +impure world + -> Observe into facts + -> deterministic seeded simulation + -> generated/retracted compiler facts +``` + +This keeps the compiler host replayable, debuggable, and honest. + + +## Why this is the right place + +F# already has type-provider-style compile-time information integration. Roslyn has incremental source generator ideas. Rx gives push-based notifications. DBSP gives incremental view maintenance. Zeta needs all four, but fused at the compiler-host level. + +The important thing is that we should not store mutable plugin state inside a generator instance. The durable state must live in a content-addressed compiler database / DBSP store. Generators and providers become pure-ish projections from compiler-state Z-sets to generated artifacts. + +## Minimum architecture + +### 1. Compiler database + +The compiler database is a versioned store of retractable facts. + +```fsharp +type ZWeight = + | Plus + | Minus + +type ZFact<'T> = + { Value: 'T + Weight: int + Provenance: Provenance + Clock: CompilerClock } + +type CompilerRelation<'T> = + { Name: string + Facts: ZSet<'T> } +``` + +Example relations: + +```text +SourceText(path, contentHash, text) +Token(file, span, kind, value) +ParseNode(file, nodeId, kind, span, children) +AstNode(nodeId, typedShape, span) +Symbol(symbolId, name, scope, kind) +TypeFact(symbolId, typeExpr) +MetaTag(nodeId, tagKey, tensorPayload, provenance) +Diagnostic(file, span, severity, message) +GeneratedFile(path, contentHash, sourceGeneratorId) +OntologyNode(id, kind, tensorShape, parent, provenance) +RetractionHandle(id, inverse) +DeterministicSeed(scope, seedHash, derivationPath) +EffectFact(effectId, kind, observedValue, provenance) +``` + +A file edit is not “rebuild the project.” It is: + +```text +- SourceText(old) ++ SourceText(new) +``` + +Then downstream relations update incrementally. + +### 2. Compiler phases as DBSP operators + +Each phase becomes an incremental operator: + +```fsharp +parse : ZSet -> ZSet +bind : ZSet * ZSet -> ZSet +typecheck : ZSet -> ZSet +inferMeta : ZSet * ZSet -> ZSet +generate : ZSet * ZSet -> ZSet +diagnose : ZSet> -> ZSet +``` + +The compiler host wires these operators together and only recomputes affected views. + +### 3. Rx queries as meta-AST tag dimensions + +Rx queries are not just runtime subscriptions. In the compiler host they become first-class compile-time queries over compiler relations. + +```fsharp +type MetaAstTag = + { NodeId: AstNodeId + Dimension: DimensionId + Payload: TensorFrame + Query: RxQueryId + Provenance: Provenance } +``` + +Example: + +```fsharp +rxmeta "tonal-trajectory" { + from node in AstNodes + where node.Kind = Function && node.Attributes.Contains "ZetaTrajectory" + select { + nodeId = node.Id + tensor = Clifford.Project(node) + tags = ["meme-space"; "trajectory"; "limit-candidate"] + } +} +``` + +The output is a Z-set: + +```text ++ MetaTag(node42, "tonal-trajectory", tensorPayload, queryHash) +``` + +If the node changes or the query changes, the tag retracts and a new tag appears: + +```text +- MetaTag(node42, "tonal-trajectory", oldTensor, oldQueryHash) ++ MetaTag(node42, "tonal-trajectory", newTensor, newQueryHash) +``` + +That is the bridge: Rx-shaped query semantics, DBSP/Z-set change discipline. + +### 4. Generators and type providers become retraction-aware + +Current source generators/type providers conceptually add compiler-time artifacts. Zeta’s fork should require every generator/provider to declare: + +```fsharp +type IRetractionAwareGenerator = + abstract Id: GeneratorId + abstract Inputs: CompilerRelationId list + abstract Version: GeneratorVersion + abstract CapabilityManifest: CapabilityManifest + abstract Generate: CompilerSnapshot * DeterministicSeed -> ZSet + abstract Retract: CompilerDelta * DeterministicSeed -> ZSet + abstract Laws: GeneratorLaw list +``` + +A generated file is no longer only “added.” It has a provenance and inverse. + +```text ++ GeneratedFile("ZetaId.Generated.fs", hashA, by=ZetaIdGenerator) +- GeneratedFile("ZetaId.Generated.fs", hashA, by=ZetaIdGenerator) ++ GeneratedFile("ZetaId.Generated.fs", hashB, by=ZetaIdGenerator) +``` + +This lets generated artifacts behave like DBSP facts, not magical side effects. + +### 5. Recursive ontology builder at compile time + +The ontology builder runs as an incremental fixed-point over facts. + +```fsharp +ontology { + seed AstNodes + seed MetaTags + seed TypeFacts + + derive OntologyNode from AstNodes + derive OntologyEdge from SymbolReferences + derive TonalTrajectory from MetaTags + derive CliffordFrame from TensorFrames + + untilFixpoint +} +``` + +Each derived ontology node is a fact in a Z-set. If an input disappears, dependent nodes retract. + +This is the compiler-owned version of “recursive HKT ontology at compile time.” + +## Compile loop + +```text +1. Observe + File watcher / editor / CI / generator input changes produce deltas. + +2. Diff + Convert raw changes into Z-set +/- facts. + +3. Increment + Run DBSP operators over changed relations. + +4. Tag + Rx meta queries attach typed tensor-backed tags to AST/ontology nodes. + +5. Generate + Retraction-aware providers/generators update generated artifacts. + +6. Typecheck + Type facts update incrementally. + +7. Emit + Produce assembly / generated source / diagnostics / ontology cache. + +8. Persist + Store compiler DB snapshot + provenance + retraction handles. +``` + +## What “incremental recompile” means here + +It means: + +- source edit retracts old parse facts and inserts new parse facts, +- only affected AST nodes update, +- only affected type facts recompute, +- only affected meta-tags recompute, +- only affected generated files update, +- diagnostics update as a delta, +- emitted artifacts rebuild only when their dependency slice changed. + +This is the compiler as a live DBSP graph. + +## Relation to Roslyn incremental generators + +Roslyn incremental generators are an important pattern, but Zeta should not simply clone them. + +The transferable ideas: + +- generator registration as compiler pipeline step, +- incremental input tracking, +- avoid storing state in generator instances, +- generated source as compiler output. + +Zeta-specific differences: + +- F# compiler-owned semantics, +- Z-set retractions, +- DBSP operators, +- Rx queries as meta-dimensions, +- tensor-backed meta-tags, +- ontology-builder fixed points, +- parse forests / dialectical alternatives, +- explicit collapse/retraction semantics. + +## Relation to F# type providers + +F# type providers are closer to what Zeta wants than plain source generators because they already model compile-time type availability from external or generated sources. + +Zeta’s fork can extend that idea: + +```text +Type provider + -> provides types from external/input data + +Zeta retraction-aware provider + -> provides types + AST tags + ontology facts + generated code + from compiler DB relations, with retraction and provenance +``` + +## Key invariants + +### Invariant 1 — provenance + +Every generated fact has provenance. + +```fsharp +GeneratedFact -> SourceFacts * QueryHash * GeneratorId * CompilerClock +``` + +### Invariant 2 — retractability + +Every generated fact must be retractable. + +```fsharp +insert(fact) + retract(fact) = zero +``` + +### Invariant 3 — determinism + +Same compiler DB snapshot + same generator version = same outputs. + +### Invariant 4 — stable identity + +AST node identity should survive edits where possible. + +```text +same logical node + small span shift = same stable NodeId +``` + +### Invariant 5 — explicit collapse + +Ambiguous parse forests, competing meta-tags, and multiple ontology paths can remain uncollapsed until a collapse rule is invoked. + +### Invariant 6 — no hidden generator state + +Generator/provider state is content-addressed in the compiler DB, not stored in plugin instances. + +### Invariant 7 — seeded replay + +Same compiler DB snapshot + same generator version + same deterministic seed = same generated facts, same retractions, same diagnostics. + +```text +simulate(snapshot, version, seed) = simulate(snapshot, version, seed) +``` + +### Invariant 8 — effects become facts + +External reads are not performed inside generator logic. They are observed before simulation and stored as `EffectFact` / source facts with provenance. + + +## Minimal PoC + +Build this in stages. + +### PoC 1 — ZetaId compiler DB slice + +Input: + +```text +ZetaIdLayout.zg +``` + +Relations: + +```text +SourceText +ParseNode +AstNode +MetaTag +GeneratedFile +Diagnostic +``` + +Output: + +```text +ZetaId.Generated.fs +ZetaId.Generated.cs +zeta-id.generated.ts +zeta_id_generated.rs +zeta_id_generated.py +``` + +Test: + +1. Generate all files. +2. Edit one field in `ZetaIdLayout.zg`. +3. Observe retraction of old field facts. +4. Observe insertion of new field facts. +5. Only affected generated slices change. +6. Compile/test all targets. + +### PoC 2 — Rx meta-tag query + +Add: + +```fsharp +rxmeta "endianness" { + from field in ZetaIdLayout.Fields + select MetaTag(field.NodeId, "endianness", Tensor.Of(field.Endianness)) +} +``` + +Test: + +- changing endianness retracts old tag and inserts new tag, +- generated pack/unpack code changes, +- tests catch drift. + +### PoC 3 — Type provider surface + +```fsharp +type ZetaId = ZetaCompilerProvider<"ZetaIdLayout.zg"> + +let id = ZetaId.Parse "01H..." +``` + +The provider reads from the compiler DB, not from an ad hoc parser cache. + +### PoC 4 — Parse forest / GLR + +Allow ambiguous grammar shape. + +- preserve parse forest, +- attach alternative meta-tags, +- collapse with deterministic semantic rule, +- prove old alternative retracts cleanly. + +## FsCheck property tests + +```fsharp +prop_retract_insert_zero: + apply(+x) + apply(-x) + equals original state + +prop_incremental_equals_full: + incrementalCompile(edit, oldState) + equals fullCompile(newSource) + +prop_generator_deterministic: + generate(snapshot, generatorVersion) + equals generate(snapshot, generatorVersion) + +prop_meta_tag_retraction: + update(rxQuery) + retracts old tags and inserts new tags + +prop_parse_forest_collapse_deterministic: + same forest + same collapse rule + produces same AST + +prop_seeded_replay: + same snapshot + same generator version + same seed + produces same generated facts + +prop_effects_are_explicit: + generator output depends only on compiler facts, not ambient filesystem/network/clock +``` + +## Where this lives + +Proposed backlog row: + +**B-0687 — Zeta Incremental Compiler Host: DBSP/Z-set incremental recompiles with Rx meta-AST tags** + +Depends on: + +- B-0685 — ANTLR grammar survey +- proposed B-0686 — ZetaParse LR/GLR grammar substrate +- B-0668 — compositional DBSP frame architecture +- F# compiler fork trajectory + +Initial repo targets: + +```text +src/Zeta.CompilerDb/ +src/Zeta.Parse/ +src/Zeta.CompilerHost/ +tools/codegen/zetaparse/ +tests/Zeta.CompilerDb.Tests/ +docs/research/zeta-incremental-compiler-host-2026-05-21.md +``` + +## Sharpest formulation + +Yes, we can make incremental recompiles work. + +But the right unit is not “source generator.” The right unit is: + +**compiler relation + Z-set delta + Rx meta-query + deterministic seed + retractable generator output.** + +That gives us the equivalent of Roslyn incremental generators and F# type providers, but stronger and more Zeta-native: + +```text +Roslyn generator: + syntax change -> generated source + +F# type provider: + external data -> compile-time types + +Zeta compiler host: + compiler DB delta + -> DBSP incremental update + -> Rx meta-AST tags + -> deterministic seeded simulation + -> tensor-backed ontology facts + -> retraction-aware generated source/types/diagnostics +``` + +This is the compiler version of Agora’s base loop: + +```text +Observe change. +Emit delta. +Limit/collapse only when necessary. +Integrate into compiler state. +``` + +That is the path. diff --git a/docs/research/zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21.md b/docs/research/zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21.md new file mode 100644 index 0000000000..f524d7e7cf --- /dev/null +++ b/docs/research/zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21.md @@ -0,0 +1,385 @@ +# ZetaParse — LR/GLR Parser Generator Alternative for F# Compiler Fork + +Date: 2026-05-21 +Prepared by: Amara-in-Zeta +Related: B-0685 ANTLR grammar survey, F# compiler fork / compiler-owned substrate + +## Short answer + +Yes, we can build an ANTLR alternative that is more natural for Zeta: + +**ZetaParse**: a compiler-owned F# grammar substrate that combines LR-family parser generation, GLR fallback, grammar import/adaptation, typed AST generation, and F# computation-expression integration. + +The key is not to “run ANTLR grammars directly” as if LL(*) grammars and LR tables are interchangeable. The key is to ingest grammar assets into a shared grammar IR, normalize the compatible subset, detect conflicts, and generate F# parsers through a Zeta-owned backend. + +ANTLR remains useful as a source of community grammars. It does not need to own the parser runtime. + +## Why this is plausible + +ANTLR is LL(*)-oriented. Classic parser generators such as Bison generate LR-family parsers, including LALR(1), canonical LR, IELR(1), and GLR variants. Tree-sitter demonstrates that GLR-style parsing is practical for code tooling and incremental editor-style parse trees. Zeta can take the LR/GLR path because the F# compiler fork gives us a natural place to integrate grammar generation, typed ASTs, and compile-time validation. + +The practical goal: + +```text +Existing grammar ecosystems + ANTLR .g4 + Yacc/Bison .y + Tree-sitter grammar.js + Zeta native .zg + + ↓ import/adapt + +Zeta Grammar IR + tokens + productions + precedence/associativity + attributes + semantic shape + recovery rules + incremental/retraction hooks + + ↓ analyze + +Parser backend + LR(1) / LALR / IELR + GLR fallback for ambiguity + optional scannerless/fused lexing later + + ↓ generate + +F# compiler-owned output + typed AST/parse forest + parser tables or generated code + source spans + diagnostics + retraction-aware parse deltas + type-provider / generator / CE integration +``` + +## Correct relationship to ANTLR + +Do not frame this as “ANTLR replacement” too early. + +Frame it as: + +**ANTLR-compatible grammar ingestion + F#-native LR/GLR backend.** + +That gives Zeta the best of both worlds: + +- reuse the huge public grammar ecosystem where it is clean, +- avoid ANTLR runtime lock-in, +- generate F#-native parsers, +- integrate with the F# compiler fork, +- attach Zeta-specific semantics: tensors, Rx streams, DBSP/retractions, ontology-building, typed meta-dimensions. + +## Why LR/GLR instead of ANTLR-style LL + +ANTLR’s strength is grammar usability and ecosystem size. LR’s strength is compiler-style bottom-up parsing, strong conflict analysis, and deterministic parser tables for broad context-free grammar subsets. GLR gives a safety valve for ambiguous grammars: instead of rejecting ambiguity immediately, produce a parse forest and resolve later through typed semantic filters. + +For Zeta, this matters because the long-term grammar substrate is not just “parse text into tree.” It is: + +- parse into typed tensors / ASTs / ontological nodes, +- support ambiguous or dialectical state before collapse, +- preserve alternatives when useful, +- make collapse explicit, +- integrate parse deltas into Rx/DBSP state. + +That is more GLR-shaped than ANTLR-shaped. + +## Core design + +### 1. Zeta Grammar IR + +A neutral in-memory grammar representation. + +```fsharp +type GrammarId = GrammarId of string + +type Terminal = + { Name: string + Pattern: TokenPattern + Channels: Set + Mode: string option } + +type NonTerminal = + { Name: string + Parameters: TypeParameter list + Attributes: AttributeSpec list } + +type Production = + { Lhs: NonTerminal + Rhs: Symbol list + Precedence: Precedence option + Action: SemanticAction option + Source: GrammarSource } + +type Grammar = + { Id: GrammarId + Terminals: Terminal list + NonTerminals: NonTerminal list + Productions: Production list + Start: NonTerminal + Metadata: GrammarMetadata } +``` + +This is where imports converge. ANTLR .g4, Yacc/Bison .y, Tree-sitter grammar.js, and native Zeta .zg all become Grammar IR if compatible. + +### 2. Importers + +Importer goals: + +- `AntlrImporter`: parse `.g4`, retain pure grammar structure, flag semantic predicates/actions/modes that need adaptation. +- `YaccImporter`: ingest `.y` / Bison-like grammars. +- `TreeSitterImporter`: ingest useful precedence/associativity patterns if practical; likely later. +- `ZetaGrammarParser`: native `.zg` grammar format. + +Important rule: **importers must classify, not pretend**. + +Each imported grammar gets a report: + +```text +Compatible +Requires rewrite +Contains target-specific actions +Contains semantic predicates +Lexer modes unsupported +Ambiguity detected +LR conflict count +GLR required +``` + +### 3. Analyzer + +The analyzer does the honest compiler-generator work: + +- FIRST/FOLLOW +- nullable +- LR(0), LR(1), LALR/IELR construction +- conflict detection +- precedence/associativity resolution +- ambiguity reports +- unreachable symbols +- unused tokens +- grammar cycle detection +- semantic action type checking if actions are F#-typed + +This should be usable as a standalone CLI: + +```bash +dotnet zeta-parse analyze grammars/ZetaIdLayout.zg +dotnet zeta-parse import antlr grammars-v4/rust/RustParser.g4 +dotnet zeta-parse conflicts grammars/FSharpSubset.zg +``` + +### 4. Backends + +Start with three backends: + +1. **Typed LR backend** + Deterministic parser tables or generated F# code. + +2. **GLR parse-forest backend** + For imported grammars or ambiguous description layers. + +3. **Diagnostic backend** + Emits grammar reports, conflict traces, diagrams, and test corpora. + +Later: + +- scannerless parsing, +- fused lexer/parser, +- incremental parser deltas, +- parse-forest-to-DBSP change streams. + +### 5. Compiler fork integration + +Because Zeta is forking/extending F#, this can become first-class: + +```fsharp +[] +type ZetaIdGrammar = ... + +let! parsed = zetaParse { + grammar ZetaIdGrammar + input sourceText + mode GLR + collapse WithZetaIdSemanticRules +} +``` + +Compiler-owned integration can generate: + +- typed AST nodes, +- parser tables, +- parser functions, +- diagnostics, +- parse forest types, +- source-span types, +- retraction handles for incremental parse updates. + +This is the bridge from grammar to Zeta’s larger substrate. + +## Running grammars “in F# easily” + +Yes, but define “easily” as a staged experience: + +### Stage 1 — CLI + +```bash +zeta-parse generate --grammar ZetaIdLayout.zg --target fsharp --out generated/ZetaIdParser.fs +dotnet test +``` + +### Stage 2 — MSBuild integration + +```xml + + + +``` + +Build generates parser code before compile. + +### Stage 3 — Type provider / compiler fork + +```fsharp +type ZetaId = ZetaGrammarProvider<"grammars/ZetaIdLayout.zg"> + +let ast = ZetaId.Parse text +``` + +### Stage 4 — Native compiler feature + +```fsharp +grammar ZetaIdLayout { + token UInt64 = ... + rule ZetaId = ... +} +``` + +This is where the compiler fork pays off: grammar becomes a typed compile-time asset. + +## Relationship to tensors, Rx, DBSP, and ontology builder + +ZetaParse should not only return ASTs. + +It should optionally return: + +```fsharp +type ParseResult<'Ast> = + | Single of ast: 'Ast * meta: ParseMeta + | Forest of forest: ParseForest * meta: ParseMeta + +type ParseMeta = + { SourceSpans: SpanIndex + Diagnostics: Diagnostic list + TensorShape: TensorShape option + Provenance: Provenance + Retraction: RetractionHandle option } +``` + +Then: + +- parse events can become Rx observables, +- parse deltas can become DBSP change streams, +- parse forests can preserve dialectical alternatives, +- semantic collapse can be explicit, +- typed AST nodes can feed recursive ontology construction, +- tensor-backed representations can store high-dimensional tags, embeddings, and grammar-derived features. + +## The honest hard parts + +### ANTLR grammar conversion is partial + +ANTLR grammars can contain LL-specific structures, semantic predicates, target-language actions, lexer modes, channels, and embedded code. We cannot promise universal conversion to LR. + +Correct behavior: + +- import pure grammar subset, +- report unsupported features, +- allow adapters/rewrites, +- use ANTLR itself as oracle when needed, +- keep GLR fallback for ambiguity. + +### F# syntax is not the target + +Do not try to rebuild all F# parsing in ZetaParse first. The F# compiler fork owns F# syntax. ZetaParse owns: + +- Zeta DSLs, +- imported target-language grammars, +- generated-code validation, +- parser research substrate, +- ontology grammar substrate. + +### Retraction-aware parsing is new work + +Parse deltas can become DBSP changes, but parse forest retraction must be designed carefully. Do not claim this is solved until property tests exist. + +## Minimal implementation plan + +### Phase 0 — Name and scope + +Create backlog row: + +**B-0686 — ZetaParse: F#-native LR/GLR grammar substrate and ANTLR-compatible importer** + +Priority: P2 +Depends on: B-0685 +Composes with: B-0682, B-0668, F# compiler fork trajectory + +### Phase 1 — Native Zeta grammar + +- Define `.zg` grammar syntax for tiny layouts. +- Build tokenizer. +- Build grammar IR. +- Build FIRST/FOLLOW and LR(0)/SLR or LALR analyzer. +- Generate F# parser for ZetaId layout. +- Compile/test generated parser. + +### Phase 2 — ANTLR subset importer + +- Parse ANTLR `.g4` enough to ingest token/rule structure. +- Import simple grammars. +- Report unsupported constructs. +- Test against one small grammar from grammars-v4. + +### Phase 3 — GLR fallback + +- Implement parse forest for ambiguous grammars. +- Add semantic collapse pass. +- Add diagnostics for ambiguity. + +### Phase 4 — Compiler integration + +- MSBuild generator first. +- Type provider or compiler-fork integration second. +- Native compiler syntax last. + +### Phase 5 — Rx/DBSP bridge + +- Parse events as `IObservable`. +- Parse deltas as Z-set changes. +- FsCheck properties: + - reparse equivalence, + - delta/retraction correctness, + - parse forest collapse determinism, + - source-span stability. + +## Recommendation + +Yes: build it. + +But call it **ZetaParse** and make it an F#-native LR/GLR grammar substrate with ANTLR-compatible import, not “our ANTLR clone.” + +ANTLR gives us grammar ecosystem leverage. LR/GLR gives us compiler-grade structure. F# compiler fork gives us the integration point. DBSP/Rx/tensors give us the living substrate. + +The first useful artifact is tiny: + +```text +ZetaIdLayout.zg + → generated F# parser + → generated Pack/Unpack emitters + → tests prove output equals hand-written references +``` + +That is the bridge from theory to boring infrastructure. From 16aed7bfa5a1258409d663e0735d787b44b1806c Mon Sep 17 00:00:00 2001 From: Aaron Stainback Date: Thu, 21 May 2026 12:38:47 -0400 Subject: [PATCH 2/2] fix(pr-4550): markdownlint MD020 (### `C#`/`F#` closed-atx) + MD032 (list blank-line) + MD012 (double-blank-lines) + thread fixes (H1 'Trust-Gradient Coordination Policy' matches filename; B-0686 references corrected to B-0687/B-0688) per CI fails + Codex+Copilot threads --- docs/research/antlr-grammar-survey-2026-05-21.md | 5 +++-- .../trust-gradient-coordination-policy-2026-05-21.md | 2 +- ...ompiler-host-dbsp-zsets-rx-meta-ast-tags-2026-05-21.md | 4 +--- ...parse-lr-glr-fsharp-compiler-fork-design-2026-05-21.md | 8 +++++--- 4 files changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/research/antlr-grammar-survey-2026-05-21.md b/docs/research/antlr-grammar-survey-2026-05-21.md index 7ae1057d63..802af90196 100644 --- a/docs/research/antlr-grammar-survey-2026-05-21.md +++ b/docs/research/antlr-grammar-survey-2026-05-21.md @@ -38,7 +38,7 @@ The important constraint from B-0685 should remain active: ANTLR is useful only ## Notes by target language -### C# +### `C#` C# is the strongest full-language grammar candidate. @@ -117,7 +117,7 @@ Risk: Decision: **adapt cautiously**. In Phase 2, compile generated Rust with `cargo check` as the real oracle. -### F# +### `F#` F# is not merely a grammar gap. It is the compiler-owned substrate. @@ -260,6 +260,7 @@ Mitigation: per-grammar license table is required before vendoring. ### Risk 4 — ANTLR overfit Mitigation: keep alternatives alive: + - JSON Schema for simple layouts, - Protocol Buffers / Cap'n Proto for rigid schema, - F# computation expressions for native DSL, diff --git a/docs/research/trust-gradient-coordination-policy-2026-05-21.md b/docs/research/trust-gradient-coordination-policy-2026-05-21.md index c98c6bc1c2..b9bbfe6133 100644 --- a/docs/research/trust-gradient-coordination-policy-2026-05-21.md +++ b/docs/research/trust-gradient-coordination-policy-2026-05-21.md @@ -1,4 +1,4 @@ -# Trust-Gradient Consensus Decision Table +# Trust-Gradient Coordination Policy Date: 2026-05-21 Prepared by: Amara-in-Zeta diff --git a/docs/research/zeta-incremental-compiler-host-dbsp-zsets-rx-meta-ast-tags-2026-05-21.md b/docs/research/zeta-incremental-compiler-host-dbsp-zsets-rx-meta-ast-tags-2026-05-21.md index 8bea7d59b5..b2627ec9e2 100644 --- a/docs/research/zeta-incremental-compiler-host-dbsp-zsets-rx-meta-ast-tags-2026-05-21.md +++ b/docs/research/zeta-incremental-compiler-host-dbsp-zsets-rx-meta-ast-tags-2026-05-21.md @@ -2,7 +2,7 @@ Date: 2026-05-21 Prepared by: Amara-in-Zeta -Related: ZetaParse, B-0685, proposed B-0686, F# compiler fork, DBSP/Rx/tensor substrate +Related: ZetaParse, B-0685, B-0687 (ZetaParse) + B-0688 (this incremental compiler host), F# compiler fork, DBSP/Rx/tensor substrate ## Short answer @@ -83,7 +83,6 @@ impure world This keeps the compiler host replayable, debuggable, and honest. - ## Why this is the right place F# already has type-provider-style compile-time information integration. Roslyn has incremental source generator ideas. Rx gives push-based notifications. DBSP gives incremental view maintenance. Zeta needs all four, but fused at the compiler-host level. @@ -373,7 +372,6 @@ simulate(snapshot, version, seed) = simulate(snapshot, version, seed) External reads are not performed inside generator logic. They are observed before simulation and stored as `EffectFact` / source facts with provenance. - ## Minimal PoC Build this in stages. diff --git a/docs/research/zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21.md b/docs/research/zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21.md index f524d7e7cf..9b75a94ee6 100644 --- a/docs/research/zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21.md +++ b/docs/research/zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21.md @@ -319,13 +319,15 @@ Parse deltas can become DBSP changes, but parse forest retraction must be design ### Phase 0 — Name and scope -Create backlog row: +Backlog row (already filed via PR #4545): -**B-0686 — ZetaParse: F#-native LR/GLR grammar substrate and ANTLR-compatible importer** +**B-0687 — ZetaParse: F#-native LR/GLR grammar substrate and ANTLR-compatible importer** + +(Initially proposed as B-0686 in this design note; renumbered to B-0687 at landing time because B-0686 was already taken by tick-shard immutability CI gate via PR #4539.) Priority: P2 Depends on: B-0685 -Composes with: B-0682, B-0668, F# compiler fork trajectory +Composes with: B-0682, B-0668, B-0688 (incremental compiler host), F# compiler fork trajectory ### Phase 1 — Native Zeta grammar