diff --git a/docs/VISION.md b/docs/VISION.md index eada865b..970fbb11 100644 --- a/docs/VISION.md +++ b/docs/VISION.md @@ -1,122 +1,252 @@ # Zeta — Long-Term Vision -> **Status:** first-draft synthesis by Kenji (architect) from -> everything Aaron has said across rounds 1-33. Aaron is the -> source of truth and will redefine this document forever. The -> `product-visionary` role (to be spawned, see -> `docs/BACKLOG.md`) will steward it after that. +> **Status:** round 33 v2 after Aaron's first pass of edits. +> Aaron is the source of truth; this document changes freely. +> The `product-visionary` role (to be spawned, see +> `docs/BACKLOG.md`) will steward it once it exists. +> +> **Lore:** Aaron almost named the project "Database." The +> codename landed on Zeta, but the ambition stayed: all DB +> technologies in one big playground, built retraction-native +> from the ground up. -## North star +## The project has two products -Zeta is the world's best retraction-native incremental-computation -library. It is: +This is load-bearing and worth stating first. Zeta is: -- **Mathematically honest.** Every operator obeys laws Milewski - would recognise. When a law cannot hold, the API tells you - so, loudly and at compile time. +1. **A full database** (the product the code produces), and +2. **A cross-platform, AI-automated software factory** (the + project that produces the code). + +Both are first-class. A decision that optimises (1) at the +cost of (2) — or vice versa — is a design-doc event. + +## Product 1 — Zeta the database + +### North star + +Zeta is a research-grade, retraction-native database. Every +DB technology worth studying lives here as a playground: the +goal is to explore the full database surface built honestly +on DBSP foundations, not to carve out a narrow niche. + +- **Mathematically honest.** Every operator obeys laws + Milewski would recognise. When a law cannot hold, the API + tells you so, loudly and at compile time. +- **Retraction-native by construction.** DBSP's differentiator + is exactly-symmetric insert/retract. Designs that break + retraction are wrong, not merely uncool. - **Publishable.** Every major feature is a research - contribution (paper target) or an explicitly named engineering - fundamental. No clever-but-unjustified abstractions. -- **First-class F#, with a C# shim.** F# is the primary - language; C# callers get `Zeta.Core.CSharp`. Polyglot over - time (Lean for proofs, Java for the Alloy driver, TypeScript - for post-install automation once we research the runtime), - but F# stays load-bearing. -- **Retraction-native.** DBSP's differentiator is exactly- - symmetric insert/retract. Designs that break retraction are - wrong by construction, not by taste. + contribution (paper target) or an explicitly named + engineering fundamental. No clever-but-unjustified + abstractions. +- **First-class F#, polyglot over time.** F# is the primary + language; C# callers get `Zeta.Core.CSharp`. Lean for + proofs, Java for the Alloy driver, TypeScript (or + researched alternative) for post-install automation. F# + stays load-bearing. - **Production-grade security.** Nation-state + supply-chain threat model. SLSA ladder L1 → L3 pre-v1.0. OpenSpec is first-class for every committed artefact, including CI. -- **Research-worthy performance.** Every hardware intrinsic we - can reach (SIMD, tensor ops, SIMD-JSON-shape parsing, +- **Research-worthy performance.** Every hardware intrinsic + we can reach (SIMD, tensor ops, SIMD-JSON-shape parsing, cache-line discipline). Benchmarks that answer specific questions, not vanity numbers. -- **Fully self-directed over time.** The software factory - itself is the project's second product. Every round improves - both Zeta and the factory. +- **Multi-node by design.** Single-node is the first + shippable subset, not the ceiling. Distribution + (control-plane + data-plane, consensus, sharding, + cross-node retractions) is explicitly in scope. + +### What the DB eventually covers + +Aaron's "all the DB technologies in one big playground" is +the long-term scope. Non-exhaustive menu: + +- **Incremental query engine** (DBSP core; already shipping). +- **Storage substrate** (DiskBackingStore, Spine family, + durability modes, Witness-Durable Commit). +- **Sketches + approximate query** (retraction-aware HLL, + CountMin, KLL, Bloom family; HyperBitBit; retraction- + native quantile sketches). +- **Streaming / CDC ingestion** (NATS JetStream, Kafka- + shaped, Arrow Flight, typed event sources). +- **Query frontends** (SQL eventually; also the native F# + DSL + plugin operator surface; possibly a query IR that + multiple frontends target). +- **Planner / optimiser** (cost model, SIMD kernel + dispatch, join ordering, predicate pushdown, adaptive + re-planning under retractions). +- **Multi-node control plane** (shape TBD — NATS vs gRPC + vs Arrow Flight vs bespoke; sharding, replication, + consensus, info-theoretic sharder). +- **CRDT / replication layer** (OrSet already shipping; + more as multi-node matures). +- **Bitemporal / time-travel** (append-dated history, + retraction-aware point-in-time queries). +- **Formal verification surface** (18+ TLA+ specs, Alloy + structural invariants, Lean proofs where the math is + foundational). + +### v1.0 subset + +What makes `Zeta.Core 1.0.0` on NuGet: + +- Operator algebra with LawRunner-verified tags (linear, + bilinear, sink-terminal, retraction-complete). +- Recursion — retraction-safe `Recursive` combinator; + `RecursiveSemiNaive` where monotonicity holds; honest + LFP termination. +- Storage layer — four durability modes (in-memory, + OS-buffered, stable-storage, witness-durable) with + honest recovery-property advertisement. +- Sketches — retraction-aware HLL, CountMin, KLL, Blocked + plus Counting Bloom. +- Query planner — cost model with SIMD / tensor kernel + dispatch. +- Plugin surface — `PluginOp<'TIn, 'TOut>` or equivalent. +- FsCheck LawRunner — `checkBilinear`, + `checkSinkTerminal`, `checkRetractionCompleteness` + already landed. +- Formal-method coverage — TLA+ / Alloy specs running + green in CI. +- CI parity + security posture — see Product 2 below. + +### Post-v1 exploration + +- Witness-Durable Commit protocol (paper-worthy). +- Retraction-aware analytic sketches (HyperBitBit, + retraction-native quantiles; publication target). +- Info-theoretic sharder (Alloy-verified). +- Multi-node deployment (control-plane shape is an open + research question in-scope for this vision, not + deferred out). +- SQL frontend on top of the algebra. +- Bitemporal query surface. + +## Product 2 — The software factory + +### Why the factory is first-class + +Aaron: "this whole factory is self-directed and fully +automated." Round 32: "the cross platform AI automated +software factory, eventually with UI too but that's way +down the road." + +The factory is not a means to an end. It's the second +product. Every round improves both Zeta-the-database AND +Zeta-the-factory; a round that ships a feature while +degrading the factory is a net-negative round. + +### Factory north star + +- **AI-automated.** Agents (personas) do the work; + humans set direction, review, and ratify. See + `AGENTS.md` + `docs/PROJECT-EMPATHY.md` for the agent + contract. +- **Cross-platform.** Dev-laptop (macOS + Linux today, + Windows via PowerShell when it lands) + CI runner + + devcontainer all bootstrap via the same source of truth. + Post-install automation moves to a single cross-platform + runtime (Bun/Deno/Python/.NET — research-pending). +- **Declarative dependencies.** Every installed tool lives + in a committed manifest. `../scratch`'s tiered shape + (`min` / `runner` / `quality` / `all`) is the ratchet + target. Non-declarative installation is a smell. +- **Research-level reproducibility.** Pin everything pinnable + (SHAs, versions, lock files). SLSA ladder. Reproducible + builds are a long-term goal gated on upstream compiler + work. Every run should be replayable. +- **Symmetry.** Configs, ignore lists, editor settings, + lint scopes — the same ignore list should appear in every + tool that needs one. Asymmetry is a bug (round-33 + vscode-was-gitignored surfacing is the canonical example). +- **Production-ready quality.** The factory itself ships + at production-grade: `0 Warning(s) / 0 Error(s)` build + gate, Semgrep-in-CI, shellcheck-in-CI, actionlint-in-CI, + markdownlint-in-CI, all green on main. A red factory is + a factory down. +- **Fully self-directed eventually.** The loop is + upstream-signals + research + novel-ideas → vision- + check → backlog → next-steps → round work → merge. + Humans set direction; agents run the loop. UI comes + "way down the road" — text-first today. + +### v1.0 subset of the factory + +- Three-way-parity install script (GOVERNANCE §24) — + done. +- CI gate with build + lint jobs — done. +- Persona memory as directories (NOTEBOOK + MEMORY + OFFTIME) — done. +- OpenSpec as first-class for every committed artefact + (GOVERNANCE §28) — done. +- Backlog scoping (GOVERNANCE §29) — done. +- `docs/VISION.md` — this document, lands round 33. +- All gap-finders (skill / openspec / static-analysis) — + in progress across rounds 33-35. +- Declarative-manifest tiered shape — ratchet across + 5-8 rounds. +- Upstream sync script + `references/upstreams/` — pending. + +### Post-v1 factory work + +- `product-visionary` role spawn (BACKLOG P1). +- Factory UI (far future — agents + human today). +- Fully self-directed loop (all gap-finders running on + cadence, feeding backlog autonomously, Aaron reviews + at round-close). +- SLSA L3 reproducibility once upstream compiler work + lands. +- Full polyglot repo-automation runtime (post-install + cross-platform scripting — researched-pending). ## What Zeta is NOT -- Not a streaming SQL database. Feldera / Materialize / Noria - cover that space; Zeta is the library layer below a - database, not the database itself. -- Not a production streaming-analytics platform. We are a - correctness-first reference implementation that happens to - perform well — not a k8s-native analytics warehouse. -- Not a clone of any upstream. `references/upstreams/` are - read-only inspiration; Zeta hand-crafts every artefact. -- Not a compliance product. Enterprise consumers certify at - their deployment layer; we provide the evidence trail. - -## What we are building (v1.0) - -The subset that makes Zeta shippable as `Zeta.Core 1.0.0` on -NuGet: - -- **Operator algebra** — Z-sets, D/I/z⁻¹/H, retraction-native - by construction. Higher-order derivatives where the math - calls for them. -- **Recursion** — retraction-safe `Recursive` combinator; - `RecursiveSemiNaive` where monotonicity holds; LFP - convergence with honest termination guarantees. -- **Storage layer** — four durability modes (in-memory, - OS-buffered, stable-storage, witness-durable) with honest - recovery-property advertisement. DiskBackingStore + - Spine family. -- **Sketches** — retraction-aware HLL, CountMin, KLL, plus - Blocked and Counting Bloom filters. Paper-worthy contribution. -- **Query planner** — cost model with SIMD / tensor-intrinsic - kernel dispatch. Research-grade. -- **Plugin surface** — external consumers write their own - operators via `PluginOp<'TIn, 'TOut>` or equivalent. -- **FsCheck LawRunner** — every operator advertised as - linear / bilinear / sink-terminal / retraction-complete - earns the tag by passing a property-based law check. -- **Formal-method coverage** — 18+ TLA+ specs, Alloy specs - for structural invariants, Lean proofs where the math is - foundational. -- **CI parity** — dev laptop + CI runner + (future) - devcontainer all bootstrap via `tools/setup/install.sh`. - All PRs green through the gate before merge. -- **Security posture** — nation-state + supply-chain threat - model documented + controls enforced (Semgrep-in-CI, - SHA-pinned actions, INCIDENT-PLAYBOOK, SDL-CHECKLIST). - -## What we are exploring (post-v1.0) - -Things on the roadmap that we intend to ship but haven't -committed shape on: - -- **Witness-Durable Commit protocol** — paper-worthy - durability mode; full impl blocked on paper peer-review - rebuttal. -- **Retraction-aware analytic sketches** — HyperBitBit + - retraction-native quantile sketches; publication target. -- **Info-theoretic sharder** — load-aware shard placement - with per-shard feedback; Alloy-verified. -- **Multi-node deployment** — the control-plane shape - (NATS? gRPC? Arrow Flight?) is an open question. -- **Consumer-facing distribution** — the v1.0 NuGet publish; - licensing, telemetry posture, documentation site. +- **Not a clone of any upstream.** `references/upstreams/` + are read-only inspiration; Zeta hand-crafts every artefact. +- **Not a compliance product.** Enterprise consumers certify + at their deployment layer; Zeta provides the evidence + trail, not the audit. +- **Not a narrow niche.** The ambition is full database; + saying "Zeta only does X" is under-promising and would + pull us off-course. +- **Not a product chasing users pre-v1.** Research first; + users follow the research. + +## Commercial posture + +Pure research / open-source is the first-class experience. +Commercial ambitions are on the table — not decided, not +required, not refused. The factory and the library are +designed to support either trajectory: + +- If Zeta stays pure-OSS: research papers + reference + implementation + NuGet distribution. +- If commercial emerges: enterprise-grade features (tenancy, + compliance attestations, hosted offering, etc.) layer on + top of the OSS core; the OSS core never degrades for the + commercial story. + +When commercial moves from "on the table" to "decided," +the `product-visionary` + `branding-specialist` pair updates +this section. ## How we decide what to build -The loop we want the `product-visionary` role to run: - -1. **Ingest signals.** Upstream reference repos ship something - interesting; a paper appears; someone has a novel idea; - an existing Zeta subsystem shows a design smell. -2. **Check against the vision.** Does this sharpen - Zeta's north star, or does it pull us toward being - something we explicitly are NOT (§What Zeta is NOT)? -3. **Propose to the backlog.** If yes, write a - `docs/BACKLOG.md` entry with the reasoning. If no, write - a `docs/WONT-DO.md` entry with the reasoning. +The loop the `product-visionary` role runs: + +1. **Ingest signals.** Upstream reference repos ship + something interesting; a paper appears; someone has a + novel idea; an existing Zeta subsystem shows a design + smell; the factory itself has a friction point. +2. **Check against the vision.** Does this sharpen Zeta's + north star across Product 1 + Product 2, or does it + pull us toward what we explicitly are NOT? +3. **Propose to the backlog.** If yes → `docs/BACKLOG.md` + entry with reasoning. If no → `docs/WONT-DO.md` entry + with reasoning. 4. **Ask Aaron if unsure.** Long-term vision drift is the - silent killer. Product-visionary asks Aaron many - questions and doesn't decide alone on - direction-shifting items. + silent killer. Product-visionary asks many questions; + direction-shifting items never get decided alone. ## Operating principles (abbreviated from AGENTS.md + GOVERNANCE) @@ -132,49 +262,54 @@ The loop we want the `product-visionary` role to run: - Docs read as current state, not history. - OpenSpec is first-class. - Every CI minute earns its slot. +- Non-declarative installation is a smell. +- Symmetry — across configs, tools, ignore lists, bootstrap paths. +- Deterministic scripts — retries and polling are last resort. ## What this document is NOT - Not the roadmap (that's `docs/ROADMAP.md`). - Not the backlog (that's `docs/BACKLOG.md`). - Not the next-steps queue (that's the `next-steps` skill). -- Not a sales pitch. Zeta has no paying customers pre-v1; - this is the vision the research + engineering work - serves. +- Not a sales pitch. ## Revision cadence The `product-visionary` role (when spawned) re-reads this doc every 5-10 rounds, proposes edits, gets Aaron's sign-off, commits. Ad-hoc edits land when a round's work -surfaces a vision-level question. - -Aaron can revise at any time without ceremony. +surfaces a vision-level question. Aaron can revise at any +time without ceremony. -## First-pass confidence + gaps +## First-pass confidence + gaps (post v2 edits) -Things in this draft Kenji is confident about because Aaron -has stated them directly (rounds 30-33): +Things Aaron has now stated directly and Kenji is confident about: -- Nation-state + supply-chain security posture. -- Retraction-native is non-negotiable. +- Zeta IS going to be a full database, not a library layer + below one. "All DB technologies in one big playground." +- Multi-node deployment is in scope. +- The software factory is a first-class product, not a + means to an end. +- Factory goals: cross-platform, AI-automated, declarative + dependencies, research-level reproducibility, symmetry, + production-ready quality, UI someday. +- Pure research/OSS is first-class; commercial is on the + table but undecided. - F# primary, polyglot over time. -- OpenSpec is first-class for every committed artefact. -- Self-directed factory as a second product. -- NuGet library shipping target at v1.0. - -Things in this draft Kenji has inferred that need Aaron's -explicit validation: - -- The "Zeta is NOT a streaming SQL database" framing. -- "Research-grade / publishable" as a first-class goal vs - a nice-to-have. -- The specific operator-algebra + sketches + storage + planner - scope for v1.0 (derived from what's shipped + on ROADMAP). -- Whether multi-node deployment is post-v1 or out-of-scope. -- Whether Zeta has ambitions toward a commercial product - around the library, or is purely a research + open-source - contribution. - -The product-visionary role's first audit, on spawn, should -walk these gaps with Aaron. +- Nation-state + supply-chain security posture. +- OpenSpec first-class for every committed artefact. +- NuGet library shipping at v1.0 (the first slice of the + full database). + +Remaining gaps for the product-visionary's first audit +with Aaron: + +- Naming: Aaron almost named it "Database" — is "Zeta" + locked in, or is a rename still possible at v1.0? +- SQL frontend: is it P1 post-v1 or much later? +- Commercial trigger: what event / milestone would move + commercial from "on the table" to "decided"? +- Bitemporal / time-travel queries: first-class v2 target + or speculative? +- Query IR: is Zeta aiming at "multiple frontends target + one IR" or "the F# DSL is the surface" long-term?