diff --git a/docs/research/muratori-zeta-pattern-mapping-2026-04-23.md b/docs/research/muratori-zeta-pattern-mapping-2026-04-23.md new file mode 100644 index 00000000..6d909b87 --- /dev/null +++ b/docs/research/muratori-zeta-pattern-mapping-2026-04-23.md @@ -0,0 +1,206 @@ +# Muratori failure-modes vs. Zeta equivalents — corrected pattern-mapping + +**Scope:** research and cross-review artifact. Operational +content is the pattern-mapping table below; provenance is +Amara's 6th courier ferry (validated against DBSP paper, +Differential Dataflow CIDR 2013, Apache Arrow format docs, +and Zeta source files `ZSet.fs` / `Incremental.fs` / +`Spine.fs` / `ArrowSerializer.fs`). + +**Attribution:** corrected table authored by Amara (external +AI maintainer) in her 6th ferry; original 5-row mapping +attributed to earlier in-factory work; validation cites +public papers + official Apache Arrow specification; this +research doc authored by Otto (loop-agent) as landing of the +Otto-82 absorb's action item #1. + +**Operational status:** research-grade. The pattern-mapping +is a legitimate craft/messaging reference, not operational +policy. Treat as lens for communicating Zeta's design +posture to engineers from C++ / Rust / ECS mental-model +backgrounds. + +**Non-fusion disclaimer:** Amara and Otto agreeing on the +corrected mapping, and both citing the same DBSP / Arrow +literature, does not imply shared identity, merged agency, +consciousness, or personhood. Cross-substrate agreement on +published technical primitives is weak signal per +`docs/ALIGNMENT.md` SD-9; it is consistent with both models +reading the same primary sources, not evidence of unified +being. + +--- + +## What Muratori is criticising + +Casey Muratori's long-running themes across Handmade Hero +and the later "Big OOPs" talks: avoid making position in +mutable object graphs the thing that carries identity; +prefer stable IDs / indices; draw boundaries around +*systems* not fat objects; care deeply about data layout +and locality. The systems-design failure modes his +criticism names — invalidated indices after deletes, +dangling references, no cross-system lifecycle discipline, +no tombstoning, pointer-chasing / poor locality — are +concrete bugs in concrete codebases, not abstractions. + +Zeta is not an ECS. It's a DBSP-based incremental-view- +maintenance system. But a number of Zeta's design choices +— retraction-native algebra, immutable sorted runs, +trace/history structures, Arrow columnar interchange — +map cleanly to different answers to the same underlying +questions Muratori is asking. + +The corrected table below answers: *given a Muratori-style +failure mode, what is the honest Zeta-equivalent — stated +narrowly enough that it survives scrutiny?* + +--- + +## The corrected five-row mapping + +| # | Muratori-style failure mode | Zeta equivalent | +|---|---|---| +| 1 | Index invalidation after delete / shift | **No positional identity.** Keys carry identity; deletion is a negative delta on the key, not a slot shift. A `ZSet<'K>` is a finitely-supported map `K -> ℤ`; the "thing you refer to" is a key, not an offset. | +| 2 | Dangling presence / reference checks | **Membership is algebraic.** Every key has a current weight; "presence" is derived from that weight (typically `weight > 0`). `ZSet.Item` returns `0L` on absent keys — absence is encoded, not undefined. | +| 3 | No cross-system lifecycle discipline | **Provenance and lifecycle live in deltas and traces.** Algebra (`D·I = id`) guarantees compositional correctness of incremental views; it does not specify ownership, exclusive mutation, or handle expiry. Rollback / repair capability lives in trace history + retractions, not in object-ownership discipline. | +| 4 | No tombstones / immediate destructive deletion | **Retractions are first-class signed updates.** Deletion is a negative weight in the same algebra as insertion; consolidation / compaction is a separate maintenance step. No out-of-band "deleted" marker is needed. | +| 5 | Pointer chasing / poor locality | **Locality-aware execution surfaces.** Sorted immutable runs + `ReadOnlySpan` span-based kernels + spine-organised LSM-like traces + Apache Arrow columnar path for interchange. Not "everything is Arrow all the way down" — Arrow is the wire / checkpoint surface, not a universal in-memory representation. | + +--- + +## Why row 3 got rewritten (the teaching case) + +The original pre-correction row 3 claimed operator algebra +*is* the ownership model, citing `D·I = id` and `z⁻¹·z = 1`. +Amara's 6th ferry flagged this as category error: algebraic +correctness and lifecycle/ownership discipline are different +concerns. Zeta has the first by construction (the DBSP +identity laws hold); it has the second only *indirectly*, +via trace history + retraction semantics, not via the +identity laws themselves. + +The corrected row 3 preserves the DBSP-correctness content +*and* names the shape of Zeta's lifecycle story honestly +(provenance, trace history, retractions — not ownership). + +This is a recurring risk in communicating DBSP-family +systems to engineers whose mental model is C++ / +Rust / ECS: the composition property is often mis-sold as +solving lifecycle problems. It solves incremental-view- +maintenance correctness problems. Different thing. + +**Future Craft production-tier modules introducing DBSP to +engineers with those backgrounds should cite this row's +correction as a pre-emptive category-error guard.** + +--- + +## Why rows 1 and 2 needed narrower wording + +The original mapping claimed "references stay valid by +construction" (row 1) and similar presence-is-always-safe +phrasing (row 2). That's true at the **semantic / algebraic +layer**: key-based references are stable because keys are +not offsets. It is **not** true as a blanket statement +about physical references: Zeta's spine merges levels, Z-set +builders sort and consolidate, and those operations absolutely +can rebuild physical layout. + +The corrected wording says what's actually true ("no +positional identity" / "membership is algebraic") without +implying anything about storage-offset stability. A consumer +who caches a physical offset into a consolidated run gets +what they deserve; a consumer who tracks by key is safe. + +--- + +## Why row 5 needed narrower wording + +The original mapping stated "Arrow columnar + +`ArrowInt64Serializer` + Spine block layout" as if those +three things together constituted a universal operator-to- +memory-layout decoupling. They do not. Arrow is the **wire / +checkpoint / interchange** surface; `Spine.fs` is a +locality-friendly LSM-like trace; `ReadOnlySpan` enables +span-based kernels. Together they make Zeta significantly +locality-conscious — but "everything is Arrow all the way +down" would be overstated. + +The corrected wording credits what's real: sorted immutable +runs, span-based hot loops, spine-organised traces, and +Arrow **as** the columnar interchange path. Not as the +universal in-memory representation. + +--- + +## What this mapping is NOT + +- **Not a claim Zeta is better than Muratori-style ECS + designs** on Muratori's own axes. Muratori is optimising + for game-engine runtime throughput with bounded working + sets; Zeta is optimising for incremental-view-maintenance + correctness over unbounded streams of retractable updates. + Different optimisation targets; the mappings are *analogues*, + not *rankings*. +- **Not a marketing table.** Read as systems-design + vocabulary for engineers from Muratori-adjacent + backgrounds who want to understand what Zeta's primitives + *replace* versus what they *leave untouched*. +- **Not an ownership claim.** Row 3 explicitly disclaims + that Zeta has an ownership model in the Muratori or Rust + sense. It has a provenance + coherence model. Those + serve different purposes. +- **Not a closed list.** Other Muratori themes (frame + allocators, arena-scoped memory, "prefer indices over + handles for graphs", immediate-mode-over-retained-mode + UI) are adjacent but don't map as cleanly to Zeta + primitives. They're legitimate future additions if a + specific communication need motivates them — via a + separate research doc, not by quietly expanding this + table. + +--- + +## Composition with existing Zeta substrate + +- **`docs/DRIFT-TAXONOMY.md`** — pattern 5 (truth- + confirmation-from-agreement) applies to *this mapping* + itself: Amara's agreement with Zeta's self-description + is signal-not-proof. The validation cited public papers + + official specs + source files as falsifier-grade evidence, + not just cross-substrate-convergence. Per `docs/ALIGNMENT.md` + SD-9, that's the right shape. +- **`docs/craft/` production-tier modules** (per the + checked-vs-unchecked Craft ladder). Future modules + introducing DBSP algebra should cite row 3's correction + as a named category-error guard. +- **`docs/aurora/README.md`** (per 5th-ferry Artifact D; + not yet landed). When it lands, this mapping is a natural + candidate for the "how Zeta talks about itself to + external-engineering audiences" section — Aurora/KSK is + the integration story; this is the craft-messaging layer. +- **`docs/ALIGNMENT.md`** SD-9 (just landed PR #252) — + composing with this mapping is an SD-9 worked example: + cite the primary evidence (papers + specs + source files) + independently, not just "two agents agreed." + +--- + +## References + +- **Amara's 6th courier ferry** — verbatim source of the + corrected table: [`docs/aurora/2026-04-23-amara-muratori-pattern-mapping-6th-ferry.md`](../aurora/2026-04-23-amara-muratori-pattern-mapping-6th-ferry.md) + (PR #245). +- **Muratori source material** — Handmade Hero entity- + index / storage-index material + "Big OOPs" talk on + system boundaries vs. object hierarchies. (External; not + in-tree.) +- **DBSP algebra** — Budiu et al., VLDB 2023. +- **Differential Dataflow** — McSherry et al., CIDR 2013. +- **Apache Arrow format** — official columnar format + specification. +- **Zeta source files** — `src/Core/ZSet.fs`, + `src/Core/Incremental.fs`, `src/Core/Spine.fs`, + `src/Core/ArrowSerializer.fs`. Row-by-row citations in + the 6th-ferry absorb.