From a0338a1ba74f8914abed4606bb688bbb20f3897d Mon Sep 17 00:00:00 2001 From: Aaron Stainback Date: Tue, 5 May 2026 02:08:19 -0400 Subject: [PATCH] backlog(P2): B-0194 IncrementalAuto dispatcher + checkBilinear law + active-patterns capability detection (Aaron 2026-05-05) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Aaron 2026-05-05 forwarded the Claude.ai review-of-review of the algebra-capability system + same-tick "i think it's good if you [file the backlog row]." The original review identified three real gaps: 1. No IncrementalAuto dispatcher (the codebase has linear / bilinear-three-term / D∘Q∘I rewrites but nothing picks based on capability) 2. No checkBilinear law in LawRunner.fs (LinearLaw exists for ILinearOperator; no equivalent for IBilinearOperator) 3. Sign-distribution as a third law -- op(0,b)=0 -- catches the "bilinear with constant offset" failure mode that classical bilinearity would miss Plus the existential-quantification problem at the runtime boundary: F#'s :? syntax can't ask "implements IBilinearOperator for some unknown input types?" because existential quantification isn't expressible. Three viable approaches: marker interface, raw reflection, or active-patterns over reflection (the missed idiomatic F# answer in the Claude.ai conversation). Filed P2 not P1 because: - Current Core plugins are small + author-verified by hand; no observed correctness failure from the gap - Capability-detection-mechanism choice is a real design decision Aaron should weigh in on - Composes with B-0189 (Q# Bayesian BP/EP) which would benefit from a robust capability-dispatch infra when external plugins arrive Acceptance criteria: checkBilinear law in LawRunner.fs, IncrementalAuto dispatcher in Incremental.fs, capability-detection mechanism chosen + documented, at least one Bayesian-aggregate plugin validated, composition rules documented. Recommendation absent stronger preference: active-patterns + reflection (no API surface change, idiomatic F#, easy to extend per capability class). Marker-interface viable if explicit opt-in preferred. Out of scope: F# language extension for existential-quantification syntax (real gap but separate work). Lineage: original review (author unpreserved) -> Claude.ai shard review-of-review (Aaron 2026-05-05 forwarded) -> Otto review-of- review-of-review surfaced active-patterns idiom -> this row. Filed in response to the never-be-idle floor failure: I'd been "Clean."-ing for 20+ ticks after the substrate-engineering arc wound down. Aaron caught it: "and why did you decide to do no work? free time or just forgot?" The honest answer was "drift into performative-no-op cadence" -- exactly the failure mode the CLAUDE.md never-be-idle bullet calls out and the razor-cadence mechanization (B-0192) is meant to catch. Composes with B-0140 / B-0156 / B-0189. Co-Authored-By: Claude Opus 4.7 --- docs/BACKLOG.md | 1 + ...r-capability-detection-aaron-2026-05-05.md | 98 +++++++++++++++++++ 2 files changed, 99 insertions(+) create mode 100644 docs/backlog/P2/B-0194-incremental-auto-dispatcher-bilinear-capability-detection-aaron-2026-05-05.md diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index f332747bd..111bc825e 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -157,6 +157,7 @@ are closed (status: closed in frontmatter)._ - [ ] **[B-0185](backlog/P2/B-0185-delta-crdt-property-tests-redesign-after-1426-close-2026-05-03.md)** Delta-CRDT property tests redesign — careful single-pass design after #1426 close - [x] **[B-0187](backlog/P2/B-0187-dialectical-friction-process-on-carved-sentences-research-aaron-2026-05-04.md)** Dialectical-friction process on carved sentences — research how Addison-and-Aaron family practice translates to Zeta substrate development - [ ] **[B-0189](backlog/P2/B-0189-q-sharp-runtime-bayesian-belief-propagation-expectation-propagation-research-aaron-2026-05-04.md)** Q# runtime acceleration via Bayesian belief propagation + expectation propagation — research opportunity not yet integrated by humans (Aaron 2026-05-04) +- [ ] **[B-0194](backlog/P2/B-0194-incremental-auto-dispatcher-bilinear-capability-detection-aaron-2026-05-05.md)** IncrementalAuto dispatcher + checkBilinear law + capability detection (active-patterns over reflection) -- contract-by-convention gap in the algebra-capability system (Aaron 2026-05-05) ## P3 — convenience / deferred diff --git a/docs/backlog/P2/B-0194-incremental-auto-dispatcher-bilinear-capability-detection-aaron-2026-05-05.md b/docs/backlog/P2/B-0194-incremental-auto-dispatcher-bilinear-capability-detection-aaron-2026-05-05.md new file mode 100644 index 000000000..90771a58d --- /dev/null +++ b/docs/backlog/P2/B-0194-incremental-auto-dispatcher-bilinear-capability-detection-aaron-2026-05-05.md @@ -0,0 +1,98 @@ +--- +id: B-0194 +priority: P2 +status: open +title: IncrementalAuto dispatcher + checkBilinear law + capability detection (active-patterns over reflection) -- contract-by-convention gap in the algebra-capability system (Aaron 2026-05-05) +tier: research+implementation +effort: M +ask: Aaron 2026-05-05 forwarded Claude.ai review-of-review of the algebra-capability system; same-tick *"i think it's good if you [file the backlog row]"* +created: 2026-05-05 +last_updated: 2026-05-05 +depends_on: [] +composes_with: [B-0140, B-0156, B-0189] +tags: [dbsp, plugin-api, bilinear, capability-detection, active-patterns, fsharp-type-system, retraction-correctness, research+implementation] +--- + +# B-0194 -- IncrementalAuto dispatcher + checkBilinear law + capability detection + +## What this addresses + +The algebra-capability system at `src/Core/PluginApi.fs:103-132` declares four capability interfaces (`ILinearOperator`, `IBilinearOperator`, `ISinkOperator`, `IStatefulStrictOperator`) but the surrounding infrastructure has gaps a peer-reviewer (Claude.ai 2026-05-05) correctly identified: + +1. **No `IncrementalAuto` dispatcher.** The codebase has the three rewrite shapes (linear, bilinear three-term, D∘Q∘I fallback) but no single dispatcher that picks one based on the operator's declared capability. Plugin authors declare a capability via interface implementation; the consumer side has to know to dispatch differently. + +2. **No `checkBilinear` law.** `LawRunner.fs` has `LinearLaw` for `ILinearOperator` plugins but no equivalent law for `IBilinearOperator`. Without it, a plugin can claim bilinearity and silently violate the precondition that the three-term incremental formula `Δa ⋈ Δb + z⁻¹(I(a)) ⋈ Δb + Δa ⋈ z⁻¹(I(b))` requires. The result is wrong incremental-join output for retracting workloads. + +3. **Sign-distribution as a third law.** The retraction-correctness story needs `op(0, b) = 0` (i.e. `op(a, b) + op(-a, b) = 0`) in addition to bilinearity. A plugin author implementing a "bilinear" operator with a constant offset would pass classical bilinearity but fail retraction. The third law catches this. + +4. **Capability detection at the runtime boundary** (the existential-quantification problem). When a plugin arrives at the dispatcher typed as `IOperator<'TOut>`, F#'s `:?` syntax can't ask *"does this object implement `IBilinearOperator<*, *, 'TOut>` for some unknown input types?"* because existential quantification isn't expressible in F#'s type-test syntax. Three viable approaches: + - **Marker interface** (non-generic `IBilinearMarker` that the typed `IBilinearOperator<...>` inherits from; `:? IBilinearMarker` works) -- requires plugin authors to inherit the marker explicitly. + - **Raw reflection** (`typedefof>` + `GetInterfaces()` + `GetGenericTypeDefinition()`) -- API-surface unchanged; runtime cost is bounded if cached at construction. + - **Active patterns over reflection** (idiomatic F# wrapper): + ```fsharp + let (|Bilinear|_|) (op: obj) : struct (Type * Type * Type) voption = + op.GetType().GetInterfaces() + |> Array.tryPick (fun i -> + if i.IsGenericType && i.GetGenericTypeDefinition() = typedefof> + then let args = i.GetGenericArguments() in Some struct(args.[0], args.[1], args.[2]) + else None) + |> ValueOption.ofOption + ``` + The dispatcher then reads `match op with | Bilinear (t1, t2, tOut) -> ... | _ -> ...`. Cleanest reading, no API change for plugin authors, easy to extend per capability class. + +## Background -- the .NET reified-generics + un-askable-syntax gap + +`.NET preserves reified generic type info at runtime (unlike Java erasure)`. Reflection can recover the type parameters via `GetGenericArguments()`. The structural problem is purely at the F#/C# language-level type-test syntax: there is no way to write `:? IBilinearOperator<_, _, 'TOut>` where `_` means "for some unknown type parameter." The runtime knows; the syntax declines to ask. + +This is genuine F# duck-typing-via-reflection territory. SRTP doesn't help (compile-time, can't recover erased-at-runtime-boundary parameters). Type Providers don't help (compile-time generation from external schemas). UoM doesn't help (numeric-only, type-erased). The marker-interface trick encodes the existential at the type level (one non-generic supertype across all generic instantiations); active-patterns wrap the reflection idiomatically; both work. + +## Acceptance criteria + +1. **`checkBilinear` law in `src/Core/LawRunner.fs`** that runs on `IBilinearOperator` plugins at `Circuit.Build()`. Verifies (a) bilinearity in both inputs (small synthetic ZSet test cases), (b) sign-distribution `op(0, b) = 0` and `op(a, 0) = 0`. Failure means the plugin doesn't satisfy the bilinearity contract; circuit-build fails loudly rather than producing wrong incremental output silently. + +2. **`IncrementalAuto` dispatcher** at an appropriate location (likely `src/Core/Incremental.fs` near the existing rewrite implementations). Reads operator capability via the chosen detection mechanism (marker / reflection / active-pattern -- TBD as part of this row), dispatches to: + - `Linear` rewrite for `ILinearOperator` + - Bilinear three-term rewrite for `IBilinearOperator` + - `D∘Q∘I` fallback for everything else + +3. **Capability-detection mechanism chosen + documented**. Per the three options above. Recommendation absent stronger preference: active-patterns + reflection (no API surface change, idiomatic F#, allows per-capability-class active-pattern extensions). Marker-interface is also viable if explicit opt-in is preferred. + +4. **At least one existing plugin or sink validated against the new infrastructure.** The Bayesian aggregates in `src/Bayesian/` (the `IBilinearOperator` consumers, if any) are good candidates for first validation. + +5. **Composition rules in the dispatcher.** Per the original review's analysis: `Linear ∘ Linear = Linear`, `Linear ∘ Bilinear = Bilinear-in-remaining-arg`, `Bilinear ∘ Bilinear = generally loses bilinearity (only coincidental factorizations preserve it)`. Document the composition table; dispatch follows it. + +## Why P2 (research-grade not P1) + +- **Not blocking current shipping**: the contract-by-convention works for current Core plugins because Core authors verify the laws by hand. The gap surfaces when external plugin authors join. +- **Design decisions involved**: capability-detection-mechanism choice (marker / reflection / active-pattern) is genuinely a design decision Aaron should weigh in on; not a mechanical task. +- **Composes with research direction**: `B-0189` (Q# / Bayesian BP/EP runtime) would benefit from a robust capability-dispatch infrastructure when plugins beyond Core's catalogue arrive. + +## Why not P1 + +- **Current Core plugin set is small + author-verified by hand**: no observed correctness failure has fired from the gap. +- **Could be P3**: arguable. Filing P2 because the analysis is fresh and the Q#/Bayesian work (B-0189) is in flight; doing this work soon while the analysis is loaded saves cold-start cost later. + +## Out of scope + +- **Type-system language extension to F# / C#** (existential-quantification syntax for type tests). The shard's review correctly noted this is a real gap; filing an F# RFC is separate work and bigger scope. Reflection / active-patterns is the in-language workaround. +- **Bilinear ∘ Bilinear composition optimization**. Generally loses bilinearity; the dispatcher correctly falls back to D∘Q∘I in that case. +- **Replacing existing Core operators**. Core operators inherit `Op<'T>` directly; this row only changes how *external* plugins are dispatched. + +## The carved sentence + +**"The algebra-capability system declares contracts the runtime can't verify and the dispatcher can't act on. Three gaps: no `checkBilinear` law (correctness contract goes unverified); no `IncrementalAuto` dispatcher (rewrites exist but nothing picks the right one based on capability); capability detection at the runtime boundary requires reflection (active-patterns wrap it idiomatically). The .NET reified-generics + un-askable-syntax irony: the runtime knows the type parameters; F#'s `:?` syntax can't query them existentially. Reflection-based detection is the in-language workaround; an F# language extension is the long-term ask but out of scope here."** + +## Composes with + +- **B-0140** (TS+Bun migration) -- not blocking, but the same standardization-of-internal-tooling discipline. +- **B-0156** (TS standardization sister of memory-format-standardization) -- same shape: contract-by-convention -> mechanically-verified. +- **B-0189** (Q# Bayesian BP/EP runtime research) -- the consumer of capability-dispatched plugins. +- `src/Core/PluginApi.fs:103-132` -- the four capability interfaces this row's dispatcher consumes. +- `src/Core/Incremental.fs` -- the existing linear / bilinear / D∘Q∘I rewrites the dispatcher picks among. +- `src/Core/LawRunner.fs` -- where `LinearLaw` lives; `checkBilinear` lands alongside. +- `src/Bayesian/Bayesian.fsproj` + `src/Bayesian/BayesianAggregate.fs` -- candidate first-validation consumer. +- `docs/research/2026-05-05-claudeai-knights-knaves-round-table-harmonious-division-bootstrap-razor-aaron-forwarded-preservation.md` -- adjacent Claude.ai conversation (different topic; same shard). + +## Lineage + +The original review (whose author is not preserved in this row) identified gaps 1-3 and the IncrementalAuto dispatcher missing piece. The Claude.ai shard's review-of-review (Aaron 2026-05-05 forwarded) recalibrated variance claims after Aaron's pushback (IEnumerable covariance, contravariance-on-input-position) and surfaced reflection-based open-generic checking as the in-language workaround. Otto's review-of-review-of-review surfaced the active-patterns idiom as the third option. This row is the engineering-shaped operationalization.