docs: v2.x specs for replay/derivation-worker/view-flip chain

docs/v2_derivation_worker.md

@@ -0,0 +1,90 @@
+# v2.x spec: derivation worker — beliefs become materialized state
+
+Spec for issue [#264](https://github.com/robotrocketscience/aelfrice/issues/264). Cascade addendum to [`design/write-log-as-truth.md`](design/write-log-as-truth.md). The actual refactor; #261/#262/#263 are scaffolding for this.
+
+Status: spec, no implementation. Recommendation included; decision is the user's.
+
+## What's being decided
+
+Four contract calls left open by the issue body:
+
+1. **Order of operations** in the new ingest path.
+2. **Idempotency check** on re-runs.
+3. **Crash-recovery semantics** when the worker dies mid-batch.
+4. **Edge derivation timing** — same pass as beliefs or a follow-up pass.
+
+After this issue ships, every ingest entry point (`scan_repo`, `ingest_turn`, `triple_extractor.ingest_triples`, MCP `tool_lock`, CLI lock, `accept_classifications`) writes to `ingest_log` only and invokes the worker. Direct `Belief()` construction outside the worker is forbidden by convention; the next issue (#265) makes it forbidden by assertion.
+
+## Recommendation
+
+**Ship at v2.x with synchronous in-process invocation, single-pass derivation (beliefs + deterministic edges together), and recover-by-replay crash semantics.**
+
+### Order of operations
+
+```
+ingest entry point:
+  1. record_ingest(log_row)         -> ingest_log INSERT (derived_*_ids = [])
+  2. derived = derive(log_row)      -> pure function, no side effects
+  3. INSERT/UPDATE beliefs from derived.beliefs
+  4. INSERT/UPDATE edges from derived.deterministic_edges
+  5. UPDATE ingest_log SET derived_belief_ids = ..., derived_edge_ids = ...
+       WHERE id = log_row.id
+  All in a single SQLite transaction.
+```
+
+Single transaction is the right call: SQLite's WAL makes this cheap, and it eliminates the "log row exists but beliefs don't" failure window. The crash-recovery story below is for the rarer case where the *transaction itself* succeeds but the worker dies before the next batch.
+
+### Idempotency check
+
+The composite check the issue body proposes is correct: a log row is **already derived** iff `derived_belief_ids` is non-empty AND every id in that list exists in `beliefs`. Worker MUST re-derive (and re-stamp) if either half fails — that is what catches the "belief was deleted post-ingest" case described in #262 as `derived_orphan`.
+
+```python
+def needs_derivation(log_row, store):
+    if not log_row.derived_belief_ids:
+        return True
+    return not all(store.belief_exists(bid) for bid in log_row.derived_belief_ids)
+```
+
+### Crash recovery
+
+If the worker dies between the SQLite transaction commit and returning control to the entry point: nothing to recover, the database state is consistent. If the worker dies *between batches* (e.g. an entry point ingests 100 rows in a loop and dies after row 50): on next worker invocation, sweep all rows where `needs_derivation(row) == True` and derive them. This is `aelf doctor --derive-pending` as a manual escape hatch and a startup hook for the next entry point.
+
+The pathological case the issue body raises ("worker dies between INSERT belief and UPDATE log row") is impossible under single-transaction operation. We adopt the constraint to make it impossible rather than recovering from it.
+
+### Edge derivation timing
+
+Same pass as beliefs. Deterministic edges (from `triple_extractor`) are a function of the same raw_text the belief is derived from; computing them in the same `derive()` call costs nothing extra and removes a class of "edges lag beliefs" bugs. Feedback-driven edges (`propagate_valence`) are NOT computed by the worker — they are written by the feedback path and live in a separate cohort per the equality contract in #262.
+
+### Concurrency
+
+Two-process race producing the same belief: `INSERT OR IGNORE` on `(content_hash)` lets one win; both ingest_log rows stamp `derived_belief_ids = [winner_bid]`. The existing `tests/test_concurrency.py` shape covers this; the new test extends it to assert both log rows converge to the same belief id.
+
+## Decision asks
+
+- [ ] **Confirm single-transaction operation.** The alternative is multi-transaction with a recovery sweep on crash. Single-transaction is simpler and SQLite-WAL-cheap; the only reason to reject is if a future async/daemon worker needs to be designed-in now (recommendation: defer to v3).
+- [ ] **Confirm "worker dies between batches" recovery semantics.** `aelf doctor --derive-pending` as both a CLI escape hatch and an automatic startup hook on next entry-point invocation. Default: automatic; `--no-auto-derive-on-startup` opt-out for diagnosing stuck states.
+- [ ] **Confirm edges in the same pass.** Alternative is a separate edge-derivation pass for parallelism / locality. Single-pass is simpler; defer the split until benchmarks force it.
+- [ ] **Worker-only insert enforcement timing.** This issue makes direct `insert_belief()` *unused* outside the worker. #265 makes it *forbidden* (raises). Confirm the split — keeping the assertion gated on #265's feature flag means this issue stays additive and bisectable.
+
+## Why this is judgment-scope
+
+The four decisions above are the design work. The body the issue lists at ~400 LOC code + ~300 LOC tests is achievable only if these are settled before implementation; otherwise the worker grows a recovery-mode flag matrix and the test suite balloons.
+
+## Downstream impact
+
+- Every ingest entry point's call shape changes. Hooks, MCP tools, CLI commands, scanner — all must route through the worker. The patch is wide but mechanical.
+- `tests/test_concurrency.py` extends with worker-race coverage.
+- `LIMITATIONS.md`: drops the "re-classification requires re-onboard" caveat (the worker handles the re-derivation path; #265 surfaces it as `aelf rebuild`).
+- `aelf doctor`: gains `--derive-pending` to manually trigger the sweep.
+
+## Out of scope (deferred)
+
+- **Async / daemon-mode worker.** v3.
+- **Multi-rule-set re-derivation.** Held for #265's `aelf rebuild --rule-set <hash>` surface.
+- **Performance work beyond the latency alarm in #205.** If the worker becomes a hotspot under realistic ingest sizes, that is a follow-up.
+
+## Provenance
+
+- Source-of-truth: [`docs/design/write-log-as-truth.md`](design/write-log-as-truth.md) §§ "What changes under the proposed contract", "Costs and risks".
+- Upstream chain: #205 → #261 → #262 → **#264 this issue** → #265.
+- Equality contract that the worker must satisfy: [`v2_replay.md`](v2_replay.md).

docs/v2_replay.md

@@ -0,0 +1,66 @@
+# v2.x spec: `replay_full_equality` — flip-readiness probe
+
+Spec for issue [#262](https://github.com/robotrocketscience/aelfrice/issues/262). Cascade addendum to [`design/write-log-as-truth.md`](design/write-log-as-truth.md). Gates the view-flip in [`v2_view_flip.md`](v2_view_flip.md).
+
+Status: spec, no implementation. Recommendation included; decision is the user's.
+
+## What's being decided
+
+Whether `replay_full_equality()` — the validation harness the design memo calls for — ships under a *shape-equality at ingest time* contract or a *bit-equality across all fields* contract, and how the report distinguishes drift caused by the rule set from drift caused by post-ingest mutation (feedback, decay).
+
+The function re-runs the deterministic derivation function (#261) over every `ingest_log` row written since v2.0 and compares synthesized beliefs to canonical `beliefs`. Output is a `FullEqualityReport` with row counts in five buckets (match / mismatch / derived_orphan / canonical_orphan / drift_examples). It is the gate the chain has to clear before view authority can flip.
+
+## Substrate / chain dependency
+
+- **Depends on:** #261 (pure derivation function). The probe has nothing to invoke until that lands.
+- **Depends on:** #263 (`legacy_unknown` migration). Pre-v2.0 stores have beliefs without a corresponding ingest_log row; without the migration synthesizing rows, every legacy belief reports as `canonical_orphan` and the probe is useless on existing user stores.
+- **Excludes from comparison:** rows whose `source_kind = legacy_unknown`. They have no raw_text the derivation function can consume; the migration backfills shape-only metadata.
+
+## Recommendation
+
+**Ship at v2.x with shape-equality at ingest time, not bit-equality.**
+
+Three reasons:
+
+1. **alpha/beta evolve via feedback.** A belief ingested at α=1, β=1 today and given two upvotes is α=3, β=1 tomorrow. Re-deriving from the log produces α=1, β=1 — the prior. Bit-equality flags every fed-back belief as drift, which is the wrong signal. The right equality is *what would derivation produce on a fresh store ingesting the same log row?* Posterior drift is **not drift** — it is feedback doing its job.
+2. **Origin can legitimately rewrite.** #224 added origin propagation; pre-#224 beliefs have `origin=NULL` while a re-derivation today produces `origin=ingest_turn` (or whichever entry point). This is a known cohort, not a bug. The report should bucket these under `legacy_origin_backfill` rather than `mismatch`.
+3. **Edge equality is partial.** `triple_extractor` edges are deterministic and replay-equal; `propagate_valence` edges are feedback-driven and won't replay. Scope edge equality to the deterministic set; report the feedback-driven subset under a separate `feedback_derived_edges` count, never as drift.
+
+### Equality contract (concrete)
+
+A canonical belief and a derived belief are **shape-equal** iff:
+- `content_hash` matches, AND
+- `type` matches, AND
+- `origin` matches OR canonical `origin IS NULL` (legacy backfill cohort), AND
+- the deterministic edge set (FROM `triple_extractor`) matches.
+
+α/β/last_retrieved_at/feedback-driven edges are explicitly out of scope for the equality check. They are tracked in separate counters for human inspection but never trigger the drift alarm.
+
+## Decision asks
+
+- [ ] **Confirm shape-equality contract.** If the user wants strict bit-equality (e.g., to catch posterior-write bugs), the report must bucket every fed-back belief separately. Default recommendation: ship shape-equality.
+- [ ] **Drift threshold for `aelf doctor --replay` exit code.** Recommendation: exit 0 if `mismatch + derived_orphan == 0` (canonical_orphan and legacy_origin_backfill are not drift); exit 1 otherwise. Configurable via `--max-drift N`.
+- [ ] **Scope of the legacy backfill cohort.** When `replay_full_equality` runs on a store with `legacy_unknown` rows, those rows are excluded from `total_log_rows`. Beliefs whose only log row is `legacy_unknown` are excluded from `canonical_orphan`. This is the right call but should be ratified — it does mean the probe is silent about pre-v2.0 ingest correctness.
+- [ ] **Drift example sample size.** Recommendation: up to 10 representative cases per drift bucket, raw_text truncated to 200 chars. Configurable.
+
+## Why this is judgment-scope
+
+The decisions above are the design work. Once equality is contracted, implementation is mechanical (~150 LOC code + ~250 LOC tests).
+
+## Downstream impact
+
+- `aelf doctor --replay` becomes the gate for #265 (view-flip). Required to exit 0 on a clean store before the flag flips default-on.
+- `LIMITATIONS.md`: section on "what re-onboarding can and can't reproduce" gets to point at the probe instead of describing the limitation in prose.
+- New CLI flags: `--max-drift N`, `--drift-examples N`, `--scope <all|since-v2>`.
+
+## Out of scope (deferred to follow-up)
+
+- **Time-travel replay** (`at_timestamp` parameter). Requires per-version derivation library. Tracked separately.
+- **Multi-rule-set replay** (replay against version N of the rule set). Same blocker.
+- **Auto-fix.** The probe reports drift; #265's view-flip is what corrects it.
+
+## Provenance
+
+- Source-of-truth: [`docs/design/write-log-as-truth.md`](design/write-log-as-truth.md) §§ "What changes under the proposed contract", "Smallest first step".
+- Substrate ratification: [`substrate_decision.md`](substrate_decision.md) (#196 Option B). Beta-Bernoulli's posterior drift behavior is what motivates shape-equality over bit-equality here.
+- Upstream chain: #205 (parallel-write phase, merged) → #261 (derivation function) → **#262 this issue** → #264 (worker) → #265 (view-flip).