feat(retrieval): hot-path touch-state storage substrate (#816, closes #748)

CHANGELOG/v3.md

@@ -10,6 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Hot-path touch state v1 storage substrate** ([#816](https://github.com/robotrocketscience/aelfrice/issues/816), closes [#748](https://github.com/robotrocketscience/aelfrice/issues/748)'s storage axis). Lands the per-(belief, session) touch sidecar specified by `experiments/hot-path/DESIGN.md` v1 (lab `6b40538`) after the R0..R7c campaign — sidecar table over wide-row columns (H1 PASS via R1/R1b), boolean "touched in last K fires" decay over exponential τ (H2 REFUTED via R2c/R2d), INJECTION-only event kind (H4 REFUTED via R4..R4e/R5; `retrieve_hit` adds zero observable surface at top-K Jaccard = 1.000). **No retrieval consumer wired in v1** — the rerank multiplier path is gated on the H3 fidelity test post-R7c (DESIGN.md v1 ship list item 7). New `belief_touches(belief_id, session_id, last_fire_idx, touch_count, event_kinds_bitmask)` table with composite PK + FK CASCADE to `beliefs` + `(session_id, last_fire_idx DESC)` index; `MemoryStore` APIs `record_touch` (INSERT ... ON CONFLICT DO UPDATE — last_fire_idx refresh, touch_count bump, event_kinds_bitmask OR-in), `read_touch_set_in_window` (boolean window predicate), `count_touches_for_session`, `list_touch_sessions`. New `src/aelfrice/hot_path.py` module with `is_hot()` pure predicate, `DEFAULT_TOUCH_WINDOW_K = 50` (R2c canonical cell), and `TOUCH_EVENT_KIND_*` bitmask constants (only `INJECTION` populated; `RETRIEVE_HIT` / `BFS_VISIT` / `USER_ACTION` bits reserved per DESIGN.md "Out of scope"). Hook integration writes touches alongside the existing #744 JSON injection ring at the UPS site, sharing the ring's monotonic `fire_idx` so both substrates track the same counter. Forward-only — the hook records only the current turn's injection set; ring entries that predate this table are not backfilled (an earlier revision did backfill via `record_touch`, but the replay was non-idempotent under `ON CONFLICT DO UPDATE` and was dropped before merge). Determinism (#605): `fire_idx` is a monotonic integer, never wall-clock; same query + same store + same fire_idx → same window contents. Federation (#661): composite PK `(belief_id, session_id)` keeps foreign federated beliefs cold every read by construction. New `aelf doctor --hot-path` read-only diagnostic surface lists every session_id with at least one touch row plus row count and max fire_idx. Fail-soft throughout the hook — touch state is opportunistic substrate; a write failure must not break the user-visible context-injection contract. 22 new tests in `tests/test_hot_path_touch_state.py` cover schema, store API round-trip, ON CONFLICT semantics, bitmask OR-in, window-boundary read, per-session isolation property, ordering of `list_touch_sessions`, determinism property, FK CASCADE on belief delete, hook forward-only writes (current-turn only; pre-substrate ring entries NOT backfilled), `touch_count`-matches-actual-inject-count regression across repeated UPS fires, and missing-DB fail-soft. Concept doc at `docs/feature-hot-path.md`.
+
 - **ζ posterior-rerank surface — bounded sigmoid contribution** ([#817](https://github.com/robotrocketscience/aelfrice/issues/817), closes [#800](https://github.com/robotrocketscience/aelfrice/issues/800)). Ships behind a default-OFF flag mirroring γ ([#796](https://github.com/robotrocketscience/aelfrice/issues/796) / [PR #807](https://github.com/robotrocketscience/aelfrice/pull/807)). Replaces γ's unbounded `(1/T)·log(p)` with `α·(σ(β·(log(p)−log(0.5)))−0.5)·scale` — bounded posterior contribution in `(-α·scale/2, +α·scale/2)`, collapses to zero at the posterior-neutral `p=0.5`. Pinned defaults from the #800 R&D campaign verdict (R0–R4 at `experiments/zeta-posterior/`, R2 head-to-head: ζ dominates γ on `rank_biased_overlap` at similar `rank_changed_fraction`): `ZETA_ALPHA_DEFAULT=1.0`, `ZETA_BETA_DEFAULT=0.25`, `ZETA_SCALE_DEFAULT=14.5`. New API: `scoring.zeta_posterior_score(bm25_raw, alpha, beta, scale, posterior_mean)` with `ZETA_POSTERIOR_FLOOR` clamp on degenerate posteriors (corrupted store row never raises math domain error at retrieval time). Retrieval-side wiring: `USE_ZETA_POSTERIOR_RERANK_FLAG`, `ENV_USE_ZETA_POSTERIOR_RERANK` (`AELFRICE_USE_ZETA_POSTERIOR_RERANK`), `_env_use_zeta_posterior_rerank_override()`, `resolve_use_zeta_posterior_rerank(explicit=None, *, start=None)` with five-path precedence (env > kwarg > TOML `[retrieval] use_zeta_posterior_rerank` > False). `_l1_hits` gains `zeta_params: tuple[float, float, float] | None`; both byte-identical short-circuits extend to require `zeta_params is None` so ζ-on always exercises the rerank loop. `retrieve()` and `retrieve_with_tiers()` resolve both γ and ζ at entry and call `_assert_gamma_zeta_mutual_exclusion(gamma_on, zeta_on)` — both flags ON raises `ValueError` per the operator decision to defer composition (#817 § "Out of scope"). Heat-rerank still dominates both γ and ζ. ζ is **not** byte-identical to γ@T=1.0 nor `partial_bayesian_score(..., 1.0)` at any non-trivial inputs (issue #817 § "Note re: cold-start byte-identity"); it is rank-equivalent to log-BM25 alone on uniform-posterior=0.5 stores. 42 new tests across `tests/test_scoring_zeta.py` (posterior-neutral point, σ-bound, monotonicity, floor clamp, determinism, non-byte-identity to γ, uniform-posterior collapse to log-BM25), `tests/test_retrieve_zeta_flag.py` (five-path resolver precedence, flag-off byte-identical, flag-on deterministic, reorders by posterior, γ + ζ mutex helper, both-flags-ON raises in retrieve and retrieve_with_tiers), and `tests/test_zeta_vs_gamma_panel.py` (panel-metric reuse with `rank_biased_overlap` / `ordered_top_k_overlap`, γ-on/ζ-on rank-identical on uniform-posterior fixture). Flip-default deferred until the labeled relevance corpus exists (same gate as γ's G3). Env / TOML knobs for `(α, β, scale)` deferred per #817 § "Out of scope".
 
 ### Fixed

docs/feature-hot-path.md

@@ -0,0 +1,175 @@
+# Hot-path touch state (v1)
+
+**Status (v3.x):** storage substrate shipped, retrieval consumer parked.
+
+The hot-path touch state is a per-(belief, session) sidecar that
+records the most-recent `fire_idx` at which each belief was injected
+into the agent's context. v1 ships the write path and inspection
+surface; the rerank consumer that reads this state is a separate PR
+gated on a lab-side post-R7c campaign round.
+
+## What it is
+
+`belief_touches` is a SQLite sidecar table next to `injection_events`
+(#779). Where `injection_events` records every (turn × belief) inject
+row for the close-the-loop relevance sweeper, `belief_touches` keeps
+only the *last* touch per (belief, session) with a touch count and an
+event-kind bitmask. The two tables answer different questions:
+
+| Table | Read shape | Cardinality | Consumer |
+|---|---|---|---|
+| `injection_events` | "did the assistant reference this belief?" | one row per (turn × belief) | #779 sweeper |
+| `belief_touches` | "was this belief recently in the prompt?" | one row per (belief × session) | (parked — post-R7c) |
+
+The intended consumer for `belief_touches` is a posterior-rerank
+multiplier that boosts beliefs touched in the last K fires of the
+current session. v1 writes the state but no production caller reads
+it.
+
+## Schema
+
+```sql
+CREATE TABLE belief_touches (
+    belief_id           TEXT    NOT NULL,
+    session_id          TEXT    NOT NULL,
+    last_fire_idx       INTEGER NOT NULL,
+    touch_count         INTEGER NOT NULL DEFAULT 0,
+    event_kinds_bitmask INTEGER NOT NULL DEFAULT 0,
+    PRIMARY KEY (belief_id, session_id),
+    FOREIGN KEY (belief_id) REFERENCES beliefs(id) ON DELETE CASCADE
+);
+CREATE INDEX idx_belief_touches_session_fire
+    ON belief_touches(session_id, last_fire_idx DESC);
+```
+
+`event_kinds_bitmask` reserves four bits:
+
+| Bit | Constant | Written in v1? |
+|---|---|---|
+| 0 | `TOUCH_EVENT_KIND_INJECTION` | yes (the only one) |
+| 1 | `TOUCH_EVENT_KIND_RETRIEVE_HIT` | no — H4 REFUTED at R4/R4e/R5 |
+| 2 | `TOUCH_EVENT_KIND_BFS_VISIT` | no |
+| 3 | `TOUCH_EVENT_KIND_USER_ACTION` | no — H4a deferred |
+
+## How it gets written
+
+On every UserPromptSubmit hook fire that produces a rebuild block, the
+hook:
+
+1. Calls `_ring_append_ids(session_id, injected_ids, ...)` — the
+   existing #744 JSON ring append. Returns the session's
+   `next_fire_idx`.
+2. Calls `_record_touches(session_id, injected_ids,
+   fire_idx=next_fire_idx - 1, ...)`. The fire_idx the touches receive
+   is the same one the ring just assigned, so the JSON ring and the
+   sidecar share a counter.
+3. Inside `_record_touches`, the supplied `injected_ids` are each
+   recorded via `record_touch`. The hook is **forward-only** — it does
+   NOT read the JSON ring or backfill pre-substrate entries. An earlier
+   revision tried a one-shot per-session migration off the JSON ring,
+   but `record_touch` uses `ON CONFLICT DO UPDATE` so the replay was
+   non-idempotent: every UPS fire re-bumped `touch_count` on every ring
+   entry. The migration is gone; ring entries that predate this table
+   are simply not represented in `belief_touches`.
+
+`record_touch` upserts: a new (belief, session) pair inserts with
+`touch_count=1` and the supplied `event_kind` bit set; an existing
+pair refreshes `last_fire_idx`, increments `touch_count` by one, and
+OR-s the event_kind bit into the bitmask.
+
+## Locked decisions honored
+
+- **Determinism (#605, `c06f8d575fad71fb`).** `last_fire_idx` is a
+  monotonic integer per session, not a wall-clock timestamp. Same
+  query + same store + same fire_idx → same window contents across
+  replays.
+- **Federation (#661, `d0c5ecdebb3f0f4d`).** The composite primary key
+  `(belief_id, session_id)` keeps foreign federated beliefs cold every
+  read by construction. No touch row crosses the federation boundary.
+- **PHILOSOPHY narrow surface (#605).** The decay shape is one
+  integer comparison (`is_hot`); no embedding, no ML, no LLM. Pure
+  stdlib.
+- **Audit-immutable `beliefs` table.** Sidecar table preserves the
+  per-turn audit invariant — touch-state updates don't mutate belief
+  rows.
+
+## Why no consumer in v1
+
+The retrieval-consumer plan is a rerank-stage posterior multiplier
+that boosts beliefs `is_hot(b, current_fire_idx, K)` returns True for.
+DESIGN.md v1 (`experiments/hot-path/DESIGN.md` in the lab) gates the
+consumer flip on two preconditions:
+
+1. **PR #782** (v3.1 `JUDGE_PROMPT_TEMPLATE` sharpening + hot_start
+   fixture widening) — landed 2026-05-14.
+2. **R7c production-posterior-temperature ρ measurement** — needs
+   ≥50 rows accumulated in `injection_events` on a real per-project
+   DB. Structurally unrunnable until the substrate has been live long
+   enough to gather rows.
+
+Until R7c reports `r`, the v1 substrate writes are the only useful
+output — they buy R7c the ability to run. The consumer flip is a
+separate PR.
+
+## What R4 measures and what it does NOT
+
+The lab campaign's R4 series proved that adding `retrieve_hit` events
+to the touch state changes the top-K rerank ordering on the corpus by
+0% (Jaccard 1.000 across the standard cells). That's why bit 1 stays
+unwritten: `retrieve_hit` adds zero observable surface to what
+consumers actually see. The decision is robust to formula choice
+(R4d/R4e), corpus dwell (R4b), event-mix frequency (R4c), and
+multiplicative-vs-additive blend (R5).
+
+R4 does **not** measure rebuilder continuation fidelity. That is H3's
+job (R3 — load-bearing, parked on R7c). v1 ships the substrate so
+H3 can be measured at all.
+
+## Inspection
+
+```
+$ aelf doctor --hot-path
+aelf doctor --hot-path: 2 session(s) with touch state.
+session_id                                            rows    max_fire_idx
+<session-A>                                             42              81
+<session-B>                                             12              17
+```
+
+Empty (cold start, no UPS fires yet under this PR):
+
+```
+$ aelf doctor --hot-path
+aelf doctor --hot-path: belief_touches is empty.
+```
+
+Read-only; always exits 0. The future consumer flip will add gate
+semantics here.
+
+## Window default
+
+`DEFAULT_TOUCH_WINDOW_K = 50` lives as a module constant in
+`src/aelfrice/hot_path.py`. The value is sourced from the lab
+campaign's R2c canonical cell. Promotion to a
+`meta:retrieval.hot_window_K` knob is an explicit follow-up if the
+consumer flip lands and motivates tuning — DESIGN.md v1 locks the
+constant per the "non-decisions" section: move it only by
+re-measurement, not by config knob.
+
+## File map
+
+| Path | Role |
+|---|---|
+| `src/aelfrice/hot_path.py` | Pure helpers (`is_hot`), constants (`DEFAULT_TOUCH_WINDOW_K`, `TOUCH_EVENT_KIND_*`). |
+| `src/aelfrice/store.py` | Schema DDL, `record_touch`, `read_touch_set_in_window`, `count_touches_for_session`, `list_touch_sessions`. |
+| `src/aelfrice/hook.py` | `_record_touches` helper; UPS call site after `_ring_append_ids`. |
+| `src/aelfrice/cli.py` | `aelf doctor --hot-path` surface. |
+| `tests/test_hot_path_touch_state.py` | Schema + store + helper + hook integration tests (21). |
+
+## Related issues
+
+- [#748](https://github.com/robotrocketscience/aelfrice/issues/748) — R&D campaign tracker (closes once consumer ships and H3 reports).
+- [#816](https://github.com/robotrocketscience/aelfrice/issues/816) — this storage substrate.
+- [#779](https://github.com/robotrocketscience/aelfrice/issues/779) — `injection_events` sibling.
+- [#744](https://github.com/robotrocketscience/aelfrice/issues/744) / [#740](https://github.com/robotrocketscience/aelfrice/issues/740) — JSON injection ring (predecessor; v1 shares its `fire_idx` counter but does NOT migrate ring entries — forward-only).
+- [#605](https://github.com/robotrocketscience/aelfrice/issues/605) — locked PHILOSOPHY (determinism, narrow surface).
+- [#661](https://github.com/robotrocketscience/aelfrice/issues/661) — locked federation decision.

src/aelfrice/cli.py

@@ -3779,6 +3779,8 @@ def _cmd_doctor(args: argparse.Namespace, out: object) -> int:
         return _cmd_doctor_prune_dormant(args, out)
     if getattr(args, "meta_beliefs", False):
         return _cmd_doctor_meta_beliefs(args, out)
+    if getattr(args, "hot_path", False):
+        return _cmd_doctor_hot_path(args, out)
     scope = getattr(args, "scope", None)
     exit_code = 0
     if scope in (None, "hooks"):
@@ -3859,6 +3861,45 @@ def _cmd_doctor_fix_hooks(
         )
 
 
+def _cmd_doctor_hot_path(args: argparse.Namespace, out: object) -> int:
+    """Surface ``belief_touches`` per-session inventory (#816 v1).
+
+    Read-only diagnostic. Lists every session_id that has at least one
+    touch row, with its row count and most-recent fire_idx. Useful for
+    operators inspecting whether the touch-state substrate is
+    accumulating data ahead of the H3 consumer flip (R7c-gated).
+
+    Always exits 0; v1 is observational. The future consumer flip will
+    add a gate path here that compares writes against expected
+    cardinality.
+    """
+    store = _open_store()
+    try:
+        sessions = store.list_touch_sessions()
+    finally:
+        store.close()
+    if not sessions:
+        print(
+            "aelf doctor --hot-path: belief_touches is empty.",
+            file=out,  # type: ignore[arg-type]
+        )
+        return 0
+    print(
+        f"aelf doctor --hot-path: {len(sessions)} session(s) with touch state.",
+        file=out,  # type: ignore[arg-type]
+    )
+    print(
+        f"{'session_id':<48}  {'rows':>8}  {'max_fire_idx':>14}",
+        file=out,  # type: ignore[arg-type]
+    )
+    for sid, n, max_fire in sessions:
+        print(
+            f"{sid:<48}  {n:>8d}  {max_fire:>14d}",
+            file=out,  # type: ignore[arg-type]
+        )
+    return 0
+
+
 def _cmd_doctor_meta_beliefs(args: argparse.Namespace, out: object) -> int:
     """Surface installed meta-belief state (#755 substrate diagnostic).
 
@@ -5726,6 +5767,17 @@ def build_parser(*, show_advanced: bool = False) -> argparse.ArgumentParser:
             "Add --json for machine-readable output."
         ),
     )
+    p_doctor.add_argument(
+        "--hot-path",
+        dest="hot_path",
+        action="store_true",
+        default=False,
+        help=(
+            "report `belief_touches` per-session inventory (#816): "
+            "session_id, row count, max fire_idx. Bypasses the "
+            "hooks/graph checks. Read-only; v1 is observational."
+        ),
+    )
     p_doctor.add_argument(
         "--json",
         dest="json_output",