feat(retrieval): live close-the-loop relevance-signal infrastructure (#779)

CHANGELOG.md

@@ -12,6 +12,8 @@ installable release; see the roadmap in [README.md](README.md).
 
 ### Added
 
+- **Live close-the-loop relevance-signal infrastructure** ([#779](https://github.com/robotrocketscience/aelfrice/issues/779), umbrella [#480](https://github.com/robotrocketscience/aelfrice/issues/480)). Closes the gap [#756](https://github.com/robotrocketscience/aelfrice/issues/756) and [#757](https://github.com/robotrocketscience/aelfrice/issues/757) flagged in their D4 deferrals: production retrieval had no live source of `referenced ∈ {0, 1}` evidence per injected belief. Three layers, all stdlib-deterministic per the locked PHILOSOPHY (#605, `c06f8d575fad71fb`). **Layer 1** — new `injection_events` SQL table with `(id, session_id, turn_id, belief_id, injected_at, source, active_consumers, referenced, referenced_at)` columns and three indexes (`session_turn`, `belief`, partial `pending`). `active_consumers` is a canonical-sorted JSON array of meta-belief keys whose retrieval consumer was env-gated ON for the turn; JSON-column shape ratified at [#779#issuecomment-4448107904](https://github.com/robotrocketscience/aelfrice/issues/779#issuecomment-4448107904) over a sidecar table (Q1). New `MemoryStore` API: `record_injection_event`, `list_pending_injection_events` (with `before_turn_id` slicing — the `{utc-compact-ts}-{hex4}` turn-id shape sorts lexicographically = chronologically), `update_injection_referenced` (idempotent at the `UPDATE ... WHERE referenced IS NULL` level). **Layer 2** — new `src/aelfrice/relevance_detection.py` with `normalize_text(s)` (NFC + casefold + whitespace collapse, fixed-point idempotent), `is_referenced(belief, response)`, and `score_references(pairs, response_text, *, strategy=STRATEGY_EXACT_SUBSTRING)`. Exact-substring full-content verbatim per the 2026-05-14 ratification (Q5/Q6); 8-char minimum normalised-belief length to prevent short-belief false positives. `STRATEGY_NGRAM_OVERLAP` is reserved but raises `ValueError` in v1. **Layer 3** — `_sweep_relevance_signal` fires at the start of every UPS hook (right after `apply_sentiment_feedback`, before this turn's retrieval). Reads pending events for the session, joins `event.belief_id → belief.content` via `get_belief()`, reads the assistant transcript via `_read_assistant_text_since(session_id, oldest.injected_at)` (filters `turns.jsonl` by session + role + ts), scores, fires `update_meta_belief(consumer_key, SIGNAL_RELEVANCE, evidence=float(referenced), ...)` once per consumer key in the event's `active_consumers`, then idempotently stamps the row. Substrate's "no-op on non-subscribed consumer" contract keeps the wiring single-sourced via the env flags. Helper `get_active_meta_belief_consumers()` in `retrieval.py` returns the sorted env-truthy set; v1 covers `#756` half-life + `#757` bm25f_anchor_weight; siblings (#758–#760) drop in here as they ship. Fail-soft on every layer — store-open, transcript-read, meta-belief-update, and event-stamp failures each print one stderr line and return; never block retrieval. 63 new tests across `tests/test_injection_events.py` (schema + store API), `tests/test_relevance_detection.py` (Layer 2 purity, normalize fixed point, encoding edge cases), `tests/test_hook_injection_events_wiring.py` (turn-id shape, active-consumer threading, batch-shares-turn-id), and `tests/test_relevance_sweeper.py` (transcript read filters, sweeper hit / miss / idempotent re-run / latency-only-consumer-no-relevance-row). Defers per Q3/Q4: no retention/GC (posteriors decay; ancient rows contribute negligible signal); UPS-only `source` enum (`'pre_compact'` deferred); n-gram detection deferred (Q5); embedding / LLM-judge out of scope (#605). Schema doc at `docs/design/relevance-signal.md`.
+
 - **v3.1 close of #592 hot-start AC — sharpened `JUDGE_PROMPT_TEMPLATE` + widened hot_start fixture to n=10 eval turns** ([#778](https://github.com/robotrocketscience/aelfrice/issues/778)). Run 2 of the multi-judge κ sweep (2026-05-12/13, see [#687](https://github.com/robotrocketscience/aelfrice/issues/687)) saw `inter_judge_kappa.min = 0.0` (gate 0.7) and `hot_start_fidelity_mean = 0.667` (gate 0.8), diagnosed as two compounding causes: (a) the prompt template's "load-bearing claim" clause was operationally under-specified — judges disagreed on which part of each reference counted as load-bearing on 2 of 3 fixture turns; (b) statistical thinness on n=3 binary samples, where a single judge flip drops the κ catastrophically. (a) is addressed by replacing the single-clause definition in `benchmarks/context-rebuilder/judges/llm_judge.py` with a three-step rubric that names the unit of scoring as "specific facts" (concrete atoms — names, paths, values, actions, numeric quantities), requires the candidate to convey every such fact, and defines `matched=true` as "a reader of the candidate alone could state each specific fact"; plus two worked examples (multi-fact and low-content) drawn from outside the hot_start fixture to avoid biasing scoring on the corpus being scored. (b) is addressed by extending `hot_start_debugging_session_001.{jsonl,meta.json}` from 14 to 28 turns, keeping the same fork (turn 8) and the same synthetic CRLF-vs-LF `_dedup_key` debugging scenario, and adding 7 new user/assistant pairs that probe pre-fork facts the prior cut did not exercise (intermittency root cause, literal code change, why-fix-in-`_dedup_key`-not-conftest rationale, dedup tuple shape, NFC normalization form, repro command, conftest file:line). New `eval_turns = [9, 11, 13, 15, 17, 19, 21, 23, 25, 27]`. Test assertion in `tests/test_context_rebuilder_eval_harness_wiring.py` updated to the new tuple. Fixture content remains fully synthetic — no `_dedup_key` or `test_ingest_jsonl_idempotent` exists in source on `github/main`. The κ sweep re-run is operator-dispatched (subagent path per locked bench convention, not direct API) and gates closing the v3.0-deferred hot-start AC ≥80% bullet.
 
 - **Fifth meta-belief consumer: adaptive `bfs_depth_budget`** ([#759](https://github.com/robotrocketscience/aelfrice/issues/759), umbrella [#480](https://github.com/robotrocketscience/aelfrice/issues/480)). Wires `meta:retrieval.bfs_depth_budget` into `retrieve_v2`'s `bfs_max_depth` resolution. Per the 2026-05-13 operator ratification on #756 (C-F template): log-linear bounded `[1, 6]` encoding lives in the consumer with a rounding-to-int step (new wrinkle vs #756/#757 — `expand_bfs` takes `int max_depth`); MVP ships with the `latency` signal only (`bfs_depth` close-the-loop signal deferred to [#779](https://github.com/robotrocketscience/aelfrice/issues/779) per the same D4-deferral pattern #756 used for `relevance`). Rationale for latency-only MVP: `bfs_depth` requires detecting which surfaced beliefs the consuming agent referenced and at what BFS hop they originated — infrastructure not yet available. `latency` is live today and is load-bearing for safety because BFS depth dominates p95 retrieval latency. New module-level constants: `META_BFS_DEPTH_BUDGET_KEY`, `BFS_DEPTH_BUDGET_FLOOR` (1), `BFS_DEPTH_BUDGET_CEIL` (6), `META_BFS_DEPTH_BUDGET_STATIC_DEFAULT` (0.5 → decodes to 2 = round(sqrt(6))), `META_BFS_DEPTH_BUDGET_POSTERIOR_DECAY_SECONDS` (30d). New API: `decode_bfs_depth_budget(v)`, `is_meta_belief_bfs_depth_budget_enabled()`, `install_bfs_depth_budget_meta_belief(store, *, now_ts)`, `resolve_bfs_depth_budget_with_meta(store, *, now_ts, explicit=None)`. Resolver precedence: explicit positive-int kwarg → meta-belief → `BFS_DEFAULT_MAX_DEPTH`; no env-var or TOML layer because `bfs_max_depth` has never had a user-facing config knob. `retrieve_v2` wires the resolver between `effective_now_ts` calculation and the `retrieve_with_tiers` call; the explicit clause passes `None` when the caller passes the default so the meta-belief layer is not inadvertently bypassed. A second independent latency-signal update fires after `retrieve_with_tiers` when the depth-budget flag is on; both the #756 half-life and #759 depth-budget updates run on every call when their respective flags are on. `BFS_DEFAULT_MAX_DEPTH` verified as 2 (not 3 as the spec estimated): `decode_bfs_depth_budget(0.5) = 2 = BFS_DEFAULT_MAX_DEPTH`, so cold-start byte-identity holds. Ships default-OFF behind `AELFRICE_META_BELIEF_BFS_DEPTH_BUDGET` (also accepts `=enabled`); flips on after the #437 corpus A/B bench gate clears. 27 new tests cover log-linear encoding boundaries + int return + bounds + monotonicity + rounding-band transition, env-flag tokens, constants pin, install idempotency + signal-class assertion, five resolver precedence paths, 100-strong-positive-evidence responsiveness, default-OFF byte-identical `retrieve_v2`, and determinism (same evidence sequence + same now_ts → same resolved depth on two fresh stores).

docs/design/relevance-signal.md

@@ -0,0 +1,148 @@
+# Relevance signal — close-the-loop infrastructure
+
+**Issue:** [#779](https://github.com/robotrocketscience/aelfrice/issues/779).
+**Umbrella:** [#480](https://github.com/robotrocketscience/aelfrice/issues/480) (adaptive meta-belief layer).
+**Substrate prereq:** [#755](https://github.com/robotrocketscience/aelfrice/issues/755) (meta-belief tables + `update_meta_belief` API).
+
+## Why this layer exists
+
+The umbrella #480 adaptive-meta-belief substrate (#755) ships four
+signal classes: `relevance`, `latency`, `bfs_depth`, `bm25_l0_ratio`.
+The first two consumers shipped (#756 half-life, #757 BM25F anchor
+weight) wired the **non-relevance** signals only — `latency` and
+`bm25_l0_ratio` respectively — because the production retrieval path
+had no live source of `referenced ∈ {0, 1}` evidence per injected
+belief.
+
+#365 had shipped offline calibration metrics (`precision_at_k`,
+`roc_auc`, `spearman_rho`), but those are bench-time scoring against
+a labeled JSONL fixture. Nothing in the production hook path
+recorded *which beliefs were injected* and detected *did the agent
+reference them next turn*.
+
+#779 closes that loop.
+
+## Three layers, three files
+
+| Layer | Owner | What it does |
+|---|---|---|
+| 1. Injection log | `MemoryStore` (`store.py`) + UPS hook (`hook.py`) | One `injection_events` row per (UPS turn × injected belief), audit-friendly. |
+| 2. Reference detection | `relevance_detection.py` | Pure-function scoring of belief content against assistant response. Exact-substring v1; n-gram is a future opt-in. |
+| 3. Sweeper | `hook.py:_sweep_relevance_signal` | At the start of every UPS hook, score the prior turn's pending events and push `relevance` evidence into each event's active consumers. |
+
+## Data flow
+
+```
+UPS turn N (user prompt)
+├── hits = retrieve(prompt)
+├── _emit_user_prompt_submit_rebuild_log(hits)         # #288 diagnostic JSONL
+├── _record_injection_events(hits, source='ups',       # #779 Layer 1 audit row
+│       active_consumers=get_active_meta_belief_consumers())
+└── render <aelfrice-rebuild> block
+       │
+       ▼  (claude generates response)
+       │
+Stop hook → transcript_logger appends to turns.jsonl
+       │      {role:'assistant', text:..., session_id:..., ts:...}
+       │
+       ▼
+UPS turn N+1 (next user prompt)
+├── apply_sentiment_feedback(...)
+├── _sweep_relevance_signal(session_id)                # #779 Layer 3
+│     ├── list_pending_injection_events(session_id)    # rows with referenced IS NULL
+│     ├── _read_assistant_text_since(session_id, oldest.injected_at)
+│     ├── join event.belief_id → belief.content
+│     ├── score_references(pairs, response_text)       # Layer 2
+│     ├── for each (event_id, referenced):
+│     │     for consumer_key in event.active_consumers:
+│     │         update_meta_belief(consumer_key, SIGNAL_RELEVANCE,
+│     │             evidence=float(referenced), ...)
+│     │     update_injection_referenced(event_id, referenced, ...)
+└── hits = retrieve(prompt)    # consumers see shifted posteriors
+```
+
+## Schema (Layer 1)
+
+```sql
+CREATE TABLE injection_events (
+    id               INTEGER PRIMARY KEY AUTOINCREMENT,
+    session_id       TEXT    NOT NULL,
+    turn_id          TEXT    NOT NULL,
+    belief_id        TEXT    NOT NULL REFERENCES beliefs(id) ON DELETE CASCADE,
+    injected_at      TEXT    NOT NULL,            -- ISO-8601 UTC
+    source           TEXT    NOT NULL,            -- 'ups' (v1); 'pre_compact' deferred
+    active_consumers TEXT    NOT NULL DEFAULT '[]',
+                                                  -- canonical-sorted JSON
+                                                  -- array of meta-belief keys
+    referenced       INTEGER,                     -- NULL / 0 / 1 (tri-state)
+    referenced_at    TEXT
+);
+```
+
+Indexes: `(session_id, turn_id)`, `(belief_id)`, partial
+`(session_id, referenced) WHERE referenced IS NULL` for the sweeper's
+hot path.
+
+## Determinism contract (#605 / `c06f8d575fad71fb`)
+
+- `normalize_text(s)` is a fixed point: NFC + casefold + whitespace
+  collapse. Running it twice yields the same bytes.
+- `score_references(pairs, response_text)` is a pure function: same
+  inputs → byte-identical output list, in input order.
+- The sweeper's wall-clock dependence is bounded to:
+  - `update_meta_belief`'s `now_ts` (caller-supplied; the substrate's
+    decay math is wall-clock-independent at the function level).
+  - `referenced_at` ISO timestamp (audit-only; never re-read).
+
+## Why `active_consumers` is a JSON column
+
+Two design questions ratified in
+[#779#issuecomment-4448107904](https://github.com/robotrocketscience/aelfrice/issues/779#issuecomment-4448107904):
+
+- **Q1 — JSON column vs sidecar table.** Every read path is "one
+  event, all its consumers." No query asks "find all events where
+  consumer X was active." JSON is the cheapest representation; a
+  sidecar `injection_event_consumers(event_id, meta_key)` would cost
+  one extra row per (event × consumer) with no observable benefit.
+- **Q2 — Don't reuse `rebuild_log`.** `rebuild_log` is JSONL on disk
+  (`~/.aelfrice/logs/rebuild/<session>.jsonl`), not SQL. Appending
+  `referenced` to existing lines is a write-in-place footgun; the
+  audit-table-in-SQL precedent (`belief_corroborations`,
+  `deferred_feedback_queue`, `meta_belief_signal_posteriors`) is the
+  right substrate for queryable lifecycle state.
+
+## Deferred (not in v1)
+
+- **GC.** No retention policy ships in v1. The Beta-Bernoulli
+  posteriors decay over time (30d half-life on every consumer), so
+  ancient injection_events rows contribute negligible signal. Add
+  GC as a sub-issue when the storage-vs-evidence trade-off is
+  measured.
+- **`source='pre_compact'`.** The PreCompact rebuilder injects via
+  the `<aelfrice-rebuild>` block but the "next user turn" reference-
+  detection semantics get fuzzy when injection happens *after* a
+  user prompt. v1 ships UPS only; the schema's TEXT-not-CHECK
+  `source` column accommodates the second source without migration.
+- **N-gram overlap detection.** `STRATEGY_NGRAM_OVERLAP` is reserved
+  in `relevance_detection.py` but the dispatch raises `ValueError`
+  in v1. Add as opt-in via `[retrieval] relevance_detection =
+  "ngram"` in a follow-up sub-issue.
+- **Embedding / LLM-judge detection.** Out of scope per locked
+  PHILOSOPHY (#605, `c06f8d575fad71fb`). Not a sub-issue path.
+
+## What this PR enables for siblings
+
+The half-life consumer (#756) subscribes to `latency` only today;
+the sweeper will already push `relevance` evidence into it via
+`update_meta_belief(META_HALF_LIFE_KEY, SIGNAL_RELEVANCE, ...)` if
+the env flag is on — the substrate silently no-ops because the
+consumer doesn't subscribe. Adding `relevance` to a consumer's
+subscription is a separate config decision (the install signature
+is immutable per `meta_beliefs.install_meta_belief`'s "config rows
+are never silently mutated" contract); a future sub-issue can
+ship a migration to re-install consumers with both signal
+subscriptions.
+
+Same applies to #757 BM25F anchor weight (`bm25_l0_ratio` only
+today). And to #758 / #759 / #760 as they ship — `active_consumers`
+in `get_active_meta_belief_consumers()` lights them up automatically.