feat(retrieval): entity-persist demotion lane onto production retrieve() (#1107 phase 3, #1096)

CHANGELOG/v3.md

@@ -10,7 +10,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- **Production retrieval converges on `retrieve_v2`; temporal-spine lane goes live end-to-end ([#1107](https://github.com/robotrocketscience/aelfrice/issues/1107), Phase 2; [#1064](https://github.com/robotrocketscience/aelfrice/issues/1064)).** Several staged retrieval lanes were wired into `retrieve_v2()` only, so their default-ON flips changed benchmark/eval consumers but never what a live host retrieved — the production hook path (`hook_search.retrieve()`, `context_rebuilder`, `mcp_server`) still called the legacy `retrieve()`. The #1107 cutover makes `retrieve()` a thin adapter over `retrieve_v2` (Phase 1, behaviour-preserving: all six staged lanes forced off, pinned byte-identical by `tests/test_retrieve_v2_equivalence.py`), then graduates lanes onto the live path one at a time behind a hot-path latency gate. The **temporal-spine lane** ([#1064](https://github.com/robotrocketscience/aelfrice/issues/1064)) is the first to graduate: the shim now passes `use_temporal_spine` through resolver-driven (env → TOML → default) and its default is flipped **ON**, so — together with the writer flip ([#1111](https://github.com/robotrocketscience/aelfrice/issues/1111)) — the +14.6pp-coverage spine is live on every host's `retrieve()`, not just the eval surface. G3 latency stays in-band (`benchmarks/temporal_spine_latency.py`, Δp95 ≤ 50 ms); the lane is a no-op on spineless stores. Opt out with `AELFRICE_TEMPORAL_SPINE=0` or `[retrieval] use_temporal_spine = false`. The other five staged lanes remain forced off pending their own latency-gated graduation. Deterministic per #605.
+- **Entity-persist demotion lane goes live on the production `retrieve()` path ([#1107](https://github.com/robotrocketscience/aelfrice/issues/1107), Phase 3; [#1096](https://github.com/robotrocketscience/aelfrice/issues/1096)).** The #1086 junk-percolation sink — the log-additive demotion that pushes beliefs grounded only in transient coordination tokens (bare PR/issue numbers, version tags) below those grounded in durable entities (file paths, symbols) — was default-ON in `retrieve_v2` since v4.0 but inert on the live hook path, which called the legacy `retrieve()`. The #1107 shim now passes `use_entity_persist_demote` resolver-driven (`None` → env→TOML→default-ON, the G2-cleared #1103 default) instead of hard-off, so the demotion reaches every host's `retrieve()`, `context_rebuilder`, and `mcp_server` — this is what makes the #1086 fix real for users rather than only bench/eval consumers. No-op on entity-free or well-grounded candidates; one batched `belief_entities` query, no candidate expansion. Opt out with `AELFRICE_ENTITY_PERSIST_DEMOTE=0` or `[retrieval] use_entity_persist_demote = false`. The `retrieve()`↔`retrieve_v2` equivalence guard now pins both graduated lanes (spine + entity-persist) resolver-driven with the remaining four forced off. Deterministic per #605.
+
+- **Production retrieval converges on `retrieve_v2`; temporal-spine lane goes live end-to-end ([#1107](https://github.com/robotrocketscience/aelfrice/issues/1107), Phase 2; [#1064](https://github.com/robotrocketscience/aelfrice/issues/1064)).** Several staged retrieval lanes were wired into `retrieve_v2()` only, so their default-ON flips changed benchmark/eval consumers but never what a live host retrieved — the production hook path (`hook_search.retrieve()`, `context_rebuilder`, `mcp_server`) still called the legacy `retrieve()`. The #1107 cutover makes `retrieve()` a thin adapter over `retrieve_v2` (Phase 1, behaviour-preserving: all six staged lanes forced off, pinned byte-identical by `tests/test_retrieve_v2_equivalence.py`), then graduates lanes onto the live path one at a time behind a hot-path latency gate. The **temporal-spine lane** ([#1064](https://github.com/robotrocketscience/aelfrice/issues/1064)) is the first to graduate: the shim now passes `use_temporal_spine` through resolver-driven (env → TOML → default) and its default is flipped **ON**, so — together with the writer flip ([#1111](https://github.com/robotrocketscience/aelfrice/issues/1111)) — the +14.6pp-coverage spine is live on every host's `retrieve()`, not just the eval surface. G3 latency stays in-band (`benchmarks/temporal_spine_latency.py`, Δp95 ≤ 50 ms); the lane is a no-op on spineless stores. Opt out with `AELFRICE_TEMPORAL_SPINE=0` or `[retrieval] use_temporal_spine = false`. The other staged lanes graduate one at a time behind their own gates (see the entity-persist entry above). Deterministic per #605.
 
 - **Origin-priority retrieval tie-break, default-off ([#1089](https://github.com/robotrocketscience/aelfrice/issues/1089), axis 2).** When two retrieval candidates tie on relevance, the higher-trust *origin* now wins — so a belief curated from a claude-memory `user`/`feedback` fact file (`origin=user_validated`) outranks one auto-captured from a chat transcript (`origin=user_transcript`). It is a pure within-tier **tie-break**, never a primary rerank term: the origin key sits *between* the relevance score and the id tie-break, so relevance always dominates and byte-identical behaviour is preserved when the flag is off. This is deliberately not a resurrected origin *rerank lane* — that was refuted on LoCoMo in #1013, where the failure was a BM25 *recall* limit (the trusted fact never became a candidate), which reranking cannot fix. The priority ladder mirrors `contradiction.precedence_class` (a sync-guard test keeps the two copies aligned) and is applied in both ranked candidate tiers: the L1 FTS rerank (`_l1_hits`) and the L2.5 entity-index overlap (`MemoryStore.lookup_entities`), so it covers both keyword-rerank and exact-entity-match hits. Flag-gated (`AELFRICE_ORIGIN_TIEBREAK` / `use_origin_tiebreak`), deterministic per #605; `benchmarks/origin_tiebreak_ablation.py` is the offline ablation. The mechanism lands default-off; the default-ON flip is gated on a LoCoMo retrieval no-regression run (note that a single-provenance corpus like LoCoMo shares one origin tier, so the tie-break is inert there and cannot regress recall) and is a separate operator call.
 

docs/user/CONFIG.md

@@ -9,7 +9,7 @@ This is the reference for power users whose project has a documentation idiom or
 A single optional TOML file at the root of a project (or any ancestor). It exposes the following power-user surfaces:
 
 - `[noise]` — onboard-time belief filter. Changes how `aelf onboard` ingests beliefs; nothing else.
-- `[retrieval]` (v1.3+) — retrieval-time tier toggles + ranking. Knobs: `entity_index_enabled` (L2.5), `bfs_enabled` (L3), `posterior_weight` (partial Bayesian-weighted L1 ranking), `l1_limit` + `token_budget` (the #1045 wide-retrieval knobs — BM25 candidate cap + token budget, default 50/2400; raise both together for multi-hop recall), `use_bm25f_anchors` (BM25F-with-anchor-text since v1.7), `use_heat_kernel` (authority scoring lane, default-on since v2.1), `use_hrr_structural` (HRR structural-query lane, default-on since v2.1), `hrr_persist` (HRR structural-index on-disk persistence, default-on since v3.0), `use_type_aware_compression` (per-belief retention-class compression, default-on since #769), `use_intentional_clustering` (co-locating related beliefs, default-on since v3.0), `expansion_gate_enabled`, `use_gamma_posterior_temperature` (default off), and `use_zeta_posterior_rerank` (default off; mutually exclusive with the γ flag — `retrieve()` raises `ValueError` when both are on), `use_temporal_spine` + `temporal_spine_budget` (the #1064 chronological-adjacency lane, default **on**/32 since v4.0 — live on the production `retrieve()` path via the #1107 cutover; pairs with `[ingest] write_temporal_spine`), `use_entity_persist_demote` (the #1096 entity-persistence demotion / organic-sink rerank modifier, default **on** in `retrieve_v2` since v4.0; not exposed on the legacy `retrieve()` path), `use_origin_tiebreak` (the #1089 origin-priority within-tier tie-break, default off). Two placeholder flags (`use_signed_laplacian`, `use_posterior_ranking`) are recognised but emit a deprecation warning if set — their lanes have not yet shipped.
+- `[retrieval]` (v1.3+) — retrieval-time tier toggles + ranking. Knobs: `entity_index_enabled` (L2.5), `bfs_enabled` (L3), `posterior_weight` (partial Bayesian-weighted L1 ranking), `l1_limit` + `token_budget` (the #1045 wide-retrieval knobs — BM25 candidate cap + token budget, default 50/2400; raise both together for multi-hop recall), `use_bm25f_anchors` (BM25F-with-anchor-text since v1.7), `use_heat_kernel` (authority scoring lane, default-on since v2.1), `use_hrr_structural` (HRR structural-query lane, default-on since v2.1), `hrr_persist` (HRR structural-index on-disk persistence, default-on since v3.0), `use_type_aware_compression` (per-belief retention-class compression, default-on since #769), `use_intentional_clustering` (co-locating related beliefs, default-on since v3.0), `expansion_gate_enabled`, `use_gamma_posterior_temperature` (default off), and `use_zeta_posterior_rerank` (default off; mutually exclusive with the γ flag — `retrieve()` raises `ValueError` when both are on), `use_temporal_spine` + `temporal_spine_budget` (the #1064 chronological-adjacency lane, default **on**/32 since v4.0 — live on the production `retrieve()` path via the #1107 cutover; pairs with `[ingest] write_temporal_spine`), `use_entity_persist_demote` (the #1096 entity-persistence demotion / organic-sink rerank modifier, default **on** since v4.0 — live on the production `retrieve()` path via the #1107 cutover), `use_origin_tiebreak` (the #1089 origin-priority within-tier tie-break, default off). Two placeholder flags (`use_signed_laplacian`, `use_posterior_ranking`) are recognised but emit a deprecation warning if set — their lanes have not yet shipped.
 - `[rebuilder]` (v1.4+) — context-rebuilder knobs: `turn_window_n` (default 50), `token_budget` (default 4000), `trigger_mode` (`manual`|`threshold`|`dynamic`, default `threshold`), `threshold_fraction` (default 0.6), and `query_strategy` (v1.7+, default `stack-r1-r3` since v3.0). `[rebuild_floor]` (v1.7+) sets the token-budget floors for the session-scoped and L1 belief lanes (`[rebuild_floor] session` and `[rebuild_floor] l1`).
 - `[onboard.llm]` (v1.3.0+) — direct-API onboard classifier gate; documented under [Keys § `[onboard.llm]`](#onboardllm-v130) below.
 - `[cadence]`, `[implicit_feedback]`, and `[hook_audit]` — feedback-cadence scoring, deferred retrieval-exposure feedback, and the per-turn hook audit log. Recognised here but documented in their module docstrings (`src/aelfrice/cadence.py`, `src/aelfrice/deferred_feedback.py`, `src/aelfrice/hook.py`).
@@ -397,13 +397,13 @@ Precedence (first decisive wins): env var `AELFRICE_POSTERIOR_WEIGHT=<float>` >
 
 ### `use_entity_persist_demote`
 
-Boolean, default `true` in `retrieve_v2` since v4.0 ([#1096](https://github.com/robotrocketscience/aelfrice/issues/1096); flipped from default-off once the G2 mixed-corpus eval [#1103] cleared the no-regression gate). The **entity-persistence demotion lane** is a deterministic *organic sink* for the #1086 junk-percolation problem (junk ranks up, not down), applied as a log-additive rerank modifier over the ranked candidate tiers. **Scope:** the lane is wired into `retrieve_v2()` only — the legacy `retrieve()` path used by the live `UserPromptSubmit` hook does not expose it, so this default governs `retrieve_v2` consumers (benchmarks, the eval suite, direct callers) until the production cutover.
+Boolean, default `true` in `retrieve_v2` since v4.0 ([#1096](https://github.com/robotrocketscience/aelfrice/issues/1096); flipped from default-off once the G2 mixed-corpus eval [#1103] cleared the no-regression gate). The **entity-persistence demotion lane** is a deterministic *organic sink* for the #1086 junk-percolation problem (junk ranks up, not down), applied as a log-additive rerank modifier over the ranked candidate tiers. **Scope:** default-ON on the production `retrieve()` path since the #1107 Phase-3 cutover (the shim passes the flag resolver-driven), so the live `UserPromptSubmit` hook, `context_rebuilder`, and `mcp_server` all run the demotion — this is what makes the #1086 junk-percolation fix reach real hosts, not just `retrieve_v2` consumers.
 
 For each entity-bearing candidate it reads a grounding score `S1 = durable / (durable + transient + 1)` from the `belief_entities` index (one batched query over the candidate set), then applies the penalty `min(0, log(S1 + ε))`. Beliefs that ground only to *transient* coordination tokens (bare PR/issue numbers, version/branch tags) are demoted below those that ground to *durable* entities (file paths, error codes, symbol identifiers). It is a **pure demotion** — well-grounded beliefs are neutral, never boosted — and touches only entity-bearing candidates, so entity-free durable content (docstrings, formulae) is never penalised. Measured separation on a 118-belief hand-labelled set: durable vs ephemeral S1 mean 0.56 vs 0.06, lifting the durable-above-ephemeral ranking AUC from 0.48 to 0.87.
 
 The sink is **content-referential, not temporal**: a time/recency decay sink was measured empirically inert for this workload (the junk is *hot*, not stale), so this lane — not cold-hibernation — is the organic sink. Deterministic per #605 (an entity-index join, no embeddings), byte-identical when unset.
 
-Precedence (first decisive wins): env var `AELFRICE_ENTITY_PERSIST_DEMOTE=1`/`0` > explicit Python kwarg `use_entity_persist_demote=<bool>` on `retrieve_v2()` > TOML `[retrieval] use_entity_persist_demote` > default `true`. Set any rung falsy to opt out for parity with the pre-flip ranking.
+Precedence (first decisive wins): env var `AELFRICE_ENTITY_PERSIST_DEMOTE=1`/`0` > explicit Python kwarg `use_entity_persist_demote=<bool>` on `retrieve_v2()` > TOML `[retrieval] use_entity_persist_demote` > default `true`. The lean production `retrieve()` shim takes no per-call kwarg — opt out there via the env var or TOML key for parity with the pre-flip ranking.
 
 ### `use_origin_tiebreak`
 

src/aelfrice/retrieval.py

@@ -2996,17 +2996,18 @@ def retrieve(
     bare `retrieve()` so `rebuild_v14`'s call site observes the
     toggle that A4 (#775) measures.
     """
-    # #1107 Phase 2: `retrieve()` is a thin adapter over `retrieve_v2`, the
+    # #1107 cutover: `retrieve()` is a thin adapter over `retrieve_v2`, the
     # single retrieval implementation the production hook path shares with
     # the benchmark/eval surface. Lanes light up in production one at a time
-    # as each clears its latency gate. The **temporal-spine** lane is the
-    # first to graduate (#1064): its flag is passed through as `None` so the
-    # production `retrieve()` path honours the `is_temporal_spine_enabled`
-    # resolver (env -> TOML -> default) exactly like the eval surface. The
-    # remaining five staged lanes (entity-persist demotion, origin tie-break,
-    # HRR-expand, intentional clustering, HRR-structural) stay forced OFF,
-    # because `retrieve()`'s historical pack loop never ran them; equivalence
-    # for those is pinned by tests/test_retrieve_v2_equivalence.py.
+    # as each clears its gate; a graduated lane is passed through as `None`
+    # so the production path honours its resolver (env -> TOML -> default)
+    # exactly like the eval surface. Graduated so far: **temporal spine**
+    # (#1064, Phase 2) and **entity-persist demotion** (#1096, Phase 3 — the
+    # #1086 junk-percolation sink, resolver default-ON since the G2 eval).
+    # The remaining four staged lanes (origin tie-break, HRR-expand,
+    # intentional clustering, HRR-structural) stay forced OFF, because
+    # `retrieve()`'s historical pack loop never ran them; equivalence for
+    # those is pinned by tests/test_retrieve_v2_equivalence.py.
     out = retrieve_v2(
         store,
         query,
@@ -3028,11 +3029,12 @@ def retrieve(
         eigenbasis_cache=eigenbasis_cache,
         use_type_aware_compression=use_type_aware_compression,
         manifest_reference_locks=manifest_reference_locks,
-        # Temporal-spine pilot lane (#1064/#1107 Phase 2): resolver-driven
-        # (env -> TOML -> default) rather than hard-off, so a production host
-        # gets the lane the moment its resolver says on.
+        # Graduated lanes (#1107): resolver-driven (env -> TOML -> default)
+        # rather than hard-off, so a production host gets the lane the moment
+        # its resolver says on. temporal spine #1064 (Phase 2), entity-persist
+        # demotion #1096 (Phase 3, resolver default-ON).
         use_temporal_spine=None,
-        use_entity_persist_demote=False,
+        use_entity_persist_demote=None,
         use_origin_tiebreak=False,
         use_hrr_expand=False,
         use_intentional_clustering=False,

tests/test_bayesian_ranking.py

@@ -99,7 +99,9 @@ def test_ac1_retrieve_and_retrieve_v2_accept_posterior_weight() -> None:
 # --- AC2: posterior_weight=0.0 is byte-identical to v1.0.x ordering ------
 
 
-def test_ac2_weight_zero_byte_identical_to_v10x() -> None:
+def test_ac2_weight_zero_byte_identical_to_v10x(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
     """At weight 0 AND with the legacy plain-BM25 (non-BM25F) path,
     retrieve() result equals what `store.search_beliefs(...)` returns
     for the L1 portion. (L0 prefix is unaffected by weight.)
@@ -109,7 +111,13 @@ def test_ac2_weight_zero_byte_identical_to_v10x() -> None:
     `use_bm25f_anchors=False` here pins the contract to the legacy
     v1.0.x BM25 path so this regression test still asserts what it
     was meant to assert.
+
+    Per #1096 / #1107 Phase 3: entity-persist demotion is default-ON on the
+    production `retrieve()` path and reranks entity-bearing candidates, so it
+    perturbs the pure-BM25 tie order too. `AELFRICE_ENTITY_PERSIST_DEMOTE=0`
+    isolates the posterior_weight=0 contract from it, mirroring the BM25F pin.
     """
+    monkeypatch.setenv("AELFRICE_ENTITY_PERSIST_DEMOTE", "0")
     s = _equal_bm25_store()
     direct = s.search_beliefs("widget", limit=50)
     weighted = retrieve(