feat(retrieval): flip entity-persistence demotion default-on in retrieve_v2 (#1096)

CHANGELOG/v3.md

@@ -38,6 +38,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- **Entity-persistence demotion flipped default-on in `retrieve_v2` ([#1096](https://github.com/robotrocketscience/aelfrice/issues/1096)).** The lane added default-off earlier this cycle now defaults on, once the G2 mixed-corpus eval ([#1103](https://github.com/robotrocketscience/aelfrice/issues/1103)) cleared the no-regression gate: durable recall held (20→20), ephemeral coordination demoted (20→3 at the tight pack budget), MRR 0.883→1.000, and the lane is recall-safe / inert on LoCoMo (whose entities are all `noun_phrase`). The resolver now also honours a `[retrieval] use_entity_persist_demote` TOML rung (precedence env → kwarg → TOML → default), so a project can opt out without an env var — matching the `use_heat_kernel` idiom and making the `CONFIG.md` precedence accurate. **Scope:** the lane is wired into `retrieve_v2()` only — the live `UserPromptSubmit` hook uses the legacy `retrieve()` path, which does not expose it, so this flip governs `retrieve_v2` consumers (benchmarks, the eval suite, direct callers) and does **not** change production hook retrieval; the `retrieve()`/`retrieve_v2()` production-path gap is filed for disposition in [#1107](https://github.com/robotrocketscience/aelfrice/issues/1107).
+
 - **Conversational capture is now a first-class `transcript` ingest source ([#1089](https://github.com/robotrocketscience/aelfrice/issues/1089), axis-2 prerequisite).** Passive/transcript capture previously recorded `source_kind=filesystem` in `ingest_log`, indistinguishable from project-file scanning; it now records a new `INGEST_SOURCE_TRANSCRIPT` kind. Behaviour is otherwise preserved — the retention default is unchanged (`RETENTION_FACT`), the `role=user → origin=user_transcript` undeflated-prior routing still fires, and replay-equality holds (a `transcript_v0_1.jsonl` replay-soak corpus covers the new kind, re-deriving on the `agent_inferred` path so it is stable under the `raw_meta=None` production replay). Side benefit: the user-transcript branch now keys on the transcript *source kind* rather than a bare `source_path=="transcript"` string match, so a project file literally named `transcript` can no longer spoof the undeflated user prior. This names the source so "rank curated claude-memory above conversational capture" (axis 2) becomes expressible; the ranking lever itself is a separate follow-up and this does **not** close #1089.
 
 ### Fixed

docs/concepts/ARCHITECTURE.md

@@ -122,12 +122,12 @@ Dedupe L1+L2.5+L3 against L0 ids
 Trim from tail until sum(estimated_tokens) ≤ token_budget
 ```
 
-Two **default-off rerank modifiers** refine the ranked (L1 / L2.5) tiers without adding a lane:
+Two **rerank modifiers** refine the ranked (L1 / L2.5) tiers without adding a lane (both wired into `retrieve_v2` only — the legacy `retrieve()` hook path exposes neither):
 
-- **Entity-persistence demotion** (#1096, `[retrieval] use_entity_persist_demote` / `AELFRICE_ENTITY_PERSIST_DEMOTE`) — the organic sink for #1086's junk-percolation problem. A log-additive **demotion** term `min(0, log(S1 + ε))`, where `S1 = durable / (durable + transient + 1)` is read from the `belief_entities` index (one batched query), down-weights candidates that ground only to *transient* coordination tokens (bare PR/issue numbers, version/branch tags) relative to *durable* entities (file paths, error codes, symbols). Pure demotion (well-grounded candidates are neutral, never boosted); applied only to entity-bearing candidates, so entity-free prose is untouched. Note the sink is **content-referential, not temporal** — a time/cold-decay sink was measured empirically inert here (the junk is *hot*, not stale). Default-ON flip is gated on a retrieval-bench no-regression check and is a separate operator call.
+- **Entity-persistence demotion** (#1096, `[retrieval] use_entity_persist_demote` / `AELFRICE_ENTITY_PERSIST_DEMOTE`) — the organic sink for #1086's junk-percolation problem. A log-additive **demotion** term `min(0, log(S1 + ε))`, where `S1 = durable / (durable + transient + 1)` is read from the `belief_entities` index (one batched query), down-weights candidates that ground only to *transient* coordination tokens (bare PR/issue numbers, version/branch tags) relative to *durable* entities (file paths, error codes, symbols). Pure demotion (well-grounded candidates are neutral, never boosted); applied only to entity-bearing candidates, so entity-free prose is untouched. Note the sink is **content-referential, not temporal** — a time/cold-decay sink was measured empirically inert here (the junk is *hot*, not stale). **Default-ON in `retrieve_v2` since v4.0**, flipped once the #1096 G2 mixed-corpus eval (#1103) cleared the no-regression gate; opt out with a falsy env/kwarg/TOML rung.
 - **Origin-priority tie-break** (#1089, `[retrieval] use_origin_tiebreak` / `AELFRICE_ORIGIN_TIEBREAK`) — a within-tier tie-break (not a rerank term): when two candidates tie on relevance, the higher-trust *origin* wins, sitting between the relevance score and the id tie-break so relevance always dominates. Byte-identical when off.
 
-Both are deterministic (#605) and byte-identical to the prior pipeline when their flags are unset.
+Both are deterministic (#605) and byte-identical to the prior pipeline when their flags resolve falsy (origin tie-break is default-off; entity-persistence demotion is default-on, so opt out explicitly for parity).
 
 Token estimate: `(len(content) + 3) // 4`. Empty query: L0 only. L0 always wins overflow.
 

docs/user/CONFIG.md

@@ -9,7 +9,7 @@ This is the reference for power users whose project has a documentation idiom or
 A single optional TOML file at the root of a project (or any ancestor). It exposes the following power-user surfaces:
 
 - `[noise]` — onboard-time belief filter. Changes how `aelf onboard` ingests beliefs; nothing else.
-- `[retrieval]` (v1.3+) — retrieval-time tier toggles + ranking. Knobs: `entity_index_enabled` (L2.5), `bfs_enabled` (L3), `posterior_weight` (partial Bayesian-weighted L1 ranking), `l1_limit` + `token_budget` (the #1045 wide-retrieval knobs — BM25 candidate cap + token budget, default 50/2400; raise both together for multi-hop recall), `use_bm25f_anchors` (BM25F-with-anchor-text since v1.7), `use_heat_kernel` (authority scoring lane, default-on since v2.1), `use_hrr_structural` (HRR structural-query lane, default-on since v2.1), `hrr_persist` (HRR structural-index on-disk persistence, default-on since v3.0), `use_type_aware_compression` (per-belief retention-class compression, default-on since #769), `use_intentional_clustering` (co-locating related beliefs, default-on since v3.0), `expansion_gate_enabled`, `use_gamma_posterior_temperature` (default off), and `use_zeta_posterior_rerank` (default off; mutually exclusive with the γ flag — `retrieve()` raises `ValueError` when both are on), `use_temporal_spine` + `temporal_spine_budget` (the #1064 chronological-adjacency lane, default off/32; pairs with `[ingest] write_temporal_spine`), `use_entity_persist_demote` (the #1096 entity-persistence demotion / organic-sink rerank modifier, default off), `use_origin_tiebreak` (the #1089 origin-priority within-tier tie-break, default off). Two placeholder flags (`use_signed_laplacian`, `use_posterior_ranking`) are recognised but emit a deprecation warning if set — their lanes have not yet shipped.
+- `[retrieval]` (v1.3+) — retrieval-time tier toggles + ranking. Knobs: `entity_index_enabled` (L2.5), `bfs_enabled` (L3), `posterior_weight` (partial Bayesian-weighted L1 ranking), `l1_limit` + `token_budget` (the #1045 wide-retrieval knobs — BM25 candidate cap + token budget, default 50/2400; raise both together for multi-hop recall), `use_bm25f_anchors` (BM25F-with-anchor-text since v1.7), `use_heat_kernel` (authority scoring lane, default-on since v2.1), `use_hrr_structural` (HRR structural-query lane, default-on since v2.1), `hrr_persist` (HRR structural-index on-disk persistence, default-on since v3.0), `use_type_aware_compression` (per-belief retention-class compression, default-on since #769), `use_intentional_clustering` (co-locating related beliefs, default-on since v3.0), `expansion_gate_enabled`, `use_gamma_posterior_temperature` (default off), and `use_zeta_posterior_rerank` (default off; mutually exclusive with the γ flag — `retrieve()` raises `ValueError` when both are on), `use_temporal_spine` + `temporal_spine_budget` (the #1064 chronological-adjacency lane, default off/32; pairs with `[ingest] write_temporal_spine`), `use_entity_persist_demote` (the #1096 entity-persistence demotion / organic-sink rerank modifier, default **on** in `retrieve_v2` since v4.0; not exposed on the legacy `retrieve()` path), `use_origin_tiebreak` (the #1089 origin-priority within-tier tie-break, default off). Two placeholder flags (`use_signed_laplacian`, `use_posterior_ranking`) are recognised but emit a deprecation warning if set — their lanes have not yet shipped.
 - `[rebuilder]` (v1.4+) — context-rebuilder knobs: `turn_window_n` (default 50), `token_budget` (default 4000), `trigger_mode` (`manual`|`threshold`|`dynamic`, default `threshold`), `threshold_fraction` (default 0.6), and `query_strategy` (v1.7+, default `stack-r1-r3` since v3.0). `[rebuild_floor]` (v1.7+) sets the token-budget floors for the session-scoped and L1 belief lanes (`[rebuild_floor] session` and `[rebuild_floor] l1`).
 - `[onboard.llm]` (v1.3.0+) — direct-API onboard classifier gate; documented under [Keys § `[onboard.llm]`](#onboardllm-v130) below.
 - `[cadence]`, `[implicit_feedback]`, and `[hook_audit]` — feedback-cadence scoring, deferred retrieval-exposure feedback, and the per-turn hook audit log. Recognised here but documented in their module docstrings (`src/aelfrice/cadence.py`, `src/aelfrice/deferred_feedback.py`, `src/aelfrice/hook.py`).
@@ -396,21 +396,21 @@ Precedence (first decisive wins): env var `AELFRICE_POSTERIOR_WEIGHT=<float>` >
 
 ### `use_entity_persist_demote`
 
-Boolean, default `false` (v4.0+, [#1096](https://github.com/robotrocketscience/aelfrice/issues/1096)). Enables the **entity-persistence demotion lane** — a deterministic *organic sink* for the #1086 junk-percolation problem (junk ranks up, not down), applied as a log-additive rerank modifier over the ranked candidate tiers.
+Boolean, default `true` in `retrieve_v2` since v4.0 ([#1096](https://github.com/robotrocketscience/aelfrice/issues/1096); flipped from default-off once the G2 mixed-corpus eval [#1103] cleared the no-regression gate). The **entity-persistence demotion lane** is a deterministic *organic sink* for the #1086 junk-percolation problem (junk ranks up, not down), applied as a log-additive rerank modifier over the ranked candidate tiers. **Scope:** the lane is wired into `retrieve_v2()` only — the legacy `retrieve()` path used by the live `UserPromptSubmit` hook does not expose it, so this default governs `retrieve_v2` consumers (benchmarks, the eval suite, direct callers) until the production cutover.
 
 For each entity-bearing candidate it reads a grounding score `S1 = durable / (durable + transient + 1)` from the `belief_entities` index (one batched query over the candidate set), then applies the penalty `min(0, log(S1 + ε))`. Beliefs that ground only to *transient* coordination tokens (bare PR/issue numbers, version/branch tags) are demoted below those that ground to *durable* entities (file paths, error codes, symbol identifiers). It is a **pure demotion** — well-grounded beliefs are neutral, never boosted — and touches only entity-bearing candidates, so entity-free durable content (docstrings, formulae) is never penalised. Measured separation on a 118-belief hand-labelled set: durable vs ephemeral S1 mean 0.56 vs 0.06, lifting the durable-above-ephemeral ranking AUC from 0.48 to 0.87.
 
 The sink is **content-referential, not temporal**: a time/recency decay sink was measured empirically inert for this workload (the junk is *hot*, not stale), so this lane — not cold-hibernation — is the organic sink. Deterministic per #605 (an entity-index join, no embeddings), byte-identical when unset.
 
-Precedence (first decisive wins): env var `AELFRICE_ENTITY_PERSIST_DEMOTE=1` > explicit Python kwarg `use_entity_persist_demote=<bool>` on `retrieve()` / `retrieve_v2()` > TOML `[retrieval] use_entity_persist_demote` > default `false`. The default-ON flip is gated on a retrieval-bench no-regression check and is a separate operator call.
+Precedence (first decisive wins): env var `AELFRICE_ENTITY_PERSIST_DEMOTE=1`/`0` > explicit Python kwarg `use_entity_persist_demote=<bool>` on `retrieve_v2()` > TOML `[retrieval] use_entity_persist_demote` > default `true`. Set any rung falsy to opt out for parity with the pre-flip ranking.
 
 ### `use_origin_tiebreak`
 
 Boolean, default `false` (v4.0+, [#1089](https://github.com/robotrocketscience/aelfrice/issues/1089)). Enables the **origin-priority tie-break**: when two ranked candidates tie on relevance, the higher-trust *origin* wins (e.g. a belief curated from a `user`/`feedback` fact file outranks one auto-captured from a chat transcript).
 
 This is a within-tier **tie-break**, never a primary rerank term — the origin key sits *between* the relevance score and the id tie-break, so relevance always dominates and byte-identical behaviour is preserved when the flag is off. It applies in both ranked tiers (the L1 FTS rerank and the L2.5 entity-index overlap). Deliberately *not* an origin *rerank lane* — that was refuted on LoCoMo in #1013 (the failure there was a BM25 recall limit, which reranking cannot fix). Deterministic per #605.
 
-Precedence (first decisive wins): env var `AELFRICE_ORIGIN_TIEBREAK=1` > explicit Python kwarg `use_origin_tiebreak=<bool>` on `retrieve()` / `retrieve_v2()` > TOML `[retrieval] use_origin_tiebreak` > default `false`. Note a single-provenance corpus (e.g. LoCoMo) shares one origin tier, so the tie-break is inert there; the default-ON flip is a separate operator call.
+Precedence (first decisive wins): env var `AELFRICE_ORIGIN_TIEBREAK=1` > explicit Python kwarg `use_origin_tiebreak=<bool>` on `retrieve_v2()` (the lane is not exposed on the legacy `retrieve()` path) > TOML `[retrieval] use_origin_tiebreak` > default `false`. Note a single-provenance corpus (e.g. LoCoMo) shares one origin tier, so the tie-break is inert there; the default-ON flip is a separate operator call.
 
 Negative values clamp to `0.0`. Non-numeric env values trace to stderr and fall through. The cache key is extended with the resolved weight (rounded to four decimals), so two callers passing different weights against the same store do not collide on a shared `RetrievalCache`.