feat(retrieval): entity-persist demotion lane onto production retrieve() (#1107 phase 3, #1096)

[robotrocketscience]

What

#1107 Phase 3 — graduate the entity-persist demotion lane onto the production retrieve() path. Operator-directed ("take the next lane"). The #1096 demotion (the #1086 junk-percolation sink) was default-ON in retrieve_v2 since v4.0 but inert on the live hook path, which called the legacy retrieve(). This exposes it on production.

Why this lane is next

Of the five staged lanes still hard-off after Phase 2, entity-persist is the clear next graduate:

• Its default-ON decision is already ratified (Entity-persistence demotion prior: a deterministic organic sink for coordination junk (follow-up to #1086/#1081) #1096 scope-A, G2 mixed-corpus eval test(retrieval): mixed-corpus G2 eval for entity-persist demote (#1096) #1103 cleared no-regression) — the resolver default is already True, so this is a single-step graduation (expose via shim; no separate flip).
• It's the lane the epic body names explicitly: "features framed as fixing user-facing retrieval (e.g. Belief scoring is one-directional: exposure promotes, no organic sink (junk percolates up, not down) #1086's junk-percolation via Entity-persistence demotion prior: a deterministic organic sink for coordination junk (follow-up to #1086/#1081) #1096) never actually reach the hook." Graduating it makes that fix real for users.
• Lowest latency risk of the five: a log-additive rerank modifier (one batched belief_entities query, no candidate expansion) — unlike the pack-reordering clustering/HRR lanes.

(The other four stay off: origin-tiebreak is a contested BM25-recall problem rerank can't reach (#1013); HRR-expand is recall-neutral (#1001); clustering / HRR-structural reorder the L1 pack and each need their own decision.)

Commits (atomic)

1. feat(retrieval): shim passes use_entity_persist_demote=None (resolver-driven, default-ON) instead of hard-off. Tests: equivalence SHIM_LANES gains use_entity_persist_demote=None; test_shim_runs_temporal_spine_lane_others_off → test_shim_runs_graduated_lanes_others_off, now proving both graduated lanes (spine + entity-persist) live and the remaining four off, non-vacuously (durable-vs-coordination tie corpus where demotion reorders). The AC2 byte-identity regression test pins AELFRICE_ENTITY_PERSIST_DEMOTE=0 to isolate the posterior_weight=0 → BM25-only contract from the now-default-on demotion — mirroring its existing use_bm25f_anchors=False pin.
2. docs: CONFIG.md (lane live on production retrieve(), not just retrieve_v2) + CHANGELOG.

Verification

• Full suite: 5849 passed / 69 skipped / 75 xfailed. The one fallout (AC2 byte-identity) was the expected interaction and is isolated in-test, matching the precedent for the BM25F default-on flip.
• Ranking ablation (benchmarks/entity_persist_ablation.py): entity-persistence AUC 1.000 vs 0.500 posterior-only (durable/ephemeral separation) — signal intact.
• No-regression: the G2 mixed-corpus gate (test(retrieval): mixed-corpus G2 eval for entity-persist demote (#1096) #1103, tests/bench_gate/test_entity_persist_g2_mixed_corpus.py) already cleared; the lane adds one batched query, no candidate expansion.

Scope

Refs #1096, #1107 (Phase 3). Does not close the #1107 epic — origin-tiebreak / HRR-expand / clustering / HRR-structural remain, each a separate gated decision.

Opt-out

AELFRICE_ENTITY_PERSIST_DEMOTE=0 or [retrieval] use_entity_persist_demote = false.

[coderabbitai]

Warning

Review limit reached

@robotrocketscience, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 17 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6ceb89da-ba56-4b22-a1af-7632881eda02

📥 Commits

Reviewing files that changed from the base of the PR and between 3a6814a and dc95c59.

📒 Files selected for processing (5)

• CHANGELOG/v3.md
• docs/user/CONFIG.md
• src/aelfrice/retrieval.py
• tests/test_bayesian_ranking.py
• tests/test_retrieve_v2_equivalence.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/1107-phase3-entity-persist-lane

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

PR-size soft cap

This PR is over the advisory size threshold:

• 142 changed lines (limit: 200)
• 5 changed files (limit: 3)

Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated attn:merge-conflict cycles (see #602). When practical, split into smaller PRs that each touch a focused surface.

This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the size:override label and this comment will be removed on the next push.

[sourcery-ai]

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[robotrocketscience]

[claim:review:Setr:2026-07-07T16:30:50Z]

[robotrocketscience]

Review: approve.

Verified:

• Functional change is exactly the described single-step graduation: use_entity_persist_demote=False → None in the retrieve() shim, so the production path honours the resolver (default-ON per Entity-persistence demotion prior: a deterministic organic sink for coordination junk (follow-up to #1086/#1081) #1096 scope-A, G2 eval test(retrieval): mixed-corpus G2 eval for entity-persist demote (#1096) #1103 cleared). No candidate expansion — one batched belief_entities query, log-additive rerank modifier.
• Equivalence net preserved. _entity_persist_penalty returns 0.0 when a belief has no extracted entities, so on the entity-free equivalence corpora the default-ON lane is a genuine no-op — SHIM_LANES with use_entity_persist_demote=None reproduces legacy retrieve() byte-for-byte.
• Test strengthening is non-vacuous. test_shim_runs_graduated_lanes_others_off now proves both graduated lanes (spine + entity-persist) live and the remaining four off, on a durable-vs-coordination tie corpus where demotion is load-bearing (prod != demote_off).
• AC2 byte-identity correctly isolated via AELFRICE_ENTITY_PERSIST_DEMOTE=0, mirroring the existing BM25F pin — the expected and only suite fallout.
• FF-clean on main, both commits SSH-signed, discretion grep clean, full CI green (pytest 3.12 + 3.13), scope correctly does not close the epic(retrieval): converge production hook onto retrieve_v2 — staged lanes are not on the live path #1107 epic.

Adding ready-to-merge.

[github-actions]

merge-train: merged dc95c59 → main via FF push.

[robotrocketscience]

[release:review:Setr:2026-07-07T16:33:29Z]