test(retrieval): mixed-corpus G2 eval for entity-persist demote (#1096)

[robotrocketscience]

What

Adds the mixed-corpus retrieval eval that the entity-persistence demotion lane (#1096) needs to exercise G2 — the evidence gate that has been unmeetable on the academic benches.

Why the existing benches can't do it: LoCoMo extracts almost entirely noun_phrase entities, so every candidate takes the same grounding-neutral penalty and nothing reorders. The lane is provably recall-safe there but its demotion half is unmeasurable (all arms within ~0.13pp). Proving G2's positive half needs a store that mixes durable technical content with ephemeral coordination content — which no bench provided.

The fixture (tests/bench_gate/_entity_persist_mixed_store.py): per topic, a durable technical belief (grounding to file_path / identifier / error_code) paired with an ephemeral coordination belief (grounding to branch / version) that shares the query vocabulary. Entities auto-populate via the real extract_entities, so it's an end-to-end exercise of extraction + S1 + retrieval, not a hand-seeded entity table. Fully deterministic, self-contained, public-safe.

Fair by construction: under control (lane off), ephemeral beliefs interleave at ranks 2–3 — BM25 alone does not separate them (the realistic exposure-inflation shape). The entity-persistence lane is the only signal that sinks them. A test_corpus_grounds_as_labeled guard fails loudly if the extractor ever drifts and breaks that premise.

Result

retrieve_v2 control vs treatment on the mixed corpus:

budget	durable off→on	ephemeral off→on	first-durable MRR off→on
250	20→20	20→3	0.883→1.000
500	20→20	20→6	0.883→1.000
1000	20→20	20→20 (reordered below durable)	0.883→1.000

• G2 negative half (recall-safe): durable recall unchanged at every budget.
• G2 positive half (demotion): ephemeral coordination hits demoted out of a tight pack (250: 20→3, an 85% reduction); at wider budgets they reorder below all durable content.
• First-durable reciprocal rank rises to 1.000.
• Neutral fillers stay absent from the persistence map (entity_persistence_scores: noun_phrase/url-only beliefs get max demotion, not neutral (follow-up to #1096/#1097) #1098) — never demoted.

Scope

• Test-only. No src/ change. Not bench_gated — runs on public CI as a repeatable G2 gate and a regression lock on the lane's behaviour.
• References Entity-persistence demotion prior: a deterministic organic sink for coordination junk (follow-up to #1086/#1081) #1096; does not close it. Entity-persistence demotion prior: a deterministic organic sink for coordination junk (follow-up to #1086/#1081) #1096 stays open as the flip tracker. This PR delivers the G2 evidence the operator's "measure before flip" clause requires; the default-ON flip remains a separate operator call.

Test

tests/bench_gate/test_entity_persist_g2_mixed_corpus.py ... 3 passed

[sourcery-ai]

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Review limit reached

@robotrocketscience, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 55 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 85a09818-c133-4424-9753-b7569cb30af4

📥 Commits

Reviewing files that changed from the base of the PR and between 8c848ff and 2f299f8.

📒 Files selected for processing (2)

• tests/bench_gate/_entity_persist_mixed_store.py
• tests/bench_gate/test_entity_persist_g2_mixed_corpus.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch test/issue-1096-g2-mixed-corpus-eval

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

PR-size soft cap

This PR is over the advisory size threshold:

• 460 changed lines (limit: 200)
• 2 changed files (limit: 3)

Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated attn:merge-conflict cycles (see #602). When practical, split into smaller PRs that each touch a focused surface.

This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the size:override label and this comment will be removed on the next push.

[robotrocketscience]

[claim:review:Setr:2026-07-06T19:47:34Z]

[robotrocketscience]

Review — LGTM, mergeable (test-only)

Reviewed for the one risk that matters in a synthetic G2 fixture: is it fair, or built to guarantee the demotion wins? It holds up well.

Strengths

• Grounding premise is locked to the real extractor. test_corpus_grounds_as_labeled asserts durable S1 ≥ 0.5 / ephemeral S1 < 0.25 via the actual entity_persistence_scores, so a drift in extract_entities fails loudly rather than silently mis-measuring — the exact failure mode a hand-seeded fixture would hide.
• Signal isolated. Uniform priors (α=β=1) across all beliefs mean μ=0.5 everywhere, so the entity-persistence lane is the only thing that can reorder — no posterior confound.
• Both G2 halves asserted, recall-safe (dur_on >= dur_off) at every budget and MRR-non-regressing, plus the tight-budget demotion (eph_on <= eph_off // 2). G1 byte-identical-off and the entity_persistence_scores: noun_phrase/url-only beliefs get max demotion, not neutral (follow-up to #1096/#1097) #1098 neutral-absent guard are both covered.
• Deterministic, self-contained, public-safe, not bench_gated — doubles as a public regression lock. Rebased onto current main (FF, signed); pytest 3.12/3.13 green.

Two minor observations (non-blocking — for the flip-decision's confidence, not this merge):

1. The control-fairness guard is only tight_eph_off > 0; the "control interleaves ephemeral at ranks 2–3" claim is stronger than what's asserted. The printed eph_off (20/20 at budget 250) substantiates it, but a rank-based control assertion would nail it down as a regression lock.
2. Some durable strings' code symbols echo the query terms (apply_demotion_penalty → apply/demotion/penalty; retrieval.py → retrieval), giving durable a small BM25 edge independent of grounding. That runs conservative (BM25 already nudges durable up, so the lane isn't over-credited) but slightly softens "grounding is the only separator." Worth a note if this table feeds the G2 sign-off.

Neither changes the verdict. This lands the G2 evidence the "measure before flip" clause needed; it does not flip the lane (test-only, #1096 stays open as the operator's flip tracker). Merging.

[github-actions]

merge-train: blocked

branch is not fast-forward on main (branch base 748a225a213c6a1a873f3d9c82c24bf0a239354a, current main 8c848ff5710e157c05989061087097bd056d974c). Rebase locally (git rebase github/main), force-push, and re-add the label.

The ready-to-merge label has been removed. Address the issue above and re-add the label when you're ready for another attempt.

[github-actions]

merge-train: merged 2f299f8 → main via FF push.

[robotrocketscience]

[release:review:Setr:2026-07-06T19:58:52Z]