feat(retrieval): ζ bounded posterior-contribution rerank surface (#817, closes #800)

[robotrocketscience]

Summary

Ships ζ rerank — bounded posterior-contribution surface — behind a default-OFF flag. Closes #800. Implementation issue #817.

Decision honoured

R&D campaign verdict (#800 R0–R4, committed to lab test/corpus-v2-a2-a4-seed): ADOPT ζ at (α=1, scale=14.5, β=0.25). H2 head-to-head panel: ζ at those parameters dominates γ on rank_biased_overlap for similar rank_changed_fraction.

This PR mirrors PR #807's γ″ scope (default-OFF flag + scoring entry + resolver + tests + docs) so ζ ships dark next to γ. Adoption-verdict (flip default) is deferred until the labeled relevance corpus (#819) lands.

Scope (verdict-aligned)

1. scoring.py — ζ entry point. zeta_posterior_score(bm25_raw, alpha, beta, *, scale, beta_param, alpha_param) returns the bounded-posterior log-additive score with the (α=1, scale=14.5, β=0.25) default. Pure function; no state.
2. retrieval.py — flag + wiring. resolve_use_zeta_posterior_rerank() (env > kwarg > TOML > False). When on, _l1_hits swaps the rerank stage. Mutual exclusion with γ + heat preserved (heat takes precedence; γ and ζ are mutually exclusive — both flags on prefers γ at runtime).
3. Tests — 19 cases across two new modules:
   • tests/test_retrieve_zeta_flag.py — flag-off baseline byte-identical, flag-on deterministic, resolver precedence.
   • tests/test_zeta_vs_gamma_panel.py — head-to-head bench panel exercising both surfaces on the same corpus shape (mirrors the compare_ranking_panel shape from feat(retrieval): γ posterior-temperature rerank + bench panel widening (#796) #807).
4. Docs — docs/feature-zeta-posterior-rerank.md matches the feature-posterior-temperature.md shape (contract, where it sits, bench-gate / ship-or-defer policy, out-of-scope follow-ups). CHANGELOG entry under Unreleased.

Out of scope (gated follow-ups)

• Flip default — gated on labeled relevance corpus ([corpus] labeled rerank-relevance corpus — unblock for #769, #724, #800 R5, #817 flip-default #819), same as γ.
• Adaptive ζ-scale / β as meta-beliefs — deferred; ζ ships with hardcoded constants for the bench gate.
• γ ⊕ ζ composition — both flags can be on but γ wins at runtime. Composing the two scoring paths is a separate scoping decision.

Verification

• uv run pytest -x -q --timeout=120 tests/test_retrieve_zeta_flag.py tests/test_zeta_vs_gamma_panel.py → 19 passed.
• Full suite: uv run pytest -x -q --timeout=120 → 4222 passed, 62 skipped, 75 xfailed in 84s.
• Discretion grep on git diff github/main...HEAD: clean.
• Rebased on top of github/main (no merge commits, FF-able).

Files changed

• src/aelfrice/scoring.py — zeta_posterior_score.
• src/aelfrice/retrieval.py — resolve_use_zeta_posterior_rerank(), env override, TOML key, _l1_hits ζ branch.
• tests/test_retrieve_zeta_flag.py, tests/test_zeta_vs_gamma_panel.py — 19 new test cases.
• docs/feature-zeta-posterior-rerank.md — feature spec.
• CHANGELOG.md — Unreleased entry.

Notes for review

• This branch was staged at 53098c31 by an earlier session, blocked on PR feat(retrieval): γ posterior-temperature rerank + bench panel widening (#796) #807 (γ surface) shipping. After PR feat(retrieval): γ posterior-temperature rerank + bench panel widening (#796) #807 merged, rebase onto current main was clean (git rebase skipped 7 already-applied commits — γ + A4 bench gate is structurally vacuous — rebuilder uses verbatim token cost, ignoring compression flag #798 follow-ups).
• One fix-up commit added during rebase: test(zeta): drop demotion_pressure kwarg from _mk_belief (#817) — Belief.__init__ no longer accepts demotion_pressure after refactor(feedback): remove unreachable CONTRADICTS-edge auto-demote path (per #605) #814 / refactor(feedback): remove unreachable contradict-edge auto-demote (#814) #820. Same fix shape PR feat(retrieval): γ posterior-temperature rerank + bench panel widening (#796) #807 needed.

Closes #800.

[sourcery-ai]

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Rate limit exceeded

@robotrocketscience has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 21 minutes and 39 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 59866c6e-8c6e-49d2-8eaa-5bd58648f8e7

📥 Commits

Reviewing files that changed from the base of the PR and between 32aa894 and ca0b09c.

📒 Files selected for processing (7)

• CHANGELOG/v3.md
• docs/feature-zeta-posterior-rerank.md
• src/aelfrice/retrieval.py
• src/aelfrice/scoring.py
• tests/test_retrieve_zeta_flag.py
• tests/test_scoring_zeta.py
• tests/test_zeta_vs_gamma_panel.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/issue-817-zeta-posterior-rerank

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

PR-size soft cap

This PR is over the advisory size threshold:

• 978 changed lines (limit: 200)
• 7 changed files (limit: 3)

Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated attn:merge-conflict cycles (see #602). When practical, split into smaller PRs that each touch a focused surface.

This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the size:override label and this comment will be removed on the next push.

[robotrocketscience]

[claim:review:bagheera:2026-05-14T23:05:29Z]

[robotrocketscience]

[claim:review:clarke:2026-05-14T23:09:36Z]

[robotrocketscience]

[release:review:clarke:2026-05-14T23:09:40Z]

[robotrocketscience]

Review: rebased on current main, ready to merge

Rebased ζ surface onto github/main (32aa894e, post-#811 CHANGELOG split). All 6 commits signed (G); branch FF on current main. Single conflict in CHANGELOG.md resolved cleanly:

• Top-level CHANGELOG.md kept the post-docs(changelog): split CHANGELOG.md by major version under CHANGELOG/ #811 thin-index shape verbatim.
• The PR's ### Added ζ entry relocated into CHANGELOG/v3.md under ## [Unreleased], before the existing ### Fixed block (rebuilder pack accounting A4 bench gate is structurally vacuous — rebuilder uses verbatim token cost, ignoring compression flag #798, subfloor noise feat(ingest): SVO-extractor min-length floor + edge-anchor demotion (#785 §3) #809). No content drift — the entry text is byte-identical to the original commit; only the host file changed.

Code-side verification

• Scoring math (scoring.py:zeta_posterior_score). Posterior-neutral at p=0.5 (log(p)−log(0.5)=0, σ(0)=0.5, bracket (0.5−0.5)·scale = 0) — uniform-posterior store collapses to log-BM25 alone. Bounded contribution ∈ (-α·scale/2, +α·scale/2) = (-7.25, +7.25) at defaults (α=1, β=0.25, scale=14.5). ZETA_POSTERIOR_FLOOR clamp prevents math domain error on degenerate posterior_mean=0 rows.
• Wiring (retrieval.py). resolve_use_zeta_posterior_rerank mirrors resolve_use_gamma_posterior_temperature precedence (env > kwarg > TOML > False). _assert_gamma_zeta_mutual_exclusion raises ValueError if both flags resolve True (operator decision per feat(retrieval): ζ bounded posterior-contribution rerank surface (closes #800) #817 §"Out of scope"). Both byte-identical short-circuits in _l1_hits (BM25F + FTS5) extended to require zeta_params is None. Heat-rerank precedence over both γ and ζ preserved.

Pre-push gates

• pytest -x -q --timeout=120 → 4222 passed, 62 skipped, 75 xfailed in 88s.
• ζ-targeted: tests/test_scoring_zeta.py + tests/test_retrieve_zeta_flag.py + tests/test_zeta_vs_gamma_panel.py → 42 passed in 5s.
• Discretion grep on git diff main...HEAD → 0 hits.
• Pre-push hook → clean push (FF on main; no relocated v3.x content in this branch's diff).

Notes

• Original PR body line "19 cases across two new modules" undercounts — actually 42 tests across 3 modules (test_scoring_zeta.py, test_retrieve_zeta_flag.py, test_zeta_vs_gamma_panel.py). Doesn't matter for merge; just calling it out.
• No CodeRabbit findings yet (rate-limited at PR-open time). Pushing the rebase will trigger a fresh review run; if findings warrant, can dispatch a follow-up.

Adding ready-to-merge. Releasing claim.

[robotrocketscience]

[release:review:bagheera:2026-05-14T23:13:51Z]

[github-actions]

merge-train: merged ca0b09c → main via FF push.