feat(retrieval): γ posterior-temperature rerank + bench panel widening (#796)

docs/feature-posterior-temperature.md

@@ -0,0 +1,132 @@
+# Feature spec: γ rerank — Boltzmann posterior temperature (#796)
+
+**Status:** implemented behind default-OFF flag at v3.x; bench panel widened (`ordered_top_k_overlap`, `rank_biased_overlap`); adoption verdict (flip default) **deferred** until a labeled relevance corpus exists.
+**Issue:** #796
+**Substrate prereqs:** #756 / #757 / #759 / #760 (meta-belief consumer pattern, shipped v3.x). Operator path-B decision 2026-05-14T16:48Z, refined to γ″ after the R&D campaign 2026-05-14T18:04Z.
+
+---
+
+## Purpose
+
+The v1.3 retrieval rerank is **log-additive**:
+
+```
+score = log(max(-bm25_raw, EPS)) + posterior_weight · log(posterior_mean)
+```
+
+`posterior_weight` is a process-wide scalar tuned at v1.3.0 (default `0.5`). Two independent operator concerns motivate the γ surface:
+
+1. The intended consumer for `meta:retrieval.posterior_temperature` (a softmax / Boltzmann temperature `T`) does not exist on `github/main` — pre-claim analysis on #758 confirmed `git grep -E 'softmax|boltzmann|temperature' github/main -- 'src/aelfrice/'` returns zero hits in src. Issue #796 is the load-bearing precursor: ship the temperature surface so `meta:retrieval.posterior_temperature` has something to bind to.
+2. PR@5 + Spearman ρ alone cannot discriminate top-K reorderings from middle-of-list churn (#796 R4). The bench panel needs `ordered_top_k_overlap` and `rank_biased_overlap` to make γ-vs-log-additive comparisons load-bearing.
+
+γ is **reparametrised log-additive**, not a new family — see *Contract* below. At `T = 1.0` it is byte-identical to `partial_bayesian_score(..., posterior_weight=1.0)`. Lower `T` sharpens the posterior contribution; higher `T` flattens toward BM25-only ranking.
+
+---
+
+## Contract
+
+```python
+from aelfrice.scoring import gamma_posterior_score
+
+score: float = gamma_posterior_score(
+    bm25_raw, alpha, beta, temperature,
+)
+```
+
+Formula:
+
+```
+score = log(max(-bm25_raw, EPS)) + (1 / T) · log(posterior_mean(α, β))
+```
+
+Equivalent to `partial_bayesian_score(bm25_raw, α, β, posterior_weight=1.0/T)` exactly — γ is a reparametrisation, not a new function family. Pure, deterministic, no store reads, no clock reads.
+
+`T <= GAMMA_TEMPERATURE_FLOOR` (1e-6) clamps upward so a misconfigured meta-belief or env override never raises at retrieval time. Negative temperatures are likewise clamped (the Boltzmann reading is undefined for `T <= 0`).
+
+---
+
+## Flag + meta-belief
+
+Resolved at `retrieve()` / `retrieve_with_tiers()` entry, once per call.
+
+| Layer | Surface | Resolver |
+|---|---|---|
+| Env | `AELFRICE_USE_GAMMA_POSTERIOR_TEMPERATURE` | `_env_use_gamma_posterior_temperature_override()` |
+| TOML | `[retrieval] use_gamma_posterior_temperature` | `_read_toml_flag_for(...)` |
+| Default | False | `resolve_use_gamma_posterior_temperature()` |
+
+When the flag resolves True, the temperature is resolved against the meta-belief substrate:
+
+```python
+T = resolve_posterior_temperature_with_meta(store, now_ts=...)
+```
+
+Bounds: `T ∈ [POSTERIOR_TEMPERATURE_FLOOR, POSTERIOR_TEMPERATURE_CEIL] = [0.5, 2.0]`. Log-linear decode from the meta-belief's `[0, 1]` posterior value; geometric mean is exactly 1.0, so the cold-start `static_default = 0.5` decodes to `T = 1.0` and a fresh install with the flag on is byte-identical to `partial_bayesian_score(..., 1.0)`. Adaptive learning of `T` (the evidence-signal loop that moves the meta-belief away from its prior) is out of scope for #796 — that is issue #758.
+
+When the flag is False, `gamma_temperature` is `None` and `_l1_hits` skips the γ branch entirely. The pre-#796 log-additive contract holds byte-for-byte.
+
+### Heat-rerank composition
+
+γ and the heat-kernel rerank (`use_heat_kernel`) are **mutually exclusive** on a given call. When both flags are on and a non-stale eigenbasis is available, the heat-rerank fires and γ is a no-op for that call. Composition was deferred to a later issue by the operator decision (R&D campaign verdict, 2026-05-14T18:04Z).
+
+---
+
+## Where γ sits
+
+`src/aelfrice/retrieval.py::_l1_hits` — the rerank loop, both branches:
+
+```python
+elif gamma_temperature is not None:
+    s = gamma_posterior_score(
+        bm25_raw, b.alpha, b.beta, gamma_temperature,
+    )
+else:
+    s = partial_bayesian_score(
+        bm25_raw, b.alpha, b.beta, posterior_weight,
+    )
+s = _hash_n_boosted(s, b.content, hash_n_literals)
+```
+
+`_hash_n_boost` runs after the rerank, unchanged from the log-additive path — the R2 / R2b finding that path-B-literal divergence was entirely the boost interaction is informational, not a code change.
+
+The byte-identical short-circuit (`posterior_weight == 0.0 and not heat_active and not hash_n_literals`) extends to require `gamma_temperature is None` so γ-on always exercises the rerank loop.
+
+---
+
+## Bench-gate / ship-or-defer policy
+
+| Gate | Status | Notes |
+|---|---|---|
+| **G1** — surface lands behind a default-OFF flag | shipped | this PR |
+| **G2** — bench panel widened with `ordered_top_k_overlap` + `rank_biased_overlap` | shipped | `src/aelfrice/calibration_metrics.py`, `src/aelfrice/eval_harness.py::compare_ranking_panel` |
+| **G3** — labeled relevance corpus exists | **pending** | corpus authoring tracked separately |
+| **G4** — γ@T=1.0 vs γ@T=0.5 / T=2.0 on labeled corpus shows discriminable rank-overlap deltas | **pending G3** | adoption verdict gate |
+| **G5** — flip default to True if G4 clears with effect size ≥ 1σ | **pending G3, G4** | follow-up PR |
+
+The bench-gate / ship-or-defer policy is the same shape as `feature-type-aware-compression.md` § *"Bench-gate / ship-or-defer policy"* and `feature-bfs-multihop.md` § *"Bench-gate posture"*. Until G3 lands, the flag is off and γ is plumbing — no behavioural change on any default code path.
+
+---
+
+## Out of scope (separate issues)
+
+- **Adaptive `T`** — the evidence-signal loop that moves the meta-belief away from its `static_default = 0.5` prior. That is #758. Until #758 ships, the meta-belief is never updated; flag-on cold installs decode to `T = 1.0` and stay there.
+- **ζ follow-up** — a bounded / sigmoid posterior-contribution parametrisation that gives `T` subtler dynamics than γ's global re-weighting. Filed as a separate R&D campaign after R&D refuted the α/β/γ trichotomy (see #800).
+- **Composition with heat-rerank** — both flags can be on but γ is a no-op on heat-active calls. Composing the two scoring paths is a separate scoping decision.
+- **Composition with `_hash_n_boost`** — boost interaction is informational, not a code change. R2 / R2b finding.
+
+---
+
+## Refs
+
+- #796 — this issue (operator path-B decision 2026-05-14T16:48Z; γ″ refinement 2026-05-14T18:04Z).
+- #758 — adaptive `T` follow-up; gated on #796 shipping.
+- #800 — ζ parametrisation R&D campaign.
+- #605 — PHILOSOPHY (deterministic, narrow surface). γ inherits.
+- #661 — federation read-only; meta-belief is local-only write state.
+- `src/aelfrice/scoring.py:gamma_posterior_score` — entry point.
+- `src/aelfrice/retrieval.py:resolve_use_gamma_posterior_temperature` — flag resolver.
+- `src/aelfrice/retrieval.py:resolve_posterior_temperature_with_meta` — decoder.
+- `src/aelfrice/retrieval.py:_l1_hits` — call site.
+- `src/aelfrice/calibration_metrics.py:ordered_top_k_overlap`, `rank_biased_overlap` — bench primitives.
+- `src/aelfrice/eval_harness.py:compare_ranking_panel` — bench-panel aggregator.
+- `tests/test_scoring_gamma.py`, `tests/test_retrieve_gamma_flag.py`, `tests/test_rank_overlap_metrics.py` — contract tests.

src/aelfrice/calibration_metrics.py

@@ -27,6 +27,8 @@
     "precision_at_k",
     "roc_auc",
     "spearman_rho",
+    "ordered_top_k_overlap",
+    "rank_biased_overlap",
 )
 
 
@@ -130,3 +132,97 @@ def spearman_rho(
     if denom == 0:
         return None
     return (n * sxy - sx * sy) / denom
+
+
+def ordered_top_k_overlap(
+    a: Sequence[object], b: Sequence[object], k: int,
+) -> float:
+    """Fraction of the top-k positions where ``a`` and ``b`` agree.
+
+    Compares the first ``k`` elements of two ranked lists position-by-
+    position; the score is the count of matching positions divided by
+    ``k``. Returns 1.0 when both lists' prefixes are identical and 0.0
+    when no top-k position matches.
+
+    Items beyond rank ``k`` are ignored. Missing slots (an input shorter
+    than ``k``) count as non-matches. Use this alongside
+    `rank_biased_overlap` to discriminate top-of-list churn from
+    middle-of-list reorderings — the #796 R4 finding was that PR@k and
+    Spearman ρ alone cannot tell those apart.
+    """
+    if k <= 0:
+        raise ValueError("k must be positive")
+    a_top = list(a[:k])
+    b_top = list(b[:k])
+    matches = sum(
+        1
+        for i in range(k)
+        if i < len(a_top) and i < len(b_top) and a_top[i] == b_top[i]
+    )
+    return matches / k
+
+
+def rank_biased_overlap(
+    a: Sequence[object], b: Sequence[object], p: float = 0.9,
+) -> float:
+    """Rank-biased overlap (RBO) — top-weighted ranking similarity.
+
+    Implements the extrapolated finite-list form (RBO_EXT) from Webber
+    et al. (2010), "A Similarity Measure for Indefinite Rankings".
+    Comparison runs up to depth ``D = min(len(a), len(b))`` and
+    extrapolates a constant agreement rate beyond ``D``:
+
+        RBO_EXT = (X_D / D) * p^D
+                + (1 - p) * sum_{d=1}^{D} p^(d-1) * X_d / d
+
+    where ``X_d = |A_d ∩ B_d|`` is the intersection size at depth d.
+    This is the conventional "RBO score" used in IR practice and
+    satisfies the unit-bound property: identical equal-length lists
+    score 1.0, fully disjoint lists score 0.0. The non-extrapolated
+    RBO_MIN underestimates identical lists by ``p^D`` and was
+    rejected here because the tests in #796 R4 rely on identical →
+    1.0 as a sanity gate.
+
+    ``p`` controls top-weight: ``p → 0`` weights rank 1 only;
+    ``p → 1`` weights the tail almost as much as the head. The #796
+    R4 finding used ``p = 0.9`` to give the top-K meaningful weight
+    without ignoring downstream rearrangements; that is the default.
+    Lists of unequal length are compared on their common prefix
+    length ``D``; tail items past ``D`` are ignored.
+
+    Properties covered by the test suite:
+      * Identical lists → 1.0.
+      * Disjoint lists → 0.0.
+      * Monotone in prefix agreement: extending a shared prefix never
+        lowers the score.
+      * Both empty → 1.0 (vacuous identity); one empty → 0.0.
+    """
+    if not 0.0 < p < 1.0:
+        raise ValueError("p must be in the open interval (0, 1)")
+    la, lb = len(a), len(b)
+    if la == 0 and lb == 0:
+        return 1.0
+    if la == 0 or lb == 0:
+        return 0.0
+    depth = min(la, lb)
+    seen_a: set[object] = set()
+    seen_b: set[object] = set()
+    overlap_count = 0
+    weighted_sum = 0.0
+    final_agreement = 0.0
+    for d in range(depth):
+        x = a[d]
+        if x in seen_b:
+            overlap_count += 1
+        seen_a.add(x)
+        y = b[d]
+        if y in seen_a:
+            overlap_count += 1
+        seen_b.add(y)
+        agreement = overlap_count / (d + 1)
+        weighted_sum += (p ** d) * agreement
+        final_agreement = agreement
+    # RBO_EXT extrapolation term: assume agreement stays at the depth-D
+    # rate for ranks beyond D. For identical equal-length lists this
+    # term equals p^D so the total converges to 1.0.
+    return final_agreement * (p ** depth) + (1.0 - p) * weighted_sum

src/aelfrice/eval_harness.py

@@ -30,7 +30,9 @@
 
 from aelfrice.calibration_metrics import (
     CalibrationReport,
+    ordered_top_k_overlap,
     precision_at_k,
+    rank_biased_overlap,
     roc_auc,
     spearman_rho,
 )
@@ -42,12 +44,23 @@
     "DEFAULT_CALIBRATION_CORPUS",
     "DEFAULT_K",
     "DEFAULT_SEED",
+    "DEFAULT_RBO_PERSISTENCE",
     "load_calibration_fixtures",
     "build_calibration_store",
     "run_calibration_on_fixtures",
     "format_calibration_report",
+    "RankComparisonReport",
+    "compare_ranking_panel",
+    "format_ranking_comparison",
 )
 
+# #796 R4 default — top-weight that gives the top-K meaningful weight
+# without ignoring tail rearrangements. Chosen so a single transposition
+# at rank 1 shifts the score noticeably while a transposition at rank
+# 10 does not. Held in this module so the γ-vs-log-additive A/B bench
+# and any later harness see the same default.
+DEFAULT_RBO_PERSISTENCE: float = 0.9
+
 DEFAULT_CALIBRATION_CORPUS = (
     Path(__file__).resolve().parent.parent.parent
     / "benchmarks"
@@ -232,3 +245,93 @@ def format_calibration_report(
     lines.append(f"ROC-AUC:      {_format_optional_float(report.roc_auc)}")
     lines.append(f"Spearman ρ:   {_format_optional_float(report.spearman_rho)}")
     return "\n".join(lines) + "\n"
+
+
+# ---------------------------------------------------------------------------
+# #796 R4 rank-comparison panel
+# ---------------------------------------------------------------------------
+# When comparing two retrieval configurations (γ rerank vs log-additive
+# baseline; bm25 vs bm25f; etc.) on the same query corpus, PR@K and
+# Spearman ρ measure each config independently against the relevance
+# label set. They cannot tell whether the configs produce the same
+# top-K order or merely the same top-K set. The R4 campaign added
+# ``ordered_top_k_overlap`` and ``rank_biased_overlap`` to discriminate
+# top-of-list churn from middle-of-list reorderings; this panel
+# averages those metrics across queries and exposes them in a stable
+# report shape. The γ-vs-log-additive bench harness is the load-
+# bearing consumer.
+
+
+from dataclasses import dataclass  # noqa: E402 — keep grouping with #796 block
+
+
+@dataclass(frozen=True)
+class RankComparisonReport:
+    """Per-query mean of rank-overlap metrics between two retrieval
+    configurations on a shared query set.
+
+    ``ordered_top_k`` is the mean of ``ordered_top_k_overlap(a_i, b_i, k)``
+    across queries. ``rbo`` is the mean of
+    ``rank_biased_overlap(a_i, b_i, p)``. Both are bounded in [0, 1];
+    1.0 means the two configs agree at the measured granularity, 0.0
+    means they disagree completely.
+
+    ``n_queries`` is the number of (a, b) pairs aggregated. Queries
+    where both rankings are empty contribute 1.0 to RBO (vacuous
+    identity) but 0.0 to ordered_top_k (no positions to match).
+    """
+
+    k: int
+    p: float
+    n_queries: int
+    ordered_top_k: float
+    rbo: float
+
+
+def compare_ranking_panel(
+    pairs: Sequence[tuple[Sequence[object], Sequence[object]]],
+    *,
+    k: int = DEFAULT_K,
+    p: float = DEFAULT_RBO_PERSISTENCE,
+) -> RankComparisonReport:
+    """Compute the #796 rank-comparison panel for paired rankings.
+
+    Each pair ``(a_i, b_i)`` is the ranked-id list from configuration A
+    and configuration B on the same query. The two metrics are computed
+    per pair and arithmetic-mean-aggregated across pairs. An empty
+    ``pairs`` list raises ``ValueError`` (no meaningful average).
+    """
+    if not pairs:
+        raise ValueError("pairs must be non-empty")
+    if k <= 0:
+        raise ValueError("k must be positive")
+    otk_vals: list[float] = []
+    rbo_vals: list[float] = []
+    for a, b in pairs:
+        otk_vals.append(ordered_top_k_overlap(a, b, k))
+        rbo_vals.append(rank_biased_overlap(a, b, p=p))
+    return RankComparisonReport(
+        k=k,
+        p=p,
+        n_queries=len(pairs),
+        ordered_top_k=sum(otk_vals) / len(otk_vals),
+        rbo=sum(rbo_vals) / len(rbo_vals),
+    )
+
+
+def format_ranking_comparison(report: RankComparisonReport) -> str:
+    """Format a ``RankComparisonReport`` as a deterministic text block.
+
+    Mirrors ``format_calibration_report``'s shape so the two panels
+    sit cleanly side-by-side in a combined bench run.
+    """
+    lines = [
+        "rank-comparison panel — γ vs log-additive (#796 R4)",
+        f"  n_queries:    {report.n_queries}",
+        f"  k:            {report.k}",
+        f"  p:            {report.p:.4f}",
+        "",
+        f"ordered_top_k@{report.k}:  {report.ordered_top_k:.4f}",
+        f"RBO(p={report.p:.2f}):      {report.rbo:.4f}",
+    ]
+    return "\n".join(lines) + "\n"