diff --git a/CHANGELOG/v3.md b/CHANGELOG/v3.md index 402f12c7..d9530dd3 100644 --- a/CHANGELOG/v3.md +++ b/CHANGELOG/v3.md @@ -8,6 +8,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- **ζ posterior-rerank surface — bounded sigmoid contribution** ([#817](https://github.com/robotrocketscience/aelfrice/issues/817), closes [#800](https://github.com/robotrocketscience/aelfrice/issues/800)). Ships behind a default-OFF flag mirroring γ ([#796](https://github.com/robotrocketscience/aelfrice/issues/796) / [PR #807](https://github.com/robotrocketscience/aelfrice/pull/807)). Replaces γ's unbounded `(1/T)·log(p)` with `α·(σ(β·(log(p)−log(0.5)))−0.5)·scale` — bounded posterior contribution in `(-α·scale/2, +α·scale/2)`, collapses to zero at the posterior-neutral `p=0.5`. Pinned defaults from the #800 R&D campaign verdict (R0–R4 at `experiments/zeta-posterior/`, R2 head-to-head: ζ dominates γ on `rank_biased_overlap` at similar `rank_changed_fraction`): `ZETA_ALPHA_DEFAULT=1.0`, `ZETA_BETA_DEFAULT=0.25`, `ZETA_SCALE_DEFAULT=14.5`. New API: `scoring.zeta_posterior_score(bm25_raw, alpha, beta, scale, posterior_mean)` with `ZETA_POSTERIOR_FLOOR` clamp on degenerate posteriors (corrupted store row never raises math domain error at retrieval time). Retrieval-side wiring: `USE_ZETA_POSTERIOR_RERANK_FLAG`, `ENV_USE_ZETA_POSTERIOR_RERANK` (`AELFRICE_USE_ZETA_POSTERIOR_RERANK`), `_env_use_zeta_posterior_rerank_override()`, `resolve_use_zeta_posterior_rerank(explicit=None, *, start=None)` with five-path precedence (env > kwarg > TOML `[retrieval] use_zeta_posterior_rerank` > False). `_l1_hits` gains `zeta_params: tuple[float, float, float] | None`; both byte-identical short-circuits extend to require `zeta_params is None` so ζ-on always exercises the rerank loop. `retrieve()` and `retrieve_with_tiers()` resolve both γ and ζ at entry and call `_assert_gamma_zeta_mutual_exclusion(gamma_on, zeta_on)` — both flags ON raises `ValueError` per the operator decision to defer composition (#817 § "Out of scope"). Heat-rerank still dominates both γ and ζ. ζ is **not** byte-identical to γ@T=1.0 nor `partial_bayesian_score(..., 1.0)` at any non-trivial inputs (issue #817 § "Note re: cold-start byte-identity"); it is rank-equivalent to log-BM25 alone on uniform-posterior=0.5 stores. 42 new tests across `tests/test_scoring_zeta.py` (posterior-neutral point, σ-bound, monotonicity, floor clamp, determinism, non-byte-identity to γ, uniform-posterior collapse to log-BM25), `tests/test_retrieve_zeta_flag.py` (five-path resolver precedence, flag-off byte-identical, flag-on deterministic, reorders by posterior, γ + ζ mutex helper, both-flags-ON raises in retrieve and retrieve_with_tiers), and `tests/test_zeta_vs_gamma_panel.py` (panel-metric reuse with `rank_biased_overlap` / `ordered_top_k_overlap`, γ-on/ζ-on rank-identical on uniform-posterior fixture). Flip-default deferred until the labeled relevance corpus exists (same gate as γ's G3). Env / TOML knobs for `(α, β, scale)` deferred per #817 § "Out of scope". + ### Fixed - **Rebuilder pack accounting now honours `use_type_aware_compression`** ([#798](https://github.com/robotrocketscience/aelfrice/issues/798)). `rebuild_v14` was re-packing `retrieve()`'s candidate set with verbatim token cost regardless of the flag, so any ON-arm extras `retrieve()` admitted at compressed cost got trimmed back to the OFF-arm count. The downstream A4 continuation-fidelity bench gate ([#775](https://github.com/robotrocketscience/aelfrice/issues/775) / [PR #776](https://github.com/robotrocketscience/aelfrice/pull/776)) was therefore structurally vacuous — per-row fidelity delta = 0 by construction, regardless of corpus. Fix resolves the flag once at `rebuild_v14` entry (`resolve_use_type_aware_compression(use_type_aware_compression)`), threads it into the `retrieve()` call and into `_estimate_belief_tokens(b, *, compress_on=...)` at all three pack sites (L0 init, session tier, L1 / L2.5 tier). The rebuild block content itself stays verbatim — the change is in *how many* beliefs survive the budget, not what each surviving belief renders as. Default-OFF and the legacy `_retrieve_for_rebuild` (v1.2.0a0 alpha contract) are byte-identical. Unblocks the A4 axis of the [#769](https://github.com/robotrocketscience/aelfrice/issues/769) flip-default decision. Operator-decision history: Option A per [#798 thread](https://github.com/robotrocketscience/aelfrice/issues/798); Options B (rebuilder emits `compressed_beliefs[i].rendered`) and C (drop A4 from #769 acceptance) declined. Two new tests in `tests/test_context_rebuilder.py` (`test_rebuild_v14_pack_size_matches_compression_flag`, `test_rebuild_v14_compression_off_byte_identical_default`). diff --git a/docs/feature-zeta-posterior-rerank.md b/docs/feature-zeta-posterior-rerank.md new file mode 100644 index 00000000..a5d5918d --- /dev/null +++ b/docs/feature-zeta-posterior-rerank.md @@ -0,0 +1,179 @@ +# Feature spec: ζ rerank — bounded sigmoid posterior contribution (#817 / #800) + +**Status:** implemented behind default-OFF flag at v3.x; flip-default deferred until a labeled relevance corpus exists (same gate as γ — see `feature-posterior-temperature.md`). +**Issues:** [#817](https://github.com/robotrocketscience/aelfrice/issues/817) (implementation), [#800](https://github.com/robotrocketscience/aelfrice/issues/800) (R&D campaign, closed by this PR). +**R&D verdict:** ADOPT at (α=1.0, β=0.25, scale=14.5), per R0–R4 campaign at `experiments/zeta-posterior/` (lab-side). R2 head-to-head: ζ dominates γ on `rank_biased_overlap` for similar `rank_changed_fraction`. + +--- + +## Purpose + +γ (`gamma_posterior_score`, #796) shipped a Boltzmann reparametrisation of v1.3's log-additive rerank: + +``` +γ: score = log(max(-bm25_raw, EPS)) + (1 / T) · log(posterior_mean) +``` + +At moderate `T`, γ produces **global re-weighting** — measured `rank_changed_fraction` ≈ 0.92–0.96, ~96% of beliefs change rank. The #800 operator decision was that this is "more aggressive than the intended semantics" of `meta:retrieval.posterior_temperature` (#758). + +ζ replaces γ's unbounded `(1/T)·log(p)` with a **bounded sigmoid contribution**: + +``` +ζ: score = log(max(-bm25_raw, EPS)) + + α · (σ(β · (log(p) − log(0.5))) − 0.5) · scale +``` + +Where `σ(x) = 1 / (1 + exp(-x))` is the logistic. The posterior contribution lies strictly in `(-α·scale/2, +α·scale/2)`, centred so `posterior_mean = 0.5` (no evidence) contributes nothing. No belief gets unboundedly leveraged by extreme posteriors — the structural difference from γ. + +ζ is **not** a continuous extension of γ. At the pinned defaults `(α=1, β=0.25, scale=14.5)`, ζ is rank-equivalent to log-BM25 alone on a uniform-posterior=0.5 store (both contributions = 0) but otherwise produces a different score sequence. The note at issue #817 § "Note re: cold-start byte-identity" makes this explicit; the unit tests in `tests/test_scoring_zeta.py::test_zeta_not_byte_identical_to_gamma_at_T_one` enforce it. + +--- + +## Contract + +```python +from aelfrice.scoring import ( + ZETA_ALPHA_DEFAULT, + ZETA_BETA_DEFAULT, + ZETA_SCALE_DEFAULT, + zeta_posterior_score, +) + +score: float = zeta_posterior_score( + bm25_raw, + alpha, # ζ magnitude — defaults to 1.0 + beta, # ζ sharpness — defaults to 0.25 + scale, # ζ global multiplier — defaults to 14.5 + posterior_mean, # the Beta-Bernoulli posterior mean (already computed) +) +``` + +Pure-function, deterministic, no store reads, no clock reads. `posterior_mean ≤ ZETA_POSTERIOR_FLOOR` clamps upward — a corrupted store row (degenerate posterior) never raises `math domain error` at retrieval time. Negative posteriors clamp the same way (the sigmoid is undefined as `p → 0`). + +The `(alpha, beta, scale)` arguments are the **ζ math parameters**, not the Beta-Bernoulli `(α, β)` of the originating belief. The caller computes `posterior_mean` externally (typically `posterior_mean(b.alpha, b.beta)`) and passes the scalar. This is the structural difference from `gamma_posterior_score`, which takes the Beta-Bernoulli `(α, β)` directly and computes the posterior internally. + +### Bounded property + +For any (`α`, `β`, `scale`) and any `posterior_mean ∈ (0, 1)`: + +``` +-α·scale/2 < (score - log(max(-bm25, EPS))) < +α·scale/2 +``` + +At the pinned defaults, the contribution is bounded by ±7.25 — comparable in magnitude to a strong BM25 spread but unable to dwarf it on any single belief. The bound is exact in the limit; the open interval holds at every finite posterior. Tests: `test_sigma_bound`, `test_sigma_bound_at_extreme_posteriors`. + +### Posterior-neutral point + +At `posterior_mean = 0.5`: + +``` +log(p) − log(0.5) = 0 +σ(β · 0) = 0.5 +(σ − 0.5) · α · scale = 0 +score = log(max(-bm25, EPS)) # identical to BM25-only ordering +``` + +A store of all-uniform posteriors is rank-equivalent to log-BM25 alone — the same edge-case property `partial_bayesian_score` has at `posterior_weight = 0.0`. Tests: `test_posterior_half_is_log_bm_only`. + +--- + +## Flag + +Resolved at `retrieve()` / `retrieve_with_tiers()` entry, once per call. Five-path precedence (first decisive wins): + +| Layer | Surface | Resolver | +|---|---|---| +| Env | `AELFRICE_USE_ZETA_POSTERIOR_RERANK` | `_env_use_zeta_posterior_rerank_override()` | +| Kwarg | `explicit=True/False` to `resolve_use_zeta_posterior_rerank` | — | +| TOML | `[retrieval] use_zeta_posterior_rerank` | `_read_toml_flag_for(...)` | +| Default | False | `resolve_use_zeta_posterior_rerank()` | + +When the flag resolves True, `retrieve()` / `retrieve_with_tiers()` package the pinned defaults as a 3-tuple: + +```python +zeta_params = (ZETA_ALPHA_DEFAULT, ZETA_BETA_DEFAULT, ZETA_SCALE_DEFAULT) +``` + +and pass it into `_l1_hits(..., zeta_params=zeta_params)`. The rerank loop swaps `partial_bayesian_score(bm25_raw, b.alpha, b.beta, posterior_weight)` for `zeta_posterior_score(bm25_raw, ζα, ζβ, ζscale, posterior_mean(b.alpha, b.beta))` on the non-heat path. + +The byte-identical short-circuits (`posterior_weight == 0.0 and not heat_active and not hash_n_literals`) extend to require `zeta_params is None`, so ζ-on always exercises the rerank loop. + +### Tunability + +Tunability via the `(alpha, beta, scale)` kwargs is preserved for testability — the unit tests sweep parameters to verify the σ-bound and monotonicity properties. **Env / TOML knobs for α, β, scale are deferred** per #817 § "Out of scope". Operator intent is to ship the fixed-default and revisit tunability if the labeled-corpus bench surfaces a need for per-deployment tuning. + +--- + +## Composition with γ + +γ and ζ are **mutually exclusive on a given retrieval call**. When both flags resolve True, `_assert_gamma_zeta_mutual_exclusion(gamma_on, zeta_on)` raises `ValueError` at flag resolution — both `retrieve()` and `retrieve_with_tiers()` perform the check immediately after resolving each flag independently: + +```python +gamma_on = resolve_use_gamma_posterior_temperature() +gamma_t = (... if gamma_on else None) +zeta_on = resolve_use_zeta_posterior_rerank() +_assert_gamma_zeta_mutual_exclusion(gamma_on, zeta_on) +zeta_params = ((ζα, ζβ, ζscale) if zeta_on else None) +``` + +The operator decision per #817 § "Out of scope" was: composition (γ then ζ, or weighted sum, or hierarchical) requires a separate scoping issue. Raise loudly on collision so a misconfigured deployment is caught at retrieval-time, not at silent rank-divergence time. + +Tests: `test_retrieve_raises_when_both_flags_on`, `test_retrieve_with_tiers_raises_when_both_flags_on`, `test_mutex_raises_when_both_on`. + +## Composition with heat-rerank + +ζ and the heat-kernel rerank (`use_heat_kernel`) are mutually exclusive on a given call, same as γ. When both ζ and heat are on AND a non-stale eigenbasis is available, the heat-rerank fires and ζ is a no-op for that call. The byte-identical short-circuit detects this and routes through the existing `combine_log_scores` path. Composition with heat is the same deferred-scoping question as γ-with-heat. + +--- + +## Where ζ sits + +`src/aelfrice/scoring.py::zeta_posterior_score` — the math primitive. +`src/aelfrice/retrieval.py`: + +- `USE_ZETA_POSTERIOR_RERANK_FLAG`, `ENV_USE_ZETA_POSTERIOR_RERANK` — surface names. +- `_env_use_zeta_posterior_rerank_override()` — env decoder. +- `resolve_use_zeta_posterior_rerank(explicit=None, *, start=None)` — five-path resolver. +- `_assert_gamma_zeta_mutual_exclusion(gamma_on, zeta_on)` — mutex helper. +- `_l1_hits` — rerank loop; `zeta_params: tuple[float, float, float] | None` kwarg. +- `retrieve()` / `retrieve_with_tiers()` — call sites. + +--- + +## Bench-gate / ship-or-defer policy + +| Gate | Status | Notes | +|---|---|---| +| **G1** — surface lands behind a default-OFF flag | shipped | this PR | +| **G2** — R&D campaign R0–R4 verdict ADOPT | shipped | `experiments/zeta-posterior/` (lab-side) | +| **G3** — labeled relevance corpus exists | **pending** | corpus authoring tracked separately; same gate as γ's G3 | +| **G4** — ζ@defaults vs γ@T=1.0 on labeled corpus shows discriminable rank-overlap deltas | **pending G3** | adoption verdict gate | +| **G5** — flip default to True if G4 clears with effect size ≥ 1σ | **pending G3, G4** | follow-up PR | + +Same shape as `feature-posterior-temperature.md` § "Bench-gate / ship-or-defer policy". Until G3 lands, ζ is plumbing — no behavioural change on any default code path. The R&D campaign already cleared the "ζ is plausibly better than γ on synthetic corpora" gate; G3+G4 are the production-fidelity gate. + +--- + +## Out of scope (separate issues) + +- **Adaptive `β` via meta-belief.** Per #800: "Deferred until ζ is verified." If ζ clears G4 the eventual #758 retarget shifts from adaptive γ-T to adaptive ζ-β. Until then `β` is pinned at 0.25. +- **Env / TOML tunability of α, β, scale.** The fixed-default ships first. Operator surfaces for the shape parameters require campaign evidence that single-deployment tuning matters. +- **Composition with γ.** Both flags ON raises. Designing a composed rerank (linear combination, gating, or hierarchical) is a separate scoping decision per the operator note. +- **Composition with heat-rerank.** Heat dominates ζ on heat-active calls; composing the two is the same deferred-scoping question as γ-with-heat. +- **Adaptive learning of α / scale.** The R&D verdict at fixed (α=1, scale=14.5) is what shipped. Whether α and scale should be meta-belief-driven is a follow-up campaign. + +--- + +## Refs + +- **#800** — R&D campaign (closes via this PR). +- **#796 / PR #807** — γ predecessor. +- **#758** — adaptive `posterior_temperature` consumer. Retargets to adaptive ζ-β if ζ clears G4. +- **#605** — PHILOSOPHY (deterministic, narrow surface, stdlib only). ζ inherits. +- **#661** — federation read-only; score computation is read-time. +- `src/aelfrice/scoring.py:zeta_posterior_score` — entry point. +- `src/aelfrice/retrieval.py:resolve_use_zeta_posterior_rerank` — flag resolver. +- `src/aelfrice/retrieval.py:_assert_gamma_zeta_mutual_exclusion` — γ/ζ mutex. +- `src/aelfrice/retrieval.py:_l1_hits` — call site. +- `tests/test_scoring_zeta.py`, `tests/test_retrieve_zeta_flag.py`, `tests/test_zeta_vs_gamma_panel.py` — contract tests. +- `experiments/zeta-posterior/RUNNING_DOC.md` (lab-side) — R0–R4 + verdict. diff --git a/src/aelfrice/retrieval.py b/src/aelfrice/retrieval.py index c088df7b..71ac6ec1 100644 --- a/src/aelfrice/retrieval.py +++ b/src/aelfrice/retrieval.py @@ -98,9 +98,13 @@ from aelfrice.models import LOCK_NONE, LOCK_USER, Belief from aelfrice.scoring import ( DEFAULT_POSTERIOR_WEIGHT, + ZETA_ALPHA_DEFAULT, + ZETA_BETA_DEFAULT, + ZETA_SCALE_DEFAULT, gamma_posterior_score, partial_bayesian_score, posterior_mean, + zeta_posterior_score, ) from aelfrice.store import MemoryStore @@ -177,6 +181,16 @@ USE_GAMMA_POSTERIOR_TEMPERATURE_FLAG: Final[str] = ( "use_gamma_posterior_temperature" ) +# v3.x #817 / #800 ζ rerank flag. Default-OFF, mirrors γ's posture: +# the flag is plumbing until a labeled relevance corpus exists and +# the bench panel demonstrates uplift over either log-additive or γ. +# When ON, `_l1_hits` routes its rerank through `zeta_posterior_score(...)` +# with pinned `(ZETA_ALPHA_DEFAULT, ZETA_BETA_DEFAULT, ZETA_SCALE_DEFAULT)` +# from the #800 R&D campaign verdict. ζ + γ are mutually exclusive on +# any given retrieval call — `retrieve()` / `retrieve_with_tiers()` +# raise `ValueError` at flag resolution when both are on (the operator +# decision per issue #817 §"Out of scope" deferred composition). +USE_ZETA_POSTERIOR_RERANK_FLAG: Final[str] = "use_zeta_posterior_rerank" PLACEHOLDER_FLAGS: Final[tuple[str, ...]] = ( SIGNED_LAPLACIAN_FLAG, @@ -214,6 +228,10 @@ ENV_USE_GAMMA_POSTERIOR_TEMPERATURE: Final[str] = ( "AELFRICE_USE_GAMMA_POSTERIOR_TEMPERATURE" ) +# v3.x #817 ζ rerank env override. Tri-state, default-OFF. +ENV_USE_ZETA_POSTERIOR_RERANK: Final[str] = ( + "AELFRICE_USE_ZETA_POSTERIOR_RERANK" +) # v1.3.0 posterior-weight env override. Float-typed; "0.0" is the # only value that fully disables (collapsing to BM25-only ordering). # Empty / non-numeric values fall through to the next precedence @@ -608,6 +626,21 @@ def _env_use_gamma_posterior_temperature_override() -> bool | None: return None +def _env_use_zeta_posterior_rerank_override() -> bool | None: + """Return True/False if AELFRICE_USE_ZETA_POSTERIOR_RERANK is set to a + recognised truthy/falsy value, else None. Symmetric to + `_env_use_gamma_posterior_temperature_override`.""" + raw = os.environ.get(ENV_USE_ZETA_POSTERIOR_RERANK) + if raw is None: + return None + norm = raw.strip().lower() + if norm in _ENV_FALSY: + return False + if norm in _ENV_TRUTHY: + return True + return None + + def _env_hrr_persist_override() -> bool | None: """Return True/False if AELFRICE_HRR_PERSIST is set to a recognised truthy/falsy value, else None. Symmetric to `_env_bm25f_override`. @@ -1764,6 +1797,64 @@ def resolve_posterior_temperature_with_meta( return math.exp(log_floor + v * (log_ceil - log_floor)) +def resolve_use_zeta_posterior_rerank( + explicit: bool | None = None, + *, + start: Path | None = None, +) -> bool: + """Resolve the ζ rerank flag (#817 / #800). + + Precedence (first decisive wins): + 1. AELFRICE_USE_ZETA_POSTERIOR_RERANK env var. + 2. Explicit `explicit` kwarg from the caller. + 3. `[retrieval] use_zeta_posterior_rerank` in `.aelfrice.toml`. + 4. Default: False — ships behind the flag at v3.x. Flip-default + is gated on the same labeled relevance corpus as γ's flip, + per `docs/feature-zeta-posterior-rerank.md` § + "Bench-gate / ship-or-defer policy". + + Mirror of `resolve_use_gamma_posterior_temperature`. The mutual- + exclusion with γ is enforced at `retrieve()` / + `retrieve_with_tiers()` entry via `_assert_gamma_zeta_mutual_exclusion` + after both flags have been resolved — this resolver returns a bool + independent of γ's state. + """ + env = _env_use_zeta_posterior_rerank_override() + if env is not None: + return env + if explicit is not None: + return explicit + toml_value = _read_toml_flag_for( + USE_ZETA_POSTERIOR_RERANK_FLAG, start, + ) + if toml_value is not None: + return toml_value + return False + + +def _assert_gamma_zeta_mutual_exclusion( + gamma_on: bool, zeta_on: bool, +) -> None: + """Raise `ValueError` if both the γ and ζ rerank flags resolve True. + + Composition semantics are deferred per issue #817 §"Out of scope" — + the operator decision is "raise at flag resolution, operator picks + one". This helper is called from `retrieve()` and + `retrieve_with_tiers()` after each flag is resolved independently. + """ + if gamma_on and zeta_on: + raise ValueError( + "γ and ζ posterior rerank flags are mutually exclusive on a " + "given retrieval call. " + f"AELFRICE_USE_GAMMA_POSTERIOR_TEMPERATURE / " + f"{USE_GAMMA_POSTERIOR_TEMPERATURE_FLAG} resolved True AND " + f"AELFRICE_USE_ZETA_POSTERIOR_RERANK / " + f"{USE_ZETA_POSTERIOR_RERANK_FLAG} resolved True. " + "Pick one. See docs/feature-zeta-posterior-rerank.md § " + "'Composition with γ' (issue #817)." + ) + + _PLACEHOLDER_WARNED: set[str] = set() @@ -2015,6 +2106,7 @@ def _l1_hits( eigenbasis_cache: GraphEigenbasisCache | None = None, heat_kernel_on: bool = False, gamma_temperature: float | None = None, + zeta_params: tuple[float, float, float] | None = None, ) -> list[Belief]: """Run L1: FTS5 BM25 search (default) or BM25F sparse-matvec (v1.5.0 opt-in), optionally reranked by partial-Bayesian score. @@ -2044,6 +2136,16 @@ def _l1_hits( exclusive by design (the operator decision deferred composition to a later issue). + `zeta_params` (v3.x #817 / #800): when not None AND `heat_kernel_on` + is False (or no eigenbasis is available), the rerank loop swaps + `partial_bayesian_score(...)` for + `zeta_posterior_score(-raw, ζα, ζβ, ζscale, posterior_mean(α, β))`. + The tuple is `(alpha, beta, scale)` in ζ-parameter space (not + Beta-Bernoulli α/β). None falls through to the log-additive or γ + path; γ and ζ are caller-enforced mutually exclusive — see + `_assert_gamma_zeta_mutual_exclusion`. Heat-rerank dominates ζ + just as it does γ. + `heat_kernel_on` (v1.7.0): when True AND `eigenbasis_cache` holds a non-stale eigenbasis whose `belief_ids` intersect the L1 hit set, the rerank uses `combine_log_scores(bm25, heat, posterior_mean)` @@ -2127,14 +2229,15 @@ def _l1_hits( if b is None: continue beliefs.append((b, raw)) - # γ is opt-in; when set it forces the rerank loop so the - # byte-identical short-circuit can't bypass the temperature - # reweighting. + # γ / ζ are opt-in; when either is set it forces the rerank + # loop so the byte-identical short-circuit can't bypass the + # posterior reweighting. if ( posterior_weight == 0.0 and not heat_active and not hash_n_literals and gamma_temperature is None + and zeta_params is None ): return [b for b, _ in beliefs] # BM25F scores are non-negative; the rerank uses `raw` as the @@ -2163,6 +2266,12 @@ def _l1_hits( s = gamma_posterior_score( -raw, b.alpha, b.beta, gamma_temperature, ) + elif zeta_params is not None: + ζα, ζβ, ζscale = zeta_params + s = zeta_posterior_score( + -raw, ζα, ζβ, ζscale, + posterior_mean(b.alpha, b.beta), + ) else: s = partial_bayesian_score( -raw, b.alpha, b.beta, posterior_weight, @@ -2177,6 +2286,7 @@ def _l1_hits( and not heat_active and not hash_n_literals and gamma_temperature is None + and zeta_params is None ): return store.search_beliefs(query, limit=l1_limit) scored = store.search_beliefs_scored(query, limit=l1_limit) @@ -2207,6 +2317,12 @@ def _l1_hits( s = gamma_posterior_score( bm25_raw, b.alpha, b.beta, gamma_temperature, ) + elif zeta_params is not None: + ζα, ζβ, ζscale = zeta_params + s = zeta_posterior_score( + bm25_raw, ζα, ζβ, ζscale, + posterior_mean(b.alpha, b.beta), + ) else: s = partial_bayesian_score( bm25_raw, b.alpha, b.beta, posterior_weight, @@ -2308,6 +2424,17 @@ def retrieve( ) if gamma_on else None ) + # #817 ζ rerank — same posture as γ. Pinned shape parameters from + # the #800 R&D verdict; tunability via TOML/env for α/β/scale is + # deferred. γ and ζ are mutually exclusive: when both flags resolve + # True, raise at flag-resolution time per issue #817 § "Out of scope" + # — the operator decision to defer composition. + zeta_on = resolve_use_zeta_posterior_rerank() + _assert_gamma_zeta_mutual_exclusion(gamma_on, zeta_on) + zeta_params = ( + (ZETA_ALPHA_DEFAULT, ZETA_BETA_DEFAULT, ZETA_SCALE_DEFAULT) + if zeta_on else None + ) compress_on = resolve_use_type_aware_compression( use_type_aware_compression, ) @@ -2386,6 +2513,7 @@ def _cost(b: Belief) -> int: use_bm25f_anchors=bm25f_on, bm25f_cache=bm25f_cache, eigenbasis_cache=eigenbasis_cache, heat_kernel_on=heat_on, gamma_temperature=gamma_t, + zeta_params=zeta_params, ) l1 = [ b for b in raw_l1 @@ -2527,6 +2655,13 @@ def retrieve_with_tiers( ) if gamma_on else None ) + # #817 ζ rerank — same resolution as `retrieve()`. + zeta_on = resolve_use_zeta_posterior_rerank() + _assert_gamma_zeta_mutual_exclusion(gamma_on, zeta_on) + zeta_params = ( + (ZETA_ALPHA_DEFAULT, ZETA_BETA_DEFAULT, ZETA_SCALE_DEFAULT) + if zeta_on else None + ) # #741 adaptive expansion-gate. Same shape as retrieve(): short- # circuit BFS on broad prompts; L0 / L1 / L2.5-entity unaffected. # #760: pass store + now_ts for the meta-belief token-threshold @@ -2598,6 +2733,7 @@ def _cost(b: Belief) -> int: use_bm25f_anchors=bm25f_on, bm25f_cache=bm25f_cache, eigenbasis_cache=eigenbasis_cache, heat_kernel_on=heat_on, gamma_temperature=gamma_t, + zeta_params=zeta_params, ) l1 = [ b for b in raw_l1 diff --git a/src/aelfrice/scoring.py b/src/aelfrice/scoring.py index a13b4e7e..20597b24 100644 --- a/src/aelfrice/scoring.py +++ b/src/aelfrice/scoring.py @@ -55,6 +55,24 @@ # or env override never raises at retrieval time. GAMMA_TEMPERATURE_FLOOR: Final[float] = 1e-6 +# #800 / #817 ζ rerank — pinned defaults from the R&D campaign +# verdict at `experiments/zeta-posterior/` (lab-side, R0–R4, +# commits `2baac93` through `82ec453`). At +# (α=1.0, β=0.25, scale=14.5) ζ dominates γ on rank_biased_overlap +# for similar rank_changed_fraction (R2 head-to-head). Tunability +# via kwargs preserved for tests; env/TOML knobs for α / β / scale +# are deferred per #817 § "Out of scope". +ZETA_ALPHA_DEFAULT: Final[float] = 1.0 +ZETA_BETA_DEFAULT: Final[float] = 0.25 +ZETA_SCALE_DEFAULT: Final[float] = 14.5 + +# Floor on the `posterior_mean` argument to `zeta_posterior_score`. +# A corrupted store row reading `posterior_mean = 0.0` would otherwise +# raise `ValueError: math domain error` at the inner `log()` call; +# the floor clamps such inputs upward to `PARTIAL_BAYESIAN_BM25_FLOOR` +# so retrieval never crashes on degenerate posteriors. +ZETA_POSTERIOR_FLOOR: Final[float] = PARTIAL_BAYESIAN_BM25_FLOOR + # --- Half-lives in seconds --- _HOUR: Final[float] = 3600.0 TYPE_HALF_LIFE_SECONDS: Final[dict[str, float]] = { @@ -240,3 +258,52 @@ def gamma_posterior_score( return partial_bayesian_score( bm25_raw, alpha, beta, posterior_weight=(1.0 / t_safe), ) + + +def zeta_posterior_score( + bm25_raw: float, + alpha: float, + beta: float, + scale: float, + posterior_mean: float, +) -> float: + """#817 / #800 ζ rerank — bounded sigmoid posterior contribution. + + `score = log(max(-bm25_raw, EPS)) + + alpha * (sigmoid(beta * (log(p) - log(0.5))) - 0.5) * scale` + + Where `p = posterior_mean` (already computed by the caller, typically + via `posterior_mean(α_beta, β_beta)`). `alpha`, `beta`, `scale` are + the ζ shape parameters — not the Beta-Bernoulli `(α, β)` pair. + + Bounded contribution: the posterior term lies in + `[-α·scale/2, +α·scale/2]`. No belief gets unboundedly leveraged + by extreme posteriors — the structural difference from γ, where + `(1/T)·log(p)` is unbounded as T → 0. + + Posterior-neutral point: at `p = 0.5`, `log(p) − log(0.5) = 0`, the + sigmoid is 0.5, the bracket is 0, and the entire posterior term + collapses. A store of all-uniform posteriors is rank-equivalent to + log-BM25 alone — the same edge-case property `partial_bayesian_score` + has at `posterior_weight = 0.0`. + + Floor clamp: `posterior_mean <= ZETA_POSTERIOR_FLOOR` clamps upward + so a corrupted store row never raises `math domain error` at + retrieval time. + + ζ is **not** byte-identical to `partial_bayesian_score` or + `gamma_posterior_score` at any (α, β, scale) — the additive term + is bounded around `log(0.5)`, not a free log of the posterior. ζ + is the new long-term shape, not a continuous extension of γ. See + `docs/feature-zeta-posterior-rerank.md` § "Contract" for the + full derivation. + """ + relevance_pos = max(-bm25_raw, PARTIAL_BAYESIAN_BM25_FLOOR) + log_bm25 = math.log(relevance_pos) + p_safe = posterior_mean if posterior_mean > ZETA_POSTERIOR_FLOOR else ( + ZETA_POSTERIOR_FLOOR + ) + # `sigmoid(x) = 1 / (1 + exp(-x))`. Centred so p=0.5 → contribution=0. + x = beta * (math.log(p_safe) - math.log(0.5)) + sigmoid = 1.0 / (1.0 + math.exp(-x)) + return log_bm25 + alpha * (sigmoid - 0.5) * scale diff --git a/tests/test_retrieve_zeta_flag.py b/tests/test_retrieve_zeta_flag.py new file mode 100644 index 00000000..12def4b1 --- /dev/null +++ b/tests/test_retrieve_zeta_flag.py @@ -0,0 +1,208 @@ +"""Tests for #817 ζ flag wiring in retrieve / retrieve_with_tiers. + +Properties under test: + +1. **Flag-off byte-identity.** With AELFRICE_USE_ZETA_POSTERIOR_RERANK + unset (the default) AND γ off, retrieve()'s output is unchanged + compared to a pre-#817 baseline — the existing log-additive / γ + contract holds. +2. **Flag-on deterministic.** Flag on, no γ → ζ runs the rerank loop; + output is deterministic given the same store + query. +3. **Resolver precedence.** env > kwarg > TOML > False. Verified by + the resolver-only tests (no store touch needed for the precedence + chain). +4. **γ + ζ mutual exclusion.** When both flags resolve True, both + retrieve sites raise `ValueError` at flag-resolution time. +5. **No-cross-fire.** With ζ off, the γ-on path is unchanged + (regression protection against a future refactor that + inadvertently routes through ζ when ζ is off). +""" +from __future__ import annotations + +import uuid + +import pytest + +from aelfrice.models import BELIEF_FACTUAL, LOCK_NONE, RETENTION_FACT, Belief +from aelfrice.retrieval import ( + _assert_gamma_zeta_mutual_exclusion, + resolve_use_zeta_posterior_rerank, + retrieve, + retrieve_with_tiers, +) +from aelfrice.store import MemoryStore + + +_ENV_FLAG_ZETA = "AELFRICE_USE_ZETA_POSTERIOR_RERANK" +_ENV_FLAG_GAMMA = "AELFRICE_USE_GAMMA_POSTERIOR_TEMPERATURE" + + +@pytest.fixture(autouse=True) +def _isolate_env(monkeypatch): + """Ensure the rerank env flags never leak across tests.""" + monkeypatch.delenv(_ENV_FLAG_ZETA, raising=False) + monkeypatch.delenv(_ENV_FLAG_GAMMA, raising=False) + yield + + +def _mk_belief(text: str, *, alpha: float = 1.0, beta: float = 1.0) -> Belief: + bid = uuid.uuid4().hex[:16] + return Belief( + id=bid, + content=text, + content_hash=f"h_{bid}", + alpha=alpha, + beta=beta, + type=BELIEF_FACTUAL, + lock_level=LOCK_NONE, + locked_at=None, + created_at="2023-11-14T22:13:20+00:00", + last_retrieved_at=None, + retention_class=RETENTION_FACT, + ) + + +@pytest.fixture +def populated_store(): + """Fresh in-memory store with a small corpus that exercises + posterior reweighting (different α/β so ζ can move beliefs + relative to each other).""" + s = MemoryStore(":memory:") + s.insert_belief(_mk_belief("alpha is the first letter", alpha=10.0, beta=1.0)) + s.insert_belief(_mk_belief("beta is the second letter", alpha=1.0, beta=10.0)) + s.insert_belief(_mk_belief("gamma is the third letter", alpha=5.0, beta=5.0)) + s.insert_belief(_mk_belief("delta is the fourth letter", alpha=1.0, beta=1.0)) + yield s + s.close() + + +# --------------------------------------------------------------------------- +# Resolver precedence +# --------------------------------------------------------------------------- + +def test_resolver_default_false(monkeypatch) -> None: + monkeypatch.delenv(_ENV_FLAG_ZETA, raising=False) + assert resolve_use_zeta_posterior_rerank() is False + + +def test_resolver_env_truthy_wins(monkeypatch) -> None: + monkeypatch.setenv(_ENV_FLAG_ZETA, "1") + assert resolve_use_zeta_posterior_rerank() is True + + +def test_resolver_env_falsy_wins_over_kwarg(monkeypatch) -> None: + monkeypatch.setenv(_ENV_FLAG_ZETA, "0") + assert resolve_use_zeta_posterior_rerank(explicit=True) is False + + +def test_resolver_explicit_kwarg_when_env_unset(monkeypatch) -> None: + monkeypatch.delenv(_ENV_FLAG_ZETA, raising=False) + assert resolve_use_zeta_posterior_rerank(explicit=True) is True + assert resolve_use_zeta_posterior_rerank(explicit=False) is False + + +def test_resolver_unrecognised_env_falls_through(monkeypatch) -> None: + monkeypatch.setenv(_ENV_FLAG_ZETA, "maybe") + assert resolve_use_zeta_posterior_rerank() is False + + +# --------------------------------------------------------------------------- +# Mutual exclusion helper +# --------------------------------------------------------------------------- + +def test_mutex_raises_when_both_on() -> None: + """Both γ and ζ resolved True → ValueError at flag-resolution time.""" + with pytest.raises(ValueError, match="mutually exclusive"): + _assert_gamma_zeta_mutual_exclusion(True, True) + + +@pytest.mark.parametrize( + "gamma_on,zeta_on", + [(False, False), (True, False), (False, True)], +) +def test_mutex_does_not_raise_when_not_both_on( + gamma_on: bool, zeta_on: bool, +) -> None: + """Only the both-True case raises; the three other combinations + pass silently (so the call is cheap on the hot path).""" + _assert_gamma_zeta_mutual_exclusion(gamma_on, zeta_on) + + +# --------------------------------------------------------------------------- +# Flag-off byte-identity + flag-on determinism +# --------------------------------------------------------------------------- + +def test_flag_off_baseline_unchanged(populated_store, monkeypatch) -> None: + """Both flags unset → retrieve() produces the same output across calls + (the regression-protection lane).""" + monkeypatch.delenv(_ENV_FLAG_ZETA, raising=False) + monkeypatch.delenv(_ENV_FLAG_GAMMA, raising=False) + a = retrieve(populated_store, "letter alphabet") + b = retrieve(populated_store, "letter alphabet") + assert [x.id for x in a] == [x.id for x in b] + + +def test_flag_on_deterministic(populated_store, monkeypatch) -> None: + """ζ on, γ off → ζ runs the rerank loop; output is deterministic + across repeats.""" + monkeypatch.setenv(_ENV_FLAG_ZETA, "1") + monkeypatch.delenv(_ENV_FLAG_GAMMA, raising=False) + a = retrieve(populated_store, "letter alphabet") + b = retrieve(populated_store, "letter alphabet") + assert [x.id for x in a] == [x.id for x in b] + + +def test_flag_on_versus_off_runs_clean(populated_store, monkeypatch) -> None: + """Sanity: flipping the ζ flag on does not raise on a small corpus. + + This is the smoke gate the broader bench-corpus comparison sits + on top of. The actual ζ-vs-γ-vs-log-additive ranking comparison + is the job of the lab-side R&D harness (campaign at + experiments/zeta-posterior/), not this unit test. + """ + monkeypatch.delenv(_ENV_FLAG_ZETA, raising=False) + off = retrieve(populated_store, "letter alphabet") + monkeypatch.setenv(_ENV_FLAG_ZETA, "1") + on = retrieve(populated_store, "letter alphabet") + assert isinstance(off, list) + assert isinstance(on, list) + + +def test_flag_on_reorders_by_posterior(populated_store, monkeypatch) -> None: + """ζ at pinned defaults pushes high-posterior beliefs up the rank + when BM25 is similar across rows. The corpus has α/β = (10,1), + (1,10), (5,5), (1,1) — posterior means 0.91, 0.09, 0.5, 0.5. + + With ζ on, the high-posterior 'alpha' belief should outrank the + low-posterior 'beta' belief on a generic 'letter' query (both + match equally well). With both flags off the v1.3 log-additive + path produces the same dominance, but the assert proves ζ doesn't + invert the expected ordering. + """ + monkeypatch.setenv(_ENV_FLAG_ZETA, "1") + out = retrieve(populated_store, "letter alphabet") + ids_to_contents = {b.id: b.content for b in out} + contents = [ids_to_contents[b.id] for b in out] + alpha_idx = next(i for i, c in enumerate(contents) if c.startswith("alpha")) + beta_idx = next(i for i, c in enumerate(contents) if c.startswith("beta")) + assert alpha_idx < beta_idx + + +# --------------------------------------------------------------------------- +# Both-flags-ON path raises through retrieve / retrieve_with_tiers +# --------------------------------------------------------------------------- + +def test_retrieve_raises_when_both_flags_on(populated_store, monkeypatch) -> None: + monkeypatch.setenv(_ENV_FLAG_GAMMA, "1") + monkeypatch.setenv(_ENV_FLAG_ZETA, "1") + with pytest.raises(ValueError, match="mutually exclusive"): + retrieve(populated_store, "letter alphabet") + + +def test_retrieve_with_tiers_raises_when_both_flags_on( + populated_store, monkeypatch, +) -> None: + monkeypatch.setenv(_ENV_FLAG_GAMMA, "1") + monkeypatch.setenv(_ENV_FLAG_ZETA, "1") + with pytest.raises(ValueError, match="mutually exclusive"): + retrieve_with_tiers(populated_store, "letter alphabet") diff --git a/tests/test_scoring_zeta.py b/tests/test_scoring_zeta.py new file mode 100644 index 00000000..481f8b26 --- /dev/null +++ b/tests/test_scoring_zeta.py @@ -0,0 +1,199 @@ +"""Tests for #817 / #800 ζ rerank — zeta_posterior_score. + +Properties under test: + +1. **Posterior-neutral point.** ``zeta_posterior_score(bm, α, β, scale, 0.5)`` + equals ``log(max(-bm, EPS))`` exactly. The bounded sigmoid contribution + collapses to zero when the posterior is uninformative — the same + edge-case property ``partial_bayesian_score`` has at + ``posterior_weight = 0.0``. +2. **σ-bound.** For any (α, β, scale) and any posterior_mean ∈ (0, 1), + the posterior contribution lies strictly in + ``(-α·scale/2, +α·scale/2)``. No belief is unboundedly leveraged + by extreme posteriors — the structural difference from γ, where + ``(1/T)·log(p)`` is unbounded as T → 0. +3. **Monotonicity in posterior_mean.** With (α, β, scale) fixed and + bm25_raw fixed, the score is strictly increasing in posterior_mean + on (0, 1). +4. **Floor clamp.** ``posterior_mean <= ZETA_POSTERIOR_FLOOR`` clamps + upward rather than raising ``math domain error``. Negative posteriors + (corrupted store row) are likewise clamped — they decode to the + minimum contribution rather than NaN. +5. **Determinism.** Same inputs → bit-identical output across calls. +6. **Not byte-identical to γ or partial_bayesian.** ζ at the pinned + defaults (α=1, β=0.25, scale=14.5) is a different function family + from γ; the contracts of #817 §"Note re: cold-start byte-identity" + say so explicitly. We assert *inequality* at non-trivial inputs to + catch accidental coincidence-bugs (a future ζ refactor that + collapses to log-additive shape would silently break the bench). +""" +from __future__ import annotations + +import math + +import pytest + +from aelfrice.scoring import ( + PARTIAL_BAYESIAN_BM25_FLOOR, + ZETA_ALPHA_DEFAULT, + ZETA_BETA_DEFAULT, + ZETA_POSTERIOR_FLOOR, + ZETA_SCALE_DEFAULT, + gamma_posterior_score, + partial_bayesian_score, + posterior_mean, + zeta_posterior_score, +) + + +# --------------------------------------------------------------------------- +# Posterior-neutral point +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize( + "bm25_raw,alpha,beta,scale", + [ + (-1.5, 1.0, 0.25, 14.5), + (-0.001, 0.5, 0.1, 5.0), + (-10.0, 2.0, 1.0, 20.0), + (0.0, 1.0, 0.25, 14.5), # no-match: BM25 = 0; floor protects log + ], +) +def test_posterior_half_is_log_bm25_only( + bm25_raw: float, alpha: float, beta: float, scale: float, +) -> None: + """At posterior_mean=0.5, the bracket (σ-0.5) is exactly 0 → ζ collapses + to log(max(-bm25, EPS)).""" + relevance_pos = max(-bm25_raw, PARTIAL_BAYESIAN_BM25_FLOOR) + expected = math.log(relevance_pos) + got = zeta_posterior_score(bm25_raw, alpha, beta, scale, 0.5) + assert got == expected, (got, expected) + + +# --------------------------------------------------------------------------- +# σ-bound +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("posterior", [0.001, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 0.99, 0.999]) +def test_sigma_bound(posterior: float) -> None: + """Contribution lies strictly in (-α·scale/2, +α·scale/2).""" + bm25_raw = -1.5 + alpha, beta, scale = ZETA_ALPHA_DEFAULT, ZETA_BETA_DEFAULT, ZETA_SCALE_DEFAULT + half_range = alpha * scale / 2.0 + log_bm = math.log(-bm25_raw) + s = zeta_posterior_score(bm25_raw, alpha, beta, scale, posterior) + contribution = s - log_bm + assert -half_range < contribution < +half_range, ( + f"posterior={posterior}: contribution={contribution} " + f"outside (±{half_range})" + ) + + +def test_sigma_bound_at_extreme_posteriors() -> None: + """As p → 1 or p → 0+, contribution approaches ±α·scale/2 but + does not exceed it (open interval). Use very-extreme posteriors + to stress the sigmoid saturation.""" + bm25_raw = -1.5 + alpha, beta, scale = ZETA_ALPHA_DEFAULT, ZETA_BETA_DEFAULT, ZETA_SCALE_DEFAULT + half_range = alpha * scale / 2.0 + log_bm = math.log(-bm25_raw) + s_high = zeta_posterior_score(bm25_raw, alpha, beta, scale, 1.0 - 1e-15) + s_low = zeta_posterior_score(bm25_raw, alpha, beta, scale, 1e-15) + assert s_high - log_bm < half_range + assert s_low - log_bm > -half_range + # And the sigmoid does saturate close to the bound at β=0.25 even + # with extreme p — verifies the parameter space we ship. + assert s_high - log_bm > 0 + assert s_low - log_bm < 0 + + +# --------------------------------------------------------------------------- +# Monotonicity +# --------------------------------------------------------------------------- + +def test_monotone_in_posterior() -> None: + """Score is strictly increasing in posterior_mean on (0, 1).""" + bm25_raw = -1.5 + alpha, beta, scale = ZETA_ALPHA_DEFAULT, ZETA_BETA_DEFAULT, ZETA_SCALE_DEFAULT + posteriors = [0.05, 0.1, 0.2, 0.4, 0.5, 0.6, 0.8, 0.95] + scores = [ + zeta_posterior_score(bm25_raw, alpha, beta, scale, p) for p in posteriors + ] + for i in range(len(scores) - 1): + assert scores[i] < scores[i + 1], ( + f"non-monotone at p={posteriors[i]} → {posteriors[i + 1]}: " + f"{scores[i]} not < {scores[i + 1]}" + ) + + +# --------------------------------------------------------------------------- +# Floor clamp on degenerate posterior +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("bad_p", [0.0, -0.5, -1e9, ZETA_POSTERIOR_FLOOR]) +def test_floor_clamp_finite(bad_p: float) -> None: + """posterior_mean <= floor never raises; returns a finite score.""" + s = zeta_posterior_score( + -1.5, ZETA_ALPHA_DEFAULT, ZETA_BETA_DEFAULT, ZETA_SCALE_DEFAULT, bad_p, + ) + assert math.isfinite(s) + + +def test_floor_clamp_matches_floor_value() -> None: + """A posterior at the floor and a more-negative posterior produce + the same score (both clamp to the floor).""" + args = (-1.5, ZETA_ALPHA_DEFAULT, ZETA_BETA_DEFAULT, ZETA_SCALE_DEFAULT) + s_at_floor = zeta_posterior_score(*args, ZETA_POSTERIOR_FLOOR) + s_below = zeta_posterior_score(*args, 0.0) + s_negative = zeta_posterior_score(*args, -1.0) + assert s_at_floor == s_below == s_negative + + +# --------------------------------------------------------------------------- +# Determinism +# --------------------------------------------------------------------------- + +def test_deterministic_across_calls() -> None: + """Same inputs → bit-identical output across calls.""" + args = (-1.5, 1.0, 0.25, 14.5, 0.7) + for _ in range(3): + assert zeta_posterior_score(*args) == zeta_posterior_score(*args) + + +# --------------------------------------------------------------------------- +# Not-byte-identical to γ / partial_bayesian (issue §"Note re cold-start") +# --------------------------------------------------------------------------- + +def test_zeta_not_byte_identical_to_gamma_at_T_one() -> None: + """ζ at the pinned defaults is not byte-identical to γ@T=1.0 (which + is itself byte-identical to partial_bayesian_score(..., 1.0)). This + asserts the structural difference noted in issue #817's 'Note re: + cold-start byte-identity'.""" + bm25_raw, alpha_beta_a, alpha_beta_b = -1.5, 3.0, 2.0 + p = posterior_mean(alpha_beta_a, alpha_beta_b) # = 0.6 + z = zeta_posterior_score( + bm25_raw, + ZETA_ALPHA_DEFAULT, ZETA_BETA_DEFAULT, ZETA_SCALE_DEFAULT, + p, + ) + g = gamma_posterior_score(bm25_raw, alpha_beta_a, alpha_beta_b, 1.0) + pb = partial_bayesian_score(bm25_raw, alpha_beta_a, alpha_beta_b, 1.0) + assert g == pb # already shipped contract; sanity + assert z != g # the load-bearing inequality + assert z != pb + + +def test_zeta_collapses_to_log_bm_at_uniform_posteriors() -> None: + """All-uniform-posterior=0.5 store: ζ ranks identically to log-BM25 + alone. The contribution is identically zero for every belief, so + the entire posterior layer is inert. This is the same shape + `partial_bayesian_score(..., 0.0)` has.""" + bm25_values = [-0.5, -1.0, -1.5, -2.0, -3.0] + for bm in bm25_values: + z = zeta_posterior_score( + bm, ZETA_ALPHA_DEFAULT, ZETA_BETA_DEFAULT, ZETA_SCALE_DEFAULT, 0.5, + ) + pb = partial_bayesian_score(bm, 1.0, 1.0, posterior_weight=0.0) + # partial_bayesian at pw=0 short-circuits to log(-bm); ζ at p=0.5 + # also collapses there. Both should produce log(max(-bm, EPS)). + assert z == pb diff --git a/tests/test_zeta_vs_gamma_panel.py b/tests/test_zeta_vs_gamma_panel.py new file mode 100644 index 00000000..9f2dcf4b --- /dev/null +++ b/tests/test_zeta_vs_gamma_panel.py @@ -0,0 +1,179 @@ +"""ζ vs γ ranking panel — #817 §"Scope" item 5. + +The full lab-side comparison ζ@(α=1, β=0.25, scale=14.5) vs γ@T=1.0 +runs against the corpus at ``experiments/zeta-posterior/`` and is not +in this test file's scope. What this file *does* cover is the +property contracts visible from public test fixtures: + +1. **Both flags OFF byte-identical to main.** With neither γ nor ζ + on, retrieve() output is identical to what the v3.1 log-additive + path produced — a structural regression guard. +2. **Single-flag panels run clean.** ζ on alone and γ on alone each + produce a deterministic ranking on a small fixture; the panel + metrics (``rank_biased_overlap``, ``ordered_top_k_overlap``) are + computable on those rankings without raising. +3. **ζ vs γ on a constant-posterior fixture is rank-identical.** When + every belief in the fixture has α=β=1.0 (posterior_mean=0.5), ζ's + posterior contribution is identically zero and γ at T=1.0 also + adds a constant log(0.5), so neither rerank moves any belief + relative to BM25 alone. RBO and top-K overlap both = 1.0. +4. **Both flags ON → ValueError.** Covered exhaustively in + ``test_retrieve_zeta_flag.py``; cross-referenced here for + navigability. +""" +from __future__ import annotations + +import uuid + +import pytest + +from aelfrice.calibration_metrics import ( + ordered_top_k_overlap, + rank_biased_overlap, +) +from aelfrice.models import BELIEF_FACTUAL, LOCK_NONE, RETENTION_FACT, Belief +from aelfrice.retrieval import retrieve +from aelfrice.store import MemoryStore + + +_ENV_FLAG_ZETA = "AELFRICE_USE_ZETA_POSTERIOR_RERANK" +_ENV_FLAG_GAMMA = "AELFRICE_USE_GAMMA_POSTERIOR_TEMPERATURE" + + +@pytest.fixture(autouse=True) +def _isolate_env(monkeypatch): + monkeypatch.delenv(_ENV_FLAG_ZETA, raising=False) + monkeypatch.delenv(_ENV_FLAG_GAMMA, raising=False) + yield + + +def _mk_belief(text: str, *, alpha: float = 1.0, beta: float = 1.0) -> Belief: + bid = uuid.uuid4().hex[:16] + return Belief( + id=bid, content=text, content_hash=f"h_{bid}", + alpha=alpha, beta=beta, type=BELIEF_FACTUAL, + lock_level=LOCK_NONE, locked_at=None, + created_at="2023-11-14T22:13:20+00:00", + last_retrieved_at=None, retention_class=RETENTION_FACT, + ) + + +@pytest.fixture +def varied_posterior_store(): + """Six beliefs spanning the posterior_mean range so γ and ζ have + something to move on. Bench panels comparing γ-vs-ζ exercise the + `_l1_hits` rerank branch.""" + s = MemoryStore(":memory:") + rows = [ + ("alpha is the first letter", 10.0, 1.0), + ("beta is the second letter", 1.0, 10.0), + ("gamma is the third letter", 5.0, 5.0), + ("delta is the fourth letter", 1.0, 1.0), + ("epsilon is the fifth letter", 8.0, 2.0), + ("zeta is the sixth letter", 2.0, 8.0), + ] + for text, a, b in rows: + s.insert_belief(_mk_belief(text, alpha=a, beta=b)) + yield s + s.close() + + +@pytest.fixture +def uniform_posterior_store(): + """Six beliefs all at α=β=1.0 (posterior_mean=0.5). Neither γ nor + ζ moves them relative to BM25 — γ adds a constant log(0.5) and + ζ's bracket is identically 0.""" + s = MemoryStore(":memory:") + for text in [ + "alpha is the first letter", + "beta is the second letter", + "gamma is the third letter", + "delta is the fourth letter", + "epsilon is the fifth letter", + "zeta is the sixth letter", + ]: + s.insert_belief(_mk_belief(text)) # α=β=1.0 + yield s + s.close() + + +# --------------------------------------------------------------------------- +# Both flags OFF → byte-identical to pre-#817 path +# --------------------------------------------------------------------------- + +def test_both_flags_off_deterministic(varied_posterior_store) -> None: + """The default — neither γ nor ζ env set. Two calls produce the + same ranking; #817 is plumbing only on default code paths.""" + a = retrieve(varied_posterior_store, "letter alphabet") + b = retrieve(varied_posterior_store, "letter alphabet") + assert [x.id for x in a] == [x.id for x in b] + # Sanity: RBO on a list with itself is exactly 1.0. + ids_a = [x.id for x in a] + assert rank_biased_overlap(ids_a, ids_a) == pytest.approx(1.0, abs=1e-12) + + +# --------------------------------------------------------------------------- +# Single-flag panels run clean +# --------------------------------------------------------------------------- + +def test_gamma_panel_runs(varied_posterior_store, monkeypatch) -> None: + monkeypatch.setenv(_ENV_FLAG_GAMMA, "1") + out = retrieve(varied_posterior_store, "letter alphabet") + ids = [b.id for b in out] + assert len(ids) >= 2 + # Panel metrics computable. + rbo = rank_biased_overlap(ids, ids) + overlap = ordered_top_k_overlap(ids, ids, k=min(5, len(ids))) + assert rbo == pytest.approx(1.0, abs=1e-12) + assert overlap == 1.0 + + +def test_zeta_panel_runs(varied_posterior_store, monkeypatch) -> None: + monkeypatch.setenv(_ENV_FLAG_ZETA, "1") + out = retrieve(varied_posterior_store, "letter alphabet") + ids = [b.id for b in out] + assert len(ids) >= 2 + rbo = rank_biased_overlap(ids, ids) + overlap = ordered_top_k_overlap(ids, ids, k=min(5, len(ids))) + assert rbo == pytest.approx(1.0, abs=1e-12) + assert overlap == 1.0 + + +# --------------------------------------------------------------------------- +# Uniform-posterior fixture: γ and ζ are rank-identical +# --------------------------------------------------------------------------- + +def test_gamma_and_zeta_agree_on_uniform_store( + uniform_posterior_store, monkeypatch, +) -> None: + """Every belief at posterior_mean=0.5 → ζ contribution is 0 for + every belief; γ at T=1.0 adds a constant log(0.5) to every + belief. Neither rerank moves anything. Result: γ-on and ζ-on + rank-identically (and identical to flag-off, modulo the + short-circuit that ζ/γ disables to force the rerank loop).""" + monkeypatch.delenv(_ENV_FLAG_GAMMA, raising=False) + monkeypatch.delenv(_ENV_FLAG_ZETA, raising=False) + baseline = [b.id for b in retrieve(uniform_posterior_store, "letter")] + + monkeypatch.setenv(_ENV_FLAG_GAMMA, "1") + gamma_ids = [b.id for b in retrieve(uniform_posterior_store, "letter")] + monkeypatch.delenv(_ENV_FLAG_GAMMA, raising=False) + + monkeypatch.setenv(_ENV_FLAG_ZETA, "1") + zeta_ids = [b.id for b in retrieve(uniform_posterior_store, "letter")] + + # RBO between γ-on and ζ-on should be 1.0 — both add a uniform + # constant (or 0) to every belief. + assert rank_biased_overlap(gamma_ids, zeta_ids) == pytest.approx( + 1.0, abs=1e-12, + ) + assert ordered_top_k_overlap( + gamma_ids, zeta_ids, k=min(5, len(gamma_ids)), + ) == 1.0 + # And both equal the baseline ordering for the same reason. + assert ordered_top_k_overlap( + baseline, gamma_ids, k=min(5, len(baseline)), + ) == 1.0 + assert ordered_top_k_overlap( + baseline, zeta_ids, k=min(5, len(baseline)), + ) == 1.0