feat: posterior-ranking eval harness — MRR uplift + ECE — slice 1 of #151 - #306
Merged
Merged
Conversation
Implements the Slice 1 eval harness scorers from docs/v2_posterior_ranking_residual.md. mrr_uplift.py: 10-round MRR uplift evaluator. Builds one in-memory MemoryStore per fixture, runs a synthetic positive-feedback loop (top-1 if it matches the known item, else the known item directly), records mrr_per_round, and checks uplift >= threshold AND no round regresses below mrr_0 - 0.01. Multi-seed runner reports (mean, ±2σ) via run_multi_seed(). ece.py: ECE calibration scorer. Buckets (alpha, beta, actual) triples into 10 equal-width [0.1) buckets, computes weighted mean absolute error between posterior_mean and empirical feedback rate. Reuses scoring.posterior_mean() for predicted probabilities. Tests: smoke, regression-detection, multi-seed reproducibility band, ECE well/poorly-calibrated, bucket sanity (no NaN, weights sum to 1.0). Refs #151 (Slice 1)
…ng eval run.py wires both scorers against a fixture file. run() builds one in-memory store per fixture, runs the MRR feedback loop and collects per-round ECE observations (alpha, beta, received_positive per retrieved belief), then delegates to run_multi_seed() and compute_ece_from_stores(). run_as_dict() serializes dataclasses for JSON output. fixtures/default.jsonl ships 7 hand-curated known-item fixtures spanning asyncio/SQLite/Bayesian/BM25/decay/FTS5/Jeffreys topics. Each fixture has 4-5 noise beliefs chosen to cover distinct retrieval lanes. Tests: runner integration (3-fixture file, structural output check), run_as_dict JSON serialization, default fixture file exists and is readable with >= 5 entries and all required keys, load_fixtures helper. Refs #151 (Slice 1)
Adds target 'posterior-residual' to _cmd_bench. Because the existing bench subparser uses nargs=REMAINDER for the rest positional, named flags after the target are parsed via a local ArgumentParser inside the handler rather than top-level argparse, matching the pattern used by other bench targets that consume positional args from args.rest. Flags: --fixtures PATH JSONL fixture file (default: default.jsonl) --seeds N multi-seed count (default 5) --mrr-threshold F MRR uplift pass gate (default 0.05) --ece-threshold F ECE calibration pass gate (default 0.10) --json emit machine-readable JSON Exit 0 when mrr.passed AND ece.passed; exit 1 otherwise. Exit 2 if benchmarks/ source tree is absent (installed wheel). Tests: exit 0 with relaxed ECE threshold on a well-structured fixture, exit 1 with impossibly tight MRR threshold, --json output parses. Refs #151 (Slice 1)
|
Important Installation incomplete: to start using Gemini Code Assist, please ask the organization owner(s) to visit the Gemini Code Assist Admin Console and sign the Terms of Services. |
Reviewer's GuideAdds a posterior-ranking evaluation harness consisting of an MRR uplift scorer, an ECE calibration scorer, and a coordinating runner; wires the harness into the Sequence diagram for the posterior-residual CLI eval harnesssequenceDiagram
actor developer
participant aelf_cli
participant bench_parser
participant posterior_runner as posterior_ranking_run
participant mrr as mrr_uplift
participant ece as ece_scorer
developer->>aelf_cli: invoke aelf bench posterior-residual [flags]
aelf_cli->>bench_parser: parse global bench args
bench_parser-->>aelf_cli: args with target=posterior-residual
aelf_cli->>aelf_cli: import benchmarks.posterior_ranking.run
aelf_cli->>aelf_cli: parse posterior-residual flags from args.rest
aelf_cli->>aelf_cli: resolve fixtures_path and thresholds
aelf_cli->>posterior_runner: run(fixtures_path, n_seeds, mrr_threshold, ece_threshold, top_k, base_seed)
posterior_runner->>posterior_runner: fixtures = load_fixtures(fixtures_path)
posterior_runner->>mrr: run_multi_seed(fixtures, n_seeds, threshold, top_k, base_seed)
mrr->>mrr: for each seed
mrr->>mrr: run_single_seed(fixtures, seed, top_k, threshold)
mrr-->>posterior_runner: MultiSeedReport
posterior_runner->>ece: _build_ece_observations(fixtures, base_seed, top_k)
ece-->>posterior_runner: observations
posterior_runner->>ece: compute_ece_from_stores(observations, ece_threshold)
ece-->>posterior_runner: ECEResult
posterior_runner->>posterior_runner: overall_pass = mrr_report.passed and ece_result.passed
posterior_runner-->>aelf_cli: {mrr: MultiSeedReport, ece: ECEResult, overall_pass}
alt json output requested
aelf_cli->>aelf_cli: json.dumps({mrr: asdict(mrr), ece: asdict(ece), overall_pass})
aelf_cli-->>developer: print machine readable JSON
else human readable output
aelf_cli->>aelf_cli: format MRR uplift and ECE summary
aelf_cli-->>developer: print human readable report
end
aelf_cli->>developer: exit code 0 if overall_pass else 1
Class diagram for posterior-ranking MRR and ECE eval harnessclassDiagram
class MRRUpliftResult {
float mrr_0
list~float~ mrr_per_round
float mrr_uplift
int seed
float pass_threshold
bool passed
+mrr_10() float
}
class MultiSeedReport {
list~MRRUpliftResult~ results
float mean_uplift
float std_uplift
float uplift_lo
float uplift_hi
float pass_threshold
bool passed
}
class BucketStat {
int bucket_idx
float bucket_lo
float bucket_hi
int count
float mean_predicted
float mean_actual
float weight
}
class ECEResult {
float ece
list~BucketStat~ buckets
int n_total
float pass_threshold
bool passed
}
class mrr_uplift_module {
<<module>>
+run_single_seed(fixtures, seed, top_k, threshold) MRRUpliftResult
+run_multi_seed(fixtures, n_seeds, threshold, top_k, base_seed) MultiSeedReport
+load_fixtures(path) list~dict~
+_build_store(fixture, seed, noise_shuffle) tuple
+_mrr_for_belief(store, query, known_content, top_k) float
}
class ece_module {
<<module>>
+compute_ece(triples, threshold) ECEResult
+compute_ece_from_stores(fixture_observations, threshold) ECEResult
+_bucket_index(predicted) int
}
class posterior_ranking_run_module {
<<module>>
+run(fixtures_path, n_seeds, mrr_threshold, ece_threshold, top_k, base_seed) dict
+run_as_dict(fixtures_path, n_seeds, mrr_threshold, ece_threshold, top_k, base_seed) dict
+_build_ece_observations(fixtures, seed, top_k) list~dict~
}
class aelf_cli_bench_posterior_residual {
<<cli_command>>
+_cmd_bench(args, out) int
+build_parser(show_advanced) ArgumentParser
}
mrr_uplift_module ..> MRRUpliftResult : creates
mrr_uplift_module ..> MultiSeedReport : creates
ece_module ..> BucketStat : aggregates
ece_module ..> ECEResult : creates
posterior_ranking_run_module ..> mrr_uplift_module : uses
posterior_ranking_run_module ..> ece_module : uses
posterior_ranking_run_module ..> MultiSeedReport : returns_in_mrr
posterior_ranking_run_module ..> ECEResult : returns_in_ece
aelf_cli_bench_posterior_residual ..> posterior_ranking_run_module : calls_run
aelf_cli_bench_posterior_residual ..> posterior_ranking_run_module : calls_run_as_dict
aelf_cli_bench_posterior_residual ..> MultiSeedReport : prints_mrr
aelf_cli_bench_posterior_residual ..> ECEResult : prints_ece
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
📝 WalkthroughWalkthroughThis change introduces a posterior-ranking benchmark suite with MRR uplift and Expected Calibration Error (ECE) evaluation metrics. It includes new evaluation modules, test fixtures, CLI integration, and comprehensive test coverage for assessing information retrieval system performance. Changes
Sequence DiagramsequenceDiagram
participant User as User/CLI
participant Runner as run.py
participant Fixtures as Fixture Loader
participant MRR as MRR Scorer
participant Retrieval as Retrieval Engine
participant ECE as ECE Scorer
User->>Runner: run(fixtures_path, n_seeds, thresholds, ...)
Runner->>Fixtures: load_fixtures(path)
Fixtures-->>Runner: list[fixture_dict]
Runner->>MRR: run_multi_seed(fixtures, n_seeds, top_k)
loop Per Seed
MRR->>Retrieval: Initialize belief store & rank
MRR->>Retrieval: Compute baseline MRR (round 0)
loop 10 Feedback Rounds
MRR->>Retrieval: Retrieve top-k beliefs
MRR->>Retrieval: Apply synthetic positive feedback
MRR->>Retrieval: Re-rank & compute MRR
end
end
MRR-->>Runner: MultiSeedReport (uplift, pass/fail)
Runner->>Retrieval: Replay all rounds with observations
loop Per Round (0-10)
Retrieval->>Retrieval: Collect (alpha, beta, received_positive)
end
Retrieval-->>Runner: observation_list
Runner->>ECE: compute_ece_from_stores(observations, threshold)
ECE->>ECE: Bin into 10 probability buckets
ECE->>ECE: Calculate weighted calibration error
ECE-->>Runner: ECEResult (ece_value, pass/fail)
Runner-->>User: {mrr: MultiSeedReport, ece: ECEResult, overall_pass: bool}
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Owner
Author
|
Self-review notes for cross-session reviewer: Slice-1 default-fixture calibration is intentionally loose. The shipped 7-entry
Synthetic feedback stream definition. Spec leaves the stream implicit; implemented as: per round, record Out of scope follow-up: Either curate a wider fixture corpus that clears 0.10, or accept the harness as instrumentation-only and gate v2.0's posterior-residual claim on the operator's own fixture file. Both are consistent with the spec. |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The synthetic feedback stream logic is duplicated between
mrr_uplift.run_single_seedand_build_ece_observations; consider factoring this into a shared helper so MRR and ECE stay in lockstep if the contract changes. - The
benchsubparser defines--fixtures/--seeds/--mrr-threshold/--ece-threshold/--jsononp_bench, but_cmd_benchre-parsesargs.restwith its ownArgumentParserand ignores those parsed values; to avoid surprising behavior and drift between help text and actual parsing, either remove the top-level options or thread the parsed values into_cmd_benchinstead of re-parsing. - In
compute_ece, you accumulate per-bucket lists of predictions and labels before averaging; if the eval set grows, you might want to switch to tracking running sums and counts per bucket to keep memory usage linear in bucket count instead of observation count.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The synthetic feedback stream logic is duplicated between `mrr_uplift.run_single_seed` and `_build_ece_observations`; consider factoring this into a shared helper so MRR and ECE stay in lockstep if the contract changes.
- The `bench` subparser defines `--fixtures/--seeds/--mrr-threshold/--ece-threshold/--json` on `p_bench`, but `_cmd_bench` re-parses `args.rest` with its own `ArgumentParser` and ignores those parsed values; to avoid surprising behavior and drift between help text and actual parsing, either remove the top-level options or thread the parsed values into `_cmd_bench` instead of re-parsing.
- In `compute_ece`, you accumulate per-bucket lists of predictions and labels before averaging; if the eval set grows, you might want to switch to tracking running sums and counts per bucket to keep memory usage linear in bucket count instead of observation count.
## Individual Comments
### Comment 1
<location path="benchmarks/posterior_ranking/mrr_uplift.py" line_range="232-241" />
<code_context>
+ )
+
+
+def run_multi_seed(
+ fixtures: list[dict[str, object]],
+ n_seeds: int = 5,
+ threshold: float = DEFAULT_MRR_THRESHOLD,
+ top_k: int = DEFAULT_TOP_K,
+ base_seed: int = 0,
+) -> MultiSeedReport:
+ """Run the uplift evaluator across n_seeds seeds.
+
+ Seeds are derived deterministically from base_seed.
+ Returns aggregated (mean, ±2σ) uplift report.
+ """
+ results: list[MRRUpliftResult] = []
+ for i in range(n_seeds):
+ seed = base_seed + i
</code_context>
<issue_to_address>
**issue:** run_multi_seed assumes n_seeds > 0; n_seeds=0 will raise due to division by zero.
If `n_seeds` is 0, `results` remains empty and `mean_uplift = sum(uplifts) / len(uplifts)` raises `ZeroDivisionError`. For a public helper, this implicit assumption is brittle. Either validate `n_seeds >= 1` up front with a clear error, or define explicit behavior for `n_seeds=0` (e.g., treat it as 1 or return an empty report with `passed=False`).
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Owner
Author
|
[claim:review:Gylf:2026-04-29T04:50:35Z] |
robotrocketscience
enabled auto-merge (squash)
April 29, 2026 04:51
Owner
Author
|
[release:review:Gylf:2026-04-29T04:51:50Z] |
Owner
Author
|
[claim:review:Kulili:2026-04-29T04:53:58Z] |
Owner
Author
|
[release:review:Kulili:2026-04-29T04:56:16Z] |
robotrocketscience
added a commit
that referenced
this pull request
Apr 29, 2026
…2 of #151) (#310) Slice 2 of #151 — heat-kernel composition into the log-additive ranking score. Builds on the Slice 1 eval harness (#306). ## What lands - `_l1_hits` in `retrieval.py` dispatches to `combine_log_scores(bm25, heat, posterior_mean)` from the heat-kernel module (#150) when the heat-kernel feature flag is on AND a non-stale `GraphEigenbasisCache` is supplied. - New `_heat_by_id` helper computes per-belief heat scores via one `eigvecs.T @ seeds` matvec per query, indexed by the eigenbasis row order. - New `eigenbasis_cache` and `heat_kernel_enabled` kwargs on `retrieve()` and `retrieve_with_tiers()`. Both default to `None`/`False` — the heat-off path is byte-identical to current `main`. - Feature flag resolution path was already in place (`is_heat_kernel_enabled()`, `AELFRICE_HEAT_KERNEL`, `[retrieval] use_heat_kernel`); now it actually does something. ## Graceful degrade Heat-kernel dispatch falls back to `partial_bayesian_score` (Slice 1 path) whenever any of these holds: - `eigenbasis_cache` is `None` - `cache.is_stale()` (any store mutation since the last `.build()`) - `cache.eigvals is None` (cache constructed but never built) - L1 hit set has no overlap with `cache.belief_ids` (every L1 belief was inserted after the last build) - Seed sum is zero This means a flag-on session with no offline eigenbasis build behaves identically to flag-off — no sharp edge. ## Tests `tests/test_posterior_ranking_heat.py` (new, 5 tests): 1. `heat_off_byte_identical_to_slice1` — flag off vs no-kwarg, identical ranking. 2. `heat_kernel_empty_eigenbasis_falls_back` — flag on with un-built cache, identical to flag off. 3. `heat_kernel_changes_ranking_on_authority_graph` — built eigenbasis, ranking diverges from flag-off when SUPPORTS edges concentrate authority on one belief. 4. `heat_kernel_cold_belief_neutral` — belief inserted after `.build()` gets the floor heat score, doesn't crash, still rankable. 5. `heat_kernel_cache_invalidation_on_store_mutation` — store mutation flips `is_stale`, retrieval falls back. ``` tests/test_posterior_ranking_heat.py: 5 passed tests/ (full suite, ignoring corpus-schema): 1905 passed, 8 skipped ``` ## Bench wedge `aelf bench posterior-residual --heat-kernel` threads `heat_kernel: bool` through `run()` → `run_multi_seed()` → `_build_ece_observations()`. Each per-seed `retrieve()` gets a fresh `GraphEigenbasisCache` with `.build()` called. Smoke run on the default fixture set (no edges in the synthetic corpus) returns the same numbers as heat-off — heat propagation degrades to the floor when there's no graph to propagate over. Real uplift demo needs graph-bearing fixtures, which is Slice 3 territory. ## Out of scope - Eigenbasis offline-build CLI (#149). - Real-corpus heat-kernel uplift demonstration / fixture rework (Slice 3). - N=50k AC6 perf measurement — needs to be captured against a real-sized store before AC6 is declared met. - `retrieve_v2()` heat kwarg — v2 path is separate. ## AC6 renegotiation #151 body was edited as part of this work: AC6 split into `≤1ms heat-off` and `≤10ms heat-on`. The original 1ms target predated the heat-kernel composition path being scoped into the issue and didn't account for the eigenbasis matvec cost (~7-8 ms at N=50k, K=200 per the heat-kernel spec § Cost). ## Acceptance criteria status (#151) - AC1, AC2, AC3 — covered by Slice 1 (`tests/test_posterior_ranking_eval.py`). - AC4 — flag-off / heat-off identity asserted in `heat_off_byte_identical_to_slice1` and `heat_kernel_empty_eigenbasis_falls_back`. - AC5 — feedback rank-improvement, Slice 1. - AC6 — heat-off path well within budget; heat-on N=50k perf measurement still owed (open as a follow-up). - AC7 — feature flag default-off in v1.x, no change. - AC8 — eigenbasis cache invalidation hook exists from #150; `heat_kernel_cache_invalidation_on_store_mutation` confirms the dispatch reads `is_stale` correctly. ## Discretion ``` git diff main..HEAD | grep -niE 'sonnet|opus|haiku|anthropic|subagent|setr|kulili|gylf|claude code' ``` returns nothing. ## Review Review needed. ## Summary by Sourcery Introduce optional heat-kernel graph authority composition into retrieval ranking and benchmarking, gated behind a feature flag with graceful degradation to existing behavior. New Features: - Add heat-kernel authority term to the L1 retrieval ranking via a compose-with-BM25 and posterior log-additive score when a non-stale eigenbasis cache is provided and the feature flag is enabled. - Expose heat-kernel controls on retrieval APIs (`retrieve`, `retrieve_with_tiers`) and posterior ranking benchmarks, including a new `--heat-kernel` CLI switch for the posterior-residual bench harness. Enhancements: - Implement eigenbasis-aware heat score computation for L1 hits that falls back to the prior partial-Bayesian ranking when caches are missing, stale, or non-overlapping. - Document the heat-kernel composition design, feature flag behavior, cost model, and degradation semantics in the Bayesian ranking spec. Tests: - Add a dedicated heat-kernel retrieval test suite validating flag-off identity, fallback behavior, authority-driven reordering, cold-belief handling, and cache invalidation behavior in the ranking pipeline. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional heat-kernel-based reranking with per-fixture caching. Disabled by default; enable via `aelf bench posterior-residual --heat-kernel` CLI flag or through function parameters. Includes automatic cache staleness detection and fallback behavior. * **Documentation** * Updated documentation describing heat-kernel specifications, computational costs, latency targets, and automatic degradation conditions. * **Tests** * Added comprehensive test coverage for heat-kernel retrieval, cache invalidation, and backward compatibility. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
robotrocketscience
added a commit
that referenced
this pull request
Apr 29, 2026
Slice 1 (eval harness — MRR uplift + ECE) shipped via #306. Slice 2 (heat-kernel composition into log-additive score) shipped via #310. Slice 3 (per-corpus weight sweep) remains residual. Adds dated status preamble and per-slice status tags so the spec stays accurate to current state without rewriting the original recommendation.
This was referenced May 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements Slice 1 of #151 (eval harness only). Slices 2 (heat-kernel composition, blocks on #150) and 3 (weight sweep, blocks on Slice 1) are out of scope for this PR.
benchmarks/posterior_ranking/mrr_uplift.py— 10-round MRR uplift evaluator: baseline retrieve, synthetic positive-feedback loop, per-round MRR series, uplift = mrr_10 - mrr_0, multi-seed (mean ±2σ) reportbenchmarks/posterior_ranking/ece.py— ECE calibration scorer: 10 equal-width buckets, weighted mean absolute error betweenposterior_mean(b)and empirical positive-feedback ratebenchmarks/posterior_ranking/run.py— runner wiring both scorers against a JSONL fixture file;run()andrun_as_dict()entry pointsbenchmarks/posterior_ranking/fixtures/default.jsonl— 7 hand-curated known-item fixtures (asyncio, SQLite WAL, Beta-Bernoulli, BM25, decay, FTS5, Jeffreys prior)src/aelfrice/cli.py—aelf bench posterior-residualtarget with--fixtures,--seeds,--mrr-threshold,--ece-threshold,--jsonflags; exit 0 if both pass, exit 1 otherwisetests/test_posterior_ranking_eval.py— 18 tests covering scorers in isolation, multi-seed reproducibility, runner integration, fixture corpus shape, and CLISpec reference:
docs/v2_posterior_ranking_residual.md(currently in PR #277, not yet merged). All four ratified decision asks from the 2026-04-29 sign-off are honored: three-slice sequencing, MRR threshold +0.05, ECE threshold 0.10, real-feedback retest deferred.Refs #151 (Slice 1)
Test plan
uv run python -m pytest tests/test_posterior_ranking_eval.py -v— 18 tests, all greenuv run python -m pytest -q(excluding known timeout-marker tests) — 1777 passed, 8 skippedaelf bench posterior-residual --fixtures benchmarks/posterior_ranking/fixtures/default.jsonl --seeds 1runs without errorSummary by Sourcery
Add a posterior-ranking evaluation harness that measures MRR uplift and calibration (ECE) over JSONL fixtures and exposes it via the benchmarking CLI.
New Features:
aelf bench posterior-residualbenchmark target with configurable thresholds, seeds, fixtures, and JSON output.Tests:
Summary by CodeRabbit
New Features
posterior-residualbenchmark target to the CLI for evaluating posterior ranking.Tests