Feat/issue 151 heat kernel composition - #316
Closed
robotrocketscience wants to merge 5 commits into
Closed
Conversation
Threads `eigenbasis_cache` + `heat_kernel_enabled` through `retrieve()` and `retrieve_with_tiers()` into `_l1_hits`. When the cache holds a non-stale eigenbasis and the flag is on, the L1 rerank dispatches to `combine_log_scores(bm25, heat, posterior_mean)` from #150's graph_spectral module instead of `partial_bayesian_score`. Heat propagation is one `eigvecs.T @ seeds` matvec per query, indexed by the eigenbasis row order. Heat-off path is byte-identical: kwargs default to None, the flag defaults False, and `_l1_hits` falls back to `partial_bayesian_score` when the cache is None / stale / empty / has no overlap with the L1 hit set. AC4 (flag-on identical to flag-off when no feedback exists) is preserved by the no-overlap fallback. Tests + bench wedge land in the next commit.
Five tests covering: - heat-off byte-identical to Slice 1 contract (AC4) - empty eigenbasis graceful degrade (AC4 fallback) - authority signal changes ranking on connected graph - cold-belief floor (newly inserted belief gets HEAT_SCORE_FLOOR) - store-mutation invalidation flips is_stale and degrades All five run against small fixtures (<=10 beliefs, K=200 eigenbasis); no N=50k bench in this commit (covered by the bench wedge in the next).
…2 of #151 Threads heat_kernel: bool through run() -> run_multi_seed() and _build_ece_observations(). When True, each per-seed retrieve() gets a fresh GraphEigenbasisCache with .build() called against that seed's store, rebuilt on stale (apply_feedback mutates the store -> invalidates cache). Heat-off remains the default; existing bench output is byte-identical when the flag is omitted. Smoke run on default.jsonl (n_seeds=2): MRR uplift +0.0000, ECE 0.1373, overall FAIL. Numerically identical to heat-off on this fixture set because the synthetic corpus carries no edges, so the heat term degrades to the floor for every belief and the ranking collapses to the BM25-only contract — graceful-degrade path is exercised end-to-end. A graph-bearing fixture set (slice 3 sweep) is what would surface real uplift.
Reproducible 50k-belief / 5k-edge benchmark for AC6. Times retrieve() heat-off vs heat-on with eigenbasis cache prebuilt, 30 calls (5 warmup, 25 measured), reports median/p90/max. Used to renegotiate AC6 from \xe2\x89\xa41ms / \xe2\x89\xa410ms to \xe2\x89\xa410ms / \xe2\x89\xa425ms against the measured retrieve() baseline (~7ms heat-off, ~19ms heat-on at N=50k).
Reviewer's GuideImplements Slice 2 of issue #151 by adding heat-kernel graph authority composition into L1 retrieval scoring behind a feature flag, wiring eigenbasis caches through retrieval APIs, extending benchmarking to measure heat-on vs heat-off behavior, documenting the new composition, and adding tests to validate graceful degradation and ranking effects. Sequence diagram for retrieve() with heat-kernel compositionsequenceDiagram
actor User
participant CLI as CLI_bench_posterior_residual
participant Bench as PosteriorRankingBench
participant Store as MemoryStore
participant Cache as GraphEigenbasisCache
participant Ret as RetrievalModule
User->>CLI: aelf bench posterior-residual --heat-kernel
CLI->>Bench: run_multi_seed(fixtures, heat_kernel=True)
Bench->>Bench: run_single_seed(..., heat_kernel=True)
Bench->>Store: _build_store(fixture, seed)
alt heat_kernel=True
Bench->>Cache: GraphEigenbasisCache(store, path)
Bench->>Cache: build()
end
loop per_round
Bench->>Bench: _refresh_caches()
Bench->>Cache: is_stale()
alt cache stale and heat_kernel=True
Bench->>Cache: build()
end
Bench->>Ret: retrieve(store, query, l1_limit, bfs_enabled=False, posterior_weight=None, heat_kernel_enabled=True, eigenbasis_cache=Cache)
activate Ret
Ret->>Ret: is_heat_kernel_enabled(heat_kernel_enabled)
Ret->>Ret: resolve_use_bm25f_anchors()
Ret->>Ret: resolve_posterior_weight()
Ret->>Ret: bfs_on, bm25f_on, heat_on
Ret->>Ret: _l1_hits(store, query, l1_limit, posterior_weight, use_bm25f_anchors=bm25f_on, bm25f_cache, eigenbasis_cache=Cache, heat_kernel_on=heat_on)
activate Ret
Ret->>Ret: heat_active = heat_kernel_on and eigenbasis_cache not None and not eigenbasis_cache.is_stale() and eigenbasis_cache.eigvals not None
alt use_bm25f_anchors=True
Ret->>Store: list_beliefs_for_bm25f()
Store-->>Ret: beliefs_with_raw_scores
else use_bm25f_anchors=False (FTS5)
Ret->>Store: search_beliefs_scored(query, l1_limit)
Store-->>Ret: beliefs_with_bm25_raw
end
alt heat_active and posterior_weight==0.0
Ret->>Ret: bm25_pos_by_id = {belief_id: bm25_pos}
Ret->>Ret: heat_map = _heat_by_id(Cache, bm25_pos_by_id)
Ret->>Ret: combine_log_scores(bm25_pos, heat, posterior)
else heat_active and posterior_weight>0.0
Ret->>Ret: bm25_pos_by_id = {belief_id: bm25_pos}
Ret->>Ret: heat_map = _heat_by_id(Cache, bm25_pos_by_id)
Ret->>Ret: combine_log_scores(bm25_pos, heat, posterior)
else not heat_active
Ret->>Ret: partial_bayesian_score(bm25_raw, alpha, beta, posterior_weight)
end
Ret-->>Bench: ranked_l1_beliefs
deactivate Ret
Bench-->>Bench: apply synthetic feedback
end
Bench->>Store: close()
Bench-->>CLI: MRR and ECE results
CLI-->>User: benchmark report
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Warning Rate limit exceeded
To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 0/1 reviews remaining, refill in 12 minutes and 9 seconds.Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The heat-kernel rerank logic in
_l1_hitsis duplicated between the BM25F and FTS5 branches; consider extracting a small helper that takes the BM25 magnitude per belief and returns the combined score to reduce divergence risk between the two paths. - In the perf and benchmark helpers (
ac6_50k.py,mrr_uplift.py,_build_ece_observations), you manually callTemporaryDirectory().cleanup(); using awith TemporaryDirectory(...) as tmp:context manager would simplify lifecycle management and avoid accidental leaks if early returns are added later. - In
benchmarks/posterior_ranking/ac6_50k.pyyou construct edges with the string literal"SUPPORTS"; using the sharedEDGE_SUPPORTSconstant there would keep the edge-type vocabulary consistent with the rest of the codebase.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The heat-kernel rerank logic in `_l1_hits` is duplicated between the BM25F and FTS5 branches; consider extracting a small helper that takes the BM25 magnitude per belief and returns the combined score to reduce divergence risk between the two paths.
- In the perf and benchmark helpers (`ac6_50k.py`, `mrr_uplift.py`, `_build_ece_observations`), you manually call `TemporaryDirectory().cleanup()`; using a `with TemporaryDirectory(...) as tmp:` context manager would simplify lifecycle management and avoid accidental leaks if early returns are added later.
- In `benchmarks/posterior_ranking/ac6_50k.py` you construct edges with the string literal `"SUPPORTS"`; using the shared `EDGE_SUPPORTS` constant there would keep the edge-type vocabulary consistent with the rest of the codebase.
## Individual Comments
### Comment 1
<location path="docs/bayesian_ranking.md" line_range="231" />
<code_context>
+
+### Bench wedge
+
+`aelf bench posterior-residual --heat-kernel` runs the MRR + ECE harness with the flag flipped on. Each per-seed `retrieve()` gets a fresh `GraphEigenbasisCache` built against that seed's in-memory store and rebuilt on stale (the synthetic feedback stream mutates the store after every round). Without `--heat-kernel`, output is byte-identical to today.
+
+### What Slice 2 still doesn't ship
</code_context>
<issue_to_address>
**suggestion (typo):** Clarify the phrase "rebuilt on stale" in the Bench wedge description.
Consider rephrasing to something like "rebuilt when stale" or "rebuilt once stale" to make the cache rebuild condition clearer.
```suggestion
`aelf bench posterior-residual --heat-kernel` runs the MRR + ECE harness with the flag flipped on. Each per-seed `retrieve()` gets a fresh `GraphEigenbasisCache` built against that seed's in-memory store and rebuilt when stale (the synthetic feedback stream mutates the store after every round). Without `--heat-kernel`, output is byte-identical to today.
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
|
|
||
| ### Bench wedge | ||
|
|
||
| `aelf bench posterior-residual --heat-kernel` runs the MRR + ECE harness with the flag flipped on. Each per-seed `retrieve()` gets a fresh `GraphEigenbasisCache` built against that seed's in-memory store and rebuilt on stale (the synthetic feedback stream mutates the store after every round). Without `--heat-kernel`, output is byte-identical to today. |
There was a problem hiding this comment.
suggestion (typo): Clarify the phrase "rebuilt on stale" in the Bench wedge description.
Consider rephrasing to something like "rebuilt when stale" or "rebuilt once stale" to make the cache rebuild condition clearer.
Suggested change
| `aelf bench posterior-residual --heat-kernel` runs the MRR + ECE harness with the flag flipped on. Each per-seed `retrieve()` gets a fresh `GraphEigenbasisCache` built against that seed's in-memory store and rebuilt on stale (the synthetic feedback stream mutates the store after every round). Without `--heat-kernel`, output is byte-identical to today. | |
| `aelf bench posterior-residual --heat-kernel` runs the MRR + ECE harness with the flag flipped on. Each per-seed `retrieve()` gets a fresh `GraphEigenbasisCache` built against that seed's in-memory store and rebuilt when stale (the synthetic feedback stream mutates the store after every round). Without `--heat-kernel`, output is byte-identical to today. |
Owner
Author
|
[claim:review:Kulili:2026-04-30T01:04:29Z] |
Owner
Author
|
Closing as superseded. Slice 2 of #151 (heat-kernel composition into log-additive ranking) already shipped via #310 (commit 4220874). This branch was opened against an older main and not rebased; merging it would delete ~3.3k lines of subsequently-landed work — deferred_feedback.py (#256), replay_full_equality (#304), per-turn hook audit log (#314), v2.0 corpus scaffold (#311), and the v2.0 posterior-ranking spec (#277). Net diff: +177 / -3292. The heat-kernel composition itself is already on main. No re-implementation needed. Once #313 (auto-rebase workflow) lands, drift this severe will be caught at PR-open time. |
Owner
Author
|
[release:review:Kulili:2026-04-30T01:04:44Z] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Linked issues
-->Type of change
feat:— new featurefix:— bug fixperf:— performance improvementrefactor:— code restructure with no behavior changetest:— test-only changedocs:— documentation-only changebuild:— build system / dependency / lockfile changeci:— CI workflow / hook changerelease:— version bump / release tagchore:— narrow housekeepingVerification
uv run pytest tests/ -x -q— all greenuv run pyright src/— strict, no new errorsuv run aelf --help— surface unchanged (or change documented)[Unreleased](if user-visible)Test plan
Notes for reviewer
Summary by Sourcery
Integrate heat-kernel graph authority into retrieval ranking behind a feature flag and wire it through retrieval, benchmarking, and CLI surfaces while preserving heat-off behavior.
New Features:
Enhancements:
Documentation:
Tests: