feat(eval-harness): wire context-rebuilder eval harness commit-1 (#592) - #596
Merged
Merged
Conversation
…#592) Replaces the NotImplementedError stub in benchmarks/context-rebuilder/ eval_harness.py with a working implementation: builds a fresh in-memory MemoryStore and ingests turns 0..fork_turn-1 from the case's JSONL via ingest_jsonl. Truncation is local — keeping it inside the harness avoids coupling the production ingest path to harness internals. Adds a sys.path shim so the script form (`python ...eval_harness.py`) can resolve aelfrice imports despite the hyphenated parent directory blocking `python -m`. 3 new tests in tests/test_context_rebuilder_eval_harness_wiring.py exercise: returns a populated MemoryStore, fork_turn boundary monotonic in belief count, fork_turn=0 returns an empty store.
Replaces the run_rebuilder stub with a real call to aelfrice.context_rebuilder.rebuild_v14. The harness's `trigger_threshold` sweep axis maps to the v1.7 per-lane composite-score floors (floor_session + floor_l1) — the closest existing rebuild-time knob to a "relevance threshold" sweep. L0 locked beliefs are unaffected by the floor per the documented v1.4 contract. Adds a `_recent_turns_pre_fork` helper that mirrors the production hook's adapter — reads pre-fork lines from the JSONL, skips compaction markers and malformed lines, returns the last DEFAULT_N_RECENT_TURNS as RecentTurn records ready for rebuild_v14. 3 new tests: returns (str, latency≥0); high floor packs ≤ low floor; empty store + empty recent_turns returns the empty block (rebuild_v14's silent-path contract).
Replaces the measure_token_cost stub with a real ratio: estimate_tokens(rebuilt) / estimate_tokens(pre_clear). Both sides use benchmarks.context_rebuilder.measure.estimate_tokens — the 4-chars-per- token heuristic that mirrors aelfrice.context_rebuilder._CHARS_PER_TOKEN so the harness measurement stays aligned with the rebuilder's own budget bookkeeping. Adds a `_pre_clear_text` helper that concatenates the `text` fields of turns 0..fork_turn-1. Empty pre-clear (fork_turn=0) returns 0.0 instead of dividing by zero — documented corner. 3 new tests: zero pre-clear → 0.0; smaller rebuilt → smaller ratio; shared-estimator parity (ratio matches estimate_tokens(rebuilt) / estimate_tokens(pre_clear) byte-for-byte).
Replaces the NotImplementedError with placeholder records so that
threshold-sweep and budget-sweep modes can run end-to-end producing
latency and token-cost numbers, even before the model-invocation client
lands.
Each placeholder is `{turn_idx, expected, actual="", matched=False,
reason=REPLAY_PENDING_REASON}`. The `expected` field is pulled from
the captured transcript at the eval-turn line index so summaries can
still surface what the agent should have continued with. score_fidelity
reads `matched=False` and returns 0.0 — no crash, just a documented
floor until the model client wires up.
The REPLAY_PENDING_REASON marker is exported as a public-ish constant
so summary readers can distinguish "skipped, judge required" from a
real fidelity miss. The model-client follow-up will swap the stub for
a real replay; the placeholder shape is preserved so downstream
consumers don't change.
4 new tests: one placeholder per eval_turn; expected pulled from
transcript; empty eval_turns → []; score_fidelity returns 0.0 instead
of crashing.
The eval harness's load_corpus walks `<corpus>/*.jsonl` and pairs each
with a same-stem `.meta.json` carrying task_type, fork_turn, and
eval_turns. The shipped synthetic fixture had no meta file, so the
harness loaded zero cases out of the box.
Adds debugging_session_001.meta.json: midpoint fork at turn 8 of the
16-turn fixture, eval_turns = the four post-fork user prompts (indices
8/10/12/14). Operators can now run:
uv run python benchmarks/context-rebuilder/eval_harness.py \\
--mode threshold-sweep \\
--corpus benchmarks/context-rebuilder/fixtures/synthetic/ \\
--out benchmarks/context-rebuilder/results/sweep.json
against the bundled fixture without setting up a separate corpus.
Two new tests cover the integration of all four wired functions (replay_to_fork, run_rebuilder, measure_token_cost, replay_post_fork stub) plus the corpus-loader path against the bundled synthetic fixture: - threshold_sweep_runs_end_to_end_on_synthetic_fixture: loads the fixture via load_corpus, runs a 2-threshold sweep, asserts the JSON output schema (mode, runs[*].fidelity/latency/token_cost_ratio, summary[task][threshold=...] metric keys). - run_one_returns_runresult_without_crashing: verifies run_one threads through replay → rebuild → replay-stub → score without raising and surfaces the REPLAY_PENDING_REASON marker on every unmatched eval_turn so summary readers can distinguish "skipped, judge required" from real fidelity misses. Locks in the harness's runnable contract before the model-invocation client lands.
Documents the three pure-Python integration points wired in this PR (replay_to_fork, run_rebuilder, measure_token_cost), the non-crashing replay_post_fork stub, and the synthetic-fixture meta.json that makes the bundled corpus resolve out of the box. Calls out the explicit follow-ups: model-invocation client and LLM judge.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @robotrocketscience, you have reached your weekly rate limit of 2500000 diff characters.
Please try again later or upgrade to continue using Sourcery
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
robotrocketscience
added
author-Maxwell
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+37
to
+42
| from aelfrice.context_rebuilder import ( # noqa: E402 | ||
| DEFAULT_N_RECENT_TURNS, | ||
| DEFAULT_REBUILDER_TOKEN_BUDGET, | ||
| RecentTurn, | ||
| rebuild_v14, | ||
| ) |
Owner
Author
|
[claim:review:Leibniz:2026-05-10T15:13:25Z] |
Owner
Author
|
[claim:review:planck:2026-05-10T15:14:25Z] |
Owner
Author
|
[release:review:planck:2026-05-10T15:14:30Z] |
Owner
Author
|
Merged via local FF push to main as 907c5a6. All CI green, 7 commits SSH-signed, discretion clean. Reviewed by Leibniz. |
Owner
Author
|
[release:review:Leibniz:2026-05-10T15:15:03Z] |
This was referenced May 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes part of #592 (commit 1 of the proposed three).
What lands
Wires three of the five
NotImplementedErrorintegration points inbenchmarks/context-rebuilder/eval_harness.py, plus a non-crashing stubfor the fourth, plus a
.meta.jsonfor the bundled synthetic fixtureso
--mode threshold-sweepruns end-to-end out of the box.replay_to_forkNotImplementedErrorMemoryStore, ingests turns 0..fork_turn-1 viaingest_jsonlrun_rebuilderrebuild_v14withfloor_session=floor_l1=trigger_threshold; returns(block, latency_ms)measure_token_costestimate_tokens(rebuilt) / estimate_tokens(pre_clear)via the shared 4-chars-per-token estimatorreplay_post_forkreason="needs_replay_client"soscore_fidelityreturns 0.0 instead of crashingscore_fidelitymatched=FalserowsThreshold-sweep semantics
The harness's
trigger_thresholdaxis maps torebuild_v14's v1.7per-lane composite-score floors (
floor_session+floor_l1). L0locked beliefs are unaffected per the documented v1.4 contract; the
sweep answers "at what relevance floor does the rebuilder still pack
useful content for this task family?" — which is the closest existing
rebuild-time knob to the harness's "threshold" wording. The original
issue body framed the threshold as a compaction-trigger, but no such
knob exists on
main; mapping to floors is the implementable v0that produces meaningful sweep curves today. This is documented in
the
run_rebuilderdocstring.Atomic commits
7 SSH-signed commits in dependency order:
feat(eval-harness): wire replay_to_fork to MemoryStore + ingest_jsonl (#592)feat(eval-harness): wire run_rebuilder to rebuild_v14 (#592)feat(eval-harness): wire measure_token_cost via shared estimator (#592)feat(eval-harness): non-crashing replay_post_fork stub (#592)feat(eval-harness): add meta.json for synthetic fixture (#592)test(eval-harness): end-to-end smoke for --mode threshold-sweep (#592)docs(changelog): unreleased entry for #592 eval-harness commit-1 wiringVerification
uv run pytest -x -q→ 3282 passed, 53 skipped (3267 baseline + 15 new).uv run python benchmarks/context-rebuilder/eval_harness.py --mode threshold-sweep --corpus benchmarks/context-rebuilder/fixtures/synthetic/ --out /tmp/sweep.jsonproduces a valid JSON report with one case, five runs (default 5 thresholds), summary keyed by task_type then threshold.Acceptance progress
--mode threshold-sweepruns to completion againsteval_corpus/-equivalent (the bundled synthetic fixture). Producesresults/sweep_<date>.json.task_type=hot_startscenario added to the eval corpus. Deferred to follow-up — depends on the<working-state>rebuilder sub-block from feat(rebuilder): post-compact hot-start — preserve working-state delta alongside retrieval #587 / PR feat(rebuilder): post-compact <working-state> hot-start (#587) #591 landing first.Out of scope (explicit follow-ups)
replay_post_forkreal impl): Anthropic SDK wiring + per-turn replay against the rebuilt block. Tracked as commit 2 of feat(eval-harness): wire context-rebuilder eval harness for #587 hot-start scoring #592.judges/llm_judge.py): cost-bounded open-ended scorer for non-string-match eval turns. Commit 3 of feat(eval-harness): wire context-rebuilder eval harness for #587 hot-start scoring #592, default-off in CI.main.run_onestore leak: the function callsreplay_to_fork(case)but never closes the returned store. With in-memory SQLite the leak is bounded by GC, but a focused fix would add atry/finallyaround the rebuild + replay calls. Filed as a separate small PR if the reviewer wants atomic-safety.