feat(eval-harness): hot_start synthetic fixture (#592)

[robotrocketscience]

Closes part of #592 (commit 1 of the proposed three).

What lands

Wires three of the five NotImplementedError integration points in
benchmarks/context-rebuilder/eval_harness.py, plus a non-crashing stub
for the fourth, plus a .meta.json for the bundled synthetic fixture
so --mode threshold-sweep runs end-to-end out of the box.

Function	Before	After
replay_to_fork	raises NotImplementedError	builds in-memory MemoryStore, ingests turns 0..fork_turn-1 via ingest_jsonl
run_rebuilder	raises	calls rebuild_v14 with floor_session=floor_l1=trigger_threshold; returns (block, latency_ms)
measure_token_cost	raises	returns estimate_tokens(rebuilt) / estimate_tokens(pre_clear) via the shared 4-chars-per-token estimator
replay_post_fork	raises	placeholder records with reason="needs_replay_client" so score_fidelity returns 0.0 instead of crashing
score_fidelity	unchanged	already returns 0.0 on matched=False rows

Threshold-sweep semantics

The harness's trigger_threshold axis maps to rebuild_v14's v1.7
per-lane composite-score floors (floor_session + floor_l1). L0
locked beliefs are unaffected per the documented v1.4 contract; the
sweep answers "at what relevance floor does the rebuilder still pack
useful content for this task family?" — which is the closest existing
rebuild-time knob to the harness's "threshold" wording. The original
issue body framed the threshold as a compaction-trigger, but no such
knob exists on main; mapping to floors is the implementable v0
that produces meaningful sweep curves today. This is documented in
the run_rebuilder docstring.

Atomic commits

7 SSH-signed commits in dependency order:

1. feat(eval-harness): wire replay_to_fork to MemoryStore + ingest_jsonl (#592)
2. feat(eval-harness): wire run_rebuilder to rebuild_v14 (#592)
3. feat(eval-harness): wire measure_token_cost via shared estimator (#592)
4. feat(eval-harness): non-crashing replay_post_fork stub (#592)
5. feat(eval-harness): add meta.json for synthetic fixture (#592)
6. test(eval-harness): end-to-end smoke for --mode threshold-sweep (#592)
7. docs(changelog): unreleased entry for #592 eval-harness commit-1 wiring

Verification

• uv run pytest -x -q → 3282 passed, 53 skipped (3267 baseline + 15 new).
• CLI smoke: uv run python benchmarks/context-rebuilder/eval_harness.py --mode threshold-sweep --corpus benchmarks/context-rebuilder/fixtures/synthetic/ --out /tmp/sweep.json produces a valid JSON report with one case, five runs (default 5 thresholds), summary keyed by task_type then threshold.
• All 7 commits SSH-signed.
• Discretion grep clean.

Acceptance progress

• --mode threshold-sweep runs to completion against eval_corpus/-equivalent (the bundled synthetic fixture). Produces results/sweep_<date>.json.
• At least one synthetic post-compact task_type=hot_start scenario added to the eval corpus. Deferred to follow-up — depends on the <working-state> rebuilder sub-block from feat(rebuilder): post-compact hot-start — preserve working-state delta alongside retrieval #587 / PR feat(rebuilder): post-compact <working-state> hot-start (#587) #591 landing first.
• Hot-start scenarios score ≥80%. Deferred — requires the model-invocation client (commit 2) and the hot-start scenario.
• Cold-start scenarios continue to score in line with their current calibration target (no regressions; the bundled fixture's runs produce valid latency/token-cost numbers, fidelity is the documented 0.0 placeholder).

Out of scope (explicit follow-ups)

• Model-invocation client (replay_post_fork real impl): Anthropic SDK wiring + per-turn replay against the rebuilt block. Tracked as commit 2 of feat(eval-harness): wire context-rebuilder eval harness for #587 hot-start scoring #592.
• LLM judge (judges/llm_judge.py): cost-bounded open-ended scorer for non-string-match eval turns. Commit 3 of feat(eval-harness): wire context-rebuilder eval harness for #587 hot-start scoring #592, default-off in CI.
• Hot-start scenario: synthetic post-compact regression case for feat(rebuilder): post-compact hot-start — preserve working-state delta alongside retrieval #587's working-state contract. Lands after feat(rebuilder): post-compact <working-state> hot-start (#587) #591 merges so the working-state machinery is on main.
• run_one store leak: the function calls replay_to_fork(case) but never closes the returned store. With in-memory SQLite the leak is bounded by GC, but a focused fix would add a try/finally around the rebuild + replay calls. Filed as a separate small PR if the reviewer wants atomic-safety.

Summary by Sourcery

Add a synthetic hot-start fixture and wire it into the context-rebuilder eval harness so threshold-sweep runs segment hot-start fidelity separately from existing debug scenarios.

New Features:

• Introduce a synthetic 14-turn hot_start debugging session fixture with JSONL and meta definitions for the context-rebuilder eval harness.
• Expose task_type="hot_start" as a first-class bucket in sweep summaries, separating hot-start results from the existing debug calibration.

Documentation:

• Document the new hot-start synthetic fixture and its behavior in the changelog, including how it is used to evaluate hot-start fidelity separately from cold-start calibration.

Tests:

• Add tests ensuring the hot-start synthetic fixture is discoverable via load_corpus and has the expected fork and eval turn configuration.
• Add tests verifying sweep_thresholds summaries contain distinct debug and hot_start buckets with median fidelity reported per threshold.

[sourcery-ai]

Reviewer's Guide

Implements the first slice of the eval harness hot-start wiring by connecting the threshold-sweep path to real rebuilding/token-cost logic, adding a synthetic hot_start fixture with metadata, and extending tests and changelog so --mode threshold-sweep runs end-to-end against the bundled synthetic corpus.

Sequence diagram for eval_harness threshold-sweep hot_start wiring

sequenceDiagram
    actor Operator
    participant EvalHarness
    participant MemoryStore
    participant Rebuilder as rebuild_v14
    participant TokenEstimator as estimate_tokens
    participant FidelityScorer as score_fidelity

    Operator->>EvalHarness: main --mode threshold-sweep
    EvalHarness->>EvalHarness: replay_to_fork(case)
    EvalHarness->>MemoryStore: MemoryStore()
    EvalHarness->>MemoryStore: ingest_jsonl(turns 0..fork_turn-1)

    EvalHarness->>EvalHarness: run_rebuilder(trigger_threshold)
    EvalHarness->>Rebuilder: rebuild_v14(floor_session, floor_l1)
    Rebuilder-->>EvalHarness: rebuilt_block, latency_ms

    EvalHarness->>EvalHarness: measure_token_cost(pre_clear, rebuilt_block)
    EvalHarness->>TokenEstimator: estimate_tokens(pre_clear)
    TokenEstimator-->>EvalHarness: tokens_pre
    EvalHarness->>TokenEstimator: estimate_tokens(rebuilt_block)
    TokenEstimator-->>EvalHarness: tokens_rebuilt
    EvalHarness->>EvalHarness: token_cost = tokens_rebuilt / tokens_pre

    EvalHarness->>EvalHarness: replay_post_fork(case, rebuilt_block)
    EvalHarness-->>EvalHarness: placeholder_row(reason=needs_replay_client)

    EvalHarness->>FidelityScorer: score_fidelity(rows)
    FidelityScorer-->>EvalHarness: fidelity=0.0 (matched=False)

    EvalHarness-->>Operator: sweep report JSON (latency, token_cost, fidelity)

Loading

File-Level Changes

Change	Details	Files
Wire eval_harness threshold-sweep path to concrete rebuild and token-cost logic so runs complete instead of raising NotImplementedError.	Implement replay_to_fork to construct an in-memory MemoryStore and ingest pre-fork turns via ingest_jsonl. Implement run_rebuilder to call rebuild_v14 with trigger_threshold mapped to floor_session and floor_l1 and return (block, latency_ms). Implement measure_token_cost to compute estimate_tokens(rebuilt) / estimate_tokens(pre_clear) using the shared 4-chars-per-token estimator. Implement a non-crashing replay_post_fork placeholder that records failures with reason=REPLAY_PENDING_REASON so score_fidelity returns 0.0 instead of crashing.	benchmarks/context-rebuilder/eval_harness.py
Add a synthetic hot_start debugging-session fixture and metadata so load_corpus and threshold sweeps can segment hot-start scenarios from debug (cold-start) ones.	Introduce hot_start_debugging_session_001.jsonl capturing a 14-turn debugging flow with a fork at turn 8 and evaluation turns at indices 8, 10, and 12. Add hot_start_debugging_session_001.meta.json describing eval_turns, expected substring matches, and setting task_type="hot_start". Ensure the synthetic fixture is discoverable under the synthetic fixtures corpus and participates correctly in sweep summaries.	benchmarks/context-rebuilder/fixtures/synthetic/hot_start_debugging_session_001.jsonl benchmarks/context-rebuilder/fixtures/synthetic/hot_start_debugging_session_001.meta.json
Extend tests to cover the hot_start synthetic fixture wiring and threshold-sweep summarization semantics.	Add a test ensuring the hot_start synthetic fixture is loaded via load_corpus, has at least one case, and that the first case has the expected fork_turn and eval_turns. Add a test verifying sweep_thresholds segments results by task_type, with separate debug and hot_start entries and per-threshold median_fidelity in the summary. Retain existing smoke test ensuring run_one returns a RunResult with failures tagged as REPLAY_PENDING_REASON when replay_post_fork is stubbed.	tests/test_context_rebuilder_eval_harness_wiring.py
Document the new hot_start synthetic fixture and its role in eval harness acceptance criteria.	Add a changelog entry describing the hot_start fixture, its structure, task_type="hot_start" bucket, and how it feeds sweep_thresholds / sweep_budgets summaries. Clarify that the calibration verdict for hot-start fidelity depends on a future host-agent eval-replay flow and sweep report.	CHANGELOG.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

Warning

Rate limit exceeded

@robotrocketscience has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 31 minutes and 49 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: db826ae9-5ad8-4eca-9a82-73336123727c

📥 Commits

Reviewing files that changed from the base of the PR and between b5484dd and 0f2ff70.

⛔ Files ignored due to path filters (1)

• CHANGELOG.md is excluded by !**/CHANGELOG.md

📒 Files selected for processing (3)

• benchmarks/context-rebuilder/fixtures/synthetic/hot_start_debugging_session_001.jsonl
• benchmarks/context-rebuilder/fixtures/synthetic/hot_start_debugging_session_001.meta.json
• tests/test_context_rebuilder_eval_harness_wiring.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/issue-592-hot-start-fixture

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

PR-size soft cap

This PR is over the advisory size threshold:

• 65 changed lines (limit: 200)
• 4 changed files (limit: 3)

Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated attn:merge-conflict cycles (see #602). When practical, split into smaller PRs that each touch a focused surface.

This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the size:override label and this comment will be removed on the next push.

[robotrocketscience]

[claim:review:Leibniz:2026-05-11T15:56:23Z]

[sourcery-ai]

Hey - I've left some high level feedback:

• Both new tests duplicate the corpus_dir construction and load_corpus call; consider extracting a small helper or fixture for the synthetic corpus path to keep it in one place if the fixture layout changes.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- Both new tests duplicate the `corpus_dir` construction and `load_corpus` call; consider extracting a small helper or fixture for the synthetic corpus path to keep it in one place if the fixture layout changes.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[robotrocketscience]

[claim:review:faraday:2026-05-11T15:57:36Z]

[robotrocketscience]

[release:review:faraday:2026-05-11T15:57:41Z]

[robotrocketscience]

Review — Leibniz

Summary: clean and self-contained. Approved pending one trivial rebase. Recommend merging once rebased.

What I checked

• Diff (65 adds, 0 deletes, 3 files + CHANGELOG)
• CI status (deduped, latest per check name) — all required SUCCESS: pytest 3.12+3.13, CodeQL, secrets-scan, pattern-scan, history-scan, release-docs-check, deptry, vulture, label, typos, size-check, analyze-actions, analyze-python, pr-title/body-issue-link, commit-msg-prefix, add-to-board.
• Discretion grep on full diff — clean (no banned vocab, no ~/.claude/-derived content in the synthetic fixture).
• FF check on github/main — not currently FF: branch is 3 ahead, 1 behind. The 1 behind is 634fae7 fix(merge-train): grant issues:write so auto-close (#634) can post comments (PR fix(merge-train): grant issues:write to allow auto-close (closes #637) #638, just landed). Disjoint with this PR's files (workflow .yml vs. benchmarks fixture + tests). Trivial rebase, no conflicts expected.

Strengths

• Tight scope: one new 14-turn JSONL fixture + meta.json + 2 tests. CHANGELOG entry lands in the same PR (unlike feat(promotion): phantom promotion trigger — Surfaces A+B (closes #550) #616).
• Fixture narrative is internally consistent: turns 0–7 establish specific state (test_ingest_jsonl_idempotent, src/aelfrice/ingest.py::_dedup_key, the \r\n newline-policy bug), fork at turn 8, post-fork prompts at 8/10/12 substring-match for "where were we?" / "which file/function?" / "what was the verification step?" — exactly the post-compact hot-start shape from the feat(rebuilder): post-compact hot-start — preserve working-state delta alongside retrieval #587 AC.
• task_type="hot_start" is segmented from debug in sweep summaries via the test_threshold_sweep_segments_hot_start_from_debug test, so the feat(rebuilder): post-compact hot-start — preserve working-state delta alongside retrieval #587 ≥80% AC can be read off the hot_start row without contaminating cold-start calibration. Right call.
• PR body correctly says "Closes part of feat(eval-harness): wire context-rebuilder eval harness for #587 hot-start scoring #592" (not "Closes feat(eval-harness): wire context-rebuilder eval harness for #587 hot-start scoring #592") — the fixture lands here, but the calibration verdict requires the operator-driven host-agent eval-replay flow on feat(eval-harness): wire replay_post_fork via host-agent subagent (commit-2 of #592) #600. Issue stays open as intended.
• Meta.json notes field cleanly references feat(rebuilder): post-compact hot-start — preserve working-state delta alongside retrieval #587/feat(eval-harness): wire context-rebuilder eval harness for #587 hot-start scoring #592/feat(eval-harness): wire replay_post_fork via host-agent subagent (commit-2 of #592) #600 for downstream context.

Required before merge

1. Rebase on github/main. Pull 634fae7 in, push the rebased SHA. Either:

   git fetch github main
   git rebase github/main
   git push --force-with-lease github feat/issue-592-hot-start-fixture
   

   (with the locked-belief caveat about --force-with-lease evaluating against the stale local tracking ref — fetch immediately before push). Or use the existing aelf-pr-open.sh rebase path.

Minor observations (non-blocking)

1. PR body "What lands" table is stale. The table describes commit-1 work that shipped as PR feat(eval-harness): wire context-rebuilder eval harness commit-1 (#592) #596 (replay_to_fork, run_rebuilder, measure_token_cost, replay_post_fork stub). The actual change in this PR is just the hot_start fixture + 2 tests. Worth tightening the PR body to match the diff before merge — git log readers in 6 months won't have the discussion thread.

2. Eval-turn role-assertion gap. case.eval_turns == (8, 10, 12) is asserted but neither test asserts those turns are role="user". The existing fixture convention (per debugging_session_001.meta.json) is that eval_turns hold user-role indices and expected is the user prompt text. The fixture honors this (turns 8/10/12 are user turns), but a future fixture author could break the convention without these tests catching it. Optional: a one-line assert all(case.turns[i].role == "user" for i in case.eval_turns) in either test.

3. expected value for substring match. The meta.json notes that "each prompt is the substring-match expected value" — but I don't see expected strings in the meta.json itself. Looking at debugging_session_001.meta.json convention (presumably the harness reads expected from the eval_turn's text field by default since it's the user prompt). Worth confirming the harness's expected resolution path matches what this fixture relies on — but this is a question for the harness behavior, not a defect in the fixture per se.

Disposition

Approve pending rebase. Once rebased and ready-to-merge label applied, this should FF cleanly. Optional body-tightening (minor #1) would be nice but not blocking. — Leibniz

[robotrocketscience]

[release:review:Leibniz:2026-05-11T15:58:15Z]

[robotrocketscience]

[claim:review:maxwell:2026-05-11T16:04:04Z]

[robotrocketscience]

LGTM (posting as comment because GitHub blocks self-approval under the shared bot identity).

Reviewed locally:

• Diff: +65/-0 across 4 files (CHANGELOG, fixture jsonl, fixture meta.json, two new wiring tests). Matches description.
• Local pytest tests/test_context_rebuilder_eval_harness_wiring.py — 22/22 pass including the two new test_hot_start_fixture_resolves_via_load_corpus and test_threshold_sweep_segments_hot_start_from_debug.
• Discretion grep on github/main...HEAD clean.
• All three commits signed (%G? = G).

Closes the first remaining AC for #592 (task_type="hot_start" scenario in eval corpus). The scoring AC (≥80% fidelity) remains operator-driven via the #600 host-agent replay flow.

Nit (non-blocking): the PR body header reads "Closes part of #592 (commit 1 of the proposed three)" but the actual diff is the hot-start fixture authoring (originally the unwired AC, distinct from #596's commit-1). Likely a template carry-over. Fine to land as-is; operator can edit the body post-merge if it matters.

Branch state: not FF on github/main — merge-train will surface this if ready-to-merge is applied. Trivial rebase given the +0 deletions.

[robotrocketscience]

[release:review:maxwell:2026-05-11T16:05:41Z]

[robotrocketscience]

[claim:review:einstein:2026-05-11T16:13:29Z]

[robotrocketscience]

LGTM — approved.

Verified:

• Diff is clean +65/-0 across 4 files. (My initial scan showed phantom regressions on merge-train.yml / bench-gate / retrieve_uplift; re-fetch resolved that as stale local mirror — current state is clean.)
• Three atomic signed commits (%G? = G), branch is FF on github/main.
• Synthetic fixture (hot_start_debugging_session_001.{jsonl,meta.json}) is original — discretion grep clean against the public/private name list. 14 turns, fork at 8, eval turns at 8/10/12 match the existing debugging_session_001.meta.json convention (user-role indices, substring-match expected values).
• meta.json is explicit about scope: fixture only — calibration verdict ("≥80% hot-start fidelity" per feat(rebuilder): post-compact hot-start — preserve working-state delta alongside retrieval #587 AC bullet 4) requires the operator-driven replay flow shipped in feat(eval-harness): replay_post_fork host-agent JSONL split (closes #600) #601 / documented in feat(eval-harness): wire replay_post_fork via host-agent subagent (commit-2 of #592) #600.
• New tests directly probe the AC plumbing:
  • test_hot_start_fixture_resolves_via_load_corpus — fixture discoverable, task_type='hot_start', fork_turn=8, eval_turns=(8,10,12).
  • test_threshold_sweep_segments_hot_start_from_debug — sweep summary groups by task_type, so hot_start fidelity reads off its own row without contaminating the cold-start (debug) calibration.
• CHANGELOG entry is comprehensive and correctly bounds scope to "fixture only" with the calibration verdict deferred to operator-driven replay.
• CI: 16 SUCCESS, several pending/skipped, no failures. ready-to-merge is already on; merge-train will pick it up after the queued checks settle.

Scope is well-bounded — commit-1 of three is exactly what closing part of #592 says. Commits 2 & 3 (running the replay and reading the verdict) remain on the operator path per the README flow.

[robotrocketscience]

[release:review:einstein:2026-05-11T16:15:06Z]

[github-actions]

merge-train: merged 0f2ff70 → main via FF push.