fix(bench): debug fixture meta — eval_turns point at canonical answers (#687)

[robotrocketscience]

Summary

Closes #687 and unblocks #592.

Two-part change for the v3.0 release gate:

1. Code: flip debugging_session_001.meta.json eval_turns from [8, 10, 12, 14] (user-prompt indices, expected = user_question_text) to [9, 11, 13, 15] (assistant-continuation indices, expected = canonical answer). Mirrors PR feat(eval-harness): hot_start synthetic fixture (#592) #639's analogous fix to the hot_start fixture.

2. Bench artifact for eval-harness: Cohen's-κ judge calibration before #592 close #687 calibration (run on the patched corpus, posted below as the closing evidence). Bench artifacts remain uncommitted per project policy.

Why

Without the fixture-meta flip, the LLM judge was being asked to compare candidate answers against user questions — the speech-act split that drove inter_judge_kappa=0.0 on the prior single-judge hot_start run, documented in #687's body and the hot_start meta notes field. The earlier hot_start patch (PR #639) noted the debug fixture would be flipped alongside once the cold-start sweep was re-ratified separately; this is that follow-up.

κ artifact — 687_hot_start_t_le_0.6 (calibrated cut)

3 independent Sonnet 4.6 judges, hot-start rows at trigger_threshold ≤ 0.6, 6 pairs aligned. --judge-model "claude-sonnet-4-6 (anchor tier per llm_judge.py)". Baseline: score_substring_exact_match.

{
  "run_id": "687_hot_start_t_le_0.6",
  "n_runs": 3,
  "n_pairs": 6,
  "judge_model": "claude-sonnet-4-6 (anchor tier per llm_judge.py)",
  "baseline": "score_substring_exact_match",
  "inter_judge_kappa": {
    "judge_run1_hot_t06_vs_judge_run2_hot_t06": 1.0,
    "judge_run1_hot_t06_vs_judge_run3_hot_t06": 1.0,
    "judge_run2_hot_t06_vs_judge_run3_hot_t06": 1.0,
    "mean": 1.0,
    "min": 1.0
  },
  "judge_vs_baseline_kappa": {
    "judge_run1_hot_t06": 0.0,
    "judge_run2_hot_t06": 0.0,
    "judge_run3_hot_t06": 0.0,
    "mean": 0.0
  },
  "per_run_hot_start_fidelity": [1.0, 1.0, 1.0],
  "hot_start_fidelity_mean": 1.0,
  "calibrated": true,
  "failure_reasons": []
}

Calibration gates

gate (per docs/BENCHMARKS.md §Eval-judge calibration + kappa.py thresholds)	result
inter_judge_kappa.min ≥ 0.70	1.0 ✓
hot_start_fidelity_mean ≥ 0.80	1.0 ✓
n_runs ≥ 3	3 ✓
calibrated	true ✓

Judge-vs-baseline κ = 0.0 across all runs because the substring baseline matched zero rows (every candidate paraphrases rather than substring-quoting the reference), while the judges agreed on 10/35 matches. This is the expected "the judge is earning its API cost" pattern documented in kappa.py's docstring — judge-vs-baseline is report-only, not a gate.

κ artifact — 687_all_35_rows (informational)

Full 35-row sweep across both fixtures and all 5 trigger_threshold values:

{
  "run_id": "687_all_35_rows",
  "n_runs": 3,
  "n_pairs": 35,
  "inter_judge_kappa": {
    "judge_responses_run1_vs_judge_responses_run2": 1.0,
    "judge_responses_run1_vs_judge_responses_run3": 1.0,
    "judge_responses_run2_vs_judge_responses_run3": 1.0,
    "mean": 1.0,
    "min": 1.0
  },
  "judge_vs_baseline_kappa": {"mean": 0.0},
  "per_run_hot_start_fidelity": [0.2857, 0.2857, 0.2857],
  "hot_start_fidelity_mean": 0.2857,
  "calibrated": false,
  "failure_reasons": ["hot_start_fidelity_mean=0.2857 < threshold 0.8"]
}

The informational run intentionally fails the calibration gate — at trigger_threshold ∈ {0.7, 0.8, 0.9} the rebuilder doesn't fire and the candidate hedges with "I don't have that in context." This is the expected behaviour the #587 AC qualifies with "at threshold ≤0.6"; the calibrated artifact above is the AC-aligned cut.

Per-(task, threshold) fidelity (mean across 3 judge runs)

config	n_rows	mean_fidelity	notes
hot_start t=0.5	3	1.00	rebuilder fires, all 3 questions answered
hot_start t=0.6	3	1.00	rebuilder fires, all 3 questions answered
hot_start t=0.7	3	0.00	rebuilder doesn't fire, candidate hedges
hot_start t=0.8	3	0.00	rebuilder doesn't fire
hot_start t=0.9	3	0.00	rebuilder doesn't fire
cold_start (debug) t=0.5	4	0.25	XML-format Q matched; idempotency/determinism/release-version Qs hedged
cold_start (debug) t=0.6	4	0.25	same
cold_start (debug) t=0.7	4	0.25	same
cold_start (debug) t=0.8	4	0.25	same
cold_start (debug) t=0.9	4	0.00	rebuilder empty, all hedged

Cold-start (debug) fidelity is 25% at low thresholds and 0% at t=0.9 — substantially below the patched-fixture pass rate the prior single-judge run reported (~75%) because the patched expected is the canonical assistant answer rather than the user prompt. This is the calibration improvement the #687 fixture-meta work was designed to deliver: cold-start scores now reflect actual continuation fidelity, not subject-match leniency. Cold-start AC isn't gated on a specific number for v3.0; this PR re-establishes the cold-start baseline at 25% at t≤0.8, 0% at t=0.9 for #592 future reference.

Methodology

1. Patched debugging_session_001.meta.json (this PR's single commit).
2. Ran the eval harness against benchmarks/context-rebuilder/fixtures/synthetic/ with --mode threshold-sweep --run-dir, producing 10 case-config dirs × 35 replay_requests.jsonl rows total.
3. Replay step: 1 Sonnet 4.6 host-subagent generated all 35 actual continuations from each row's rebuilt_block + user_turn under explicit no-cross-row-contamination instructions per judges/llm_judge.py §"Contamination protocol".
4. Re-ran harness to score replay (substring step: 0 substring-exact matches; all 35 rows tagged needs_llm_judge).
5. Dispatched 3 independent Sonnet 4.6 host-subagents over the same judge_requests.jsonl (35 rows × 3 = 105 judge calls). Each saw only (turn_idx, expected, actual) per the contamination protocol.
6. Built deterministic baseline via benchmarks/qa_scoring.score_substring_exact_match.
7. Ran python -m benchmarks.context_rebuilder.kappa on the AC-aligned subset (hot-start at t≤0.6) and on the full 35-row sweep.

All 3 judge runs produced identical verdicts (10/35 matched, same turn_idx set: 1, 5, 9, 13, 20, 21, 22, 23, 24, 25). The κ=1.0 across all run-pairs reflects that determinism within the Sonnet 4.6 judge tier for these prompts — agents on this fixture's signal/noise distribution land on the same verdict.

Sample-size caveat

The docs/BENCHMARKS.md §Eval-judge calibration doc already flags: at N=3 over ~18 deduplicated pairs the 95% CI on a 0.70 point estimate spans ≈ 0.45–0.90. The hot-start calibrated cut here is 6 pairs (3 turns × 2 thresholds), even tighter. A pre-release N=5 ratification run is still recommended as future work; this PR ships the gate-pass for v3.0 only.

Local verification

uv run pytest tests/test_context_rebuilder_eval_harness_wiring.py tests/test_context_rebuilder_harness.py tests/test_continuation_fidelity_scorer.py tests/test_rebuilder_triggers.py tests/test_eval_harness.py tests/test_eval_baseline.py -q
99 passed in 1.02s

No assertion in tests references the debug fixture's eval_turns shape (only the hot_start shape is asserted at test_context_rebuilder_eval_harness_wiring.py:534); the patch is invariant under the existing test suite.

Refs

• Closes eval-harness: Cohen's-κ judge calibration before #592 close #687
• Unblocks feat(eval-harness): wire context-rebuilder eval harness for #587 hot-start scoring #592 (gate-prereq satisfied)
• Mirrors PR feat(eval-harness): hot_start synthetic fixture (#592) #639 fixture-meta methodology
• Tracker line: v3.0 Milestone tracker — wonder lifecycle complete + HRR persistence + compression + design cuts #608 "Bench-only items [retrieval] HRR structural-query lane (bind/probe over outgoing edges) #152, feat(eval-harness): wire context-rebuilder eval harness for #587 hot-start scoring #592 (gated on eval-harness: Cohen's-κ judge calibration before #592 close #687), test(hrr): cold-start bench gate at N=50k (#553 sub-task) #697 have captured run artifacts"
• Release PR: release: v3.0.0 #717 substantive checklist item

Bench artifacts (run dirs, judge JSONLs, kappa JSONs) are local-only at /tmp/592_run/ per project policy of "uncommitted run artifacts".

Summary by CodeRabbit

• Chores
  • Updated internal test fixture metadata to reflect changes in evaluation indexing semantics.

[sourcery-ai]

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c4c118f0-4745-4a53-9dfb-bfa5f2836252

📥 Commits

Reviewing files that changed from the base of the PR and between 147b484 and 0cc251f.

📒 Files selected for processing (1)

• benchmarks/context-rebuilder/fixtures/synthetic/debugging_session_001.meta.json

---

📝 Walkthrough

Walkthrough

Updated eval_turns indices from [8, 10, 12, 14] to [9, 11, 13, 15] and revised fixture notes in debugging_session_001.meta.json to align with post-fork evaluation-turn indexing semantics used by the eval harness.

Changes

Fixture metadata alignment

Layer / File(s)	Summary
eval_turns indices and notes update benchmarks/context-rebuilder/fixtures/synthetic/debugging_session_001.meta.json	Updated eval_turns from [8, 10, 12, 14] to [9, 11, 13, 15] and revised notes to describe the new post-fork expected-answer extraction behavior and user-turn indexing semantics.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Possibly related issues

• feat(eval-harness): wire context-rebuilder eval harness for #587 hot-start scoring #592: The fixture metadata change updates eval_turns indices and indexing semantics that are directly used by the eval-harness turn-extraction logic referenced in that issue.

Suggested labels

attn:review

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	Title accurately summarizes the main change: flipping eval_turns indices in the debug fixture metadata to point to canonical answers rather than user prompts, aligning with issue #687 requirements.
Description check	✅ Passed	Description comprehensively covers the change with clear sections (Summary, Why, calibration artifacts, methodology, verification), maps to template sections, includes linked issues, test results, and detailed evidence supporting the fix.
Linked Issues check	✅ Passed	The PR fulfills all acceptance criteria from #687: multi-judge protocol (3 independent Sonnet 4.6 judges), zero-LLM baseline (score_substring_exact_match), Cohen's-κ reporting (inter_judge_kappa=1.0, calibrated=true), and hot-start fidelity ≥80% (1.0 at t≤0.6).
Out of Scope Changes check	✅ Passed	Changes are entirely in scope: single-file metadata update (debugging_session_001.meta.json eval_turns and notes), directly addressing issue #687's fixture-meta flip requirement. No unrelated changes to rebuilder retrieval, vendor SDK imports, or convention rework.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/issue-687-debug-fixture-canonical-expected

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[robotrocketscience]

approved LGTM

[github-actions]

merge-train: merged 0cc251f → main via FF push.