feat(eval-harness): v3.1 close — sharpen JUDGE_PROMPT_TEMPLATE + widen hot_start fixture (#778)

CHANGELOG.md

@@ -12,6 +12,8 @@ installable release; see the roadmap in [README.md](README.md).
 
 ### Added
 
+- **v3.1 close of #592 hot-start AC — sharpened `JUDGE_PROMPT_TEMPLATE` + widened hot_start fixture to n=10 eval turns** ([#778](https://github.com/robotrocketscience/aelfrice/issues/778)). Run 2 of the multi-judge κ sweep (2026-05-12/13, see [#687](https://github.com/robotrocketscience/aelfrice/issues/687)) saw `inter_judge_kappa.min = 0.0` (gate 0.7) and `hot_start_fidelity_mean = 0.667` (gate 0.8), diagnosed as two compounding causes: (a) the prompt template's "load-bearing claim" clause was operationally under-specified — judges disagreed on which part of each reference counted as load-bearing on 2 of 3 fixture turns; (b) statistical thinness on n=3 binary samples, where a single judge flip drops the κ catastrophically. (a) is addressed by replacing the single-clause definition in `benchmarks/context-rebuilder/judges/llm_judge.py` with a three-step rubric that names the unit of scoring as "specific facts" (concrete atoms — names, paths, values, actions, numeric quantities), requires the candidate to convey every such fact, and defines `matched=true` as "a reader of the candidate alone could state each specific fact"; plus two worked examples (multi-fact and low-content) drawn from outside the hot_start fixture to avoid biasing scoring on the corpus being scored. (b) is addressed by extending `hot_start_debugging_session_001.{jsonl,meta.json}` from 14 to 28 turns, keeping the same fork (turn 8) and the same synthetic CRLF-vs-LF `_dedup_key` debugging scenario, and adding 7 new user/assistant pairs that probe pre-fork facts the prior cut did not exercise (intermittency root cause, literal code change, why-fix-in-`_dedup_key`-not-conftest rationale, dedup tuple shape, NFC normalization form, repro command, conftest file:line). New `eval_turns = [9, 11, 13, 15, 17, 19, 21, 23, 25, 27]`. Test assertion in `tests/test_context_rebuilder_eval_harness_wiring.py` updated to the new tuple. Fixture content remains fully synthetic — no `_dedup_key` or `test_ingest_jsonl_idempotent` exists in source on `github/main`. The κ sweep re-run is operator-dispatched (subagent path per locked bench convention, not direct API) and gates closing the v3.0-deferred hot-start AC ≥80% bullet.
+
 - **PR-smoke fixtures + CI job for benchmark adapters** ([#476](https://github.com/robotrocketscience/aelfrice/issues/476)). Lands `tests/fixtures/bench_smoke/` (synthetic, schema-matching micro-fixtures for LoCoMo, MAB, LongMemEval, StructMemEval), `tests/test_bench_smoke.py` (exercises each adapter's `--retrieve-only` CLI path against its fixture, asserts non-zero retrieved-belief counts per item), and `.github/workflows/bench-smoke.yml` (runs on every PR that touches `src/`, `benchmarks/`, the smoke test, or the fixtures; 120s per-test budget; installs `[benchmarks]` extras). Per the activation-time license review on this issue, no fixture is derived from any upstream dataset — LoCoMo (CC BY-NC 4.0), MAB (MIT), LongMemEval (MIT), and StructMemEval (no upstream LICENSE file) are all covered by schema-matching synthetic data. Companion change: `mab_adapter.py` gains a `--data PATH` flag and `load_mab_from_file()` so it can be exercised offline (LoCoMo / LongMemEval / StructMemEval adapters already had a `--data` flag). Real-data shape coverage continues to come from the nightly `bench-canonical` cron; this smoke path is dispatcher-contract validation only.
 
 - **`aelf export-canvas` — JSON Canvas 1.0 exporter as Obsidian-mapping probe** ([#763](https://github.com/robotrocketscience/aelfrice/issues/763)). Hidden subcommand (not in default `--help`; surfaces via `--advanced`). Reuses the same seed-selection + BFS expansion path as `aelf reason` and emits a deterministic [JSON Canvas 1.0](https://github.com/obsidianmd/jsoncanvas/blob/main/spec/1.0.md) file Obsidian can open directly. Encoding: node color from posterior-mean buckets (locked → cyan, mu>=0.75 → green, mu<0.25 → red, mid → no color); edge color from `EDGE_VALENCE` sign (positive → green, negative → red, zero → no color); edge label = edge type plus optional truncated `anchor_text`. Positions are computed from belief-id sha256 hash on concentric rings by BFS depth, so re-running on the same store yields byte-identical output (honors `c06f8d575fad71fb` deterministic-narrow-surface lock). One-way DB → canvas; no import path. Probe scope only — the vault exporter and frontmatter-namespace decisions are deferred siblings.

benchmarks/context-rebuilder/fixtures/synthetic/hot_start_debugging_session_001.jsonl

@@ -12,3 +12,17 @@
 {"schema_version":1,"ts":"2026-05-04T09:43:05.000Z","role":"assistant","text":"src/aelfrice/ingest.py, function _dedup_key. Add a `sentence = sentence.rstrip('\\r\\n').rstrip()` line before the unicodedata.normalize('NFC', sentence) call.","session_id":"20260504T091200Z-h001","turn_id":"20260504T094305Z-0012","context":{"cwd":"/Users/synthetic/aelfrice"}}
 {"schema_version":1,"ts":"2026-05-04T09:43:32.000Z","role":"user","text":"Got it. After that, what was the verification step you suggested?","session_id":"20260504T091200Z-h001","turn_id":"20260504T094332Z-0013","context":{"cwd":"/Users/synthetic/aelfrice"}}
 {"schema_version":1,"ts":"2026-05-04T09:43:50.000Z","role":"assistant","text":"Run pytest -xvs tests/test_ingest.py::test_ingest_jsonl_idempotent to confirm the targeted test passes, then run pytest tests/test_ingest.py to confirm the rest of the ingest tests still pass. Once both pass, run the full suite to confirm no regression.","session_id":"20260504T091200Z-h001","turn_id":"20260504T094350Z-0014","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:44:30.000Z","role":"user","text":"Refresh me — what was the root cause of the intermittency?","session_id":"20260504T091200Z-h001","turn_id":"20260504T094430Z-0015","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:44:50.000Z","role":"assistant","text":"Sentences ending in '\\r\\n' versus '\\n'. The failing run passed sentences with CRLF line endings into _dedup_key; the passing run had LF only. unicodedata.normalize did not strip them, so the same logical sentence produced different dedup keys.","session_id":"20260504T091200Z-h001","turn_id":"20260504T094450Z-0016","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:45:20.000Z","role":"user","text":"What was the exact one-line fix I added to _dedup_key?","session_id":"20260504T091200Z-h001","turn_id":"20260504T094520Z-0017","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:45:42.000Z","role":"assistant","text":"sentence = sentence.rstrip('\\r\\n').rstrip(), inserted before the unicodedata.normalize('NFC', sentence) call.","session_id":"20260504T091200Z-h001","turn_id":"20260504T094542Z-0018","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:46:15.000Z","role":"user","text":"Remind me why we fixed _dedup_key rather than the conftest.","session_id":"20260504T091200Z-h001","turn_id":"20260504T094615Z-0019","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:46:40.000Z","role":"assistant","text":"Because the production ingest path needs to be robust to JSONL files arriving over the wire with '\\r\\n' line endings, not just to in-process conftest fixtures. Fixing the conftest would paper over the same bug in production.","session_id":"20260504T091200Z-h001","turn_id":"20260504T094640Z-0020","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:47:10.000Z","role":"user","text":"What is the dedup tuple shape _dedup_key uses?","session_id":"20260504T091200Z-h001","turn_id":"20260504T094710Z-0021","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:47:25.000Z","role":"assistant","text":"(session_id, turn_id, source, sentence).","session_id":"20260504T091200Z-h001","turn_id":"20260504T094725Z-0022","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:47:55.000Z","role":"user","text":"Which unicode normalization form does _dedup_key apply?","session_id":"20260504T091200Z-h001","turn_id":"20260504T094755Z-0023","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:48:08.000Z","role":"assistant","text":"NFC. The call is unicodedata.normalize('NFC', sentence).","session_id":"20260504T091200Z-h001","turn_id":"20260504T094808Z-0024","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:48:38.000Z","role":"user","text":"What was the debug command you suggested I add to see the intermittency?","session_id":"20260504T091200Z-h001","turn_id":"20260504T094838Z-0025","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:49:00.000Z","role":"assistant","text":"print(repr(sentence)) inserted directly before _dedup_key returns, then rerun pytest -xvs tests/test_ingest.py::test_ingest_jsonl_idempotent in both the isolated and full-suite contexts.","session_id":"20260504T091200Z-h001","turn_id":"20260504T094900Z-0026","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:49:30.000Z","role":"user","text":"And where in conftest was the test-side authoring bug — file location?","session_id":"20260504T091200Z-h001","turn_id":"20260504T094930Z-0027","context":{"cwd":"/Users/synthetic/aelfrice"}}
+{"schema_version":1,"ts":"2026-05-04T09:49:48.000Z","role":"assistant","text":"conftest.py line 47, in the tmp_path fixture that calls Path.write_text(content) on a string authored with '\\r\\n' line endings.","session_id":"20260504T091200Z-h001","turn_id":"20260504T094948Z-0028","context":{"cwd":"/Users/synthetic/aelfrice"}}

benchmarks/context-rebuilder/fixtures/synthetic/hot_start_debugging_session_001.meta.json

@@ -1,6 +1,17 @@
 {
   "task_type": "hot_start",
   "fork_turn": 8,
-  "eval_turns": [9, 11, 13],
-  "notes": "Post-compact hot-start scenario for #587 / #592 AC. Pre-fork (turns 0..7) establishes specific working state: failing test `test_ingest_jsonl_idempotent`, file `src/aelfrice/ingest.py`, function `_dedup_key`. The fork at turn 8 simulates a /clear-induced compact. Post-fork user prompts at indices 8, 10, 12 ask 'where were we?', 'which file/function?', 'what was the verification step?'; the harness pulls `user_turn` from each eval_turn's most-recent-user-role text (8/10/12) and `expected` from the eval_turn itself (assistant continuations at 9/11/13). This makes `expected` the canonical answer the post-rebuild agent should produce, so judge-vs-baseline κ compares answer-against-answer (calibration ratification #687). Prior shape used eval_turns=[8,10,12] which set expected=user_question_text: substring path always missed and the judge prompt split on Q/A speech-act mismatches (inter-judge κ=0.0, fidelity 0.67 — see #687 for the failure evidence). Sibling fixture debugging_session_001.meta.json keeps its [8,10,12,14] shape until the cold-start sweep is re-ratified separately."
+  "eval_turns": [
+    9,
+    11,
+    13,
+    15,
+    17,
+    19,
+    21,
+    23,
+    25,
+    27
+  ],
+  "notes": "Post-compact hot-start scenario for #587 / #592 AC, widened to n=10 eval turns per #778 (v3.1 close of #592). Pre-fork (turns 0..7) establishes specific working state: failing test `test_ingest_jsonl_idempotent`, file `src/aelfrice/ingest.py`, function `_dedup_key`, intermittency root cause (CRLF vs LF newline policy), the literal fix (`sentence.rstrip('\\r\\n').rstrip()` before `unicodedata.normalize('NFC', sentence)`), the why-fix-_dedup_key-not-conftest rationale (production robustness), the dedup tuple shape `(session_id, turn_id, source, sentence)`, the NFC normalization form, the repro command (`print(repr(sentence))` before `_dedup_key` returns), and the conftest location (line 47, `Path.write_text`). The fork at turn 8 simulates a /clear-induced compact. Post-fork user prompts at indices 8, 10, 12, 14, 16, 18, 20, 22, 24, 26 ask one canonical-recall question each; the harness pulls `user_turn` from each eval_turn's most-recent-user-role text and `expected` from the eval_turn itself (assistant continuations at 9, 11, 13, 15, 17, 19, 21, 23, 25, 27). Each new eval probes a different specific fact from pre-fork turns 0..7 not previously tested by the n=3 shape, so the judge prompt's specific-facts rubric (#778) gets exercised across distinct atom kinds (test name, file path, function name, root cause, code change, rationale, data structure, normalization form, command, file:line location). Prior shapes: n=3 with eval_turns=[9,11,13] (per #687 PR #705) — passed substring path but Run 2 κ sweep saw inter_judge_kappa.min=0.0 / fidelity_mean=0.667 against the [9,11,13] cut (#592 v3.0 deferral, locked belief cea8ba7708ae3ca3). n=3 with eval_turns=[8,10,12] (pre-#687) — set expected=user_question_text, substring path always missed and judge split on Q/A speech-act mismatches. Sibling fixture debugging_session_001.meta.json keeps its [8,10,12,14] shape until the cold-start sweep is re-ratified separately."
 }

benchmarks/context-rebuilder/judges/llm_judge.py

@@ -80,14 +80,43 @@
 CANDIDATE:
 {actual}
 
+## How to score
+
+1. Identify the specific facts the reference commits to. A specific
+   fact is a concrete atom — a name, path, value, action, or numeric
+   quantity — that the reference is asserting. A reference may commit
+   to one fact or several. Surrounding prose (restating the question,
+   tooling tips, error explanations) is supporting material, not a
+   fact for scoring purposes.
+2. Check whether the candidate, read alone, conveys EVERY specific
+   fact from the reference. Synonyms, paraphrases, equivalent
+   restatements, and reordered phrasing all count as conveying the
+   same fact.
+3. `matched` is true iff a reader of the candidate alone — given no
+   other context — could correctly state each specific fact the
+   reference commits to.
+
+## Worked examples
+
+REFERENCE: "Use binary search; runs in O(log n)."
+Specific facts: "binary search" AND "O(log n)".
+- CANDIDATE "Binary search, which is O(log n)." → matched=true
+- CANDIDATE "Use binary search." → matched=false (omits complexity)
+- CANDIDATE "It's an O(log n) algorithm." → matched=false (omits name)
+
+REFERENCE: "Yes."
+Specific facts: none — the reference commits only to affirmation.
+- CANDIDATE "Yes, that should work." → matched=true
+- CANDIDATE "No." → matched=false (contradicts)
+- CANDIDATE "" → matched=false (empty)
+
+## Verdict
+
 Reply with a single JSON object on one line:
   {{"matched": true|false, "rationale": "<= 240 chars"}}
 
-`matched` is true when the candidate conveys the same factual content
-as the reference (paraphrase, partial restatement, or equivalent
-phrasing all count). It is false when the candidate omits the
-reference's load-bearing claim, contradicts it, or is empty. Do not
-penalize style differences.
+Do not penalize style differences, missing supporting material, or
+extra phrasing not present in the reference.
 """
 """Prompt the host-side dispatcher should send to each off-band call.
 Anchors the judge in a no-retrieval-context posture (contamination

tests/test_context_rebuilder_eval_harness_wiring.py

@@ -528,10 +528,13 @@ def test_hot_start_fixture_resolves_via_load_corpus(
     assert len(hot_start) >= 1, "hot_start fixture must resolve via load_corpus"
     case = hot_start[0]
     assert case.fork_turn == 8
-    # eval_turns point at assistant continuations (9, 11, 13) so `expected`
-    # is the canonical answer text, not the user prompt — see #687 calibration
-    # ratification and the meta `notes` for the prior-shape rationale.
-    assert case.eval_turns == (9, 11, 13)
+    # eval_turns point at assistant continuations (9, 11, 13, 15, 17, 19, 21,
+    # 23, 25, 27) so `expected` is the canonical answer text, not the user
+    # prompt — see #687 calibration ratification and the meta `notes` for the
+    # prior-shape rationale. Widened from n=3 to n=10 per #778 (v3.1 close
+    # of #592 hot-start AC) to address statistical thinness diagnosed in
+    # Run 2's multi-judge κ sweep.
+    assert case.eval_turns == (9, 11, 13, 15, 17, 19, 21, 23, 25, 27)
 
 
 def test_threshold_sweep_segments_hot_start_from_debug(