feat(eval-harness): judges/llm_judge.py — opt-in LLM-judge stage (#592)

CHANGELOG.md

@@ -24,6 +24,8 @@ installable release; see the roadmap in [README.md](README.md).
 
 - **`aelf doctor` flags per-project DBs on pre-v1.x schema (no `origin` column)** ([#589](https://github.com/robotrocketscience/aelfrice/issues/589)). Scans all `~/.aelfrice/projects/*/memory.db` and appends a `legacy-schema per-project DBs detected` block listing each flagged DB with belief row count and idle days. DBs on the old schema cannot participate in the v2.x lifecycle (`agent_remembered`, `user_validated`, calibrated weights, `aelf:promote`). Block is quiet when no legacy DBs are found (parity with #557 quietness). `aelf migrate` is now visible in `--help` output (previously `argparse.SUPPRESS`) so the fix line in the nag block is discoverable.
 
+- **Context-rebuilder eval harness — opt-in LLM-judge stage for open-ended turns** ([#592](https://github.com/robotrocketscience/aelfrice/issues/592)). Commit-3 of #592. New `benchmarks/context-rebuilder/judges/llm_judge.py` adds a host-polymorphic judge that scores open-ended replay rows (those tagged `reason="needs_llm_judge"` by the deterministic substring path). `write_judge_requests(rows, run_dir, max_judge_calls=N)` writes `judge_requests.jsonl` carrying strictly `(turn_idx, expected, actual)` — the rebuilt block and user turn are deliberately excluded per `docs/BENCHMARKS.md` Pass 1 / Pass 2 separation; letting the judge see retrieval context would inflate fidelity. Default `max_judge_calls=0` disables the stage so CI runs are free and outbound model traffic is opt-in. The judge tier label is `"anchor"` — the operator maps this to their host CLI's calibration-baseline tier (the tier the prior Cohen's-κ disagreement methodology was measured at); a cheaper-tier knob is intentionally not exposed because cheaper judges weaken comparability on the open-ended turns this stage exists to score. Dispatch is operator-driven: the host CLI reads the request file, issues one off-band model call per row at the anchor tier with `JUDGE_PROMPT_TEMPLATE`, writes `judge_responses.jsonl`. `read_judge_responses` + `apply_judge_verdicts` then fold verdicts back into the replay rows (clearing `reason`, setting `matched`, adding `judge_rationale`). No provider SDK import; matches `/aelf:onboard`'s polymorphic-classify pattern. 15 deterministic tests in `tests/test_context_rebuilder_eval_judge.py` cover the contamination boundary, cost-cap binding, response round-trip, malformed-line skip, fold-verdicts purity, and the no-SDK guard. Harness-side wiring (so the stage rides on `eval_harness.py --run-dir` rather than the standalone helpers) joins onto PR #601's run-dir plumbing in a follow-up.
+
 - **Context-rebuilder eval harness — three integration points wired** ([#592](https://github.com/robotrocketscience/aelfrice/issues/592)). `benchmarks/context-rebuilder/eval_harness.py` was a skeleton with five `NotImplementedError` stubs since v1.2.0; this lands the three pure-Python wirings that make `--mode threshold-sweep` and `--mode budget-sweep` runnable end-to-end. `replay_to_fork` populates a fresh in-memory `MemoryStore` via `ingest_jsonl` against a tempfile holding the first `fork_turn` lines of the case JSONL. `run_rebuilder` calls `rebuild_v14()` with `floor_session=floor_l1=trigger_threshold` (mapping the harness's threshold-sweep axis onto the v1.7 per-lane composite-score floor) and returns `(rebuilt_block, latency_ms)` measured via `time.monotonic()`. `measure_token_cost` returns `estimate_tokens(rebuilt) / estimate_tokens(pre_clear)` using `benchmarks.context_rebuilder.measure.estimate_tokens` (the 4-chars-per-token heuristic that mirrors `aelfrice.context_rebuilder._CHARS_PER_TOKEN`), so harness measurements stay aligned with the rebuilder's own budget bookkeeping. `replay_post_fork` is a non-crashing stub that returns one placeholder per `eval_turn` (`{turn_idx, expected, actual="", matched=False, reason="needs_replay_client"}`) so the harness produces valid latency + token-cost numbers before the model-invocation client lands; `score_fidelity` reads `matched=False` and returns 0.0 instead of raising. Adds `debugging_session_001.meta.json` next to the bundled 16-turn synthetic fixture (midpoint fork at turn 8, eval_turns at indices 8/10/12/14) so `--corpus benchmarks/context-rebuilder/fixtures/synthetic/` resolves out of the box. 15 new tests in `tests/test_context_rebuilder_eval_harness_wiring.py` cover each wired function plus an end-to-end threshold-sweep smoke against the bundled fixture. The model-invocation client + LLM judge stages are explicit follow-ups; this commit lands the harness's runnable contract before fidelity scoring engages.
 
 - **Session-end Stop hook prompts to lock session corrections** ([#582](https://github.com/robotrocketscience/aelfrice/issues/582)). New `aelf-stop-hook` (default-on) fires once per assistant-turn end, walks the store for unlocked correction-class beliefs created in the current `session_id`, and emits a `<aelfrice-session-end>` block to stderr listing each candidate with a pre-filled `aelf lock --statement '<...>'` command. Candidate filter: `session_id == current AND lock_level != LOCK_USER AND (type == BELIEF_CORRECTION OR origin in {agent_inferred, agent_remembered})`. Hook is informational by default; setting `AELF_AUTOLOCK_CORRECTIONS=1` in the environment makes it auto-lock the candidates instead (logs each lock to stderr for transparency). Wired into `aelf setup` / `aelf unsetup` (`--no-stop-hook` opts out) and into `aelf doctor` as a fourth default-on auto-capture hook the v2.1 nag flags when missing. Coexists with the existing transcript-ingest Stop entry as a separate entry under the same `hooks.Stop` event key. **Note**: the issue's spec assumed a `aelf correct` CLI command and a `feedback_history.kind=correct` marker that don't exist on `main` (no historical implementation); the v0 detection signal is correction-class beliefs in the current session instead. Future work to ship `aelf correct` + `feedback_history.kind` would let this hook also surface beliefs that were *modified* (not only newly-created) in the session.

benchmarks/context-rebuilder/README.md

@@ -277,6 +277,85 @@ These remain open for the fidelity scorer (#138):
 4. Whether augment-mode loses fidelity vs. suppress-mode (matters
    for v2.x suppress-mode promotion decision).
 
+## LLM-judge stage (commit-3 of #592)
+
+Open-ended replay rows that the deterministic substring scorer
+cannot settle land in the run report tagged
+`reason="needs_llm_judge"`. The judge stage ships as a separate,
+opt-in pass that compares each candidate answer against the
+reference using an off-band model call at the host CLI's anchor
+tier.
+
+The stage is **default-off** (`max_judge_calls=0`) so CI never
+issues a model call. Operators opt in by passing a positive cap;
+the cap binds before any request file is written, so a
+misconfigured run cannot exceed its budget.
+
+### Contamination boundary
+
+The judge sees only `(turn_idx, expected, actual)`. Retrieval
+context -- the rebuilt block and the user turn -- never reaches
+the judge prompt, per [`docs/BENCHMARKS.md`][b] Pass 1 / Pass 2
+separation. Letting the judge see the rebuilt block would allow
+it to patch the candidate using context the candidate did not in
+fact produce, inflating fidelity.
+
+[b]: ../../docs/BENCHMARKS.md
+
+### Operator flow
+
+The harness (or an operator-driven helper) writes a per-run
+`judge_requests.jsonl`; the host CLI's dispatcher issues one
+off-band call per row at the anchor tier (`JUDGE_MODEL_TIER`)
+and writes `judge_responses.jsonl` back to the same directory;
+a follow-up call joins by `turn_idx` and folds the verdicts back
+into the run report.
+
+```
+# 1. Replay produced rows tagged reason="needs_llm_judge". Write
+#    the request file (replace 50 with your call budget):
+uv run python - <<'PY'
+from pathlib import Path
+import json, sys
+sys.path.insert(0, "benchmarks/context-rebuilder")
+from judges import llm_judge
+
+rows = [json.loads(l) for l in
+        Path("run_dir/replay_results.jsonl").read_text().splitlines()
+        if l.strip()]
+n = llm_judge.write_judge_requests(rows, Path("run_dir"),
+                                    max_judge_calls=50)
+print(f"wrote {n} judge requests")
+PY
+
+# 2. Host-side dispatch (one off-band model call per row at the
+#    anchor tier, prompt = llm_judge.JUDGE_PROMPT_TEMPLATE rendered
+#    with the row's `expected` and `actual`). Write
+#    run_dir/judge_responses.jsonl with rows of shape
+#    {"turn_idx": <int>, "matched": <bool>, "rationale": "<str>"}.
+
+# 3. Fold the verdicts back into the run report:
+uv run python - <<'PY'
+from pathlib import Path
+import json, sys
+sys.path.insert(0, "benchmarks/context-rebuilder")
+from judges import llm_judge
+
+rows = [json.loads(l) for l in
+        Path("run_dir/replay_results.jsonl").read_text().splitlines()
+        if l.strip()]
+responses = llm_judge.read_judge_responses(Path("run_dir"))
+folded = llm_judge.apply_judge_verdicts(rows, responses)
+Path("run_dir/replay_results.jsonl").write_text(
+    "\n".join(json.dumps(r) for r in folded) + "\n"
+)
+PY
+```
+
+Wiring this stage into the harness's main flow -- so steps 1 and
+3 ride on the same `eval_harness.py` invocation -- is a follow-up
+that joins onto the `--run-dir` plumbing landing in PR #601.
+
 ## Status
 
 * **#136 -- harness scaffolding:** SHIPPED in v1.4.0. Replay

benchmarks/context-rebuilder/judges/__init__.py

@@ -0,0 +1,8 @@
+"""Judge stages for the context-rebuilder eval harness.
+
+Each judge module writes a per-run request file and reads a per-run
+response file. The host CLI (or operator skill / MCP host) does the
+model dispatch in its own context — aelfrice and the benchmarks code
+never import a provider SDK directly. Mirrors `/aelf:onboard`'s
+polymorphic LLM-classify pattern.
+"""

benchmarks/context-rebuilder/judges/llm_judge.py

@@ -0,0 +1,246 @@
+"""LLM-judge stage for the context-rebuilder eval harness.
+
+Open-ended eval turns produce free-text continuations that substring
+match (the deterministic path in #600's `score_fidelity`) misclassifies
+as wrong. Those rows surface with `reason="needs_llm_judge"`. This
+module turns each such row into a request file the host CLI dispatches
+off-band, then folds the verdicts back into the run.
+
+## Contamination protocol
+
+`docs/BENCHMARKS.md` is explicit: generation and scoring run as separate
+passes, and **the judge never sees the retrieval context.** The request
+schema therefore carries only `(turn_idx, expected, actual)` — not the
+rebuilt block, not the user turn. Including the rebuilt block in the
+judge prompt would let the judge "patch" the prediction with details it
+sees in the retrieval window, inflating fidelity.
+
+## Cost posture
+
+`max_judge_calls` defaults to 0, which disables the stage entirely so
+CI runs are free. Operators opt in by passing a positive cap; the cap
+binds before any request file is written, so even a misconfigured run
+cannot exceed the budget.
+
+The chosen judge tier is the host CLI's anchor model — the tier the
+prior calibration work (Cohen's-κ disagreement against a zero-LLM
+baseline) was measured against. A cheaper-tier knob is intentionally
+not exposed: dropping to the small model below the anchor weakens
+comparability on the open-ended turns this stage exists to score.
+
+## Polymorphic dispatch (no SDK in this module)
+
+This module imports nothing from any provider SDK. The flow is:
+
+  1. Replay phase tags non-substring-match rows `needs_llm_judge`.
+  2. `write_judge_requests(...)` writes `judge_requests.jsonl` into the
+     per-run directory, one strict-schema row per eligible turn.
+  3. Operator runs the host-side dispatch — a host-CLI skill, MCP host
+     equivalent, or hand-driven `gh`/`uv run` call — which reads the
+     requests, issues one off-band model call per row using
+     `JUDGE_PROMPT_TEMPLATE`, and writes `judge_responses.jsonl`.
+  4. `read_judge_responses(...)` joins the responses back by turn_idx.
+  5. `apply_judge_verdicts(...)` updates the replay rows in place.
+
+Steps 2/4/5 are pure-Python and exercised by CI. Step 3 is operator-
+driven; CI never invokes a real model.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Final, Iterable
+
+JUDGE_MODEL_TIER: Final[str] = "anchor"
+"""Logical tier label for the judge model. The operator maps this to a
+concrete model name in their host CLI's dispatch step; the value
+"anchor" means the host's calibration-baseline tier (not the small /
+cheaper tier below it). Encoding the tier as a label keeps this module
+free of vendor-specific model identifiers and decoupled from any one
+host CLI's catalog."""
+
+JUDGE_REASON: Final[str] = "needs_llm_judge"
+"""Reason tag carried by replay rows that this stage processes. Set by
+the deterministic `score_fidelity` path when an `actual` is present
+but no substring match was found."""
+
+JUDGE_REQUESTS_FILENAME: Final[str] = "judge_requests.jsonl"
+JUDGE_RESPONSES_FILENAME: Final[str] = "judge_responses.jsonl"
+
+JUDGE_PROMPT_TEMPLATE: Final[str] = """\
+You are scoring whether one free-text answer means the same thing as
+a reference answer for a memory-system continuation eval. You see only
+the two answers — no retrieval context, no user prompt, no history.
+
+REFERENCE:
+{expected}
+
+CANDIDATE:
+{actual}
+
+Reply with a single JSON object on one line:
+  {{"matched": true|false, "rationale": "<= 240 chars"}}
+
+`matched` is true when the candidate conveys the same factual content
+as the reference (paraphrase, partial restatement, or equivalent
+phrasing all count). It is false when the candidate omits the
+reference's load-bearing claim, contradicts it, or is empty. Do not
+penalize style differences.
+"""
+"""Prompt the host-side dispatcher should send to each off-band call.
+Anchors the judge in a no-retrieval-context posture (contamination
+protocol) and constrains the response to a one-line JSON verdict so
+`read_judge_responses` can parse without ambiguity."""
+
+
+@dataclass(frozen=True)
+class JudgeRequest:
+    """One judge request. Strict schema — `turn_idx` is the join key,
+    `expected` is the reference answer, `actual` is the candidate.
+    No `rebuilt_block`, no `user_turn` — that is the contamination
+    boundary."""
+    turn_idx: int
+    expected: str
+    actual: str
+
+
+@dataclass(frozen=True)
+class JudgeResponse:
+    """One judge response. `matched` is the verdict; `rationale` is a
+    short free-text explanation surfaced in the run report."""
+    turn_idx: int
+    matched: bool
+    rationale: str = ""
+
+
+def _eligible_rows(replay_results: Iterable[dict]) -> list[dict]:
+    """Filter the replay rows the judge stage processes.
+
+    Eligibility: `reason == JUDGE_REASON` AND `actual` is a non-empty
+    string AND `expected` is a non-empty string. Rows missing either
+    field have nothing to compare; rows with the wrong reason were
+    already settled by the deterministic path.
+    """
+    out: list[dict] = []
+    for r in replay_results:
+        if r.get("reason") != JUDGE_REASON:
+            continue
+        expected = r.get("expected")
+        actual = r.get("actual")
+        if not isinstance(expected, str) or not expected:
+            continue
+        if not isinstance(actual, str) or not actual:
+            continue
+        out.append(r)
+    return out
+
+
+def write_judge_requests(
+    replay_results: Iterable[dict],
+    run_dir: Path,
+    *,
+    max_judge_calls: int = 0,
+) -> int:
+    """Write eligible replay rows to `<run_dir>/judge_requests.jsonl`.
+
+    Returns the number of rows written. `max_judge_calls` caps the
+    request count and defaults to 0, which is the disabled state —
+    no file is written when no calls are budgeted. Operators opt in
+    by passing a positive cap.
+
+    Each row's on-disk shape is exactly the three fields of
+    `JudgeRequest`. Asserted by the test suite to prevent retrieval
+    context from leaking into the judge prompt.
+    """
+    if max_judge_calls <= 0:
+        return 0
+    eligible = _eligible_rows(replay_results)
+    if not eligible:
+        return 0
+    run_dir.mkdir(parents=True, exist_ok=True)
+    path = run_dir / JUDGE_REQUESTS_FILENAME
+    written = 0
+    with path.open("w", encoding="utf-8") as f:
+        for row in eligible:
+            if written >= max_judge_calls:
+                break
+            req = JudgeRequest(
+                turn_idx=int(row["turn_idx"]),
+                expected=row["expected"],
+                actual=row["actual"],
+            )
+            f.write(json.dumps(asdict(req), ensure_ascii=False) + "\n")
+            written += 1
+    return written
+
+
+def read_judge_responses(run_dir: Path) -> dict[int, JudgeResponse]:
+    """Read `<run_dir>/judge_responses.jsonl` into a turn_idx-keyed dict.
+
+    Returns an empty dict if the file is absent (the operator hasn't
+    run the dispatch step yet) or if every line is malformed. Lines
+    that fail to parse, or that lack the required fields, are silently
+    skipped — partial responses are preferable to a full run failure
+    when the harness is iterating across many turns.
+    """
+    path = run_dir / JUDGE_RESPONSES_FILENAME
+    if not path.exists():
+        return {}
+    out: dict[int, JudgeResponse] = {}
+    with path.open("r", encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                obj = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+            if not isinstance(obj, dict):
+                continue
+            turn_idx = obj.get("turn_idx")
+            matched = obj.get("matched")
+            if not isinstance(turn_idx, int) or not isinstance(matched, bool):
+                continue
+            rationale = obj.get("rationale", "")
+            if not isinstance(rationale, str):
+                rationale = ""
+            out[turn_idx] = JudgeResponse(
+                turn_idx=turn_idx, matched=matched, rationale=rationale
+            )
+    return out
+
+
+def apply_judge_verdicts(
+    replay_results: list[dict],
+    responses: dict[int, JudgeResponse],
+) -> list[dict]:
+    """Fold judge verdicts back into the replay rows.
+
+    Pure function — returns a new list with judge-eligible rows
+    updated. For each row tagged `JUDGE_REASON` with a matching
+    response: `matched` becomes the judge verdict, `reason` is
+    cleared (set to empty string), and a `judge_rationale` field is
+    added. Rows without a response keep `reason=JUDGE_REASON` so the
+    next harness invocation can retry.
+
+    Non-eligible rows pass through untouched.
+    """
+    updated: list[dict] = []
+    for row in replay_results:
+        if row.get("reason") != JUDGE_REASON:
+            updated.append(dict(row))
+            continue
+        turn_idx = row.get("turn_idx")
+        if not isinstance(turn_idx, int) or turn_idx not in responses:
+            updated.append(dict(row))
+            continue
+        verdict = responses[turn_idx]
+        new_row = dict(row)
+        new_row["matched"] = verdict.matched
+        new_row["reason"] = ""
+        new_row["judge_rationale"] = verdict.rationale
+        updated.append(new_row)
+    return updated