diff --git a/CHANGELOG.md b/CHANGELOG.md index f3e2c0d35..8d9de9018 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,8 @@ installable release; see the roadmap in [README.md](README.md). - **`aelf doctor` flags per-project DBs on pre-v1.x schema (no `origin` column)** ([#589](https://github.com/robotrocketscience/aelfrice/issues/589)). Scans all `~/.aelfrice/projects/*/memory.db` and appends a `legacy-schema per-project DBs detected` block listing each flagged DB with belief row count and idle days. DBs on the old schema cannot participate in the v2.x lifecycle (`agent_remembered`, `user_validated`, calibrated weights, `aelf:promote`). Block is quiet when no legacy DBs are found (parity with #557 quietness). `aelf migrate` is now visible in `--help` output (previously `argparse.SUPPRESS`) so the fix line in the nag block is discoverable. +- **Context-rebuilder eval harness — opt-in LLM-judge stage for open-ended turns** ([#592](https://github.com/robotrocketscience/aelfrice/issues/592)). Commit-3 of #592. New `benchmarks/context-rebuilder/judges/llm_judge.py` adds a host-polymorphic judge that scores open-ended replay rows (those tagged `reason="needs_llm_judge"` by the deterministic substring path). `write_judge_requests(rows, run_dir, max_judge_calls=N)` writes `judge_requests.jsonl` carrying strictly `(turn_idx, expected, actual)` — the rebuilt block and user turn are deliberately excluded per `docs/BENCHMARKS.md` Pass 1 / Pass 2 separation; letting the judge see retrieval context would inflate fidelity. Default `max_judge_calls=0` disables the stage so CI runs are free and outbound model traffic is opt-in. The judge tier label is `"anchor"` — the operator maps this to their host CLI's calibration-baseline tier (the tier the prior Cohen's-κ disagreement methodology was measured at); a cheaper-tier knob is intentionally not exposed because cheaper judges weaken comparability on the open-ended turns this stage exists to score. Dispatch is operator-driven: the host CLI reads the request file, issues one off-band model call per row at the anchor tier with `JUDGE_PROMPT_TEMPLATE`, writes `judge_responses.jsonl`. `read_judge_responses` + `apply_judge_verdicts` then fold verdicts back into the replay rows (clearing `reason`, setting `matched`, adding `judge_rationale`). No provider SDK import; matches `/aelf:onboard`'s polymorphic-classify pattern. 15 deterministic tests in `tests/test_context_rebuilder_eval_judge.py` cover the contamination boundary, cost-cap binding, response round-trip, malformed-line skip, fold-verdicts purity, and the no-SDK guard. Harness-side wiring (so the stage rides on `eval_harness.py --run-dir` rather than the standalone helpers) joins onto PR #601's run-dir plumbing in a follow-up. + - **Context-rebuilder eval harness — three integration points wired** ([#592](https://github.com/robotrocketscience/aelfrice/issues/592)). `benchmarks/context-rebuilder/eval_harness.py` was a skeleton with five `NotImplementedError` stubs since v1.2.0; this lands the three pure-Python wirings that make `--mode threshold-sweep` and `--mode budget-sweep` runnable end-to-end. `replay_to_fork` populates a fresh in-memory `MemoryStore` via `ingest_jsonl` against a tempfile holding the first `fork_turn` lines of the case JSONL. `run_rebuilder` calls `rebuild_v14()` with `floor_session=floor_l1=trigger_threshold` (mapping the harness's threshold-sweep axis onto the v1.7 per-lane composite-score floor) and returns `(rebuilt_block, latency_ms)` measured via `time.monotonic()`. `measure_token_cost` returns `estimate_tokens(rebuilt) / estimate_tokens(pre_clear)` using `benchmarks.context_rebuilder.measure.estimate_tokens` (the 4-chars-per-token heuristic that mirrors `aelfrice.context_rebuilder._CHARS_PER_TOKEN`), so harness measurements stay aligned with the rebuilder's own budget bookkeeping. `replay_post_fork` is a non-crashing stub that returns one placeholder per `eval_turn` (`{turn_idx, expected, actual="", matched=False, reason="needs_replay_client"}`) so the harness produces valid latency + token-cost numbers before the model-invocation client lands; `score_fidelity` reads `matched=False` and returns 0.0 instead of raising. Adds `debugging_session_001.meta.json` next to the bundled 16-turn synthetic fixture (midpoint fork at turn 8, eval_turns at indices 8/10/12/14) so `--corpus benchmarks/context-rebuilder/fixtures/synthetic/` resolves out of the box. 15 new tests in `tests/test_context_rebuilder_eval_harness_wiring.py` cover each wired function plus an end-to-end threshold-sweep smoke against the bundled fixture. The model-invocation client + LLM judge stages are explicit follow-ups; this commit lands the harness's runnable contract before fidelity scoring engages. - **Session-end Stop hook prompts to lock session corrections** ([#582](https://github.com/robotrocketscience/aelfrice/issues/582)). New `aelf-stop-hook` (default-on) fires once per assistant-turn end, walks the store for unlocked correction-class beliefs created in the current `session_id`, and emits a `` block to stderr listing each candidate with a pre-filled `aelf lock --statement '<...>'` command. Candidate filter: `session_id == current AND lock_level != LOCK_USER AND (type == BELIEF_CORRECTION OR origin in {agent_inferred, agent_remembered})`. Hook is informational by default; setting `AELF_AUTOLOCK_CORRECTIONS=1` in the environment makes it auto-lock the candidates instead (logs each lock to stderr for transparency). Wired into `aelf setup` / `aelf unsetup` (`--no-stop-hook` opts out) and into `aelf doctor` as a fourth default-on auto-capture hook the v2.1 nag flags when missing. Coexists with the existing transcript-ingest Stop entry as a separate entry under the same `hooks.Stop` event key. **Note**: the issue's spec assumed a `aelf correct` CLI command and a `feedback_history.kind=correct` marker that don't exist on `main` (no historical implementation); the v0 detection signal is correction-class beliefs in the current session instead. Future work to ship `aelf correct` + `feedback_history.kind` would let this hook also surface beliefs that were *modified* (not only newly-created) in the session. diff --git a/benchmarks/context-rebuilder/README.md b/benchmarks/context-rebuilder/README.md index fc2f57c23..6d1010d78 100644 --- a/benchmarks/context-rebuilder/README.md +++ b/benchmarks/context-rebuilder/README.md @@ -277,6 +277,85 @@ These remain open for the fidelity scorer (#138): 4. Whether augment-mode loses fidelity vs. suppress-mode (matters for v2.x suppress-mode promotion decision). +## LLM-judge stage (commit-3 of #592) + +Open-ended replay rows that the deterministic substring scorer +cannot settle land in the run report tagged +`reason="needs_llm_judge"`. The judge stage ships as a separate, +opt-in pass that compares each candidate answer against the +reference using an off-band model call at the host CLI's anchor +tier. + +The stage is **default-off** (`max_judge_calls=0`) so CI never +issues a model call. Operators opt in by passing a positive cap; +the cap binds before any request file is written, so a +misconfigured run cannot exceed its budget. + +### Contamination boundary + +The judge sees only `(turn_idx, expected, actual)`. Retrieval +context -- the rebuilt block and the user turn -- never reaches +the judge prompt, per [`docs/BENCHMARKS.md`][b] Pass 1 / Pass 2 +separation. Letting the judge see the rebuilt block would allow +it to patch the candidate using context the candidate did not in +fact produce, inflating fidelity. + +[b]: ../../docs/BENCHMARKS.md + +### Operator flow + +The harness (or an operator-driven helper) writes a per-run +`judge_requests.jsonl`; the host CLI's dispatcher issues one +off-band call per row at the anchor tier (`JUDGE_MODEL_TIER`) +and writes `judge_responses.jsonl` back to the same directory; +a follow-up call joins by `turn_idx` and folds the verdicts back +into the run report. + +``` +# 1. Replay produced rows tagged reason="needs_llm_judge". Write +# the request file (replace 50 with your call budget): +uv run python - <<'PY' +from pathlib import Path +import json, sys +sys.path.insert(0, "benchmarks/context-rebuilder") +from judges import llm_judge + +rows = [json.loads(l) for l in + Path("run_dir/replay_results.jsonl").read_text().splitlines() + if l.strip()] +n = llm_judge.write_judge_requests(rows, Path("run_dir"), + max_judge_calls=50) +print(f"wrote {n} judge requests") +PY + +# 2. Host-side dispatch (one off-band model call per row at the +# anchor tier, prompt = llm_judge.JUDGE_PROMPT_TEMPLATE rendered +# with the row's `expected` and `actual`). Write +# run_dir/judge_responses.jsonl with rows of shape +# {"turn_idx": , "matched": , "rationale": ""}. + +# 3. Fold the verdicts back into the run report: +uv run python - <<'PY' +from pathlib import Path +import json, sys +sys.path.insert(0, "benchmarks/context-rebuilder") +from judges import llm_judge + +rows = [json.loads(l) for l in + Path("run_dir/replay_results.jsonl").read_text().splitlines() + if l.strip()] +responses = llm_judge.read_judge_responses(Path("run_dir")) +folded = llm_judge.apply_judge_verdicts(rows, responses) +Path("run_dir/replay_results.jsonl").write_text( + "\n".join(json.dumps(r) for r in folded) + "\n" +) +PY +``` + +Wiring this stage into the harness's main flow -- so steps 1 and +3 ride on the same `eval_harness.py` invocation -- is a follow-up +that joins onto the `--run-dir` plumbing landing in PR #601. + ## Status * **#136 -- harness scaffolding:** SHIPPED in v1.4.0. Replay diff --git a/benchmarks/context-rebuilder/judges/__init__.py b/benchmarks/context-rebuilder/judges/__init__.py new file mode 100644 index 000000000..e8551aa0f --- /dev/null +++ b/benchmarks/context-rebuilder/judges/__init__.py @@ -0,0 +1,8 @@ +"""Judge stages for the context-rebuilder eval harness. + +Each judge module writes a per-run request file and reads a per-run +response file. The host CLI (or operator skill / MCP host) does the +model dispatch in its own context — aelfrice and the benchmarks code +never import a provider SDK directly. Mirrors `/aelf:onboard`'s +polymorphic LLM-classify pattern. +""" diff --git a/benchmarks/context-rebuilder/judges/llm_judge.py b/benchmarks/context-rebuilder/judges/llm_judge.py new file mode 100644 index 000000000..ffb314424 --- /dev/null +++ b/benchmarks/context-rebuilder/judges/llm_judge.py @@ -0,0 +1,246 @@ +"""LLM-judge stage for the context-rebuilder eval harness. + +Open-ended eval turns produce free-text continuations that substring +match (the deterministic path in #600's `score_fidelity`) misclassifies +as wrong. Those rows surface with `reason="needs_llm_judge"`. This +module turns each such row into a request file the host CLI dispatches +off-band, then folds the verdicts back into the run. + +## Contamination protocol + +`docs/BENCHMARKS.md` is explicit: generation and scoring run as separate +passes, and **the judge never sees the retrieval context.** The request +schema therefore carries only `(turn_idx, expected, actual)` — not the +rebuilt block, not the user turn. Including the rebuilt block in the +judge prompt would let the judge "patch" the prediction with details it +sees in the retrieval window, inflating fidelity. + +## Cost posture + +`max_judge_calls` defaults to 0, which disables the stage entirely so +CI runs are free. Operators opt in by passing a positive cap; the cap +binds before any request file is written, so even a misconfigured run +cannot exceed the budget. + +The chosen judge tier is the host CLI's anchor model — the tier the +prior calibration work (Cohen's-κ disagreement against a zero-LLM +baseline) was measured against. A cheaper-tier knob is intentionally +not exposed: dropping to the small model below the anchor weakens +comparability on the open-ended turns this stage exists to score. + +## Polymorphic dispatch (no SDK in this module) + +This module imports nothing from any provider SDK. The flow is: + + 1. Replay phase tags non-substring-match rows `needs_llm_judge`. + 2. `write_judge_requests(...)` writes `judge_requests.jsonl` into the + per-run directory, one strict-schema row per eligible turn. + 3. Operator runs the host-side dispatch — a host-CLI skill, MCP host + equivalent, or hand-driven `gh`/`uv run` call — which reads the + requests, issues one off-band model call per row using + `JUDGE_PROMPT_TEMPLATE`, and writes `judge_responses.jsonl`. + 4. `read_judge_responses(...)` joins the responses back by turn_idx. + 5. `apply_judge_verdicts(...)` updates the replay rows in place. + +Steps 2/4/5 are pure-Python and exercised by CI. Step 3 is operator- +driven; CI never invokes a real model. +""" + +from __future__ import annotations + +import json +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Final, Iterable + +JUDGE_MODEL_TIER: Final[str] = "anchor" +"""Logical tier label for the judge model. The operator maps this to a +concrete model name in their host CLI's dispatch step; the value +"anchor" means the host's calibration-baseline tier (not the small / +cheaper tier below it). Encoding the tier as a label keeps this module +free of vendor-specific model identifiers and decoupled from any one +host CLI's catalog.""" + +JUDGE_REASON: Final[str] = "needs_llm_judge" +"""Reason tag carried by replay rows that this stage processes. Set by +the deterministic `score_fidelity` path when an `actual` is present +but no substring match was found.""" + +JUDGE_REQUESTS_FILENAME: Final[str] = "judge_requests.jsonl" +JUDGE_RESPONSES_FILENAME: Final[str] = "judge_responses.jsonl" + +JUDGE_PROMPT_TEMPLATE: Final[str] = """\ +You are scoring whether one free-text answer means the same thing as +a reference answer for a memory-system continuation eval. You see only +the two answers — no retrieval context, no user prompt, no history. + +REFERENCE: +{expected} + +CANDIDATE: +{actual} + +Reply with a single JSON object on one line: + {{"matched": true|false, "rationale": "<= 240 chars"}} + +`matched` is true when the candidate conveys the same factual content +as the reference (paraphrase, partial restatement, or equivalent +phrasing all count). It is false when the candidate omits the +reference's load-bearing claim, contradicts it, or is empty. Do not +penalize style differences. +""" +"""Prompt the host-side dispatcher should send to each off-band call. +Anchors the judge in a no-retrieval-context posture (contamination +protocol) and constrains the response to a one-line JSON verdict so +`read_judge_responses` can parse without ambiguity.""" + + +@dataclass(frozen=True) +class JudgeRequest: + """One judge request. Strict schema — `turn_idx` is the join key, + `expected` is the reference answer, `actual` is the candidate. + No `rebuilt_block`, no `user_turn` — that is the contamination + boundary.""" + turn_idx: int + expected: str + actual: str + + +@dataclass(frozen=True) +class JudgeResponse: + """One judge response. `matched` is the verdict; `rationale` is a + short free-text explanation surfaced in the run report.""" + turn_idx: int + matched: bool + rationale: str = "" + + +def _eligible_rows(replay_results: Iterable[dict]) -> list[dict]: + """Filter the replay rows the judge stage processes. + + Eligibility: `reason == JUDGE_REASON` AND `actual` is a non-empty + string AND `expected` is a non-empty string. Rows missing either + field have nothing to compare; rows with the wrong reason were + already settled by the deterministic path. + """ + out: list[dict] = [] + for r in replay_results: + if r.get("reason") != JUDGE_REASON: + continue + expected = r.get("expected") + actual = r.get("actual") + if not isinstance(expected, str) or not expected: + continue + if not isinstance(actual, str) or not actual: + continue + out.append(r) + return out + + +def write_judge_requests( + replay_results: Iterable[dict], + run_dir: Path, + *, + max_judge_calls: int = 0, +) -> int: + """Write eligible replay rows to `/judge_requests.jsonl`. + + Returns the number of rows written. `max_judge_calls` caps the + request count and defaults to 0, which is the disabled state — + no file is written when no calls are budgeted. Operators opt in + by passing a positive cap. + + Each row's on-disk shape is exactly the three fields of + `JudgeRequest`. Asserted by the test suite to prevent retrieval + context from leaking into the judge prompt. + """ + if max_judge_calls <= 0: + return 0 + eligible = _eligible_rows(replay_results) + if not eligible: + return 0 + run_dir.mkdir(parents=True, exist_ok=True) + path = run_dir / JUDGE_REQUESTS_FILENAME + written = 0 + with path.open("w", encoding="utf-8") as f: + for row in eligible: + if written >= max_judge_calls: + break + req = JudgeRequest( + turn_idx=int(row["turn_idx"]), + expected=row["expected"], + actual=row["actual"], + ) + f.write(json.dumps(asdict(req), ensure_ascii=False) + "\n") + written += 1 + return written + + +def read_judge_responses(run_dir: Path) -> dict[int, JudgeResponse]: + """Read `/judge_responses.jsonl` into a turn_idx-keyed dict. + + Returns an empty dict if the file is absent (the operator hasn't + run the dispatch step yet) or if every line is malformed. Lines + that fail to parse, or that lack the required fields, are silently + skipped — partial responses are preferable to a full run failure + when the harness is iterating across many turns. + """ + path = run_dir / JUDGE_RESPONSES_FILENAME + if not path.exists(): + return {} + out: dict[int, JudgeResponse] = {} + with path.open("r", encoding="utf-8") as f: + for line in f: + line = line.strip() + if not line: + continue + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if not isinstance(obj, dict): + continue + turn_idx = obj.get("turn_idx") + matched = obj.get("matched") + if not isinstance(turn_idx, int) or not isinstance(matched, bool): + continue + rationale = obj.get("rationale", "") + if not isinstance(rationale, str): + rationale = "" + out[turn_idx] = JudgeResponse( + turn_idx=turn_idx, matched=matched, rationale=rationale + ) + return out + + +def apply_judge_verdicts( + replay_results: list[dict], + responses: dict[int, JudgeResponse], +) -> list[dict]: + """Fold judge verdicts back into the replay rows. + + Pure function — returns a new list with judge-eligible rows + updated. For each row tagged `JUDGE_REASON` with a matching + response: `matched` becomes the judge verdict, `reason` is + cleared (set to empty string), and a `judge_rationale` field is + added. Rows without a response keep `reason=JUDGE_REASON` so the + next harness invocation can retry. + + Non-eligible rows pass through untouched. + """ + updated: list[dict] = [] + for row in replay_results: + if row.get("reason") != JUDGE_REASON: + updated.append(dict(row)) + continue + turn_idx = row.get("turn_idx") + if not isinstance(turn_idx, int) or turn_idx not in responses: + updated.append(dict(row)) + continue + verdict = responses[turn_idx] + new_row = dict(row) + new_row["matched"] = verdict.matched + new_row["reason"] = "" + new_row["judge_rationale"] = verdict.rationale + updated.append(new_row) + return updated diff --git a/tests/test_context_rebuilder_eval_judge.py b/tests/test_context_rebuilder_eval_judge.py new file mode 100644 index 000000000..1e0329335 --- /dev/null +++ b/tests/test_context_rebuilder_eval_judge.py @@ -0,0 +1,329 @@ +"""Tests for the LLM-judge stage of the context-rebuilder eval harness. + +Round-trip only. CI never invokes a real model — the judge response +file is hand-authored in each test fixture, exactly as it would be +hand-authored by an operator running the host-side dispatch. + +Two contracts are load-bearing and asserted explicitly: + + 1. The contamination boundary (`docs/BENCHMARKS.md`): the judge + request file carries only `(turn_idx, expected, actual)` — no + rebuilt block, no user turn. + + 2. The cost cap: `max_judge_calls=0` (the default) writes no file + and dispatches no calls, so CI runs are free. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import pytest + +# The benchmarks/context-rebuilder directory is hyphenated, so `python -m` +# imports do not work. Resolve the module via path hop, the same shim the +# eval harness uses. +_HARNESS_DIR = Path(__file__).resolve().parents[1] / "benchmarks" / "context-rebuilder" +if str(_HARNESS_DIR) not in sys.path: + sys.path.insert(0, str(_HARNESS_DIR)) + +from judges import llm_judge # noqa: E402 + + +def _replay_row( + *, turn_idx: int, reason: str, expected: str, actual: str, +) -> dict: + return { + "turn_idx": turn_idx, + "expected": expected, + "actual": actual, + "matched": False, + "reason": reason, + } + + +# --- 1. contamination boundary ------------------------------------------- + +def test_judge_request_schema_carries_only_turn_idx_expected_actual( + tmp_path: Path, +) -> None: + """The request file must not surface retrieval context to the judge. + + `docs/BENCHMARKS.md` requires that generation and scoring run as + separate passes and that the judge never sees the rebuilt context. + This test fails loud if a future edit widens the schema to include + `rebuilt_block`, `user_turn`, or any other retrieval-pass field. + """ + rows = [ + _replay_row( + turn_idx=4, reason=llm_judge.JUDGE_REASON, + expected="cached locks are demoted after 30 days idle", + actual="locks are evicted by the idle-30d sweep", + ), + ] + n = llm_judge.write_judge_requests(rows, tmp_path, max_judge_calls=5) + assert n == 1 + path = tmp_path / llm_judge.JUDGE_REQUESTS_FILENAME + assert path.exists() + line = path.read_text(encoding="utf-8").splitlines()[0] + obj = json.loads(line) + assert set(obj.keys()) == {"turn_idx", "expected", "actual"}, ( + f"judge request schema widened to {sorted(obj.keys())} — would leak " + "retrieval context into the judge prompt" + ) + + +def test_judge_request_writer_skips_non_judge_rows(tmp_path: Path) -> None: + """Rows tagged with other reasons (substring-match settled, + pending_replay, needs_replay_client) must not appear in the judge + request file. Only rows the deterministic path could not settle + are the judge's concern.""" + rows = [ + _replay_row( + turn_idx=1, reason="", # already settled by substring match + expected="x", actual="x", + ), + _replay_row( + turn_idx=2, reason="pending_replay", + expected="y", actual="", + ), + _replay_row( + turn_idx=3, reason=llm_judge.JUDGE_REASON, + expected="z", actual="paraphrased z", + ), + ] + n = llm_judge.write_judge_requests(rows, tmp_path, max_judge_calls=5) + assert n == 1 + payload = (tmp_path / llm_judge.JUDGE_REQUESTS_FILENAME).read_text( + encoding="utf-8" + ) + objs = [json.loads(line) for line in payload.splitlines() if line.strip()] + assert [o["turn_idx"] for o in objs] == [3] + + +def test_judge_request_writer_skips_empty_expected_or_actual( + tmp_path: Path, +) -> None: + """A judge-tagged row with no `expected` text or no `actual` text + has nothing to compare. Including it would waste a call and + confuse the verdict.""" + rows = [ + _replay_row( + turn_idx=1, reason=llm_judge.JUDGE_REASON, + expected="", actual="something", + ), + _replay_row( + turn_idx=2, reason=llm_judge.JUDGE_REASON, + expected="something", actual="", + ), + ] + n = llm_judge.write_judge_requests(rows, tmp_path, max_judge_calls=5) + assert n == 0 + assert not (tmp_path / llm_judge.JUDGE_REQUESTS_FILENAME).exists() + + +# --- 2. cost cap --------------------------------------------------------- + +def test_judge_disabled_by_default_writes_no_file(tmp_path: Path) -> None: + """With `max_judge_calls=0` (the default), no file is written and + no requests are produced even when eligible rows are present. + This is what keeps CI free.""" + rows = [ + _replay_row( + turn_idx=i, reason=llm_judge.JUDGE_REASON, + expected=f"ref-{i}", actual=f"cand-{i}", + ) + for i in range(3) + ] + n = llm_judge.write_judge_requests(rows, tmp_path) + assert n == 0 + assert not (tmp_path / llm_judge.JUDGE_REQUESTS_FILENAME).exists() + + +def test_judge_cost_cap_binds_before_writing(tmp_path: Path) -> None: + """`max_judge_calls` is the hard ceiling on requests written, not + just an advisory. With 5 eligible rows and cap=2, exactly 2 rows + land in the file. The cap binds before write to protect cost + even if eligibility logic over-counts.""" + rows = [ + _replay_row( + turn_idx=i, reason=llm_judge.JUDGE_REASON, + expected=f"ref-{i}", actual=f"cand-{i}", + ) + for i in range(5) + ] + n = llm_judge.write_judge_requests(rows, tmp_path, max_judge_calls=2) + assert n == 2 + payload = (tmp_path / llm_judge.JUDGE_REQUESTS_FILENAME).read_text( + encoding="utf-8" + ) + assert len(payload.splitlines()) == 2 + + +# --- 3. response round-trip --------------------------------------------- + +def test_judge_response_reader_joins_by_turn_idx(tmp_path: Path) -> None: + """Pre-write a hand-authored response file (as an operator would + after running the host-side dispatch), read it back, assert each + response is keyed by turn_idx and carries the verdict.""" + path = tmp_path / llm_judge.JUDGE_RESPONSES_FILENAME + path.write_text( + "\n".join( + [ + json.dumps({"turn_idx": 3, "matched": True, "rationale": "paraphrase"}), + json.dumps({"turn_idx": 7, "matched": False, "rationale": "omits load-bearing fact"}), + ] + ) + + "\n", + encoding="utf-8", + ) + responses = llm_judge.read_judge_responses(tmp_path) + assert set(responses.keys()) == {3, 7} + assert responses[3].matched is True + assert responses[3].rationale == "paraphrase" + assert responses[7].matched is False + + +def test_judge_response_reader_returns_empty_when_file_absent( + tmp_path: Path, +) -> None: + """No response file → empty dict, not an exception. The harness + runs the read-step optimistically; the operator may not have run + the dispatch yet.""" + assert llm_judge.read_judge_responses(tmp_path) == {} + + +def test_judge_response_reader_skips_malformed_lines(tmp_path: Path) -> None: + """A partially-corrupted response file still returns the well-formed + rows. Better partial coverage than full failure when iterating on + many turns.""" + path = tmp_path / llm_judge.JUDGE_RESPONSES_FILENAME + path.write_text( + "\n".join( + [ + "not-json", + json.dumps({"turn_idx": "wrong-type", "matched": True}), + json.dumps({"turn_idx": 5, "matched": True, "rationale": "ok"}), + json.dumps({"turn_idx": 6}), # missing matched + ] + ) + + "\n", + encoding="utf-8", + ) + responses = llm_judge.read_judge_responses(tmp_path) + assert set(responses.keys()) == {5} + + +# --- 4. fold verdicts into replay rows ---------------------------------- + +def test_apply_judge_verdicts_clears_reason_and_sets_matched() -> None: + """For each judge-tagged row with a matching response: `matched` + becomes the verdict, `reason` clears, and `judge_rationale` is + added. Non-judge rows pass through untouched. Rows without a + response keep their needs_llm_judge tag for the next retry.""" + rows = [ + _replay_row( + turn_idx=1, reason="", + expected="x", actual="x", + ), + _replay_row( + turn_idx=2, reason=llm_judge.JUDGE_REASON, + expected="ref-2", actual="cand-2", + ), + _replay_row( + turn_idx=3, reason=llm_judge.JUDGE_REASON, + expected="ref-3", actual="cand-3", + ), + ] + responses = { + 2: llm_judge.JudgeResponse(turn_idx=2, matched=True, rationale="ok"), + # 3 has no response — operator hasn't run dispatch for it yet. + } + out = llm_judge.apply_judge_verdicts(rows, responses) + assert out[0]["matched"] is False # unchanged (non-judge row) + assert out[0]["reason"] == "" + assert out[1]["matched"] is True + assert out[1]["reason"] == "" + assert out[1]["judge_rationale"] == "ok" + assert out[2]["matched"] is False # unchanged (no response yet) + assert out[2]["reason"] == llm_judge.JUDGE_REASON + + +def test_apply_judge_verdicts_does_not_mutate_input() -> None: + """The function is pure — mutating callers would couple it to + whatever upstream owns the replay-row list.""" + rows = [ + _replay_row( + turn_idx=1, reason=llm_judge.JUDGE_REASON, + expected="x", actual="y", + ), + ] + snapshot = json.dumps(rows, sort_keys=True) + llm_judge.apply_judge_verdicts( + rows, + {1: llm_judge.JudgeResponse(turn_idx=1, matched=True)}, + ) + assert json.dumps(rows, sort_keys=True) == snapshot + + +# --- 5. round-trip end-to-end ------------------------------------------- + +def test_end_to_end_round_trip(tmp_path: Path) -> None: + """Eligible rows → write → operator writes responses (here, + hand-authored) → read → apply → assert verdicts folded in.""" + rows = [ + _replay_row( + turn_idx=1, reason=llm_judge.JUDGE_REASON, + expected="lock posterior is Beta-Bernoulli", + actual="locks use a Beta-Bernoulli posterior", + ), + _replay_row( + turn_idx=2, reason=llm_judge.JUDGE_REASON, + expected="locks survive demotion sweeps", + actual="the demotion sweep ignores locks", + ), + ] + n = llm_judge.write_judge_requests(rows, tmp_path, max_judge_calls=10) + assert n == 2 + + # Operator dispatches host-side and writes the response file. + (tmp_path / llm_judge.JUDGE_RESPONSES_FILENAME).write_text( + "\n".join( + [ + json.dumps({"turn_idx": 1, "matched": True, "rationale": "paraphrase"}), + json.dumps({"turn_idx": 2, "matched": True, "rationale": "equivalent"}), + ] + ) + + "\n", + encoding="utf-8", + ) + + responses = llm_judge.read_judge_responses(tmp_path) + folded = llm_judge.apply_judge_verdicts(rows, responses) + assert all(r["matched"] for r in folded) + assert all(r["reason"] == "" for r in folded) + + +# --- 6. anchor constants -------------------------------------------------- + +def test_judge_model_tier_is_anchor() -> None: + """The judge tier label is "anchor" — the operator maps this to + the host CLI's calibration-baseline tier, not the cheaper small- + model tier below it. Prior calibration work (Cohen's-κ against a + zero-LLM baseline) was measured at the anchor tier; dropping to + a cheaper tier weakens comparability. This is a flag-flip canary; + intentional changes should update the constant *and* this test + together.""" + assert llm_judge.JUDGE_MODEL_TIER == "anchor" + + +@pytest.mark.parametrize( + "field", + ["expected", "actual"], +) +def test_judge_prompt_template_contains_required_field(field: str) -> None: + """The prompt template must surface the expected and actual fields + so the host-side dispatcher can render concrete prompts.""" + assert "{" + field + "}" in llm_judge.JUDGE_PROMPT_TEMPLATE