feat(eval-harness): judges/llm_judge.py — opt-in LLM-judge stage (#592) - #613
Merged
Conversation
Adds the LLM-judge stage for commit-3 of #592. Open-ended replay rows that the deterministic score path tags `reason="needs_llm_judge"` are written to a per-run `judge_requests.jsonl`; the host CLI dispatches one off-band model call per row at the anchor tier and writes `judge_responses.jsonl`; the harness joins the verdicts back by `turn_idx`. Contamination protocol (docs/BENCHMARKS.md): the judge sees only `(turn_idx, expected, actual)`. Retrieval context (rebuilt block, user turn) never reaches the judge, per the Pass 1 / Pass 2 separation required for memory-system eval integrity. Cost posture: `max_judge_calls=0` by default — the stage is disabled and writes nothing in CI. Operators opt in by passing a positive cap; the cap binds before file I/O, so misconfigured runs cannot exceed their budget. Tier label is "anchor"; the operator maps that to their host CLI's calibration-baseline tier (not the small-model tier below it) so comparability against the prior calibration work holds. No provider SDK import. Dispatch is host-polymorphic, matching /aelf:onboard's LLM-classify pattern and #600's replay split. Refs #592 (commit-3 of three). Independent of PR #601 (commit-2); their contracts join at the `reason="needs_llm_judge"` tag.
…#592) 14 tests covering the judge stage's contracts: - contamination boundary: request file carries only (turn_idx, expected, actual); a future schema widening that adds rebuilt_block / user_turn fails this test loudly. - cost cap: max_judge_calls=0 (default) writes no file; positive caps bind before write so misconfigured runs cannot exceed budget. - eligibility: only `reason="needs_llm_judge"` rows with non-empty expected and actual are surfaced for judging. - response round-trip: malformed lines skipped; absent file returns {}. - fold verdicts: pure function, clears `reason` on matched rows, preserves needs_llm_judge tag when no response present yet. - anchor constants: JUDGE_MODEL_TIER=='anchor' canary, prompt-template carries the two judgeable fields. CI invokes no real model; the operator-side dispatch is hand-mocked via pre-authored judge_responses.jsonl, identical in shape to what a host CLI would write.
README adds a self-contained "LLM-judge stage (commit-3 of #592)" section near the bottom: contamination-boundary rationale, the three-step operator flow (write requests -> host-side dispatch -> fold verdicts), and a forward pointer to the harness-level wiring that joins onto PR #601's --run-dir plumbing. Section placement is deliberately near the end of the file so the diff does not touch PR #601's likely edit region (the existing "Why LLM-judge is parked" / "Method toggle" subsections under "Continuation-fidelity scoring"). CHANGELOG entry under [Unreleased] / ### Added, ahead of the existing #592 commit-1 entry. Documents the contamination boundary + default-off cost posture + anchor-tier judge decision.
Reviewer's GuideAdds an opt-in, file-based LLM-judge stage for open-ended context-rebuilder eval rows plus tests and docs, without touching the main harness flow. Sequence diagram for the opt-in LLM-judge operator flowsequenceDiagram
actor Operator
participant EvalHarness as EvalHarness_or_helper
participant LLMJudge as llm_judge_module
participant FS as FileSystem
participant HostCLI as Host_CLI_dispatcher
Operator->>EvalHarness: Run eval_harness.py (produces replay_results.jsonl)
EvalHarness-->>Operator: replay_results.jsonl with reason=needs_llm_judge rows
Operator->>LLMJudge: write_judge_requests(rows, run_dir, max_judge_calls=N)
LLMJudge->>LLMJudge: _eligible_rows(filter by reason, expected, actual)
alt max_judge_calls <= 0 or no eligible rows
LLMJudge-->>Operator: 0 (no file written)
else
LLMJudge->>FS: Write judge_requests.jsonl (turn_idx, expected, actual)
LLMJudge-->>Operator: n_requests
end
alt Operator opts in to judge stage
Operator->>HostCLI: Read judge_requests.jsonl
HostCLI->>HostCLI: For each row, call anchor-tier model with JUDGE_PROMPT_TEMPLATE
HostCLI->>FS: Write judge_responses.jsonl (turn_idx, matched, rationale)
end
Operator->>LLMJudge: responses = read_judge_responses(run_dir)
LLMJudge->>FS: Read judge_responses.jsonl
LLMJudge-->>Operator: dict[turn_idx] = JudgeResponse
Operator->>LLMJudge: folded = apply_judge_verdicts(rows, responses)
LLMJudge->>LLMJudge: For each row with reason=needs_llm_judge and response
LLMJudge-->>Operator: updated rows with matched, judge_rationale, cleared reason
Operator->>EvalHarness: Use folded rows in subsequent analysis
Class diagram for the new LLM-judge module types and helpersclassDiagram
class llm_judge_module {
<<module>>
+JUDGE_MODEL_TIER : str
+JUDGE_REASON : str
+JUDGE_REQUESTS_FILENAME : str
+JUDGE_RESPONSES_FILENAME : str
+JUDGE_PROMPT_TEMPLATE : str
+_eligible_rows(replay_results)
+write_judge_requests(replay_results, run_dir, max_judge_calls)
+read_judge_responses(run_dir)
+apply_judge_verdicts(replay_results, responses)
}
class JudgeRequest {
<<dataclass>>
+turn_idx : int
+expected : str
+actual : str
}
class JudgeResponse {
<<dataclass>>
+turn_idx : int
+matched : bool
+rationale : str
}
llm_judge_module ..> JudgeRequest : writes
llm_judge_module ..> JudgeResponse : reads
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
robotrocketscience
added
the
attn:review
PR-size soft capThis PR is over the advisory size threshold:
Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- In
_eligible_rows, you currently only checkexpected/actualfor non-empty strings; consider treating whitespace-only values as ineligible (e.g.,if not expected.strip(): continue) to avoid wasting judge calls on effectively empty content. read_judge_responsessilently drops malformed lines and type mismatches; adding at least a lightweight counter or debug hook (e.g., returning the number of skipped lines or logging at debug level) would make it easier to detect operator/dispatch issues without changing the happy-path behavior.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `_eligible_rows`, you currently only check `expected`/`actual` for non-empty strings; consider treating whitespace-only values as ineligible (e.g., `if not expected.strip(): continue`) to avoid wasting judge calls on effectively empty content.
- `read_judge_responses` silently drops malformed lines and type mismatches; adding at least a lightweight counter or debug hook (e.g., returning the number of skipped lines or logging at debug level) would make it easier to detect operator/dispatch issues without changing the happy-path behavior.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Owner
Author
|
[claim:review:noether:2026-05-10T23:48:45Z] |
Owner
Author
|
[release:review:noether:2026-05-10T23:49:52Z] |
This was referenced May 10, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes part of #592 (commit-3 of three: LLM-judge stage). Commit-1 shipped as #596; commit-2 (replay split) is in flight as PR #601.
What lands
A standalone judge module under
benchmarks/context-rebuilder/judges/llm_judge.pythat takes the open-ended replay rows the deterministic substring scorer cannot settle (reason="needs_llm_judge") and routes them through an opt-in, off-band judging stage. The harness's main flow is untouched in this PR; the integration commit that wires--run-dirto the judge stage joins onto PR #601's plumbing in a follow-up.Three commits, atomic:
feat(eval-harness): judges/llm_judge.py for open-ended fidelity (#592)— the module:JudgeRequest/JudgeResponsedataclasses,write_judge_requests,read_judge_responses,apply_judge_verdicts, the prompt template, and the tier-label constant. No provider SDK import; dispatch is host-polymorphic per/aelf:onboard's established pattern.test(eval-harness): judges/llm_judge round-trip + contamination guard (#592)— 14 deterministic tests covering the contamination boundary, the cost cap, round-trip parsing, malformed-line skip, fold-verdicts purity, and the anchor-tier canary.docs(eval-harness): LLM-judge stage operator flow + CHANGELOG (#592)— README section near the bottom ofbenchmarks/context-rebuilder/README.md(deliberately placed away from PR feat(eval-harness): replay_post_fork host-agent JSONL split (closes #600) #601's likely edit region) plus an unreleased CHANGELOG entry under### Added.Two design decisions diverge from the issue draft
The issue body's draft sketches the LLM-judge stage but two pieces of it conflict with prior-decision artifacts in the repo. Both divergences were confirmed before claim.
Judge prompt shape. Issue draft proposed
(rebuilt_block, user_turn, expected, actual). The shippeddocs/BENCHMARKS.mdis explicit ("Generation and scoring are separate passes. The judge never sees the retrieval context"), and lists "LLM self-judging with answer visible" as one of three contamination modes that produces 0% results. Including the rebuilt block in the judge prompt would let the judge patch the candidate with details the candidate didn't produce, inflating fidelity. This PR ships the strict shape:(turn_idx, expected, actual)only. Asserted intest_judge_request_schema_carries_only_turn_idx_expected_actual— a future widening fails loudly.Judge model tier. Issue draft proposed the cheaper small-model tier. The repo's prior calibration work (Cohen's-κ disagreement methodology against a zero-LLM baseline) was measured at the host CLI's anchor tier. Dropping to the small tier weakens comparability on exactly the open-ended turns this stage exists to score. This PR encodes the tier via a logical label (
JUDGE_MODEL_TIER = "anchor") so the operator's host-CLI catalog can map it to the appropriate concrete model without baking vendor identifiers into the module.Cost posture
max_judge_calls=0by default — the stage is disabled, no file is written, no calls are issued. CI runs are free and outbound traffic is opt-in. When operators pass a positive cap, the cap binds before any file I/O so a misconfigured run cannot exceed the call budget. The harness layer that automates the dispatch step lands later (it depends on PR #601's--run-dircontract).Operator flow
Three pure-Python helpers plus an operator-driven host-CLI dispatch step in the middle. README has the worked example with paste-ready scripts:
llm_judge.write_judge_requests(rows, run_dir, max_judge_calls=N)— produces<run_dir>/judge_requests.jsonl.<run_dir>/judge_responses.jsonl.llm_judge.apply_judge_verdicts(rows, llm_judge.read_judge_responses(run_dir))— folds verdicts back into the replay rows, clearsreasonon matched rows, addsjudge_rationale.Why this doesn't conflict with PR #601
PR #601 (Planck, in
attn:merge-conflictat time of writing) closes #600 — thereplay_post_forkhost-CLI split and the substring half ofscore_fidelity. Its body explicitly out-of-scopes the LLM-judge ("LLM-judge for open-ended turns — commit-3 of #592, separate cost-bounded surface"). This PR's only contact with PR #601's edit surface is the sharedreason="needs_llm_judge"tag, which is a string constant — no source-file overlap witheval_harness.pyorscore_fidelityproper.The README is technically a shared file but the new section is appended near the bottom, away from the "Why LLM-judge is parked" / "Method toggle" subsections #601 will likely touch.
Test plan
uv run pytest tests/test_context_rebuilder_eval_judge.py -v— 14 passed locallypytest (3.12)andpytest (3.13)Summary by Sourcery
Introduce an optional LLM-based judging stage for open-ended context-rebuilder eval turns, with request/response helpers, strict contamination boundary, and operator-driven dispatch, plus supporting docs and tests.
New Features:
Enhancements:
Documentation:
Tests: