Record why a StructuredJson run is rejected (3.7) - #190
Merged
Conversation
Executed the fresh-base run this task was waiting on: 2 questions under --judge-protocol structured-json. It was rejected, and the reason is more useful than the comparability caveat the task was written around. The plan expected the risk to be incomparability with a free-text base. The actual blocker is that our own run validator is FreeText-shaped and refuses a StructuredJson run outright, on two independent counts. Call accounting: it observed 4 LLM calls, classified 2 as diagnostic judge retries, and was left with 0 base judge calls against an expected 2-6. The StructuredJson judge's call pattern is not the arithmetic that guard encodes. Correctness reconciliation: "AgentEval judge verdict and recorded correctness disagree" fired for every question in both arms, which says our recorded correctness is still derived on the free-text path regardless of which protocol the judge ran under. Not fixed by loosening the guard. Widening the call bounds and relaxing the reconciliation would produce a green run and a number that looks measured because the thing that would have objected was tuned away -- the exact failure this evaluation track exists to prevent. Bounds loose enough to admit both judge shapes admit real anomalies too. The deliverable is therefore a specification rather than a green run, written down in docs/reviews rather than only in the gitignored plan: teach the validator the StructuredJson call shape, read correctness from the structured verdict, then re-run on a fresh base and report it as a protocol change. The run was worth its cost for turning a guessed-at caveat into that list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
There was a problem hiding this comment.
Pull request overview
Adds a review note documenting why JudgeVerdictProtocol.StructuredJson runs are currently rejected by the existing (FreeText-shaped) run validator, and enumerates the concrete follow-up work needed to make structured-JSON judging evaluatable in this repo.
Changes:
- Documents the two validator blockers encountered in the first
--judge-protocol structured-jsonrun (call accounting + correctness reconciliation). - Specifies the required validator/correctness changes and why simply loosening guards is not acceptable for this evaluation track.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| @@ -0,0 +1,60 @@ | |||
| # StructuredJson judge protocol: reachable, and currently rejected | |||
Comment on lines
+59
to
+60
| Estimated S–M against AgentEval's contract. The run was worth its cost for turning a guessed-at | ||
| comparability caveat into this list. |
This was referenced Aug 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ran the fresh-base run this task was waiting on — 2 questions under
--judge-protocol structured-json. Rejected, and the reason is more useful than the comparability caveat the task was written around.The plan expected the risk to be incomparability with a free-text base. The actual blocker: our own run validator is FreeText-shaped and refuses a StructuredJson run outright, on two independent counts.
Call accounting — 4 LLM calls observed, 2 classified as diagnostic judge retries, leaving 0 base judge calls against an expected 2–6. The StructuredJson judge's call pattern is not the arithmetic that guard encodes.
Correctness reconciliation —
AgentEval judge verdict and recorded correctness disagreefired for every question in both arms. Our recorded correctness is still derived on the free-text path regardless of which protocol ran.Not fixed by loosening the guard. Widening the bounds and relaxing the reconciliation gives a green run and a number that looks measured because the thing that would have objected was tuned away — the exact failure this track exists to prevent. Bounds loose enough to admit both judge shapes admit real anomalies too.
So the deliverable is a specification, written to
docs/reviews/rather than only the gitignored plan: teach the validator the StructuredJson call shape, read correctness from the structured verdict, re-run on a fresh base, report as a protocol change. Docs only — no source changes.🤖 Generated with Claude Code
https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE