feat: judge failure handling

[tdene]

A zero reward from a judge model has three distinct causes that are currently indistinguishable:

• The response was malformatted, and thus never judged.
• The judge's judging response was itself malformatted.
• The judge genuinely provided a score of 0.

The middle case leads to severe training run degradation, should the judge model become corrupted - which is not uncommon.

This PR merely surfaces the failure reason, emitting judge_unparseable if the problem is with the judge, allowing upstream training infrastructure to decide how to best handle such judge degradation.

[tdene]

/ok to test 66bb6a5

[tdene]

/claude review

[claude]

SHIP — clean, well-scoped, well-tested.

This PR decomposes proof_judge's zero-rewards into a named failure_reason so training pipelines can distinguish policy-format failures (automatic 0) from judge-unparseable failures (judge itself broke). Reviewed for scoring correctness, request-contract compatibility, and test coverage.

Reward math is unchanged. The refactor extract_boxed_score(...) or 0.0 → score if score is not None else 0.0 is numerically identical: extract_boxed_score only ever returns {0, 0.5, 1, None}, and 0.0 or 0.0 == 0.0. The change only adds the judge_unparseable signal on top — no reward drift into RLHF signal. Verified for both beta==0 and beta!=0 paths, including the r_meta_score is None case.

Request contract — problem now required (was problem: str = ""). A missing problem now 422s instead of silently verifying against an empty problem string, which is the correct fail-loud behavior for a verifier — an empty problem would have produced a garbage judge prompt and a meaningless score. Confirmed all in-repo datasets carry a top-level problem (data/example.jsonl, rollouts, and materialized inputs all 5/5), and prepare_data.py always emits it. The group-zeroing path (_maybe_zero_incorrect_group_reward) preserves details["reason"] via spread, so failure_reason survives that branch. Operational note for the author: any external caller or cached/replayed verify request lacking a top-level problem will now 422 where it previously scored 0 — intended, but worth a heads-up if such callers exist outside this repo.

Response schema — failure_reason: Optional[str] = None is an additive optional field; backward compatible with existing consumers.

Tests — good coverage: parametrized cases assert reward + failure_reason + judge call-count for genuine-verdict / judge-unparseable / policy-format-failure, plus a required-problem rejection test. asyncio_mode = "auto" is set, so the async tests execute.

One NOTE inline re: the "422 if empty" comment being slightly inaccurate (str rejects missing, not empty). Author's call.

[claude]

NOTE: The comment says "Force a 422 error if the problem is empty," but problem: str only rejects a missing field — an empty string "" still validates. If you actually want to reject empty problems, use problem: str = Field(min_length=1). Otherwise reword the comment to "if the problem is missing." Not a correctness issue given all in-repo datasets supply a non-empty problem.

[tdene]

Fixed, thank you!

[tdene]

/ok to test 7307b03

[tdene]

/claude review

[claude]

SHIP — no reliability concerns.

Reviewed resources_servers/proof_judge/app.py and its tests.

• Correctness fix (the point of the PR): replacing extract_boxed_score(...) or 0.0 with an explicit None check correctly separates a genuine 0.0 judge verdict from an unparseable/refused judge response. Reward values are numerically unchanged; the new judge_unparseable → failure_reason signal is what lets downstream training/eval distinguish "judge said 0" from "judge failed to answer." This was a real latent conflation. Both beta == 0 and beta != 0 paths handle it, and the meta-verifier path checks both r_y and r_meta.
• problem: str = Field(min_length=1): now returns 422 on missing/empty problem instead of silently defaulting to "". Deliberate and documented; the dataset carries problem at top level, and an empty problem would previously have produced a meaningless verifier prompt — failing loud is the better behavior.
• Tests: cover the three zero-reward branches (judged verdict, judge-unparseable, format failure) and assert judge call counts to confirm the judge isn't consulted on format failures. asyncio_mode = auto is set, so the async tests execute.

No async/HTTP-client, public-API, dependency, or config-convention issues. Reward aggregation math is untouched.

[e-dobrowolska]

I think this is not needed anymore, now that we set problem: str = Field(min_length=1)

[tdene]

Oops. I knee-jerk addressed the claude review without thinking. Fixed!

[e-dobrowolska]

This PR looks correct in a sense that it will do its job without breaking anything else. However, the previous PR #2113 introduced a unification on how we handle judge failures, so making changes just to one harness does not seem right.

As for now, the unified approach assumes that unparsable responses are not judge failures. This means that the judge had a fair chance of giving a verdict (i.e. we received a response from the API) and it just happened to be not what we expected. It was not considered a judge failure because judge failures currently mean the call itself failed (eg. due to rate limiting, auth issues, timeouts, HTTP errors) so we never got a verdict at all. We could change this approach to include unparsable responses as judge failures but this change should be applied uniformly, to other benchmarks as well.

[tdene]

This PR looks correct in a sense that it will do its job without breaking anything else. However, the previous PR #2113 introduced a unification on how we handle judge failures, so making changes just to one harness does not seem right.

As for now, the unified approach assumes that unparsable responses are not judge failures. This means that the judge had a fair chance of giving a verdict (i.e. we received a response from the API) and it just happened to be not what we expected. It was not considered a judge failure because judge failures currently mean the call itself failed (eg. due to rate limiting, auth issues, timeouts, HTTP errors) so we never got a verdict at all. We could change this approach to include unparsable responses as judge failures but this change should be applied uniformly, to other benchmarks as well.

I agree in bulk, and I think we're on the same page.

As for now, the unified approach assumes that unparsable responses are not judge failures.

I don't consider this PR changing that contract. The point of this PR is just to surface judge_unparseable under some kind of readable field, that we call failure_reason here. But failure_reason is just a field that gets returned along with a normal 200 response.

The idea is that the training backend can continue to receive 200 responses in the same cases from Gym, but has an extra field they can use to make judgment calls. In this case, my team is treating repeated judge_unparseable 200s as a sign that the judge has been corrupted, which sometimes happens and requires a re-host.

this change should be applied uniformly, to other benchmarks as well.

I 100% agree; I wasn't familiar enough with the codebase to realize this should/could be done. I've put a draft PR at #2552; do you think this is sufficient?

I actually have this exact failure_reason mechanic in 3 of my PRs: #2384 #2385 #2410. So some kind of unified approach sounds really good to me. I'm just not sure what that looks like.

---

Also, the reason I ended up doing things this way is because failure_reason already exists across pre-existing harnesses: swe_pivot, text_to_sql, bird_sql, terminus_judge, spider2_lite. And they all use it the same way I am: they emit failure_reason on 200s.

[tdene]

This PR looks correct in a sense that it will do its job without breaking anything else. However, the previous PR #2113 introduced a unification on how we handle judge failures, so making changes just to one harness does not seem right.

As for now, the unified approach assumes that unparsable responses are not judge failures. This means that the judge had a fair chance of giving a verdict (i.e. we received a response from the API) and it just happened to be not what we expected. It was not considered a judge failure because judge failures currently mean the call itself failed (eg. due to rate limiting, auth issues, timeouts, HTTP errors) so we never got a verdict at all. We could change this approach to include unparsable responses as judge failures but this change should be applied uniformly, to other benchmarks as well.

Done in #2552; merged.

[ananthsub]

after #2552 this re-declaration isnt needed