feat(v1): configurable agentic-judge grading policy, uploads, and reward weights

[mikasenghaas]

Summary

Makes the built-in agentic-judge env a complete out-of-the-box judge, configurable end-to-end from TOML/CLI so domain-specific setups (SWE, search, ...) are pure .md + .toml artifacts:

• One box, shared — the box is provisioned once from the solver's runtime policy (--env.solver.runtime.type docker|prime, checked at construction); the solver plays the task in it and the judge then inspects the work exactly as the agent left it. The judge pins only its model; its runtime policy is unused. (Borrowed boxes are never retried into, so per-agent rollout retries don't apply; episode retries still do.)
• [env.task] (JudgeTaskConfig) — the judge's minted task:
  • prompt: grading-policy override, one field — a value ending in .md/.txt is read from disk, anything else is inline policy text (a missing file fails at env construction). A policy replaces only the judging instructions: the verdict contract and workspace note are always appended, so it can never break verdict parsing. {prompt} interpolates the solver task's prompt; if absent, the task statement is appended.
  • hint: optional hints injected as their own section (inline text or a .md/.txt file) — task-family pointers into the trace or box, e.g. where the reference answer lives for math, or "diff the repo / read info.patch" for SWE.
  • rubric: criteria the judge grades against — a .toml/.json criteria file in the plugged rubric judge's format ({name, text, weight=1, choices worst→best}; logic mirrored, not imported), defaulting to a single built-in solved criterion. The judge writes {"verdicts": [{"name", "reason", "verdict"}, ...]} to /tmp/verdict.json; finalize validates strictly (name set must match, only on-menu answers), records judge/<name> metrics on the solver trace, and rewards the weighted mean.
  • The solver's COMPLETE raw trace record always lands at /tmp/trace.json (not configurable) — messages, tool calls, trace.info artifacts (e.g. a captured patch), and the task row with whatever scores/reference material it carries; nothing is stripped. Independence is prompted, not enforced: the workspace note tells the judge recorded scores and references are context, never the standard, and warns that the record can be very large — peek selectively, never dump it whole. A policy that needs a record field as a file instructs the judge to extract it.
• [env.score] (ScoreConfig) — reward composition on the solver's trace: task_weight (default 0: judge-only reward) rescales the taskset's own rewards, judge_weight (default 1) scales the verdict.

Verification

• uv run eval @ configs/agentic_judge.toml (gsm8k, docker, glm-5.2), re-smoked after every design round (rubric verdicts, shared-only box, full-record upload): trace records show the judge on the borrowed solver box, verdicts land as judge/solved + the judge reward, and the default score.task_weight=0 zeroes the task's own reward.
• scaleswe-v1 through --env.id agentic-judge (prime + docker, bash seats), and a 64-instance dogfood: the judge flipped 8/9 manually-identified narrow-test false negatives to ≥0.7 while keeping 7/8 pytest-passing controls at 1.0; on a re-run with rubric verdicts and judge-side patch extraction it flipped 17/23 pytest-failures and kept 36/40 passes, agreeing with the earlier judge on every audited case. Gap experiment, run twice on 256 fixed-seed instances x {glm-4.5-air, glm-5.2}: judge scoring widened the model gap from +0.206 to +0.290 (continuous 0-10 verdicts; 243 pairs) and, on the rubric-verdict re-run, from +0.210 to +0.380 (binary solved; 224 pairs; delta +0.170, 95% bootstrap CI [+0.089, +0.246]). (Experiment builds predate the shared-box/full-record round.)
• ruff check, ruff format --check, repo ty check (diagnostic count unchanged vs main), full non-e2e tests/v1 green; test_env_id_agentic_judge covers the config surface (needs PRIME_API_KEY + docker).

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit ca25ca9. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR adds new feature capabilities including configurable rubric criteria, custom prompt/hint files, weighted reward composition, and changes the architecture so the judge shares the solver's container. These new features and runtime behavior changes warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}