[faresobeid]

Privileged grader materials (revision 91508f8)

• The grader gets the inner task's privileged materials under /grade/privileged/ (verifiers Task.privileged, feat(v1): group-grade taskset (GAR grader) verifiers#2757 4fd6beac). For r2e (prime-envs#961 e091967c) these are the hidden tests, the gold patch and the expected per-test results. The hidden tests are also written into every restored candidate box, so the grader can run bash run_tests.sh there.
• Leak safety. Privileged data is collected and written only inside the grade episode: the grade env server and a frozen grader model. It never touches the policy's boxes (candidate boxes are restored copies) and never enters training samples. The grade trace records only file names and sizes (info.privileged). The grader's own transcript can still quote what it read; that trace is logged (kind grade) but never trained on. Grade traces share the run's traces.jsonl with train/eval episodes, so tools/convert_traces_to_hf_dataset.py always skips traces with info.kind == "grade" and prints how many it skipped; they cannot reach an SFT/HF dataset export (b3c7b3c). That tool is prime-rl's only trace→dataset export path.
• deps/verifiers → 4fd6beac, deps/prime-envs → e091967c. Unit tests: 687 passed.

Advantage mode (revision b9f1c16)

[orchestrator.train.algo] mode = "advantages" (default "rewards" = the existing behavior, unchanged code path).

• What. Rewards are not used. Every group with ≥ 2 candidates is graded. The grader sees no rewards. It ranks every non-confirmed-hack candidate into tiers (ties allowed) and gives each gap between adjacent tiers a margin: slight (default), clear or large. Confirmed hacks (same citation rule) form a tier below the last, with a large gap.
• Mapping (rank_advantages, pure; revision 91508f8). The margins are anchored to the GRPO pass-vs-fail gap: slight = 1/4, clear = 1/2, large = 1, ties = 0. The bottom tier scores 0 and each tier above adds its gap's margin, so the scores are implied rewards in reward units. The advantages are those scores centered to zero mean, i.e. exactly GRPO's advantages for them (over candidates; all of a candidate's traces get its advantage). Example: [c2] slight [c1, c4] large [c3] scores 1.25 / 1 / 1 / 0, giving +0.44 / +0.19 / +0.19 / −0.81. Two tiers with a large gap are GRPO's 0/1 advantages. The prompt defines the levels: slight = same quality with minor cleanliness/minimality/style differences; clear = noticeably better approach or correctness, both reasonable; large = a real solution over a broken/wrong/harmful/non-attempt. It tells the grader to default to slight. (This replaces the earlier (|T_j|+|T_{j+1}|)/n step.)
• No signal / failure. A single tier gives all-zero advantages, pruned by the train sink. Grader error, invalid verdict, timeout or stale also give all-zero advantages (there is no reward to fall back on); info.gar.fallback_reason and the gar/fallback/* metrics record why. A hack whose citation fails is in no tier and gets 0.
• Logging. gar/win_rate per candidate (win rate over all tiers, hacks included). It is not written as the reward. info.gar has tier, margins, w, advantage, hack, evidence.
• Reward consumers:
  • Reward metrics keep logging the env's own reward (0 if the env has none); they work without rewards.
  • The difficulty_pool sampler (samples on rewards) is rejected in config validation for this mode.
  • Advantage-range gates work on the advantages, as usual.
  • Length penalty is rejected (gar already rejected it in both modes; the message names the mode). There is no cost penalty on this branch.
• Safeguards. Frozen, strong judge; ties as the default when unsure; held-out eval with a different judge or the env's own reward to catch judge-pleasing drift.
• Cost. Every group is graded, not only mixed ones (≈ 1 grader episode per group).
• Later. Per-turn / per-segment advantages are a possible next step.
• verifiers side: GroupGradeData.mode, from_traces(..., passed=None), GroupVerdict.margins, and mode-specific ranking rules in rank_passing.md / rank_all.md (feat(v1): group-grade taskset (GAR grader) verifiers#2757, d28c6be8).
• Tests: test_gar_rank_advantages (the example above, large two-tier = GRPO, hacks lowest, unranked 0, zero mean, one tier → 0). pytest tests/unit -m "not gpu": 687 passed. Config checks were run in a throwaway script: length penalty rejected, difficulty_pool rejected, checkpoint_on_finish set on the gar source's agent.

Candidates' boxes from checkpoints (revision 4501f0d)

The grader now works for every env, not only SWE: see PrimeIntellect-ai/verifiers#2757 (revision 18142383) for the design, the cost note and the smoke run.

• Config. A gar train source sets verifiers agent.checkpoint_on_finish = true on every agent of its env (OrchestratorConfig.checkpoint_gar_candidates). There is no new knob. Each rollout checkpoints its Prime box when its agent finishes, and the grader's candidate_shell tool restores a candidate's box on demand. Runtimes without checkpoints and text-only envs record nothing; those candidates are graded from their transcripts and files.
• Cleanup. GARAlgorithm.score_group now runs grading inside try/finally and deletes every checkpoint of the group (trace.info["checkpoint"] of all its traces) when grading ends, however it ends: all-equal groups (right away), graded, fallback (error/invalid/timeout/stale), and cancellation. Ids the platform refuses for now (409 while a box restored from them is alive, e.g. a cancelled grader still running) or that fail to delete are kept and retried once at shutdown through a new no-op hook Algorithm.close(), which the orchestrator teardown calls. Ids that still fail are logged.
• Metrics (per graded group): gar/checkpointed (share of candidates with a checkpoint), gar/checkpoint_seconds (mean request → durable), gar/restores and gar/restore_seconds (from the grading trace).
• Failure handling. A failed checkpoint means no info.checkpoint. The candidate is then graded without a box, and the group is still graded.
• deps/verifiers → 1814238 (feat/group-grade). uv.lock: prime-sandboxes 0.4.1 → 0.4.2 (checkpoint API).
• prime-envs#961 is no longer needed for the workspace: its stage_verifier staged hidden tests in the old git box, and the boxes now carry no hidden tests. Its info.test_output part still feeds tests.txt.
• Docs (docs/algorithms.md, GAR) are updated.

Simplified (revision e90d5af)

Line count of the PR: +480/−16 → +477/−16 (algo/gar.py 295 → 288 lines). Most of the simplification is in the verifiers PR (+616 → +517); see PrimeIntellect-ai/verifiers#2757.

• Eq. 3 works beyond binary rewards. The binary-reward assert is gone. A group is mixed when its rewards are not all equal. P, the rollouts whose credit the ranking rescales, are the rollouts with a positive advantage (reward above the group mean) instead of reward == 1.0. For 0/1 rewards this is the same set as before.
• Hack rule: a confirmed hack's reward becomes the group's minimum reward (before the group statistics). Why this rule: for a mixed 0/1 group the minimum is 0, so binary behavior is unchanged. For other rewards it needs no assumption about reward bounds (unlike "0", which is only correct on [0, 1]), and the hack never ends above an honest rollout of the group.
• Smaller verdict. The grader writes only hack levels with evidence, a tiered ranking, and free-text notes. confidence and CONFIDENCE_SHRINK are removed: a tie in the ranking expresses uncertainty. Axes and hack_kind are removed from info.gar.
• Smaller grader input. The grader ranks the candidates above the mean (passed in the row) and gets their full transcripts. Every candidate is still audited for hacks. The other candidates' transcripts keep only their first and last 10 turns (the others read [turn elided]). The citation check runs against exactly what the grader was given (Candidate.turns), so a confirmed hack on such a candidate must quote a turn the grader saw.
• Edge case: a candidate can enter P only after a hack's reward drops (it was below the original mean). The ranking does not cover it, so it gets the mid-table win rate ½, the same as a pass whose confirmed hack failed its citation.
• deps/verifiers now pins 32e35e04 (feat/group-grade).
• Kept: citation check, background finalization, deadlines, fallback reasons and metrics (no confidence metric existed), info.gar annotations, grade episode logging.

Stacked on #3802 (background group finalization). Adds gar (Groupwise Advantage Redistribution, MiMo-V2.6 §4.3.2). It is a GRPO subclass that sends each mixed group (rewards not all equal) to an agentic grader in one sandbox, drops confirmed hacks to the group's minimum reward, and redistributes the above-mean rollouts' credit by the grader's ranking (Eq. 3). It is off unless algo.type = "gar".

Landing order

1. Companion PRs: verifiers feat(v1): group-grade taskset (GAR grader) verifiers#2757 (group-grade taskset) and prime-envs r2e-gym 0.1.5: keep the test run's tail in trace.info[test_output] prime-envs#961 (r2e-gym info.test_output and stage_verifier).
2. prime-rl feat(orchestrator): opt-in background group finalization #3802 (background group finalization).
3. This PR.

deps/verifiers (32e35e04) and deps/prime-envs (3b987c87) are pinned to the companion branches so the stack works end to end. The pins must move to the merged commits before landing. uv.lock only bumps r2e-gym 0.1.4 → 0.1.5.

What

• Config (configs/algorithm.py): GARAlgoConfig(GRPOAlgoConfig).
  • grader: model (a FrozenModelConfig), env (a verifiers env config, narrowed by id), serve, timeout (default 1800 s) and max_concurrent (default 32).
  • f_min (default 0.5) and lambda_max (default 1.5).
  • length_penalty is rejected, because GAR replaces it.
  • No grader model is baked in.
• Launcher: TrainSourceConfig.grade_source turns a gar source's grader into an EnvConfig named after the source. env_sources adds it as split grade. The local launcher already loops over all splits. The multi-node sbatch template gets a grade_env_names loop.
• Algorithm (algo/gar.py):
  • Opts in to feat(orchestrator): opt-in background group finalization #3802's background finalization (finalize_in_background = True). The default cap of 64 finalizing groups per env applies; train dispatch pauses at the cap.
  • Owns its grader the way OPD owns its teacher: a frozen model pool plus an EnvClient onto the grade env server. The orchestrator wires in the step counter, max_off_policy_steps and the grader address file through GARAlgorithm.bind(...).
  • Any rewards work. Only mixed groups (rewards not all equal) are graded; the others are plain GRPO.
  • Each episode is one candidate: its trace with info.patch.
  • GroupGradeData.from_traces shuffles and anonymizes the candidates, seeded by the group id.
  • Deadline: grader.timeout covers both the wait for a grader slot (max_concurrent) and the grading itself. The group also falls back once progress.step reaches policy.start + max_off_policy_steps + 1, the last step whose batch can still take it. So a grading group falls back to GRPO before it would go stale. It is not dropped.
  • The verdict is validated with GroupVerdict.model_validate plus check().
  • Citation check: a confirmed hack is downgraded to suspected unless every evidence quote passes quote_found. That function holds the one turn definition, in verifiers. A quote must be at least 20 characters or 4 words. It must appear in text the policy itself wrote (assistant content or tool calls), never in tool output. Suspected hacks are never applied.
  • Hacks: R = the group's minimum reward, before the group statistics.
  • Eq. 3: P = rollouts with A = R − mean(R) > 0. f = f_min + (1 − f_min)·w, where w is the win rate among the ranked candidates (a tie counts ½). A member of P the ranking leaves out gets w = ½. A lone ranked candidate gets f = 1. Then λ = min(λ_max, Σ_P A / Σ_P f A), A' = λ f A on P, and the group is re-centered.
  • Fallback to plain GRPO advantages on stale, timeout, error (request failure or failed grader episode) or invalid (verdict missing or rejected). A cancel from feat(orchestrator): opt-in background group finalization #3802 (stale at step advance, drain, stop) arrives as CancelledError and propagates.
• Evidence:
  • Each candidate trace gets info.gar = {label, hack, evidence, tier, w, f, lambda, fallback_reason, grader_trace_id}. It is set on the trace and also emitted right away as a trace annotation (monitors.log_annotations), so dropped groups carry it too.
  • The grader episode is logged to the trace stream with the new Kind = "grade", plus info.group_id and info.candidates (label → trace id). The logged copy leaves out the candidates' transcripts (task.data.candidates); the candidate traces are already in the stream. The index accepts grade.
  • Trace metrics are aggregated by the existing per-agent metrics under train/all/<agent>/metrics/gar/...: graded, fallback/{stale,timeout,error,invalid}, latency, grader_tokens, grader_cost (when the provider reports it), lambda, all_fail, f, hack_confirmed, hack_suspected.
• Docs: a GAR section in docs/algorithms.md. Example: examples/advanced/glm-5.3/swe-gar.toml, an overlay that does not change swe.toml.

What I ran

• This revision: pytest tests/unit/orchestrator tests/unit/test_configs.py: 302 passed, with verifiers at 32e35e04. ruff check and ruff format --check: clean.
  • The only test added is one function in the existing test_advantage.py: test_gar_redistribution (win rates, Eq. 3, the cap with re-centering, and now a non-binary group where P is the above-mean rollouts). No new test files.
  • test_load_configs parses the new overlay.
  • Ruff check and format are clean.
• Uncommitted throwaway checks (earlier revision; not re-run for this one):
  • swe.toml @ swe-gar.toml resolves to a GroupGradeConfig with inner r2e-gym. env_sources holds ('train','r2e') and ('grade','r2e'). The written envs/grade/r2e.json round-trips through EnvServerConfig.
  • score_group against a fake grader:
    • a confirmed hack with a cited policy quote: −0.5;
    • ranked passes: 0.667 / 0.417 / 0.417 (λ = 4/3, group sum 0);
    • a hack whose quote is not in the transcript becomes suspected and keeps a positive advantage;
    • invalid verdict, timeout, stale deadline and a uniform group each fall back to GRPO with the right fallback_reason and metrics;
    • a non-binary reward raises an error that names the env;
    • the logged grade episode has no candidates (about 3.5 KB in the check).

GPU / e2e checks (not run here: no GPU or sandboxes)

• The grade env server starts and publishes its address, both locally (rl) and with the multi-node sbatch.
• A cancelled or timed-out grader episode (timeout, stale, stop) tears down its sandbox on the env server.
• network_allow=[] is enforced in the grader box: no egress beyond the interception and MCP routes.
• Verdict validity rate, grading latency (p50/p90) and the stale-fallback rate of mixed groups, under max_off_policy_steps = 8.
• Grader request size and trace-stream growth per step at GLM-5.3 transcript lengths.
• finalizing size and orchestrator RSS over many steps.
• A real r2e GLM-5.3 run with a served grader: grader box setup (branches, staged /testbed/r2e_tests); info.gar and grade episodes in the trace stream and dashboard.

Open questions

• The dashboard UI does not yet have a filter or link for kind = "grade" traces. The data is there (info.gar.grader_trace_id, info.candidates).
• Not implemented: tie fraction and Spearman(f, length/turns) metrics. Both can be computed offline from info.gar and the trace stream.
• The grader uses default sampling (vf.SamplingConfig()). Add a sampling knob only if a grader model needs one.