[xeophon]

Provide one rollout planner that matches accepted saved episodes to selected task occurrences. Prime-RL uses this Verifiers library function to resume changed tasksets without reusing stale content or collapsing repeated selections into one group.

Before

With the eval CLI removed, Verifiers exposed no library-level planner returning kept episodes and missing counts per selected occurrence. Prime-RL implemented that bookkeeping itself: resume.plan() produced separate episode, count, and group maps, then EvalSource.restore() and trigger() reconstructed the work to schedule. Task matching and occurrence allocation were spread across those stages.

flowchart TD
    Tasks["Prime-RL selected tasks"] --> Plan["Prime-RL resume.plan"]
    Saved["Saved episodes"] --> Plan
    Plan --> Kept["Kept episode list"]
    Plan --> Counts["Owed counts by task key"]
    Plan --> Groups["Group IDs by task key"]
    Counts --> Restore["EvalSource.restore"]
    Groups --> Restore
    Restore --> Trigger["Trigger reconstructs rollout requests"]

Loading

After

flowchart TD
    Tasks["Prime-RL selected task occurrences and rollout target"] --> Planner["Verifiers plan_rollouts"]
    Accepted["Saved episodes accepted by the source environment"] --> Planner
    Planner --> Match["Match task content hashes and deduplicate episode IDs"]
    Match --> Allocate["Allocate kept episodes up to each occurrence's target"]
    Allocate --> Plan["Per occurrence: task, kept episodes, missing count"]
    Plan --> Prime["Prime-RL owns groups, dispatch, and persistence"]

Loading

plan_rollouts() returns plain (task, kept_episodes, missing_rollouts) tuples. It owns content identity and rollout counts; callers own file parsing, environment completion checks, execution, and persistence. Fresh evaluations use the same function with no completed episodes.

Consumed by Prime-RL #3808. When combined with #2610, retain the library planner and drop this branch's changes to the deleted eval CLI call sites. The architecture above assumes that removal.

---

Note

Medium Risk
Resume behavior changes how saved rollouts are matched and pruned relative to the current task list, which can drop previously retained traces; core eval scheduling paths are refactored but logic is localized to planning and CLI resume.

Overview
Introduces shared plan_rollouts in verifiers/v1/utils/eval.py so fresh and resumed evals use the same per–task-occurrence plan: match saved episodes by task content hash (derive hash when missing), dedupe by episode id, and return (task, kept_episodes, missing_rollouts) for each selected task.

resume.load now accepts task objects instead of string keys, validates trace rows and applies the completion verdict before planning, atomically rewrites traces.jsonl to only rows for kept episodes (by episode.id), and returns that plan instead of a separate owed-by-key map. run_eval builds finished slots from kept episodes and schedules only missing counts; it drops the distribute path.

On resume, episodes that do not match the current task selection or exceed the per-occurrence rollout target are discarded rather than kept globally by key count.

^{Reviewed by Cursor Bugbot for commit 761573e. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Share rollout planning across eval runners via plan_rollouts

Adds a shared plan_rollouts planner in eval.py that groups completed episodes by task content hash, drops duplicate episode IDs, and allocates saved episodes to each selected task occurrence up to num_rollouts.

• resume.load in resume.py now takes task objects, validates all trace rows, applies the completion verdict, and returns a per-task plan with retained episodes and a missing rollout count.
• run_eval in runner.py uses this shared plan for both resumed and fresh runs: retained episodes become finished slots, and only missing counts are scheduled.
• Behavioral Change: accepted resume rows that do not match the current task selection or exceed the rollout target are now discarded; legacy episodes without a stored task hash get a derived hash.

^{Macroscope summarized 761573e.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-04T01:09:08.499695Z	761573e	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — The refactor adds a shared planner to the production evaluation path and changes resume allocation, deduplication, and rerun counts. Because these decisions control which substantial rollouts execute, human review is warranted despite the localized diff and lack of schema or security changes.

No code changes detected at 761573e. Prior analysis still applies.

^{You can add or adjust custom eligibility rules. Learn more.}