refactor(eval): remove vf-eval by mikasenghaas · Pull Request #2610 · PrimeIntellect-ai/verifiers

mikasenghaas · 2026-09-15T17:24:53Z

Summary

Remove the vf-eval console script and its stack: verifiers/v1/cli/eval/ (main, runner, resume, deprecation hint), the Rich eval dashboard (cli/dashboard/eval.py), and verifiers.v1.configs.cli.eval.
Type output_path with the RunOutput protocol only. Drop the per-attempt log helpers that only the eval CLI used.
Point docs, AGENTS.md, and env docstrings at vf-validate for model-free checks, at vf-run for a few quick episodes, and at prime-rl's uv run eval for full evaluations. Delete docs/evaluation.md and its nav entry.

vf-run (#3047) replaces vf-eval in CI. prime-envs and prime-envs-private already call vf-run.

Breaking

vf-eval no longer exists. Use vf-run for quick runs, or prime-rl's uv run eval for full evaluations.
vf-eval --resume and the Rich eval dashboard go away with it. vf-run --dry-run still writes the resolved config.
verifiers.v1.configs.cli.eval is gone. Import RunConfig and default_run_name from verifiers.v1.configs.cli.run.

Verification

pytest tests/unit: passes.
uv run vf-run --help: works; no vf-eval references remain in code, docs, or configs.

🤖 Generated with Claude Code

Note

High Risk
This is a breaking CLI removal: anything still invoking vf-eval, --dry-run, or --resume on eval must migrate to prime-rl, vf-run, or vf-validate.

Overview
Removes the bundled vf-eval command and its stack: console entry in pyproject.toml, verifiers/v1/cli/eval/* (main, runner, resume), EvalConfig, the Rich live eval dashboard (cli/dashboard/eval.py), and eval-only log helpers in cli/output.py. Run directory naming now goes through a RunOutput protocol plus shared RunConfig / default_run_name in configs/cli/run (still used by vf-gepa, vf-validate, and vf-run).

Docs and contributor guidance drop docs/evaluation.md and retarget workflows: vf-validate for setup/gold/no-op checks, vf-run for a few in-process model episodes, and prime-rl's uv run eval for full evaluations. CLI count in getting started goes from six to five commands.

Tests and comments shed EvalConfig dependencies (e.g. replay fixtures build minimal JSON on disk). GEPA and client docstrings stop referencing run_eval / vf-eval.

^{Reviewed by Cursor Bugbot for commit 2e0ab91. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Remove the `vf-eval` CLI command and its eval modules

Deletes the verifiers.v1.cli.eval package, including run_eval, resume helpers, the in-process runner, and the eval dashboard; removes the vf-eval console script from pyproject.toml
Deletes EvalConfig and RichConfig from the eval CLI config module, and removes per-attempt log helpers (create_attempt_log_dir, attempt_log_file) from output.py
Updates docs and contributor guidance to point at vf-validate and vf-run instead; deletes the standalone evaluation guide and its nav entries
Risk: Behavioral Change: vf-eval no longer exists after install; check remaining imports of EvalConfig, PRIME_RL_HINT, and the dashboard eval export if you have out-of-tree callers. test_replay.py now writes a minimal resolved-config JSON instead of an EvalConfig

^{Macroscope summarized 2e0ab91.}

macroscopeapp · 2026-09-15T18:44:07Z

Approvability

Verdict: Approved at 2e0ab91

Macroscope's review found this PR approvable — Removes the superseded vf-eval CLI and points users to vf-run, vf-validate, or prime-rl. Legacy invocations must migrate, but existing saved results and the underlying environment execution remain unaffected.

^{You can add or adjust custom eligibility rules. Learn more.}

## Summary - Served episodes stream turn by turn over the env-serve wire (`verifiers/v1/serve/delta.py`). `Trace.notify` fires at every phase change and after every recorded turn; the worker's `DeltaStreamer` diffs the run's live traces against what it already sent and ships the new part (header once, appended nodes/calls/errors, semantic links, changed scalars), so every byte crosses the wire once. A cursor advances only after its delta is on the wire. `EnvClient.run` assembles the episode from the deltas (`EpisodeAssembly`, which copies the nodes it keeps so a later link never mutates a delta the caller holds), validates it once, and takes `on_delta` so a caller can relay or persist the stream; the client's receive loop survives a malformed frame. - `pending` preview: the interception server hands each model request's uncommitted tail (tool results, user turns) to `Trace.preview`, so a watcher sees a tool result before the model has answered; the committed turn replaces it and the finished record never carries it. - The eval CLI is `uv run vf-eval` and runs in-process only. Serving an env to many consumers is prime-rl's job (its `uv run eval` runs env servers and consumes this stream), so the CLI keeps the quick local path and never assembles deltas; its TUI footer and console log point at prime-rl's `eval`. The env-server e2e fixture drives a worker pool through `EnvClient` directly. Based on `main`; the eval CLI stays for a transition period (#2610 removes it later). prime-rl PrimeIntellect-ai/prime-rl#3471 consumes the stream for its live traces. ## Breaking - Env-serve wire: replies are 4 frames (`client_id, request_id, kind, data`) with `kind` = `delta` | `reply`; `RunResponse` carries the episode head and per-trace counts instead of the traces. A client and server must be on the same side of this change. - Every console script is prefixed: `vf-eval`, `vf-validate`, `vf-debug`, `vf-replay`, `vf-init`, `vf-gepa`. Generic names collide with the scripts of a workspace that installs verifiers as a dependency (prime-rl's `eval`), and the winner is whichever package was installed last. `vf-eval` has no `[serve]` / `--serve.*` / `--no-serve`: rollouts always run in-process; use prime-rl's `uv run eval` for env-server runs. Existing venvs: `uv sync --reinstall-package verifiers --reinstall-package prime-rl` regenerates the scripts. - `Env.run_slot` takes `on_trace`; `Trace` gains `watch`/`notify`/`preview`/`clear_preview`. ## Verification - `tests/v1/test_serve_delta.py`: turns, a link onto an earlier node, a retried attempt and the final reward round-trip through `DeltaStreamer` → `EpisodeAssembly` → `WireEpisode`; a refused send is diffed again; the pending preview streams, is replaced by the committed turn and never reaches the record; assembling never mutates a delta; the field lists cover every serialized `Trace` field. - `uv run pytest tests/v1 -m "not e2e"` green apart from the config-parse cases of tasksets not installed in the venv; ruff clean. - End to end through prime-rl: gsm8k single turn, terminal-bench-2 fix-git with bash and with the rlm harness delegating to a sub-agent (semantic links `subagent_call`/`subagent_return`), kuhn-poker two-seat episodes, 5-step reverse-text RL runs with token ids and logprobs, including router replay plus sampling replay on a MoE; no assembly mismatch. 🤖 Generated with [Claude Code](https://claude.com/claude-code)    > [!NOTE] > ### Stream served episodes turn by turn with delta frames > - Adds `DeltaStreamer` in [delta.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2512/files#diff-106a565d15f6f5c86f7357854d512768b3f8a4775026a0eae32c69b828f5da31) that watches trace changes during `run_slot` and sends incremental deltas; the client `EpisodeAssembly` reconstructs a `WireEpisode` from those deltas plus a final episode head and `TraceSummary` counts. > - Changes the wire protocol to three-frame messages (request ID, kind, data). Delta frames are relayed by the pool and client without closing the request; reply frames close it. Updated in [server.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2512/files#diff-64422f6a8749fcc024415b2756129bc98000e830e74b3b5245de50a555480a22), [pool.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2512/files#diff-0c9c59578214be2580d340bc524321e352f48038207a68945637d00fc399600c), and [client.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2512/files#diff-ef98a21052536967b8f8723e36022f69e539f4e8b13794791e8254434ac4696e). > - Adds a trace watcher/preview API in [trace.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2512/files#diff-834820c3aa80d87ee1c1c57fb54c74f09c66ac727036d964e6005c8613c00b5e). `Env.run_slot` gains an `on_trace` callback; `Rollout.open`/`close` and `InterceptionServer` notify the trace on phase changes and pending turn previews. > - `PendingTurn.commit`/`commit_prompt` now clear the trace's pending preview after committing. > - Risk: `RunResponse` in [types.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2512/files#diff-8e290c0dfc583c92b20ee9f20fa2ecd0df82459ad63f8e819e87aa856de97cf4) removes the serialized `WireEpisode` field and replaces it with an episode head and `TraceSummary` list — any out-of-tree consumer expecting the full episode in the run response will break. > >  > #### Changes since #2512 opened > > - Renamed console script entry points in `pyproject.toml` and updated all documentation and usage strings to use `vf-` prefixed command names [02a6eff] > - Refactored trace preview system from unkeyed to keyed architecture [55500a3] > - Updated all trace preview and clear operations to use keyed API with PendingTurn instances as keys [55500a3] > - Added abandon mechanism for cleaning up previews when turns fail or are cancelled [55500a3] > - Changed `verifiers.v1.trace.Trace` to store preview entries by `id()` of the key object rather than by the key object itself [73d602d] > - Updated `PRIME_RL_HINT` constant in the `hint.py` module [fc7e45b] >  > >  > > <a href="https://app.macroscope.com">Macroscope</a> summarized fe771d8. >  >   --- > [!NOTE] > **High Risk** > Breaking env-serve wire protocol and `RunResponse` shape require matching client/server versions; CLI renames and removal of eval’s serve path change how users run evaluations. > > **Overview** > This PR **streams env-server episodes incrementally** instead of returning one fat `RunResponse`. Workers use **`DeltaStreamer`** to diff live traces on `Trace.notify()` and send **`delta`** frames; clients **`EpisodeAssembly`** rebuilds traces and validate once against a final **head + `TraceSummary` counts**. The wire format is now **`[request_id, kind, data]`** with `kind` = `delta` | `reply`; the pool relays deltas without closing the request. > > **`Trace`** gains **`watch` / `notify` / keyed `preview` / `clear_preview`**; rollouts and the interception server fire notifications (including **pending tool/user tails** before commit). **`Env.run_slot`** adds **`on_trace`** so the server can subscribe traces to the streamer. > > **`vf-eval` is renamed and narrowed**: console scripts are **`vf-*`** to avoid clashing with prime-rl’s **`eval`**, docs/config comments follow suit, and **`vf-eval` always runs in-process**—**`[serve]` / `--no-serve` are removed** from eval config—with dashboard/log hints to use **prime-rl** for env-server scale and live dashboards. > > Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit fc7e45b. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).  --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

## Summary Migrates the eval CLI from verifiers into the prime-rl ecosystem: `uv run eval` is prime-rl's entrypoint for evaluating any model on any environment, standalone or as the online evals of a training run. Why: - consolidation of features: adaptive concurrency, the local dashboard, resume from the trace stream, live rollouts - easier code sharing between the CLI entrypoints (`rl`, `sft`, `eval`) - direct access to the pre-installed environments (`prime-envs` submodule) - one workspace for post-training: training and evals Important: - the verifiers eval CLI stays for a transition period as the quick in-process path, renamed `uv run vf-eval` so prime-rl owns `eval` (PrimeIntellect-ai/verifiers#2610 removes it later); env servers and their streamed traces (PrimeIntellect-ai/verifiers#2512, merged; the submodule points at verifiers `main`) are consumed here only - evals run through env servers, for scalability; the launcher (`eval`, `sft`, `rl`) owns them - `--resume` continues an interrupted run from its trace stream: landed episodes rejoin the epoch, only the owed rollouts run; `num_examples`/`group_size` may change, the model, sampling and env config may not - multi-env capable (`[[source]]` blocks); the single-env command shorthands are preserved - adaptive concurrency only against local deployments (it reads vLLM `/metrics`); against an API, concurrency is pinned via `min_inflight = max_inflight` (the default pins 128) - prime-rl's core dependencies are plain CPU; install them and the environments with `uv sync --all-packages` - Prime runs and traces integrate through the single monitor abstraction (`--monitors.prime`, `--monitors.wandb`, the file monitor) Examples: ```bash uv run eval gsm8k # default model on Prime Inference uv run eval gsm8k -n 32 -r 4 -m openai/gpt-5.6-luna # 32 tasks × 4 rollouts, another model uv run eval terminal-bench-2 --env.taskset.tasks '["fix-git"]' --env.agent.harness.id bash -n 1 -r 2 uv run eval @ configs/debug/eval/single-turn.toml # gsm8k, null harness uv run eval @ configs/debug/eval/multi-turn.toml # 16 terminal-bench-2 tasks, bash harness uv run eval @ configs/debug/eval/multi-env.toml # terminal-bench-2 under bash and rlm side by side uv run eval @ configs/debug/eval/aime2026.toml # AIME 2026, avg@16 uv run eval @ configs/debug/eval/tb2.toml # Terminal-Bench 2, avg@4 uv run eval @ configs/debug/eval/resume.toml --run.name x # then: --resume uv run eval gsm8k --monitors.prime # stream the evaluation to the platform ``` ## Breaking - `uv run evals` is `uv run eval` (`EvalsConfig` → `EvalConfig`); config shape: `[eval.client]` → `[client]`, `[eval.concurrency]` → `[concurrency]`, `[[eval.source]]` → `[[source]]`, `eval.num_examples`/`eval.group_size` → top level. `[online]` is gone: online evals are spawned by `sft`. - `eval` has no `[ckpt]` block; `--resume` is a switch that reads the run's trace stream. - SFT online evals: no public entrypoint (`uv run sft` with an `[eval]` block spawns it); `logs/attempt_N/evals.log` → `eval.log`, `evals.json` → `eval.json`, W&B label `online-eval`; `EvalsEvalConfig` → `SFTOnlineEvalConfig`. - `ratio` is accepted on training sources only, `interval` on online eval sources only; a standalone eval's `[[source]]` takes neither. - `orchestrator.env_server_base_port` and the eval entrypoint's `env_server_base_port` are removed: launcher-managed env servers bind an OS-assigned port and publish it (`configs/attempt_N/resolved/envs/<split>/<name>.address`); `serve.address` pins one. `uv run env-server` without `serve.address` binds an OS-assigned port (`address_file` publishes it). - `configs/evals/swe.toml` is removed (`uv run eval swebench-verified --env.agent.harness.id bash`). ## Verification - Unit: `uv run pytest tests/unit` green (configs, orchestrator, eval, monitors incl. the live reader). - Debug configs: single-turn, multi-turn, multi-env and aime2026 run to completion against Prime Inference. Resume: single-env and multi-env runs interrupted with SIGTERM and resumed hold exactly examples × group_size unique episodes with the plan and epoch metrics over the full set; resuming with a larger `group_size` runs only the extra rollouts; resuming with another model is refused. - Core install (`uv sync` without extras) runs `uv run eval`; an SFT run with online evals starts one env server per source. - Live streaming, on the final code: gsm8k 16 single-turn episodes, terminal-bench-2 fix-git with bash (13-14 turns, the pending preview alternating with committed turns in the live files), kuhn-poker 4 two-seat episodes (8 traces), a 5-step reverse-text RL run (768 train episodes with token ids, logprobs and masks); no assembly mismatch. - Platform: `--monitors.prime` on a two-rollout fix-git eval opened the evaluation 9 s after launch, streamed both rollouts and closed it; the dashboard linked to it throughout. - Dashboard behaviour checked in headless Chromium: live rows and viewer follow, handover in place, metrics pane per env, filter menus, platform button states. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  --- > [!NOTE] > **Medium Risk** > Breaking CLI and config renames affect existing scripts; env-server address discovery replaces fixed ports across `rl`, `sft`, and `eval` launch paths. > > **Overview** > **Replaces `uv run evals` with `uv run eval`** as the standalone evaluation entrypoint, sharing the orchestrator eval pipeline (per-source env servers, adaptive or pinned concurrency, file/W&B/Prime monitors). Config is **flattened** (`[[source]]`, `[client]`, `[concurrency]` at top level); **`--resume`** continues from the trace stream instead of a `[ckpt]` cursor; SFT **online eval** is renamed and typed as `SFTOnlineEvalConfig` (no public `[online]` block). > > **Env server wiring changes:** fixed `env_server_base_port` ranges are removed—launcher-managed servers bind **OS-assigned loopback ports** and publish addresses via **`address_file`** / `configs/.../envs/<split>/<name>.address`. Training **`ratio`** and online-eval **`interval`** are scoped to the right source types only. > > **Docs and examples** add `docs/eval.md`, per-example `eval.toml` files, and `configs/debug/eval/*` smokes; **`configs/evals/swe.toml` is deleted**. CI adds a **gsm8k eval** integration test on VM runners. > > **Dashboard** gains live rollout APIs and eval-centric metrics (plan-driven progress, platform links, eval run status), renames **`evals.log` → `eval.log`**, and tightens static asset revalidation. > > Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 82f2f0e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).  --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

…2760) Provide one rollout planner that matches accepted saved episodes to selected task occurrences. Prime-RL uses this Verifiers library function to resume changed tasksets without reusing stale content or collapsing repeated selections into one group. ## Before With the eval CLI removed, Verifiers exposed no library-level planner returning kept episodes and missing counts per selected occurrence. Prime-RL implemented that bookkeeping itself: `resume.plan()` produced separate episode, count, and group maps, then `EvalSource.restore()` and `trigger()` reconstructed the work to schedule. Task matching and occurrence allocation were spread across those stages. ```mermaid flowchart TD Tasks["Prime-RL selected tasks"] --> Plan["Prime-RL resume.plan"] Saved["Saved episodes"] --> Plan Plan --> Kept["Kept episode list"] Plan --> Counts["Owed counts by task key"] Plan --> Groups["Group IDs by task key"] Counts --> Restore["EvalSource.restore"] Groups --> Restore Restore --> Trigger["Trigger reconstructs rollout requests"] ``` ## After ```mermaid flowchart TD Tasks["Prime-RL selected task occurrences and rollout target"] --> Planner["Verifiers plan_rollouts"] Accepted["Saved episodes accepted by the source environment"] --> Planner Planner --> Match["Match task content hashes and deduplicate episode IDs"] Match --> Allocate["Allocate kept episodes up to each occurrence's target"] Allocate --> Plan["Per occurrence: task, kept episodes, missing count"] Plan --> Prime["Prime-RL owns groups, dispatch, and persistence"] ``` `plan_rollouts()` returns plain `(task, kept_episodes, missing_rollouts)` tuples. It owns content identity and rollout counts; callers own file parsing, environment completion checks, execution, and persistence. Fresh evaluations use the same function with no completed episodes. Consumed by [Prime-RL #3808](PrimeIntellect-ai/prime-rl#3808). When combined with [#2610](#2610), retain the library planner and drop this branch's changes to the deleted eval CLI call sites. The architecture above assumes that removal.  --- > [!NOTE] > **Medium Risk** > Resume behavior changes how saved rollouts are matched and pruned relative to the current task list, which can drop previously retained traces; core eval scheduling paths are refactored but logic is localized to planning and CLI resume. > > **Overview** > Introduces shared **`plan_rollouts`** in `verifiers/v1/utils/eval.py` so fresh and resumed evals use the same per–task-occurrence plan: match saved episodes by task content hash (derive hash when missing), dedupe by episode id, and return `(task, kept_episodes, missing_rollouts)` for each selected task. > > **`resume.load`** now accepts task objects instead of string keys, validates trace rows and applies the completion verdict before planning, atomically rewrites `traces.jsonl` to only rows for kept episodes (by `episode.id`), and returns that plan instead of a separate owed-by-key map. **`run_eval`** builds finished slots from kept episodes and schedules only missing counts; it drops the **`distribute`** path. > > On resume, episodes that do not match the current task selection or exceed the per-occurrence rollout target are **discarded** rather than kept globally by key count. > > Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 761573e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).     > [!NOTE] > ### Share rollout planning across eval runners via `plan_rollouts` > Adds a shared `plan_rollouts` planner in [eval.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2760/files#diff-ace02ecf8ccf2025b9b4dd16e224128977d5752d066aa2fb9e84453aaa64ed46) that groups completed episodes by task content hash, drops duplicate episode IDs, and allocates saved episodes to each selected task occurrence up to `num_rollouts`. > - `resume.load` in [resume.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2760/files#diff-fa57536955ac6c644296a0f3317271ffdc94b2611e7ad3d0c2266c436b96a4f8) now takes task objects, validates all trace rows, applies the completion verdict, and returns a per-task plan with retained episodes and a missing rollout count. > - `run_eval` in [runner.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2760/files#diff-65be4a576330571f2729a3bbcdbe92ac81bbcba450c313597c1bdca3a1c3f2e4) uses this shared plan for both resumed and fresh runs: retained episodes become finished slots, and only missing counts are scheduled. > - Behavioral Change: accepted resume rows that do not match the current task selection or exceed the rollout target are now discarded; legacy episodes without a stored task hash get a derived hash. > >  > > <a href="https://app.macroscope.com">Macroscope</a> summarized 761573e. >  >

prime-rl's `uv run eval` replaces it: the `eval` console script, its runner, resume, rich dashboard, platform upload and example configs go, along with the `prime-runs` dependency. `RunConfig` moves to `configs/cli/run.py` for the run-directory CLIs that stay (validate, gepa), and the e2e suite runs tasksets in-process.

The scaffold installs only verifiers, which has no eval executable since the eval CLI moved to prime-rl. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Re-apply the eval CLI removal on main's new layout (flat docs/, tests/unit + tests/integration). The integration conftest builds a test-local RunnerConfig, and the examples smoke test runs in-process through the `run` fixture instead of shelling out to vf-eval. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts: # verifiers/v1/cli/eval/runner.py

## Summary - Add `vf-run`, a minimal in-process runner: `uv run vf-run <taskset> [-m <model>] [-n <tasks>] [-r <rollouts>] [-c <concurrency>] [-o <dir>] [--dry-run]`. - It runs the selected episodes in one process and appends each one to `<output_dir>/<run.dir>/traces.jsonl` as it completes. - It prints one summary line (`3/4 episodes ok, mean reward 0.750: <run-dir>`) and exits `1` when any episode is not ok. - It has no dashboard, resume, attempt logs, or platform push. - When one rollout fails, the taskset raises, or the run is interrupted, it cancels the other rollouts and waits for them to unwind before serving teardown. - It refuses a run directory that already holds episodes, in the CLI and in `run_episodes`. - `--dry-run` resolves the typed config (plugin discovery included), writes `configs/resolved/run.json`, and exits. - Run the integration suite through it. The `run` fixture calls `run_episodes`, and the examples smoke test runs the `vf-run` CLI. - Move `RunConfig` and `default_run_name` to `verifiers/v1/configs/cli/run.py`, and type `output_path` with a `RunOutput` protocol. `vf-eval`, `vf-validate`, and `vf-gepa` keep working unchanged. Merge order: 1. This PR. Its merge publishes a dev release with `vf-run`. 2. PrimeIntellect-ai/prime-envs#1053 and PrimeIntellect-ai/prime-envs-private#373 switch the env smoke tests to `vf-run`. 3. #2610 removes `vf-eval`. ## Verification - `pytest tests/unit`: passes, including the new `tests/unit/cli/test_run.py`. - `pytest tests/integration/test_examples.py -k "reverse_text or interception or kuhn_poker"` through the `vf-run` CLI: 6 passed. - Fixture-based integration tests (`test_single_turn`, `test_multi_turn`, `test_parallel`, `-k "subprocess or null"`): 6 passed. - End-to-end CLI scenarios against Prime Inference (`internal/deepseek-v4.1-flash`), 22 checks: - `reverse-text -n 2 -r 2` (null harness): exit 0, 4 episodes, mean reward 1.0. The summary line, `configs/resolved/run.json`, and one `eval` run id on every episode and trace are all present. - `kuhn-poker -n 1` (multi-agent): exit 0, one episode with several traces. - Unknown model: exit 1, `0/2 episodes ok`, both errored episodes saved. - Existing run directory: refused, traces unchanged. - `--dry-run` writes only the config, and `vf-run @ <run-dir>/configs/resolved/run.json` re-runs it. A TOML `@ run.toml` also runs. - An infinite taskset without `-n` is refused. No arguments and a missing taskset print the usage. - `vf-replay <run-dir>` re-scores `vf-run` output. - SIGINT during a 120-rollout run (bash harness, subprocess runtime): exit 130, cleanup ran, the 8 finished episodes were kept, and no processes were orphaned. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts: # docs/getting_started.md # pyproject.toml # tests/integration/conftest.py # tests/integration/test_examples.py # verifiers/v1/configs/cli/eval.py # verifiers/v1/configs/cli/run.py

## Summary - Remove the `vf-replay` console script and its stack: `verifiers/v1/cli/replay.py`, `verifiers/v1/configs/cli/replay.py` (`ReplayConfig`), the Rich replay dashboard (`cli/dashboard/replay.py`), and `tests/unit/cli/test_replay.py`. - Drop `saved_config_path` from `cli/output.py`, which only `vf-replay` used. - Remove `vf-replay` from the command table in `docs/getting_started.md` and from `docs/debugging.md`. The "Re-score without rerunning the agent" section now documents `Task.rescore(trace)`, which stays as the library API. Stacked on #2610 (removes `vf-eval`). Land after it. ## Breaking - `vf-replay` no longer exists. To score a saved trace again, call `Task.rescore(trace)` on it. - `verifiers.v1.configs.cli.replay` is gone. ## Verification - `pytest tests/unit`: passes. - No `vf-replay`, `ReplayConfig`, or replay dashboard references remain in verifiers or prime-rl. 🤖 Generated with [Claude Code](https://claude.com/claude-code)    > [!NOTE] > ### Remove the `vf-replay` CLI and its replay workflow > Deletes the replay CLI module, its `ReplayConfig` model, the replay dashboard renderer, and related tests. The console-script entry for `vf-replay` is removed from [pyproject.toml](https://github.com/PrimeIntellect-ai/verifiers/pull/3052/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711). Docs now describe offline re-scoring via `Task.rescore(trace)` instead: it clears scores and runs scoring hooks and judges that need only the saved trace, skipping setup, finalize, and runtime-dependent hooks ([debugging.md](https://github.com/PrimeIntellect-ai/verifiers/pull/3052/files#diff-6019563fa28fa370cdbd0814bd23650a70b73f126348779a20892f710a2424a4), [getting_started.md](https://github.com/PrimeIntellect-ai/verifiers/pull/3052/files#diff-7627439dcd69d95a42bb123ac4cfb7682f86da5d6aee037206d9857459906243)). > - Behavioral Change: `vf-replay` no longer installs as a console command; `verifiers.v1.cli.replay`, `run_replay`, `ReplayConfig`, and `ReplayProgress` are gone. Use `Task.rescore(trace)` for offline re-scoring. > >  > > <a href="https://app.macroscope.com">Macroscope</a> summarized ab141b6. >  >   --- > [!NOTE] > **Medium Risk** > Breaking CLI removal changes how users re-score saved runs, though the supported path is the existing `Task.rescore` API rather than deleted scoring logic. > > **Overview** > **Removes the `vf-replay` CLI** and its supporting code: the `verifiers.v1.cli.replay` entrypoint, `ReplayConfig`, the Rich replay dashboard, unit tests, and the `vf-replay` console script in `pyproject.toml`. **`saved_config_path`** in `cli/output.py` is dropped because only replay used it. > > Docs now count **four** CLI commands instead of five. The debugging guide no longer documents batch re-scoring from a run directory; offline re-scoring is described as calling **`Task.rescore(trace)`** on a saved trace (same behavior replay relied on: clear scores, run trace-only hooks and judges, skip runtime-dependent setup). > > **Breaking:** `uv run vf-replay …` is gone; re-score traces in application code via `Task.rescore`. > > Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit ab141b6. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).  Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…main removed vf-eval verifiers main removed vf-eval (PrimeIntellect-ai/verifiers#2610) for vf-run, which takes the same config, saves the traces and exits non-zero when an episode isn't ok, as the masked row is. The test uses vf-eval when it's on PATH, as on 0.4.0, with --no-push after the config when its help names it, and vf-run otherwise, then judges both by the traces; the workflow puts whichever of the two the installed verifiers has on PATH. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This was referenced Sep 15, 2026

feat: stream served episodes turn by turn #2512

Merged

feat: consolidate eval cli PrimeIntellect-ai/prime-rl#3471

Merged

mikasenghaas marked this pull request as ready for review September 15, 2026 18:39

mikasenghaas requested review from hallerite, samsja and xeophon September 15, 2026 18:39

macroscopeapp Bot reviewed Sep 15, 2026

View reviewed changes

Comment thread verifiers/v1/cli/init.py Resolved

kcoopermiller mentioned this pull request Sep 21, 2026

feat(monitors): attach eval runs to a platform-created evaluation PrimeIntellect-ai/prime-rl#3619

Merged

xeophon changed the title ~~feat: remove the eval CLI~~ refactor(eval): remove the Verifiers eval CLI and move evaluation to Prime-RL Oct 4, 2026

xeophon mentioned this pull request Oct 4, 2026

refactor(eval): plan resumed rollouts by task content and occurrence #2760

Merged

mikasenghaas and others added 2 commits October 7, 2026 18:12

scaffold README points eval at prime-rl

37b6e11

The scaffold installs only verifiers, which has no eval executable since the eval CLI moved to prime-rl. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

xeophon force-pushed the feat/drop-eval-cli branch from 067fe68 to 37b6e11 Compare October 8, 2026 01:13

macroscopeapp Bot previously approved these changes Nov 3, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into drop-eval-cli-merge

1c0ef55

# Conflicts: # verifiers/v1/cli/eval/runner.py

mikasenghaas dismissed macroscopeapp[bot]’s stale review via 1c0ef55 November 3, 2026 00:50

mikasenghaas changed the title ~~refactor(eval): remove the Verifiers eval CLI and move evaluation to Prime-RL~~ refactor(eval): remove vf-eval Nov 3, 2026

macroscopeapp Bot previously approved these changes Nov 3, 2026

View reviewed changes

hallerite previously approved these changes Nov 3, 2026

View reviewed changes

mikasenghaas mentioned this pull request Nov 3, 2026

feat: add vf-run, a minimal in-process runner #3047

Merged

Merge remote-tracking branch 'origin/main' into drop-eval-cli-main-merge

2e0ab91

# Conflicts: # docs/getting_started.md # pyproject.toml # tests/integration/conftest.py # tests/integration/test_examples.py # verifiers/v1/configs/cli/eval.py # verifiers/v1/configs/cli/run.py

mikasenghaas dismissed stale reviews from hallerite and macroscopeapp[bot] via 2e0ab91 November 3, 2026 04:02

macroscopeapp Bot approved these changes Nov 3, 2026

View reviewed changes

mikasenghaas mentioned this pull request Nov 3, 2026

feat: remove the vf-replay cli #3052

Merged

mikasenghaas merged commit 1bcdb8b into main Nov 3, 2026
64 of 65 checks passed

mikasenghaas deleted the feat/drop-eval-cli branch November 3, 2026 04:14

earakely-scale mentioned this pull request Nov 3, 2026

feat(bundle): run a bundle from the Hugging Face Hub with agent-env run hf:// scaleapi/agentenv-framework#460

Merged

salma2vec mentioned this pull request Nov 3, 2026

fix(trace_store): exclude raw training arrays from saved episodes #2957

Open

Navigation Menu

Repository navigation

refactor(eval): remove vf-eval - #2610

refactor(eval): remove vf-eval#2610
mikasenghaas merged 5 commits into
mainfrom
feat/drop-eval-cli

mikasenghaas commented Sep 15, 2026 •

edited by macroscopeapp Bot

Loading

Uh oh!

Uh oh!

macroscopeapp Bot commented Sep 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mikasenghaas commented Sep 15, 2026 • edited by macroscopeapp Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Breaking

Verification

Remove the vf-eval CLI command and its eval modules

Uh oh!

Uh oh!

macroscopeapp Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approvability

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mikasenghaas commented Sep 15, 2026 •

edited by macroscopeapp Bot

Loading

Remove the `vf-eval` CLI command and its eval modules

macroscopeapp Bot commented Sep 15, 2026 •

edited

Loading