[mikasenghaas]

Summary

• Add vf-run, a minimal in-process runner: uv run vf-run <taskset> [-m <model>] [-n <tasks>] [-r <rollouts>] [-c <concurrency>] [-o <dir>] [--dry-run].
  • It runs the selected episodes in one process and appends each one to <output_dir>/<run.dir>/traces.jsonl as it completes.
  • It prints one summary line (3/4 episodes ok, mean reward 0.750: <run-dir>) and exits 1 when any episode is not ok.
  • It has no dashboard, resume, attempt logs, or platform push.
  • When one rollout fails, the taskset raises, or the run is interrupted, it cancels the other rollouts and waits for them to unwind before serving teardown.
  • It refuses a run directory that already holds episodes, in the CLI and in run_episodes.
  • --dry-run resolves the typed config (plugin discovery included), writes configs/resolved/run.json, and exits.
• Run the integration suite through it. The run fixture calls run_episodes, and the examples smoke test runs the vf-run CLI.
• Move RunConfig and default_run_name to verifiers/v1/configs/cli/run.py, and type output_path with a RunOutput protocol. vf-eval, vf-validate, and vf-gepa keep working unchanged.

Merge order:

1. This PR. Its merge publishes a dev release with vf-run.
2. feat: run the env smoke tests through vf-run prime-envs#1053 and PrimeIntellect-ai/prime-envs-private#373 switch the env smoke tests to vf-run.
3. refactor(eval): remove vf-eval #2610 removes vf-eval.

Verification

• pytest tests/unit: passes, including the new tests/unit/cli/test_run.py.
• pytest tests/integration/test_examples.py -k "reverse_text or interception or kuhn_poker" through the vf-run CLI: 6 passed.
• Fixture-based integration tests (test_single_turn, test_multi_turn, test_parallel, -k "subprocess or null"): 6 passed.
• End-to-end CLI scenarios against Prime Inference (internal/deepseek-v4.1-flash), 22 checks:
  • reverse-text -n 2 -r 2 (null harness): exit 0, 4 episodes, mean reward 1.0. The summary line, configs/resolved/run.json, and one eval run id on every episode and trace are all present.
  • kuhn-poker -n 1 (multi-agent): exit 0, one episode with several traces.
  • Unknown model: exit 1, 0/2 episodes ok, both errored episodes saved.
  • Existing run directory: refused, traces unchanged.
  • --dry-run writes only the config, and vf-run @ <run-dir>/configs/resolved/run.json re-runs it. A TOML @ run.toml also runs.
  • An infinite taskset without -n is refused. No arguments and a missing taskset print the usage.
  • vf-replay <run-dir> re-scores vf-run output.
  • SIGINT during a 120-rollout run (bash harness, subprocess runtime): exit 130, cleanup ran, the 8 finished episodes were kept, and no processes were orphaned.

🤖 Generated with Claude Code

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9276d0d. Configure here.}

[macroscopeapp]

Approvability

Verdict: Approved at 1bbc5f6

Macroscope's review found this PR approvable — Adds an opt-in in-process runner that saves traces and reports episode outcomes, reusing existing environment and storage APIs. Existing execution behavior remains unchanged, with result-directory protection and cancellation cleanup limiting the new path’s risk.

^{You can add or adjust custom eligibility rules. Learn more.}