feat: run v0 environments on the eval CLI (legacy bridge)

[mikasenghaas]

Summary

• Backwards-compat: the v1 uv run eval entrypoint can now evaluate a classic v0 environment via --id <env-id>, bridged to v1 Traces — same mechanism the orchestrator uses (rollout_output_to_trace), run in-process for eval (no env server).
• All v0 glue stays in verifiers/v1/legacy.py (run_legacy_eval, reusing rollout_output_to_trace): loads the v0 env, runs num_rollouts per task with bounded concurrency via env.run_rollout, maps each RolloutOutput → Trace, writes the same results.jsonl / config.toml a native run does.
• Selector lives on vf.EnvConfig, grouped under an explicit legacy (v0) separator, matching prime-rl's shape so EvalConfig / EnvServerConfig inherit it:
  • id — v0 env id (load_environment(id, **args))
  • args — construction kwargs
  • extra_env_kwargs — post-load knobs applied via env.set_kwargs(**…) (e.g. max_total_completion_tokens, max_seq_len, timeout_seconds)
  • is_legacy (id set, no taskset) / env_id properties
• The eval CLI branches on config.is_legacy; --rich (v1 Rollout state) is auto-off for legacy runs.

Alignment with prime-rl

Mirrors prime-rl's legacy wiring: prime-rl's EnvConfig(vf.EnvConfig) exposes id/args/extra_env_kwargs/is_legacy/env_id, and its env server does if env.is_legacy: LegacyEnvServer(env_id, env_args, extra_env_kwargs). extra_env_kwargs is on prime-rl main but missing from feat/nano-as-v1 — adding it to the base vf.EnvConfig fills that gap and lets prime-rl drop its duplicates and inherit. Both legacy entry points (run_legacy_eval + LegacyEnvServer) apply it via the v0 env.set_kwargs.

Usage

uv run eval --id reverse-text --num_tasks 2
uv run eval --id wordle --args.use_think true -m <model>
uv run eval --id reverse-text --extra_env_kwargs.max_total_completion_tokens 256

--id (no taskset) makes is_legacy true; taskset/harness/runtime are ignored. Dict fields use dotted CLI syntax (--args.<k> <v>, --extra_env_kwargs.<k> <v>) or a @ config.toml.

Verification

$ uv run eval --id reverse-text --extra_env_kwargs.max_total_completion_tokens 256 --num_tasks 2 --rich false
INFO running 2x1 v0 rollouts on deepseek/deepseek-v4-flash (legacy: reverse-text)
# results.jsonl -> 2 traces: reward 0.92 / 1.0, turns=1, stop=max_turns_reached, errors=0

EvalConfig(id='reverse-text') → is_legacy=True, env_id='reverse-text'; native (taskset) → is_legacy=False. set_kwargs applies cleanly. Native v1 path unchanged (gsm8k-v1 subprocess → reward 1.0). ruff check/format clean.

Notes

• Legacy runner uses a plain v0 openai_chat_completions client (eval needs no token ids), unlike the training bridge's renderer client.
• --id org/name[@version] installs the v0 env from the Environments Hub on demand (via feat: resolve taskset / harness / env ids from the Environments Hub #1597's ensure_installed); a local --id <name> must already be importable (uv pip install -e environments/<name>).
• Follow-up (prime-rl): drop the now-duplicated id/args/extra_env_kwargs/is_legacy/env_id from prime-rl's EnvConfig and inherit from vf.EnvConfig; pass extra_env_kwargs to LegacyEnvServer in env_server.py (+ port main's auto-populate of max_total_completion_tokens / timeout_seconds / max_seq_len).

---

Note

Medium Risk
Introduces a parallel eval execution path and shared legacy config on EnvConfig, but native v1 eval is unchanged when a taskset is used; risk is mainly misconfiguration (legacy vs v1) and dependency on lazy v0 imports.

Overview
Adds backwards-compatible v0 environment evaluation on the v1 uv run eval CLI via --id <env-id> (no taskset), aligned with prime-rl’s legacy env shape.

EnvConfig gains legacy fields: id, args, extra_env_kwargs, plus is_legacy / env_id so eval and env-server configs can select a classic load_environment env instead of a v1 taskset/harness.

The eval CLI treats --id as a valid entry (alongside taskset or @ file.toml), branches on config.is_legacy to call run_legacy_eval instead of the native v1 path, and disables --rich for legacy runs (v1 rollout dashboard doesn’t apply).

verifiers/v1/legacy.py wires extra_env_kwargs through LegacyEnvServer via env.set_kwargs, and adds run_legacy_eval: in-process v0 rollouts with bounded concurrency, RolloutOutput → Trace mapping, and the same results.jsonl / config.toml output layout as native eval (no env server / interception).

^{Reviewed by Cursor Bugbot for commit adbcf95. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add legacy v0 environment support to the eval CLI via --id flag

• Adds --id <env-id> flag to the eval CLI (eval.py) that triggers a legacy v0 evaluation path, bypassing the v1 taskset/harness and suppressing the rich dashboard.
• Implements run_legacy_eval in legacy.py to run v0 environments in-process: loads the env, samples tasks with optional deterministic shuffle and concurrency limits, runs rollouts, and persists results as v1 Trace objects to outputs/<env>--<model>--legacy/<uuid>/results.jsonl.
• Extends EnvConfig in env.py with id, args, extra_env_kwargs fields and is_legacy/env_id properties to represent v0 environments alongside v1 taskset configs.
• Behavioral Change: when --id is passed, SIGTERM handling and the v1 environment setup are skipped entirely.

^{Macroscope summarized adbcf95.}

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit adbcf95. Configure here.}

[cursor]

Skips v0 group scoring

Medium Severity

When num_rollouts is greater than one, run_legacy_eval schedules separate env.run_rollout calls per rollout. Classic v0 evaluation uses env.run_group so rubric.score_group runs; isolated rollouts only get per-rollout scoring, so group-reward environments return incorrect rewards and metrics.

 

^{Reviewed by Cursor Bugbot for commit adbcf95. Configure here.}

[cursor]

Uses train split not eval

Medium Severity

Legacy eval builds task indices from env.get_dataset(), which uses the training dataset. Standard v0 evaluate() uses get_eval_dataset() (with train fallback only when no eval split exists), so environments with a separate eval split can be scored on the wrong examples.

 

^{Reviewed by Cursor Bugbot for commit adbcf95. Configure here.}

[cursor]

Missing evaluation documentation

Medium Severity

This PR adds a user-facing legacy uv run eval --id path, EnvConfig legacy fields, and extra_env_kwargs wiring, but the diff does not update the evaluation docs, reference material, or affected skills to describe the new CLI contract.

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit adbcf95. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces new feature functionality (legacy v0 environment bridge) and has two unresolved medium-severity bug reports: using training dataset instead of eval dataset, and skipping group scoring for multi-rollout evaluations. These correctness concerns require human review.

^{You can customize Macroscope's approvability policy. Learn more.}