chore: add v1 end-to-end eval test suite

[mikasenghaas]

Summary

End-to-end tests for the v1 stack under tests/v1/ — real eval runs (the same path as uv run eval) with the smallest settings that still exercise it, asserting on the resulting Trace, rather than unit-testing components. Also folds in the framework fixes the suite surfaced (user-simulator placement/reachability, a SUPPORTS_USER_SIM capability flag).

Tests

• test_e2e.py — reward-1 runs on trivial tasksets across the harness × runtime matrix (harness × runtime fixtures): built-in harnesses (default, rlm) × runtimes (subprocess/docker/prime; modal excluded).
  • single-turn (echo), multi-turn (echo-multi, user-sim), multi-turn + tools (glossary), agentic (echo-agentic, bash).
  • capability-gated: the tools test expects a harness without SUPPORTS_TASK_TOOLS (rlm) to be rejected (raise); the multi-turn test skips a harness without SUPPORTS_USER_SIM (rlm).
  • server-runtime matrix: test_task_tools_own_runtime / test_user_own_runtime run a tool server / the user-sim in its own runtime (subprocess/docker/prime), agent on subprocess.
• test_legacy.py — v0 backwards-compat: echo-v0 (single) + echo-multi-v0 (multi) run through the legacy bridge and match a native v1 run's output shape.
• test_configs.py — every root configs/*.toml parses into EvalConfig.
• tests/v1/fixtures/ — deterministic, self-contained fixture tasksets/envs (no dataset/Dockerfile/install), resolved by id via pytest's pythonpath: echo-v1, echo-multi-v1 (container-safe user-sim), echo-agentic-v1 (bash writes a file, verified in the runtime), echo-v0, echo-multi-v0.

Running the tests

These are real eval runs, so they need a model API key (PRIME_API_KEY / OPENAI_API_KEY); without one the e2e-marked tests skip (config parsing still runs).

# Fast (default tier): default harness × subprocess + v0 legacy + config parsing (~20s)
uv run pytest tests/v1 -m "not slow" -n auto

# Full matrix: adds docker + prime runtimes, the rlm harness, and own-runtime tool/user-sim servers
# (docker needs the daemon — if you're not in the `docker` group, wrap it: sg docker -c '...')
sg docker -c 'uv run pytest tests/v1 -n auto'

# Slices
uv run pytest tests/v1/test_configs.py          # config parsing only (no API key needed)
uv run pytest tests/v1 -m prime                 # only the prime-runtime variants
uv run pytest tests/v1 -k single_turn -n auto   # one task across the matrix

-n auto runs in parallel (pytest-xdist). Markers: e2e (needs a model key), slow (docker/prime/rlm), prime (prime-runtime variants).

Framework changes folded in

• SUPPORTS_USER_SIM — a Harness capability flag (default harness opts in; rlm can't drive a user-sim — single-instruction interface, no message history).
• User-simulator runs in its own runtime, reached host-side. It's driven by the framework on the host, so it ran colocated in the agent's runtime with a localhost URL — unreachable when the agent is a remote prime sandbox (ConnectError / 421). Now serve_user runs it in its own runtime (host subprocess by default, or its own sandbox via TasksetConfig.user.runtime) and publishes the port back to the host (serve_tools host_reachable: a remote sandbox's public_url, else localhost).
• UserConfig(runtime) on TasksetConfig; serve_tools host_reachable.
• New pytest markers: e2e, prime.

Notes

• harbor hello-world can't be used here — it ships a Dockerfile, and harbor_v1 refuses to build Dockerfiles; the echo-agentic fixture covers the agentic case instead.
• A tool/user-sim in its own prime sandbox needs a region that supports port exposure; the default doesn't, so those server-runtime cases skip (skip_if_unexposable) with a TODO at prime.public_url to lift the limit.

Verification

• Full matrix (subprocess/docker/prime, all tasks): 34 passed, 3 skipped (rlm user-sim), 0 failed (~40s, -n auto).
• docker server-runtime cases pass; prime server-runtime cases skip (port-exposure region). ruff check + ruff format --check clean.

---

Note

Medium Risk
Rollout/user-sim wiring changes affect multi-turn evals on remote runtimes; new e2e tests need API keys and optional docker/prime infra but do not alter production eval configs by default.

Overview
Adds tests/v1/ end-to-end coverage that runs real evals (same path as uv run eval) on tiny fixture tasksets and asserts on Trace outcomes, plus pytest wiring (pythonpath for fixtures, e2e / prime markers, API-key skip).

Framework fixes the suite drove: user simulators no longer run colocated in the agent runtime—they use TasksetConfig.user.runtime (default host subprocess), with serve_tools(..., host_reachable=True) so the host can reach them when the agent is remote. Adds harness SUPPORTS_USER_SIM (default harness opts in). Rollout passes config.user.runtime into serve_user.

E2E matrix spans harness × runtime (single/multi-turn, tools, bash agentic, tool/user in own runtime); legacy bridge shape tests and root EvalConfig TOML parsing included.

^{Reviewed by Cursor Bugbot for commit c4f8300. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add v1 end-to-end eval test suite with configurable user simulator runtimes

• Adds a comprehensive e2e test suite under tests/v1/ covering single-turn, multi-turn, agentic, and legacy v0-bridge scenarios using parameterized runtimes and harnesses.
• Introduces fixture tasksets (echo_v1, echo_multi_v1, echo_agentic_v1) and legacy v0 environments (echo_v0, echo_multi_v0) used as test targets.
• Changes serve_user in user.py to accept a RuntimeConfig and spin up a dedicated runtime for the user simulator, rather than colocating it with the agent runtime.
• Adds UserConfig to TasksetConfig in taskset.py so tasksets can declare a separate user runtime (defaults to subprocess).
• Behavioral Change: user simulators now run in their own runtime and are exposed via a host-reachable URL; previously they ran inside the agent runtime.

^{Macroscope summarized c4f8300.}

[macroscopeapp]

Approvability

Verdict: Needs human review

While labeled as adding a test suite, this PR includes production changes that fundamentally alter how user simulators are hosted - they now run in their own isolated runtime rather than being colocated with the agent. This new runtime isolation capability and the associated configuration changes (UserConfig, host_reachable parameter) warrant human review to verify the architectural design.

^{You can customize Macroscope's approvability policy. Learn more.}