feat(v1): Harbor separate verifier environments

[rasdani]

Wires up Harbor's [verifier].environment_mode = "separate" and [verifier.environment], which we currently ignore and hard-reject respectively — so a task that asks to be graded away from the agent is graded inside the box the agent just had root in. Design and prior art are @xeophon's (#2067).

Why

A grader running in the agent's box is attackable by the thing it grades. Verified on r2e-gym: an agent-planted /testbed/conftest.py forcing pytest's results scores reward 1.0 with no fix applied. It never touches the grading script or the tests, so hiding those does nothing — the fix is grading in a different box.

How

HarborEnv (tasksets/harbor/env.py), now the taskset's default env:

• Shared-verifier tasks (Harbor's default, and every task that says nothing): unchanged — one agent trace, graded in the box it worked in.
• Separate-verifier tasks: the solver plays as usual; its declared artifacts and the /logs/artifacts/ convention dir are collected while its box is alive. finalize then provisions a fresh box from the task's verifier declaration, restores those artifacts, stages tests/ fresh, runs the verifier there, and records its rewards onto the solver's trace (extra reward.json keys become metrics).
• --env.verifier-runtime.* places the grading box (default: derived from the solver's runtime policy; a network-restricted verifier on Prime needs vm true). --env.verifier-retries (default 2) retries provisioning/grading without re-running the solve; exhausted retries fail the episode — a grading box that can't be reached never reads as reward 0.
• Under any other env, a separate-verifier task fails closed instead of silently grading in the agent's box. --taskset.ignore-separate-verifier forces shared grading.

Which image the verifier boots follows Harbor:

task.toml	verifier image
environment_mode = "separate", no [verifier.environment]	fresh copy of the task's own image
[verifier.environment] with docker_image	that image
[verifier.environment] without docker_image	rejected at load — verifiers never builds tests/Dockerfile; build and push it, or ignore_dockerfile

Also in here

• test.sh runs by absolute path in the configured workdir, like Harbor — real tasks grade the agent's work at $PWD, and the old cd /tests graded the wrong tree
• reward-file cleanup moved into test staging, which raises on failure — a planted, unremovable reward file fails the rollout instead of being read as the score
• runtime.read(path, max_bytes=...): bounded reads with the cap enforced inside the box, for grading inputs whose size nothing guarantees (replaces read_bounded)
• one task_resources helper instead of two copies of the Harbor resource conversion

Earlier revisions did this with a Task.scoring_runtime hook, then an env verifier seat; review moved it to plain env code — see the threads.

Verification

No public dataset exercises this feature (104k cached task.tomls, Harbor Hub, and laude-institute/harbor-datasets declare zero separate-mode tasks), so: local fixtures driven through the real HarborEnv.finalize against live Docker. Fails closed outside the env; the declared artifact travels while an undeclared leftover does not; stale /tests is wiped; keyed rewards land on the solver's trace; a declared verifier image is actually used (python 3.12 vs the task's 3.11); shared grading unchanged. Full test suite, ruff, and ty green. Not verified on Prime.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes eval integrity and sandbox lifecycle (isolated grading, artifact restore, retries); misconfiguration could fail episodes or weaken isolation via ignore_separate_verifier, but agent auth/payments are untouched.

Overview
Harbor tasks with [verifier].environment_mode = "separate" can now be graded in a second sandbox the agent never used. The new default HarborEnv runs the solver as today, collects artifacts in the agent box, then in finalize provisions a verifier box (from [verifier.environment] or a copy of [environment]), restores artifacts, wipes and stages tests/, runs test.sh, and writes rewards onto the solver trace. --env.verifier-runtime.* and --env.verifier-retries control placement and infra retries; outside this env, separate-verifier tasks error unless --taskset.ignore-separate-verifier forces shared grading.

Parsing no longer rejects separate verifier environments: VerifierConfig, verifier_box_data, and graded_elsewhere() skip in-agent scoring when the env owns grading. Declared verifier envs without a pullable docker_image fail at load (or warn with ignore_dockerfile).

Runtime/helpers: shared provision_runtime context manager; Runtime.read(..., max_bytes=...) with in-sandbox cap (backends implement _read). Harbor scoring runs bash /tests/test.sh from the task workdir, clears stale reward files during staging (fail loud), and caps reward file reads at 1 MiB.

^{Reviewed by Cursor Bugbot for commit 01f0e76. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add separate verifier environment support for Harbor tasks

• Harbor tasks can now grade in a separate verifier box after the agent box is confirmed deleted, preventing cross-contamination between agent and scoring environments.
• A new Task.scoring_runtime hook and provision_runtime context manager in runtimes/init.py manage the verifier box lifecycle; harness scoring runs on the agent box first, then task scoring runs in the verifier box.
• taskset.py parses a VerifierConfig from Harbor task config (image, resources, workdir, network allowlist, fresh_copy flag) and builds a matching Docker or Prime runtime config for the verifier box.
• HarborConfig gains an ignore_separate_verifier flag to force grading in the agent box regardless of task config.
• Runtime.read gains an optional max_bytes cap enforced via head -c inside the sandbox; reward file reads use this cap. stop_confirmed and teardown_confirmed are added to Runtime so the rollout can verify the agent box is gone before starting the verifier.
• Risk: rollouts that borrow a runtime or use shared tool servers will raise TaskError if a separate scoring box is requested.

Changes since #2152 opened

• Introduced a two-seat environment flow for Harbor tasks with separate verifier environments [1213839]
• Removed task-managed separate scoring runtime infrastructure across the verifier framework [1213839]
• Removed confirmed teardown methods from the Runtime API [1213839]
• Added a no-operation harness implementation [1213839]
• Removed NoopHarness and NoopHarnessConfig classes from the verifiers.v1.harnesses.noop module and eliminated their exports from the verifiers.v1.harnesses package [be6573f]
• Replaced the verifier agent seat model in HarborEnv with direct grading execution during finalize [be6573f]
• Removed the HarborVerifierTask class from verifiers.v1.tasksets.harbor.taskset module [be6573f]
• Added verifier_box_data utility function to derive verifier box configuration from task data [be6573f]
• Updated Harbor documentation to reflect the architectural change from verifier seat to env-based grading [be6573f]
• Moved verifier box teardown outside the scoring timeout in HarborEnv.finalize method [59b9bfd]
• Fixed error propagation in Runtime.read method when max_bytes is provided [59b9bfd]
• Added validation constraint for verifier_retries and type checking for task parameter [59b9bfd]
• Bounded the fallback reward file read operation in HarborTask.finalize method [01f0e76]

^{Macroscope summarized f29df85.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a significant new feature (separate verifier environments) with new configuration options, new grading workflows, and retry logic. Additionally, there is an unresolved bug report about teardown failures potentially discarding valid scores. The scope and unresolved issue warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}

clanker approved without my consent

[mikasenghaas]

bunch of stuff in here that is not harbor related no?

[mikasenghaas]

wait also, we dont use agentic judge env for this right but reward funcs? we should defo use our multi-agent machinery for this kinda stuff

edit: @xeophon tells me we do. can we add a section in the pr desc which shows how to wire up and eval config for a harbor taskset w/ agentic judge through agentic judge env and results on actually running this e2e

[mikasenghaas]

btw at some point i think we could have smth like Runtime.restore(artifacts: Artifacts), prob implemented using upload primitives in the base runtime but fine for now

[hallerite]

yeah, good point

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 59b9bfd. Configure here.}

[cursor]

Teardown failure discards graded score

Medium Severity

return scores still exits through AsyncExitStack, so a raising stop()/teardown is caught by the broad retry except and discards a score already in hand. The scoring timeout was moved off teardown, but teardown exceptions still retrigger a full reprovision and regrade, and can fail the episode despite a successful grade.

 

^{Reviewed by Cursor Bugbot for commit 59b9bfd. Configure here.}