Add ViBench resources server + agent (PRD-to-web-app benchmark)

[deepbuilder]

Wraps ViBench as a NeMo Gym environment: a model builds a web app from a PRD, and grading stands the app up for real, seeds it through its own UI, and drives it in a browser against a human-written test plan. The score reflects what a user can reach, not what the diff contains.

Adds resources_servers/vibench and responses_api_agents/vibench_agent.

Design

The agent owns the build sandbox and copies the finished app out as a tarball; the resources server unpacks it and grades it in a fresh compose project, one per test plan, via ViBench's run-seed.py → run-evaluate-post-seeding.py.

The sandbox is never shared. Reaching into the agent's box needs serialize()/connect(), which only the OpenSandbox provider implements, so the shape swebench uses cannot run on Docker, Apptainer or enroot. See #2082 for the design discussion; this PR has no dependency on it or on #2085.

vibench_agent subclasses OpenCodeSandboxedAgent and overrides only sandbox acquisition and harvesting.

Reward

One row is one (app, artifact) pair: built once, graded across that artifact's test plans. Reward is the mean normalized score, continuous rather than binary, with per-plan values in reward_components. build_failed, seeding_failure_rate and plans_graded_rate are reported separately, because a mean reward cannot distinguish a weak app from one that never built or could not be seeded.

REVERIFY_MODE is UNSUPPORTED — grading depends on live app and database state.

Validation

Five example tasks graded end to end on a Docker host. Scores spread from 0.00 to 1.00 with partial credit in between, and every zero carried a real scorecard with per-step diagnoses (HTTP 500s, SQL errors) rather than a harness failure.

The reward tracks capability: a weaker policy model scored 0.0 on a task with 0/19 steps, its app unreachable, where a stronger model scored 1.0 on the same task with the same graders.

Scores also vary across repeats of the same model, since it does not build the same app twice. Quantifying that — and separating it from any variance in the LLM-driven verifier — is what reward profiling is for, so verified: false stands.

data/example_rollouts.jsonl holds a five-task run.

Reviewer notes

• network: host is deliberately not used. The in-sandbox harness reaches the policy model at get_server_url(...) → 127.0.0.1:<port>, which inside a bridged container is the container itself; the harness then makes zero LLM calls and exports an empty app with nothing logged. configs/docker.yaml instead adds host.docker.internal via host-gateway and the agent rewrites loopback model URLs to it. Only policy_model binds 0.0.0.0; the servers holding grader credentials stay on loopback. Residual: the sandbox can reach host-published TCP ports, which inference requires.
• Grading timeouts interrupt rather than terminate. ViBench tears its compose project down in a finally, which CPython does not unwind on SIGTERM. Escalation is SIGINT → SIGTERM → SIGKILL. Verified with a forced timeout on a real host: SIGTERM leaked two containers and a network, SIGINT cleaned up fully.
• artifact_dir is a filesystem path shared by agent and resources server. Not a new constraint — grading already shells into a local Docker daemon.
• Requires docker-compose on PATH. ViBench's scripts use the legacy name; Docker 29.x ships only the plugin. The failure misleads, so it is called out in the README.
• Task fields are top-level, not under verifier_metadata, following swebench since seed_session and verify share one typed request model. Happy to move them.
• Grading runs on the resources server's Docker daemon, not in a Gym sandbox, because ViBench's grading stack is multi-container. Single-image packaging is the prerequisite for multi-host.
• mvp artifacts only. ViBench ships reference implementations for 21 of 24 apps, so feature-ri is a follow-up; it needs a starting codebase staged into the sandbox the way the PRD already is.
• Row size: rollout rows carry the harness's own stdout/stderr, so example_rollouts.jsonl is ~500 KB for five tasks. Happy to trim.

Security

The tarball comes from a box the model controlled, so it is untrusted: unpacking refuses members and links resolving outside the app dir and passes filter="data", and artifact paths outside artifact_dir are refused and never deleted. Grader credentials are scrubbed from captured output before it is stored, since that output ships in the rollout JSONL. Test plans and test_assets/ are never staged into the build box.

Testing

gym env test --resources-server vibench passes. 75 unit tests (60 resources server, 15 agent), coverage 99%. pytest tests/unit_tests/ 2621 pass. pre-commit run --all-files clean.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally.
• Pre-commit checks pass locally (pre-commit run --all-files).
• All commits have DCO sign-off (git commit -s).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

/claude review

[claude]

SHIP WITH CARE — one RISK finding (metrics contract), otherwise a well-built environment with strong test coverage.

The reward path itself is sound: verify() computes reward in-process, isolates each test-plan failure into a zeroed PlanResult instead of 500-ing the rollout, counts every plan toward the denominator (seeding failures drag the mean down rather than vanishing), and guards div-by-zero on full_points. Async is correct — subprocess grading via create_subprocess_exec, asyncio.Semaphore bounding concurrent compose stacks, no httpx/ray.get(). Security posture is careful: tar-member and symlink escape checks, artifact-path validation before any unlink, credential redaction on captured output, and grader keys kept off the builder. New-file SPDX headers present; verified: false correctly retained.

RISK (inline, app.py:616): the compute_metrics/get_key_metrics overrides use the wrong AggregateMetricsMixin signatures. In standard eval they're never reached (the OpenCode-derived agent doesn't proxy /aggregate_metrics to the resources server), so the documented failure-breakdown metrics silently never appear; on the reverify path they're posted to the resources server and raise AttributeError/TypeError. No effect on reward — this is observability. See inline for the fix.

Notes (author's call): _grader_env caches secrets in _cached_grader_env; fine given process lifetime. The Docker-only docker-compose requirement and stochastic LLM verifier are documented as P0 limitations with verified: false, which is the right call.

[claude]

RISK — custom metrics don't work against the framework contract; silently absent in eval, crash on reverify.

AggregateMetricsMixin (nemo_gym/reward_profile.py:590) defines compute_metrics(self, tasks: List[List[Dict]]) — verify responses grouped by task — and get_key_metrics(self, agent_metrics: Dict) (one positional arg). Both overrides here have the wrong signature: compute_metrics(self, verify_responses) iterates the argument as flat dicts, and get_key_metrics(self) takes no arg.

Two consequences:

1. Silently absent in standard eval. _call_aggregate_metrics (nemo_gym/rollout_collection.py:976) POSTs /aggregate_metrics to the agent server. vibench_agent → OpenCodeSandboxedAgent does not proxy /aggregate_metrics to the resources server (contrast simple_agent app.py:354, which does). So the base agent's default compute_metrics ({}) runs and these methods are never invoked — the whole point of this override (build_failure_rate, mean_seeding_failure_rate, plans_graded_rate, and the get_key_metrics headline set) never reaches aggregate metrics.

2. Crashes on reverify. nemo_gym/rollout_reverification.py:577 POSTs /aggregate_metrics to the resources server. There compute_metrics receives List[List[Dict]]; r.get("reward") on a list → AttributeError, and get_key_metrics(self) called as get_key_metrics_fn(serialized_agent) → TypeError. Aggregation 500s.

FIX: (a) match the mixin signatures — flatten the grouped tasks in compute_metrics, and accept agent_metrics in get_key_metrics returning a dict subset; and (b) add an aggregate_metrics proxy on vibench_agent that forwards to the resources server (as simple_agent does), or these metrics never surface regardless of signature. Reward itself is unaffected (computed in verify), so this is observability, not score corruption.

[deepbuilder]

Fixed in e1e4bad — both parts confirmed against the framework before changing anything.

Signatures. compute_metrics now takes tasks: List[List[Dict]] and flattens the per-task groups; get_key_metrics(agent_metrics) returns a dict subset and preserves the mean/* default so nothing standard disappears.

Proxy. Added aggregate_metrics on vibench_agent forwarding to the resources server, as simple_agent does. You were right that without it the signatures were moot — build_failure_rate, mean_seeding_failure_rate and plans_graded_rate never ran at all.

Reward is computed in verify() and was unaffected, as you noted.

Tests now call these the way compute_aggregate_metrics does, including a conformance check that asserts the signatures match the mixin via inspect. The old tests passed throughout precisely because they called the methods the way my code expected — worth guarding against directly.

[deepbuilder]

Thanks — the RISK finding was right on both counts, and I verified each against the framework before touching anything. Fixed in e1e4bad.

Signatures now match AggregateMetricsMixin: compute_metrics(tasks: List[List[Dict]]) flattens the per-task groups, and get_key_metrics(agent_metrics) returns a dict subset while preserving the mean/* default.

Proxy added on vibench_agent forwarding /aggregate_metrics to the resources server, as simple_agent does. This was the more important half — without it the metrics never ran regardless of signature.

Reward is computed in verify() and was unaffected.

Tests now exercise these the way compute_aggregate_metrics calls them, plus a conformance check asserting the signatures match the mixin via inspect. The previous tests passed the whole time because they called the methods the way my code expected rather than the way the framework does; the conformance check guards that class of mistake directly.

On the two notes: _cached_grader_env holding secrets for the process lifetime is deliberate — it replaced a synchronous subprocess.run on the async path that also re-derived the env six times per rollout. Happy to scope it down if you would prefer.

Gates: gym env test --resources-server vibench passes, 79 unit tests (63 resources server, 16 agent), pre-commit run --all-files clean.

[cmunley1]

Claude suggestion, please take a look:

[P1] This method no longer matches the parent on main

This branch forked at 354babf. Since then #2120 changed the parent to take a request:

main   opencode_sandboxed_agent/app.py:503   async def _create_opencode_config(self, request: Request)
here                                               def _create_opencode_config(self)

Rebased on origin/main, the agent tests go to 1 failed / 15 passed with
TypeError: _create_opencode_config() missing 1 required positional argument: 'request'.
The resources server tests stay green at 61 passed / 2 skipped.

This one is worth catching before merge because of what the method does. It is what rewrites the model
URL for the sandbox. Without the rewrite the harness talks to itself, makes no LLM calls, and exports an empty app with no error. Nothing reports that as a failure, including
build_failed, because the build technically succeeded.

There is a second problem in the same method. Main now builds the URL through base_url_for_run
(nemo_gym/base_responses_api_agent.py:147), which adds /ng-rollout/<id> and the token capture suffix.
Line 150 throws that away and rebuilds from get_server_url(), which quietly turns off the token capture
path #2120 added. It is off by default, so nothing breaks today.

Both fix with one change. Take the parent's URL and rewrite only the host.
rewrite_loopback_url_for_docker (line 90) already does that and keeps the path. _origin only strips a
trailing slash or /v1, so add /v1 back:

async def _create_opencode_config(self, request: Request) -> Dict[str, Any]:
    config = await super()._create_opencode_config(request)
    options = ((config.get("provider") or {}).get("nemo_gym") or {}).get("options")
    if isinstance(options, dict) and isinstance(options.get("baseURL"), str):
        options["baseURL"] = rewrite_loopback_url_for_docker(options["baseURL"]) + "/v1"
    return config

One thing to check. This drops the sandbox_model_base_url branch, so add it back if anyone uses
that option.

[cmunley1]

if the opencode_sandboxed_agent keeps changing under you, it may be more stable to implement all here, cc @bxyu-nvidia as author of the base agent class.

[deepbuilder]

Confirmed and fixed in c49183d. Rebased onto current main first — reproduced exactly what you described: 1 failed / 15 passed with TypeError: _create_opencode_config() missing 1 required positional argument.

Took your suggested shape: the override is now async def _create_opencode_config(self, request) and rewrites only the host of the parent's URL, so the base_url_for_run path survives. Good catch on the token-capture suffix — I was rebuilding from get_server_url() and silently dropping /ng-rollout/<id>.

Kept sandbox_model_base_url: it replaces the origin but now preserves the path suffix, so an explicit address cannot drop the capture path either.

Added a conformance test asserting this signature matches the parent's via inspect. Both failure modes here are invisible at runtime — the harness just talks to itself and exports an empty app that build_failed does not catch — so a mismatch should fail loudly in CI rather than in a rollout.

Verified on the rebased tree: 63 resources server + 17 agent tests, gym env test --resources-server vibench 61 passed, pre-commit run --all-files clean.

[deepbuilder]

Fair concern — this is the second time a silent drift in the parent has bitten this branch (the other was compute_metrics against AggregateMetricsMixin).

My read is that inlining trades one failure mode for a worse one. Subclassing inherits the whole OpenCode install-and-drive path — version pinning, remote install script, config generation, export parsing — and ViBench overrides only two things: where the sandbox comes from, and rewriting the model URL host. Copying the rest would be several hundred lines that then drift from the base silently and permanently, rather than failing at a signature boundary.

What made both incidents dangerous was that they were invisible at runtime, not that they happened. So I have added conformance tests asserting the overrides match the parent signatures via inspect — a future change like #2120 now fails in CI instead of producing an empty app with no error.

Happy to go either way if @bxyu-nvidia would rather the base class not be subclassed externally — in that case the cleaner split might be a small extension point on OpenCodeSandboxedAgent for the sandbox URL, which would serve any provider whose boxes cannot reach host loopback, not just ViBench.

[bxyu-nvidia]

/ok to test ace821a

[bxyu-nvidia]

/ok to test 8719abe