feat(sandbox): add sandbox server for cross-server sandbox sharing - #2085
Draft
ananthsub wants to merge 4 commits into
Draft
feat(sandbox): add sandbox server for cross-server sandbox sharing#2085ananthsub wants to merge 4 commits into
ananthsub wants to merge 4 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
2 tasks
ananthsub
force-pushed
the
ansubramania/sandbox-server
branch
from
July 20, 2026 17:27
e53ea9f to
707f0f7
Compare
ananthsub
marked this pull request as ready for review
July 20, 2026 17:39
ananthsub
force-pushed
the
ansubramania/sandbox-server
branch
from
July 20, 2026 17:41
707f0f7 to
ec8f8fb
Compare
github-actions
Bot
added
the
sla:review-overdue
bxyu-nvidia
previously approved these changes
Jul 23, 2026
Glorf
previously approved these changes
Jul 24, 2026
ananthsub
force-pushed
the
ansubramania/sandbox-server
branch
from
July 28, 2026 20:03
ec8f8fb to
e3ce083
Compare
hemildesai
previously approved these changes
Jul 29, 2026
cmunley1
reviewed
Jul 29, 2026
| responses_api_agents: Dict[str, BaseRunServerTypeConfig] = Field(min_length=1, max_length=1) | ||
|
|
||
|
|
||
| class SandboxServerTypeConfig(BaseServerTypeConfig): |
Contributor
There was a problem hiding this comment.
should there be a SandboxServerInstanceConfig so that gym env start / validate passes?
I think GlobalConfigDictParser.filter_for_server_instance_configs() might fail
cmunley1
reviewed
Jul 29, 2026
| return StatusResponse(status=(await entry.sandbox.status()).value) | ||
|
|
||
| async def grant_lease(self, sandbox_id: str, body: LeaseRequest, request: Request) -> dict: | ||
| entry = self._check(request, sandbox_id) |
Contributor
There was a problem hiding this comment.
should we pass require_owner=True?
cmunley1
reviewed
Jul 29, 2026
| async def release_lease(self, sandbox_id: str, request: Request) -> ReleaseResponse: | ||
| entry = self._check(request, sandbox_id) | ||
| async with self._lock: | ||
| entry.leases = max(0, entry.leases - 1) |
Contributor
There was a problem hiding this comment.
does this only decrement a counter but leave the token valid? should we reject released tokens in _check() ?
Contributor
|
could we also add docs or an example environment using it? |
github-actions
Bot
removed
the
sla:review-overdue
…ct facade Introduce an optional, runtime_checkable ConnectableProvider protocol (serialize_handle / connect), separate from the base SandboxProvider, so a provider whose sandbox is reachable by id can hand a descriptor to another process. Add AsyncSandbox.serialize and AsyncSandbox.connect, which delegate to the capability and raise for providers that do not support it. No in-tree provider implements the capability yet; a fake provider in the tests exercises the serialize/connect round-trip and the negative cases. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
AsyncSandbox.connect rebuilds the spec from descriptor["workdir"] and exec
defaults cwd to it. The remote provider's SandboxRef already includes workdir,
but a provider whose descriptor omits it (e.g. OpenSandbox's {"sandbox_id"})
would lose the working directory on reattach. Have serialize() annotate the
descriptor with the sandbox's workdir when the provider did not, so exec on a
reattached sandbox defaults to the same directory for every provider.
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…ider
OpenSandbox sandboxes are reachable by id, so add serialize_handle (returns
{"sandbox_id": ...}) and connect (rebuilds a handle via Sandbox.connect(id)).
This lets an OpenSandbox-backed box be operated from another process without
a sandbox server. Tests mock the SDK and assert the provider satisfies the
ConnectableProvider protocol and round-trips a descriptor.
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Add a fourth Gym server type (sandbox_servers/) that owns physical sandboxes and lends them across servers by reference. A box created by one server (an agent running a rollout) can be operated by another (a resources server that scores it): the same live box, or a fresh eval box. - SandboxRef: serializable, signed lease capability (server_url, sandbox_id, scope); owner leases may destroy the box, operate co-leases may not; leases bind to the rollout id. - RemoteSandboxProvider: forwards the provider protocol to a sandbox server over an injected SandboxHttpTransport, so nemo_gym.sandbox never imports the server framework. It implements ConnectableProvider (serialize/connect). - nemo_gym.sandbox_client: the one bridge that injects Gym's aiohttp client (make_remote_provider, connect_sandbox). - sandbox_servers/sandbox_server: the server (create/exec/upload/download/ status + lease grant/release, in-process registry, max_concurrent admission, ttl reaping) and registration of the fourth server type in config_types. Hermetic tests use a fake in-process provider to drive a full in-process HTTP round-trip against the real server. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
ananthsub
force-pushed
the
ansubramania/sandbox-server
branch
from
July 30, 2026 17:37
e3ce083 to
3ac4e67
Compare
ananthsub
added a commit
that referenced
this pull request
Jul 30, 2026
…ct facade (#2083) ## Summary Part of #2082 Bottom of a 3-PR stack for cross-server sandbox sharing. Introduces the optional provider capability and the sandbox-facade methods that use it. No in-tree provider implements the capability yet; the next PRs add OpenSandbox and the sandbox server. Stack: 1. **#2083 (this PR)** — ConnectableProvider + serialize/connect facade 2. #2084 — OpenSandbox implements ConnectableProvider 3. #2085 — sandbox server ## What's here - **`ConnectableProvider`** — an optional `runtime_checkable` protocol (`serialize_handle` / `connect`), separate from the base `SandboxProvider` (which is unchanged). Capability membership is checked with `isinstance`, so new capabilities can be added as their own protocols instead of growing one interface. - **`AsyncSandbox.serialize(scope=...)` / `AsyncSandbox.connect(descriptor, *, provider=...)`** — facade methods that delegate to the capability and raise for providers that do not support it. There is one live type (`SandboxHandle`); a descriptor is just its serialized form. ## Test plan - [x] `tests/unit_tests/test_sandbox_connect.py`: a fake `ConnectableProvider` exercises the `serialize` -> `connect` round-trip; `serialize`/`connect` raise for a provider without the capability; `isinstance` membership check. - [x] Existing sandbox tests pass; ruff clean. --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
OlegSudakov
pushed a commit
to OlegSudakov/Gym
that referenced
this pull request
Aug 7, 2026
…ct facade (NVIDIA-NeMo#2083) ## Summary Part of NVIDIA-NeMo#2082 Bottom of a 3-PR stack for cross-server sandbox sharing. Introduces the optional provider capability and the sandbox-facade methods that use it. No in-tree provider implements the capability yet; the next PRs add OpenSandbox and the sandbox server. Stack: 1. **NVIDIA-NeMo#2083 (this PR)** — ConnectableProvider + serialize/connect facade 2. NVIDIA-NeMo#2084 — OpenSandbox implements ConnectableProvider 3. NVIDIA-NeMo#2085 — sandbox server ## What's here - **`ConnectableProvider`** — an optional `runtime_checkable` protocol (`serialize_handle` / `connect`), separate from the base `SandboxProvider` (which is unchanged). Capability membership is checked with `isinstance`, so new capabilities can be added as their own protocols instead of growing one interface. - **`AsyncSandbox.serialize(scope=...)` / `AsyncSandbox.connect(descriptor, *, provider=...)`** — facade methods that delegate to the capability and raise for providers that do not support it. There is one live type (`SandboxHandle`); a descriptor is just its serialized form. ## Test plan - [x] `tests/unit_tests/test_sandbox_connect.py`: a fake `ConnectableProvider` exercises the `serialize` -> `connect` round-trip; `serialize`/`connect` raise for a provider without the capability; `isinstance` membership check. - [x] Existing sandbox tests pass; ruff clean. --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
deepbuilder
pushed a commit
to deepbuilder/Gym
that referenced
this pull request
Aug 22, 2026
The first cut had the resources server create the build sandbox and hand the agent a sandbox_handle, copying swebench. That cannot work off OpenSandbox: only OpenSandboxProvider implements serialize()/connect(), so on Docker the rollout died with 'provider docker does not support serialize()/connect()'. That is by design, not a gap. NVIDIA-NeMo#2082 excludes node-local providers (Docker, Apptainer, enroot) from ConnectableProvider because the box has no network identity, and routes them through the sandbox server instead -- which is still unmerged in NVIDIA-NeMo#2085. The same issue classifies ViBench correctly: it is 'case 2', where the rollout copies an artifact out and the verifier grades it in a fresh box, so the sandbox never needs sharing. The first cut had accidentally made it case 3, the one shape that does need the sandbox server. So sandbox ownership moves to a new responses_api_agents/vibench_agent, which subclasses OpenCodeSandboxedAgent and overrides only sandbox acquisition and harvesting; installing and driving OpenCode is inherited. It stages the PRD via SandboxSpec.files, tars the built app into a shared artifact_dir, and passes the path to /verify. The resources server now creates no sandbox: seed_session returns PRD text, and verify unpacks the tarball. The tarball comes from a box the model controlled, so unpacking rejects members and links that escape the app dir, and artifact paths outside artifact_dir are refused. Works on Docker today with no dependency on NVIDIA-NeMo#2085. Signed-off-by: raghavendran ramakrishnan <raramakrishn@nvidia.com>
5 tasks
ananthsub
marked this pull request as draft
August 25, 2026 18:26
deepbuilder
pushed a commit
to deepbuilder/Gym
that referenced
this pull request
Aug 27, 2026
The first cut had the resources server create the build sandbox and hand the agent a sandbox_handle, copying swebench. That cannot work off OpenSandbox: only OpenSandboxProvider implements serialize()/connect(), so on Docker the rollout died with 'provider docker does not support serialize()/connect()'. That is by design, not a gap. NVIDIA-NeMo#2082 excludes node-local providers (Docker, Apptainer, enroot) from ConnectableProvider because the box has no network identity, and routes them through the sandbox server instead -- which is still unmerged in NVIDIA-NeMo#2085. The same issue classifies ViBench correctly: it is 'case 2', where the rollout copies an artifact out and the verifier grades it in a fresh box, so the sandbox never needs sharing. The first cut had accidentally made it case 3, the one shape that does need the sandbox server. So sandbox ownership moves to a new responses_api_agents/vibench_agent, which subclasses OpenCodeSandboxedAgent and overrides only sandbox acquisition and harvesting; installing and driving OpenCode is inherited. It stages the PRD via SandboxSpec.files, tars the built app into a shared artifact_dir, and passes the path to /verify. The resources server now creates no sandbox: seed_session returns PRD text, and verify unpacks the tarball. The tarball comes from a box the model controlled, so unpacking rejects members and links that escape the app dir, and artifact paths outside artifact_dir are refused. Works on Docker today with no dependency on NVIDIA-NeMo#2085. Signed-off-by: raghavendran ramakrishnan <raramakrishn@nvidia.com>
deepbuilder
pushed a commit
to deepbuilder/Gym
that referenced
this pull request
Aug 31, 2026
The first cut had the resources server create the build sandbox and hand the agent a sandbox_handle, copying swebench. That cannot work off OpenSandbox: only OpenSandboxProvider implements serialize()/connect(), so on Docker the rollout died with 'provider docker does not support serialize()/connect()'. That is by design, not a gap. NVIDIA-NeMo#2082 excludes node-local providers (Docker, Apptainer, enroot) from ConnectableProvider because the box has no network identity, and routes them through the sandbox server instead -- which is still unmerged in NVIDIA-NeMo#2085. The same issue classifies ViBench correctly: it is 'case 2', where the rollout copies an artifact out and the verifier grades it in a fresh box, so the sandbox never needs sharing. The first cut had accidentally made it case 3, the one shape that does need the sandbox server. So sandbox ownership moves to a new responses_api_agents/vibench_agent, which subclasses OpenCodeSandboxedAgent and overrides only sandbox acquisition and harvesting; installing and driving OpenCode is inherited. It stages the PRD via SandboxSpec.files, tars the built app into a shared artifact_dir, and passes the path to /verify. The resources server now creates no sandbox: seed_session returns PRD text, and verify unpacks the tarball. The tarball comes from a box the model controlled, so unpacking rejects members and links that escape the app dir, and artifact paths outside artifact_dir are refused. Works on Docker today with no dependency on NVIDIA-NeMo#2085. Signed-off-by: raghavendran ramakrishnan <raramakrishn@nvidia.com>
deepbuilder
pushed a commit
to deepbuilder/Gym
that referenced
this pull request
Aug 31, 2026
The first cut had the resources server create the build sandbox and hand the agent a sandbox_handle, copying swebench. That cannot work off OpenSandbox: only OpenSandboxProvider implements serialize()/connect(), so on Docker the rollout died with 'provider docker does not support serialize()/connect()'. That is by design, not a gap. NVIDIA-NeMo#2082 excludes node-local providers (Docker, Apptainer, enroot) from ConnectableProvider because the box has no network identity, and routes them through the sandbox server instead -- which is still unmerged in NVIDIA-NeMo#2085. The same issue classifies ViBench correctly: it is 'case 2', where the rollout copies an artifact out and the verifier grades it in a fresh box, so the sandbox never needs sharing. The first cut had accidentally made it case 3, the one shape that does need the sandbox server. So sandbox ownership moves to a new responses_api_agents/vibench_agent, which subclasses OpenCodeSandboxedAgent and overrides only sandbox acquisition and harvesting; installing and driving OpenCode is inherited. It stages the PRD via SandboxSpec.files, tars the built app into a shared artifact_dir, and passes the path to /verify. The resources server now creates no sandbox: seed_session returns PRD text, and verify unpacks the tarball. The tarball comes from a box the model controlled, so unpacking rejects members and links that escape the app dir, and artifact paths outside artifact_dir are refused. Works on Docker today with no dependency on NVIDIA-NeMo#2085. Signed-off-by: raghavendran ramakrishnan <raramakrishn@nvidia.com>
bxyu-nvidia
pushed a commit
that referenced
this pull request
Aug 31, 2026
Wraps [ViBench](https://github.com/ViBench/vibench-public) as a NeMo Gym environment: a model builds a web app from a PRD, and grading stands the app up for real, seeds it through its own UI, and drives it in a browser against a human-written test plan. The score reflects what a user can reach, not what the diff contains. Adds `resources_servers/vibench` and `responses_api_agents/vibench_agent`. ## Design The agent owns the build sandbox and copies the finished app out as a tarball; the resources server unpacks it and grades it in a fresh compose project, one per test plan, via ViBench's `run-seed.py` → `run-evaluate-post-seeding.py`. The sandbox is never shared. Reaching into the agent's box needs `serialize()`/`connect()`, which only the OpenSandbox provider implements, so the shape `swebench` uses cannot run on Docker, Apptainer or enroot. See [#2082](#2082) for the design discussion; this PR has no dependency on it or on #2085. `vibench_agent` subclasses `OpenCodeSandboxedAgent` and overrides only sandbox acquisition and harvesting. ## Reward One row is one `(app, artifact)` pair: built once, graded across that artifact's test plans. Reward is the mean normalized score, continuous rather than binary, with per-plan values in `reward_components`. `build_failed`, `seeding_failure_rate` and `plans_graded_rate` are reported separately, because a mean reward cannot distinguish a weak app from one that never built or could not be seeded. `REVERIFY_MODE` is `UNSUPPORTED` — grading depends on live app and database state. ## Validation Five example tasks graded end to end on a Docker host. Scores spread from 0.00 to 1.00 with partial credit in between, and every zero carried a real scorecard with per-step diagnoses (HTTP 500s, SQL errors) rather than a harness failure. The reward tracks capability: a weaker policy model scored 0.0 on a task with 0/19 steps, its app unreachable, where a stronger model scored 1.0 on the same task with the same graders. Scores also vary across repeats of the same model, since it does not build the same app twice. Quantifying that — and separating it from any variance in the LLM-driven verifier — is what reward profiling is for, so `verified: false` stands. `data/example_rollouts.jsonl` holds a five-task run. ## Reviewer notes - **`network: host` is deliberately not used.** The in-sandbox harness reaches the policy model at `get_server_url(...)` → `127.0.0.1:<port>`, which inside a bridged container is the container itself; the harness then makes zero LLM calls and exports an empty app with nothing logged. `configs/docker.yaml` instead adds `host.docker.internal` via `host-gateway` and the agent rewrites loopback model URLs to it. Only `policy_model` binds `0.0.0.0`; the servers holding grader credentials stay on loopback. Residual: the sandbox can reach host-published TCP ports, which inference requires. - **Grading timeouts interrupt rather than terminate.** ViBench tears its compose project down in a `finally`, which CPython does not unwind on SIGTERM. Escalation is SIGINT → SIGTERM → SIGKILL. Verified with a forced timeout on a real host: SIGTERM leaked two containers and a network, SIGINT cleaned up fully. - **`artifact_dir` is a filesystem path shared by agent and resources server.** Not a new constraint — grading already shells into a local Docker daemon. - **Requires `docker-compose` on `PATH`.** ViBench's scripts use the legacy name; Docker 29.x ships only the plugin. The failure misleads, so it is called out in the README. - **Task fields are top-level, not under `verifier_metadata`**, following `swebench` since `seed_session` and `verify` share one typed request model. Happy to move them. - **Grading runs on the resources server's Docker daemon**, not in a Gym sandbox, because ViBench's grading stack is multi-container. Single-image packaging is the prerequisite for multi-host. - **`mvp` artifacts only.** ViBench ships reference implementations for 21 of 24 apps, so `feature-ri` is a follow-up; it needs a starting codebase staged into the sandbox the way the PRD already is. - **Row size**: rollout rows carry the harness's own stdout/stderr, so `example_rollouts.jsonl` is ~500 KB for five tasks. Happy to trim. ## Security The tarball comes from a box the model controlled, so it is untrusted: unpacking refuses members and links resolving outside the app dir and passes `filter="data"`, and artifact paths outside `artifact_dir` are refused and never deleted. Grader credentials are scrubbed from captured output before it is stored, since that output ships in the rollout JSONL. Test plans and `test_assets/` are never staged into the build box. ## Testing `gym env test --resources-server vibench` passes. 75 unit tests (60 resources server, 15 agent), coverage 99%. `pytest tests/unit_tests/` 2621 pass. `pre-commit run --all-files` clean. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally. - [x] Pre-commit checks pass locally (`pre-commit run --all-files`). - [x] All commits have DCO sign-off (`git commit -s`). --------- Signed-off-by: raghavendran ramakrishnan <raramakrishn@nvidia.com> Co-authored-by: raghavendran ramakrishnan <raramakrishn@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #2082
Add a fourth Gym server type (sandbox_servers/) that owns physical sandboxes and lends them across servers by reference. A box created by one server (an agent running a rollout) can be operated by another (a resources server that scores it): the same live box, or a fresh eval box.
Hermetic tests use a fake in-process provider to drive a full in-process HTTP round-trip against the real server.
Signed-off-by: Ananth Subramaniam ansubramania@nvidia.com