Add sandbox API and mini SWE agent 2 OpenSandbox evaluation by hemildesai · Pull Request #1368 · NVIDIA-NeMo/Gym

hemildesai · 2026-05-19T15:43:50Z

Summary

This PR adds the sandbox-backed SWE evaluation path centered on OpenSandbox and mini SWE agent 2.

Features

Adds the provider-neutral nemo_gym.sandbox API with sync/async facades, sandbox handles/specs, provider registration, image rewrite support, and operation observability.
Adds an OpenSandbox provider with create/connect/exec/file/close support, SDK pool batch creation, retry handling, create probes, explicit sandbox.opensandbox.* event names, and grouped provider config sections: connection, create, probe, operations, and pool.
Adds sandbox observability recorders/exporters for local artifacts and OpenTelemetry traces, with config-driven setup for mini SWE agent 2.
Adds responses_api_agents/mini_swe_agent_2, a sandbox-backed mini-swe-agent v2 integration for SWE-bench style eval, including per-task sandbox config, resource profiles, rollout metadata, and ng_collect_rollouts examples.
Updates mini SWE agent 2 README into a unified eval-focused guide with server launch, rollout collection, profiling, observability, and sandbox environment adapter docs.
Moves sandbox dependencies into a separate nemo-gym[sandbox] optional extra and removes the provider-local OpenSandbox requirements file.
Removes the old sandbox diagnostics helper path and references.

User Impact

Users can run mini SWE agent 2 evaluations through the public Gym sandbox facade instead of embedding provider-specific runtime logic in the agent. OpenSandbox settings are now grouped by concern, which keeps the mini SWE config easier to scan and avoids misleading batch_create_* create retry names.

Validation

Completed locally:

uv run ruff check nemo_gym/sandbox/providers/opensandbox/provider.py nemo_gym/sandbox/providers/opensandbox/__init__.py tests/unit_tests/test_opensandbox_provider.py tests/unit_tests/test_sandbox.py
uv run pytest tests/unit_tests/test_opensandbox_provider.py tests/unit_tests/test_sandbox.py responses_api_agents/mini_swe_agent_2/tests/test_app.py responses_api_agents/mini_swe_agent_2/tests/test_sandbox_environment.py -q
- Result: 57 passed, 2 warnings
uv run coverage run --source=nemo_gym.sandbox,responses_api_agents.mini_swe_agent_2 -m pytest tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py responses_api_agents/mini_swe_agent_2/tests/test_app.py responses_api_agents/mini_swe_agent_2/tests/test_sandbox_environment.py -q
uv run coverage report --include='nemo_gym/sandbox/*,responses_api_agents/mini_swe_agent_2/*' --fail-under=90
- Result: 91% focused source coverage
git diff --check
Parsed responses_api_agents/mini_swe_agent_2/configs/mini_swe_agent_opensandbox.yaml and instantiated OpenSandboxProvider(**kwargs) from its nested provider config.

Validation plan before marking ready for review:

Run the focused unit test and coverage suite again in CI or a clean dev image.
Launch the mini SWE agent 2 stack with the OpenSandbox config and observability enabled:

ng_run \
  "+config_paths=[responses_api_agents/mini_swe_agent_2/configs/mini_swe_agent_opensandbox.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]" \
  +mini_swe_agent_2.responses_api_agents.mini_swe_agent_2.observability.enabled=true \
  +mini_swe_agent_2.responses_api_agents.mini_swe_agent_2.observability.output_dir=results/mini_swe_agent_2_observability/{trajectory_id}

Smoke-test rollout collection on a small SWE-bench Verified shard with traces enabled:

ng_collect_rollouts \
  +agent_name=mini_swe_agent_2 \
  +input_jsonl_fpath=/mnt/rl-workspace/hemild/gym_eval/refactor/inputs/mini_swe_verified_smoke8.jsonl \
  +output_jsonl_fpath=results/mini_swe_agent_2/swebench_verified_smoke8.rollouts.jsonl \
  +num_repeats=8 \
  +num_samples_in_parallel=64 \
  '+responses_create_params={max_output_tokens: 32768, temperature: 0.6, top_p: 0.95, metadata: {chat_template_kwargs: "{\"enable_thinking\": true}"}}'

Profile the smoke shard and inspect observability artifacts:

ng_reward_profile \
  +input_jsonl_fpath=/mnt/rl-workspace/hemild/gym_eval/refactor/inputs/mini_swe_verified_smoke8.jsonl \
  +rollouts_jsonl_fpath=results/mini_swe_agent_2/swebench_verified_smoke8.rollouts.jsonl \
  +output_jsonl_fpath=results/mini_swe_agent_2/swebench_verified_smoke8.profiled.jsonl \
  +pass_threshold=1.0

Final merge-gate validation: run the full SWE-bench Verified workflow with observability enabled, collect rollouts for the full Verified input JSONL, profile rewards, and confirm every trajectory has the expected sandbox/OpenSandbox spans and trace artifacts under results/mini_swe_agent_2_observability/{trajectory_id}.

Signed-off-by: Hemil Desai <hemild@nvidia.com>

copy-pr-bot · 2026-05-19T15:43:55Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Hemil Desai <hemild@nvidia.com>

cmunley1 · 2026-05-20T04:39:15Z

+        yield
+        return
+    async with recorder.span(name, phase=phase, attributes=attributes):
+        yield


do we need all this otel stuff immediately?

Agreed, breaking this down to a smaller PR without the observability module.

cmunley1 · 2026-05-20T04:39:39Z

+
+A NeMo Gym Responses API agent that integrates
+[mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent) v2 for evaluating
+language models on SWE-bench style software engineering tasks through the public


can we have a simple hello world like math with python tool?

cmunley1 · 2026-05-20T04:40:30Z

+    return AsyncRetrying, retry_if_exception, stop_after_attempt, wait_random_exponential
+
+
+def _httpx_retryable_types() -> tuple[type[BaseException], ...]:


could we use aiohttp instead or this is a hard opensandbox requirement?

cmunley1 · 2026-05-20T04:42:03Z

+
+This module is the boundary Gym code should use when it needs a sandbox.
+Provider packages implement the lower-level async protocol; callers use
+``AsyncSandbox`` in async code and ``Sandbox`` in synchronous integrations.


maybe default should be async?

cmunley1

could you check the steps here, such as example.jsonl and example_rollouts.jsonl

https://docs.nvidia.com/nemo/gym/main/contribute/environments/new-environment/#contribution-workflow

cmunley1 · 2026-05-20T04:49:14Z

    "results/*",
    "/tmp/*",
    "benchmarks/*",
+    "nemo_gym/sandbox/observability/recorder.py",


why exclude?

cmunley1 · 2026-05-20T04:50:47Z

+
+
+class _ObservedModel:
+    """Add an OTel span around each mini-SWE model query."""


will each sandbox-using env have to do this?

No, this is only to capture llm calls in otel telemetry and is optional

cmunley1 · 2026-05-20T04:52:57Z

+            )
+
+        print(f"[EVAL]{instance_id} Running eval", flush=True)
+        eval_report = _run_eval_v2(


could this verify step become a resources server so that eval logic can be reused across agent harness? maybe this is out of scope for sandbox poc though.

For SWE I think it needs to run on a particular image and each task will likely have a different image IIUC?

hemildesai · 2026-05-20T20:16:38Z

We are going to break this down into smaller PRs for easier review. The first split keeps the scope to the sandbox API, OpenSandbox provider, and Mini SWE Agent 2 evaluation integration, without the observability module.

## Summary Parent PR: #1368 Refs #1337 This is the first smaller PR split out from #1368. It keeps the scope to the provider-neutral sandbox API, the OpenSandbox provider, and the Mini SWE Agent 2 evaluation integration. Observability is intentionally left out for a follow-up PR. ### Features - Adds the public `nemo_gym.sandbox` facade with async and sync sandbox clients, provider registration, image rewrite support, sandbox specs/handles, and batch create support. - Adds the OpenSandbox provider with create/connect/exec/file/close operations, SDK pool-backed batch creation, retry handling, create probes, direct exec endpoint support, and nested provider configuration sections. - Adds `responses_api_agents/mini_swe_agent_2`, a sandbox-backed mini-swe-agent v2 integration for SWE-bench style evals, including sandbox resource profiles, task metadata propagation, reward aggregation, and `ng_collect_rollouts` usage docs. - Adds focused unit coverage for the sandbox facade, provider registry, OpenSandbox provider behavior, Mini SWE Agent 2 run/aggregation behavior, and sandbox environment adapter. - Moves sandbox-related dependencies behind the `nemo-gym[sandbox]` optional extra. ### Notes - This PR does not include the sandbox observability module from #1368. - The Mini SWE Agent 2 README avoids internal deployment names and user-specific paths; examples use placeholders and local `data/` / `results/` paths. ## Validation Completed on the squashed commit: ```bash uv run ruff check nemo_gym/sandbox responses_api_agents/mini_swe_agent_2 tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py ``` Result: `All checks passed!` ```bash uv run pytest tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py responses_api_agents/mini_swe_agent_2/tests/test_app.py responses_api_agents/mini_swe_agent_2/tests/test_sandbox_environment.py -q ``` Result: `44 passed, 2 warnings` ```bash uv run coverage run --source=nemo_gym.sandbox,responses_api_agents.mini_swe_agent_2 -m pytest tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py responses_api_agents/mini_swe_agent_2/tests/test_app.py responses_api_agents/mini_swe_agent_2/tests/test_sandbox_environment.py -q uv run coverage combine results uv run coverage report --include='nemo_gym/sandbox/*,responses_api_agents/mini_swe_agent_2/*' --fail-under=90 ``` Result: focused coverage `92%`. Kubernetes smoke validation: - Model: `Qwen/Qwen3.5-27B`, served by SGLang with DFLASH draft model `z-lab/Qwen3.5-27B-DFlash`. - Launched the Mini SWE Agent 2 stack through the documented `ng_collect_rollouts` path. - Ran 8 SWE-bench Verified samples with 8 repeats and 64-way rollout concurrency against the OpenSandbox internal service path. - Result: 64/64 rollout rows, `pass@8=0.875`, 7/8 tasks resolved, mean reward `0.765625`, eval error rate `0.0`, reward profile completion `100%`. - Cleanup completed with no leftover sandboxes for the successful internal-service run. Full SWE-bench Verified validation: - Model: `Qwen/Qwen3.5-27B`, served by SGLang with DFLASH draft model `z-lab/Qwen3.5-27B-DFlash`. - Ran 500 SWE-bench Verified samples with pass@1, 500-way rollout concurrency, `step_limit=250`, and OpenSandbox cleanup metadata. - Result: 500/500 rollout rows, `pass@1=0.698`, 349/500 tasks resolved, mean reward `0.698`, eval error rate `0.6`, tests status rate `99.0`, reward profile completion `100%`. - Job duration: `4h6m`; rollout collection duration: `4h04m`. - Cleanup left no sandboxes with the run labels `run_family=mini-swe2-firstpr-q35-cell-full-p1-r9` or `cleanup_id=full-p1-r9-single-20260521-053410`. --------- Signed-off-by: Hemil Desai <hemild@nvidia.com>

## Summary Parent PR: #1368 Refs #1337 This is the first smaller PR split out from #1368. It keeps the scope to the provider-neutral sandbox API, the OpenSandbox provider, and the Mini SWE Agent 2 evaluation integration. Observability is intentionally left out for a follow-up PR. ### Features - Adds the public `nemo_gym.sandbox` facade with async and sync sandbox clients, provider registration, image rewrite support, sandbox specs/handles, and batch create support. - Adds the OpenSandbox provider with create/connect/exec/file/close operations, SDK pool-backed batch creation, retry handling, create probes, direct exec endpoint support, and nested provider configuration sections. - Adds `responses_api_agents/mini_swe_agent_2`, a sandbox-backed mini-swe-agent v2 integration for SWE-bench style evals, including sandbox resource profiles, task metadata propagation, reward aggregation, and `ng_collect_rollouts` usage docs. - Adds focused unit coverage for the sandbox facade, provider registry, OpenSandbox provider behavior, Mini SWE Agent 2 run/aggregation behavior, and sandbox environment adapter. - Moves sandbox-related dependencies behind the `nemo-gym[sandbox]` optional extra. ### Notes - This PR does not include the sandbox observability module from #1368. - The Mini SWE Agent 2 README avoids internal deployment names and user-specific paths; examples use placeholders and local `data/` / `results/` paths. ## Validation Completed on the squashed commit: ```bash uv run ruff check nemo_gym/sandbox responses_api_agents/mini_swe_agent_2 tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py ``` Result: `All checks passed!` ```bash uv run pytest tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py responses_api_agents/mini_swe_agent_2/tests/test_app.py responses_api_agents/mini_swe_agent_2/tests/test_sandbox_environment.py -q ``` Result: `44 passed, 2 warnings` ```bash uv run coverage run --source=nemo_gym.sandbox,responses_api_agents.mini_swe_agent_2 -m pytest tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py responses_api_agents/mini_swe_agent_2/tests/test_app.py responses_api_agents/mini_swe_agent_2/tests/test_sandbox_environment.py -q uv run coverage combine results uv run coverage report --include='nemo_gym/sandbox/*,responses_api_agents/mini_swe_agent_2/*' --fail-under=90 ``` Result: focused coverage `92%`. Kubernetes smoke validation: - Model: `Qwen/Qwen3.5-27B`, served by SGLang with DFLASH draft model `z-lab/Qwen3.5-27B-DFlash`. - Launched the Mini SWE Agent 2 stack through the documented `ng_collect_rollouts` path. - Ran 8 SWE-bench Verified samples with 8 repeats and 64-way rollout concurrency against the OpenSandbox internal service path. - Result: 64/64 rollout rows, `pass@8=0.875`, 7/8 tasks resolved, mean reward `0.765625`, eval error rate `0.0`, reward profile completion `100%`. - Cleanup completed with no leftover sandboxes for the successful internal-service run. Full SWE-bench Verified validation: - Model: `Qwen/Qwen3.5-27B`, served by SGLang with DFLASH draft model `z-lab/Qwen3.5-27B-DFlash`. - Ran 500 SWE-bench Verified samples with pass@1, 500-way rollout concurrency, `step_limit=250`, and OpenSandbox cleanup metadata. - Result: 500/500 rollout rows, `pass@1=0.698`, 349/500 tasks resolved, mean reward `0.698`, eval error rate `0.6`, tests status rate `99.0`, reward profile completion `100%`. - Job duration: `4h6m`; rollout collection duration: `4h04m`. - Cleanup left no sandboxes with the run labels `run_family=mini-swe2-firstpr-q35-cell-full-p1-r9` or `cleanup_id=full-p1-r9-single-20260521-053410`. --------- Signed-off-by: Hemil Desai <hemild@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>

hemildesai added 22 commits May 18, 2026 11:41

feat(sandbox): add public sandbox API for mini swe

f132f35

Signed-off-by: Hemil Desai <hemild@nvidia.com>

feat(sandbox): refine otel observability

40bab24

Signed-off-by: Hemil Desai <hemild@nvidia.com>

fix(sandbox): configure command span titles

615e45f

Signed-off-by: Hemil Desai <hemild@nvidia.com>

fix(mini-swe): remove routing wrappers from sandbox path

e56ea4c

Signed-off-by: Hemil Desai <hemild@nvidia.com>

fix(mini-swe): restore llm observability spans

5342f8b

Signed-off-by: Hemil Desai <hemild@nvidia.com>

chore(sandbox): drop unrelated core config diffs

84fbffb

Signed-off-by: Hemil Desai <hemild@nvidia.com>

refactor(mini-swe): split v2 sandbox agent

491403d

Signed-off-by: Hemil Desai <hemild@nvidia.com>

fix(mini-swe-2): rely on v2 environment class loading

5b68ca2

Signed-off-by: Hemil Desai <hemild@nvidia.com>

chore(mini-swe-2): trim to verified sandbox path

08ed54f

Signed-off-by: Hemil Desai <hemild@nvidia.com>

refactor(mini-swe): use response toolcall model

11225fc

Signed-off-by: Hemil Desai <hemild@nvidia.com>

refactor(mini-swe): remove sandbox ready barrier

c1d9791

Signed-off-by: Hemil Desai <hemild@nvidia.com>

refactor(mini-swe): use raw response output

423e2e2

Signed-off-by: Hemil Desai <hemild@nvidia.com>

fix(mini-swe): normalize response api input

cf12f05

Signed-off-by: Hemil Desai <hemild@nvidia.com>

fix(mini-swe): use direct responses endpoint

c695092

Signed-off-by: Hemil Desai <hemild@nvidia.com>

fix(mini-swe): use chat completions tool calls

b177297

Signed-off-by: Hemil Desai <hemild@nvidia.com>

Remove sandbox diagnostics helpers

265edbd

Signed-off-by: Hemil Desai <hemild@nvidia.com>

Unify mini SWE agent 2 README

617bfb9

Signed-off-by: Hemil Desai <hemild@nvidia.com>

Update mini SWE agent 2 eval docs and observability

57906dd

Signed-off-by: Hemil Desai <hemild@nvidia.com>

Add sandbox optional dependency extra

e9ed815

Signed-off-by: Hemil Desai <hemild@nvidia.com>

Improve sandbox coverage and mini SWE config

d026e97

Signed-off-by: Hemil Desai <hemild@nvidia.com>

Namespace OpenSandbox observability events

8353e33

Signed-off-by: Hemil Desai <hemild@nvidia.com>

Refactor OpenSandbox provider config

c8b1bb5

Signed-off-by: Hemil Desai <hemild@nvidia.com>

hemildesai added 2 commits May 19, 2026 08:52

Fix sandbox CI checks

ff178ee

Signed-off-by: Hemil Desai <hemild@nvidia.com>

Fix optional sandbox coverage in core CI

d335a2a

Signed-off-by: Hemil Desai <hemild@nvidia.com>

cmunley1 reviewed May 20, 2026

View reviewed changes

hemildesai mentioned this pull request May 20, 2026

Add sandbox API and mini swe agent 2 resource agent #1377

Merged

hemildesai closed this May 20, 2026

sephmard mentioned this pull request Jun 1, 2026

Reconcile NeMo Gym base environment with Gymnasium API as part of Evaluator → Gym consolidation #1487

Open

4 tasks

anwithk mentioned this pull request Jun 23, 2026

NeMo RL Software Architecture Update: PR Tracking NVIDIA-NeMo/RL#2905

Open

29 tasks

Navigation Menu

Uh oh!

Add sandbox API and mini SWE agent 2 OpenSandbox evaluation - #1368

Add sandbox API and mini SWE agent 2 OpenSandbox evaluation#1368
hemildesai wants to merge 24 commits into
NVIDIA-NeMo:mainfrom
hemildesai:hemil/sandbox-api

hemildesai commented May 19, 2026

Uh oh!

copy-pr-bot Bot commented May 19, 2026

Uh oh!

cmunley1 May 20, 2026

Uh oh!

hemildesai May 20, 2026

Uh oh!

cmunley1 May 20, 2026

Uh oh!

cmunley1 May 20, 2026

Uh oh!

cmunley1 May 20, 2026

Uh oh!

cmunley1 left a comment

Uh oh!

cmunley1 May 20, 2026

Uh oh!

cmunley1 May 20, 2026

Uh oh!

hemildesai May 20, 2026

Uh oh!

cmunley1 May 20, 2026

Uh oh!

hemildesai May 20, 2026

Uh oh!

hemildesai commented May 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		return AsyncRetrying, retry_if_exception, stop_after_attempt, wait_random_exponential


		def _httpx_retryable_types() -> tuple[type[BaseException], ...]:



		class _ObservedModel:
		"""Add an OTel span around each mini-SWE model query."""

Uh oh!

Conversation

hemildesai commented May 19, 2026

Summary

Features

User Impact

Validation

Uh oh!

copy-pr-bot Bot commented May 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmunley1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hemildesai commented May 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants