Add sandbox API and mini swe agent 2 resource agent

[hemildesai]

Summary

Parent PR: #1368

Refs #1337

This is the first smaller PR split out from #1368. It keeps the scope to the provider-neutral sandbox API, the OpenSandbox provider, and the Mini SWE Agent 2 evaluation integration. Observability is intentionally left out for a follow-up PR.

Features

• Adds the public nemo_gym.sandbox facade with async and sync sandbox clients, provider registration, image rewrite support, sandbox specs/handles, and batch create support.
• Adds the OpenSandbox provider with create/connect/exec/file/close operations, SDK pool-backed batch creation, retry handling, create probes, direct exec endpoint support, and nested provider configuration sections.
• Adds responses_api_agents/mini_swe_agent_2, a sandbox-backed mini-swe-agent v2 integration for SWE-bench style evals, including sandbox resource profiles, task metadata propagation, reward aggregation, and ng_collect_rollouts usage docs.
• Adds focused unit coverage for the sandbox facade, provider registry, OpenSandbox provider behavior, Mini SWE Agent 2 run/aggregation behavior, and sandbox environment adapter.
• Moves sandbox-related dependencies behind the nemo-gym[sandbox] optional extra.

Notes

• This PR does not include the sandbox observability module from Add sandbox API and mini SWE agent 2 OpenSandbox evaluation #1368.
• The Mini SWE Agent 2 README avoids internal deployment names and user-specific paths; examples use placeholders and local data/ / results/ paths.

Validation

Completed on the squashed commit:

uv run ruff check nemo_gym/sandbox responses_api_agents/mini_swe_agent_2 tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py

Result: All checks passed!

uv run pytest tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py responses_api_agents/mini_swe_agent_2/tests/test_app.py responses_api_agents/mini_swe_agent_2/tests/test_sandbox_environment.py -q

Result: 44 passed, 2 warnings

uv run coverage run --source=nemo_gym.sandbox,responses_api_agents.mini_swe_agent_2 -m pytest tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py responses_api_agents/mini_swe_agent_2/tests/test_app.py responses_api_agents/mini_swe_agent_2/tests/test_sandbox_environment.py -q
uv run coverage combine results
uv run coverage report --include='nemo_gym/sandbox/*,responses_api_agents/mini_swe_agent_2/*' --fail-under=90

Result: focused coverage 92%.

Kubernetes smoke validation:

• Model: Qwen/Qwen3.5-27B, served by SGLang with DFLASH draft model z-lab/Qwen3.5-27B-DFlash.
• Launched the Mini SWE Agent 2 stack through the documented ng_collect_rollouts path.
• Ran 8 SWE-bench Verified samples with 8 repeats and 64-way rollout concurrency against the OpenSandbox internal service path.
• Result: 64/64 rollout rows, pass@8=0.875, 7/8 tasks resolved, mean reward 0.765625, eval error rate 0.0, reward profile completion 100%.
• Cleanup completed with no leftover sandboxes for the successful internal-service run.

Full SWE-bench Verified validation:

• Model: Qwen/Qwen3.5-27B, served by SGLang with DFLASH draft model z-lab/Qwen3.5-27B-DFlash.
• Ran 500 SWE-bench Verified samples with pass@1, 500-way rollout concurrency, step_limit=250, and OpenSandbox cleanup metadata.
• Result: 500/500 rollout rows, pass@1=0.698, 349/500 tasks resolved, mean reward 0.698, eval error rate 0.6, tests status rate 99.0, reward profile completion 100%.
• Job duration: 4h6m; rollout collection duration: 4h04m.
• Cleanup left no sandboxes with the run labels run_family=mini-swe2-firstpr-q35-cell-full-p1-r9 or cleanup_id=full-p1-r9-single-20260521-053410.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ananthsub]

looks great! a few clarifying questions added inline

[cmunley1]

can we add sandbox api docs draft in a separate PR

cc @cwing-nvidia @lbliii

[cmunley1]

and if there is a simpler example, such as hello world (math + sandboxed python tool?), would be great

[Glorf]

What is snapshot_id?

[hemildesai]

Good point. snapshot_id is not a provider-neutral sandbox concept. It is an OpenSandbox-specific create option for starting from a pre-existing snapshot, so I moved it out of SandboxSpec and into provider_options["snapshot_id"]. Generic callers no longer see it as part of the public API. The relevant change is in a93f703.

[Glorf]

I'd appreciate to precise the contract here - will the sandbox handle be returned only when the sandbox is fully up & healthy, or do we expect it to just "create" the sandbox with some status param which can be eg. "STARTING". I mean sth like a pod definition. The later could be better for observability - if the sandbox fails to start what do we know about the failure type?

[hemildesai]

The intended contract is that create returns only when the sandbox is ready enough to run commands and transfer files, otherwise it raises a create error. I kept STARTING as a status value for providers that expose lifecycle status after start, but not as a successful create result. Startup failures should be surfaced as typed create or verification errors with provider detail rather than returning a half-ready handle. The contract text and status support are in a93f703.

[Glorf]

could we have some status marker? either as a property or a function, see https://github.com/NVIDIA-NeMo/Evaluator/blob/main/src/nemo_evaluator/sandbox/base.py#L178 for ref

[hemildesai]

Yes. I added a provider-neutral SandboxStatus enum and a minimal public status() method on both AsyncSandbox and Sandbox. I also removed the extra is_running convenience wrapper so the public surface stays small while still allowing callers to inspect lifecycle state. The status API was added in a93f703 and then minimized in 97b3210.

[Glorf]

what does it mean? should it be more of an internal primitive perhaps, preceeding other operations like exec or upload?

[hemildesai]

Agreed. Connect should not be part of the public sandbox API. It is only an OpenSandbox provider implementation detail for obtaining a loop-local SDK object before provider operations. I removed the public attach and connect methods entirely, so callers now start a sandbox and then use exec, upload, download, status, and stop on that object. The public API trim is in 97b3210.

[ananthsub]

(not blocking) but for users to plug in their own providers, we should also support an entry_points configuration

[cwing-nvidia]

given the time pressure for 0.4. id recommend we create a separate issue to track this targeted for 0.5

[ananthsub]

this config shape makes it hard to swap out for new providers. ideally we should provide an example of each of these provider configs as their own yamls and make it easy to swap in for different agent implementations.

e.g.

# nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml
sandbox_main:                       # the name the agent references
  opensandbox:                       # registry key -> provider class
    connection:
        domain: ${oc.env:OPENSANDBOX_DOMAIN}
        api_key: ${oc.env:OPENSANDBOX_API_KEY}
    create:     { timeout_s: 1200, skip_health_check: true, retries: 10 }
    probe:      { timeout_s: 60, deadline_s: 180, stable_count: 2 }
    operations: { retries: 5, command_retries: 3, close_timeout_s: 30 }
    options:    { platform: { os: linux, arch: amd64 } }   # run-level provider-specific defaults

then users could have other sandbox configs like

# nemo_gym/sandbox/providers/ecs/configs/ecs_fargate.yaml
sandbox_main:
  ecs_fargate:
    # define provider specific configs

the agent config can reference this in its config:

      env: sandbox
      sandbox_provider: sandbox_main  # inlined from the reference above, makes it easy to swap out for different providers
      sandbox_spec:
        timeout_s: 18000
        ready_timeout_s: 1200
        resources:
          cpu: "2"
          memory: 8Gi
          ephemeral-storage: 20Gi
        metadata:
          benchmark: swebench-verified
          harness: mini-swe-agent
          sandbox-api: opensandbox-sdk
      sandbox_environment_kwargs:
        cwd: /testbed
        conda_env: testbed
        activate_conda: true
        user: root
        delete: true
      run_golden: false
      step_timeout: 600
      eval_timeout: 1800
      skip_if_exists: false
      step_limit: 250

AGENT=responses_api_agents/mini_swe_agent_2/configs/mini_swe_agent_opensandbox.yaml
MODEL=responses_api_models/vllm_model/configs/vllm_model.yaml
# OpenSandbox
ng_run "+config_paths=[$AGENT, nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml, $MODEL]"
# ECS: swap one path
ng_run "+config_paths=[$AGENT, nemo_gym/sandbox/providers/runpod/configs/ecs_fargate.yaml, $MODEL]"

[hemildesai]

@ananthsub agreed on the config direction. I am going to keep this PR scoped to the sandbox API/provider lifecycle cleanup, and follow up in a separate PR with the swappable provider config YAMLs and agent config wiring so providers can be exchanged by swapping a config path. Leaving this thread open for that follow-up rather than treating it as addressed here.

[cwing-nvidia]

strong agree on this, I would like to decouple sandbox and agent config as much as possible

[ananthsub]

resolving in #1708

[ananthsub]

should the individual provider options configs be represented with frozen dataclasses here? otherwise it's hard for users to see the supported options + any validation logic for it

[Glorf]

This dumps the fully-resolved agent config to disk per instance, and self.config.sandbox_provider includes connection.api_key. Server configs are built with OmegaConf.to_container(..., resolve=True) (see server_utils.py), so ${oc.env:OPENSANDBOX_API_KEY} is already the real key by the time we get here — yaml.safe_dump writes the secret in cleartext to results/<subset>/<model>/_configs/<instance_id>.sandbox.yaml, once per task. This also bypasses the framework's own _recursively_hide_secrets masking. Could we mask/strip connection.api_key before dumping and re-inject it in the Ray worker from the env var (or pass it via Ray runtime_env.env_vars)? It's gitignored, but it's still cleartext on disk and could leak via artifact upload.

[Glorf]

exec() retries through the same policy as create/file ops, and _is_retryable_sdk_operation_error treats ConnectionError/OSError as retryable (the shipped config sets command_retries: 3). For arbitrary agent commands that isn't safe: if the connection drops after the command already ran server-side, the retry re-executes it and duplicates side effects (file writes, git apply, etc.). Suggest defaulting command_retries to 0 (so only create/connect/file ops retry) and/or documenting that command retries assume idempotency.

[Glorf]

cwd is hardcoded to /, and the configured working dir is only applied via the cd <cwd> that _command() injects — but _command() returns the bare command when activate_conda is off (lines 124–125). So with conda disabled, exec_cwd is silently dropped and every command runs in /. The default SWE-bench config uses conda so it works today, but it's a latent footgun. Suggest passing cwd=exec_cwd to exec() and letting the conda branch skip its own cd.

[Glorf]

If a provider call hangs longer than 5s, join(timeout=5) returns while the loop is still running and self._loop.close() then raises RuntimeError: Cannot close a running event loop from inside Sandbox.stop()'s finally. Also call()/run() block on future.result() with no timeout (lines 174/179), so a wedged loop thread hangs the caller indefinitely. Consider only calling loop.close() when the thread actually joined, and/or a bounded future.result(timeout=...).

[Glorf]

@hemildesai can we merge? WDYT?