feat(sandbox): add E2B sandbox provider

[max-sudolabs]

Summary

This continues the E2B sandbox provider work started by @hemildesai in #2165, moved to a fork so I can iterate on the implementation directly.

The current draft:

• adds the E2B SDK dependency with a supported >=2.36.0,<3.0.0 range;
• implements create, connect, exec, file transfer, status, and close through the public E2B SDK;
• fixes sandbox cleanup, reconnect timeout propagation, and close failure handling;
• adds nemo-gym/<version> integration attribution through ConnectionConfig.set_integration();
• keeps OCI image provisioning separate from sandbox creation with a deterministic template builder and CLI;
• keeps the E2B API key out of generated mini-swe worker configs, mirroring the existing opensandbox behavior, while preserving the SDK's own E2B_API_KEY env fallback;
• adds the provider README (nemo_gym/sandbox/providers/e2b/README.md) and a Fern docs page (fern/versions/latest/pages/infrastructure/sandbox/e2b.mdx);
• adds deterministic unit tests without creating live E2B resources.

Thanks @hemildesai for the initial implementation and for inviting me to continue the work.

Validation

• 152 E2B provider, builder, and mini-swe agent tests passed with 96.33% coverage.
• Current sandbox facade and connect tests passed against the latest main interfaces.
• Ruff check and format check passed for the changed Python files.
• uv lock --check passed with the repository-pinned uv version.
• Offline wheel and sdist builds passed and include the E2B provider configuration.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[hemildesai]

Hi @max-sudolabs thanks a lot for your PR. Can you run a SWE Bench Verified eval using https://github.com/NVIDIA-NeMo/Gym/tree/main/responses_api_agents/mini_swe_agent_2 and the E2B provider and post the results. You can pick any small OSS model like Qwen 3.6 27b and run it via openrouter to make things easier.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-max-sudolabs-e2b-sandbox-provider.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/infrastructure/sandbox/e2b
• nemo/gym/infrastructure/sandbox

[sephmard]

This is great @max-sudolabs! I'm on the Product Team for Nemo Gym. Its great to see the collaboration.

One request when this work merges: would you be willing to add a note on the E2B docs side as well? Since this adds official NeMo Gym support for E2B sandboxes, it would be great to have it reflected in E2B's own documentation (integration page, supported frameworks, or wherever E2B surfaces partner integrations).

[OndrejDrapalik]

Hi @max-sudolabs thanks a lot for your PR. Can you run a SWE Bench Verified eval using https://github.com/NVIDIA-NeMo/Gym/tree/main/responses_api_agents/mini_swe_agent_2 and the E2B provider and post the results. You can pick any small OSS model like Qwen 3.6 27b and run it via openrouter to make things easier.

Hi @hemildesai, I take care of it! Will reports results back.

[OndrejDrapalik]

This is great @max-sudolabs! I'm on the Product Team for Nemo Gym. Its great to see the collaboration.

One request when this work merges: would you be willing to add a note on the E2B docs side as well? Since this adds official NeMo Gym support for E2B sandboxes, it would be great to have it reflected in E2B's own documentation (integration page, supported frameworks, or wherever E2B surfaces partner integrations).

We're excited as well! Once PR merge we draft a docs page on our side! Will crosslink it to your docs as well.

[max-sudolabs]

@hemildesai Hey, I've been running tests lately and everything passes. I can see there is an error in CI - "Secrets detector". I checked it and it seems like False Positive. Can you take a look on that as well, please? And what are the next steps, so we can finally merge it?

[hemildesai]

@hemildesai Hey, I've been running tests lately and everything passes. I can see there is an error in CI - "Secrets detector". I checked it and it seems like False Positive. Can you take a look on that as well, please? And what are the next steps, so we can finally merge it?

@max-sudolabs I will take a look at the Secrets detector error. Apart from that things look good, and we can move forward once @OndrejDrapalik reports results on SWE Bench verified via mini swe agent 2.

[hemildesai]

Hi @max-sudolabs @OndrejDrapalik, just following up to see if you were able to run SWE-Bench verified via mini swe agent 2? We would love to have this merged. Let me know if I can help in any way.

[OndrejDrapalik]

Ran the SWE-bench Verified smoke set (the committed 5 rows in responses_api_agents/mini_swe_agent_2/data/example.jsonl) end-to-end on production E2B with this PR's code at 07a53cf, model qwen/qwen3.6-27b via OpenRouter as suggested. Same setup pattern as the enroot provider validation in #1952.

1. Build the 5 task templates with this PR's builder (provisioning step, emits the provider's template_map):

python -m nemo_gym.sandbox.providers.e2b.build \
    --image docker.io/swebench/sweb.eval.x86_64.django_1776_django-10973:latest \
    --image docker.io/swebench/sweb.eval.x86_64.pylint-dev_1776_pylint-4551:latest \
    --image docker.io/swebench/sweb.eval.x86_64.sphinx-doc_1776_sphinx-8595:latest \
    --image docker.io/swebench/sweb.eval.x86_64.sympy_1776_sympy-20916:latest \
    --image docker.io/swebench/sweb.eval.x86_64.scikit-learn_1776_scikit-learn-14141:latest \
    --cpu-count 2 --memory-mb 8192 --output template_map.yaml

2. Wrap the emitted mapping into a config overlay (e2b_overlay.yaml) so the provider can resolve the SWE-bench image refs:

sandbox:
  e2b:
    create:
      template_map:
        docker.io/swebench/sweb.eval.x86_64.django_1776_django-10973:latest: sweb-eval-x86_64-django_1776_django-10973-latest__ea28a12e4b22
        # ... (5 entries, straight from the builder output)

3. Start the servers (agent + this PR's e2b provider config + model server pointed at OpenRouter):

E2B_API_KEY=... gym env start \
    --config mini_swe_agent_2_e2b.yaml \
    --config nemo_gym/sandbox/providers/e2b/configs/e2b.yaml \
    --config e2b_overlay.yaml \
    --config responses_api_models/vllm_model/configs/vllm_model.yaml \
    '++policy_base_url=https://openrouter.ai/api/v1' \
    "++policy_api_key=$OPENROUTER_API_KEY" \
    '++policy_model_name=qwen/qwen3.6-27b'

(mini_swe_agent_2_e2b.yaml is the stock agent config minus sandbox_spec.provider_options.resource_requests — that block is OpenSandbox-specific and the e2b provider rejects unknown provider options by design; config merge can't delete keys, hence the copy. Might be worth a line in the provider README's swap instructions.)

4. Run the eval:

gym eval run --no-serve \
    --agent mini_swe_agent_2 \
    --input responses_api_agents/mini_swe_agent_2/data/example.jsonl \
    --output results/mini_swe_agent_2_e2b.jsonl \
    --limit 5 \
    --num-repeats 1 \
    --concurrency 5 \
    --temperature 0.6 \
    --top-p 0.95 \
    --max-output-tokens 16384

After running the above, the result is as follows:

Key metrics for mini_swe_agent_2:

{
  "pass@1/accuracy": 60.0,
  "pass@1[avg-of-1]/accuracy": 60.0,
  "mean/reward": 0.6,
  "resolved_task_count": 3,
  "task_count": 5,
  "resolved_task_rate": 60.0,
  "eval_error_rate": 0.0,
  "tests_status_rate": 100.0
}

Per instance:

instance_id	resolved	reward
django__django-10973	✅	1.0
sympy__sympy-20916	✅	1.0
scikit-learn__scikit-learn-14141	✅	1.0
pylint-dev__pylint-4551	❌	0.0
sphinx-doc__sphinx-8595	❌	0.0

Both misses are genuine model failures, not infra: patches applied cleanly in-sandbox, FAIL_TO_PASS tests judged them insufficient. Zero eval/infrastructure errors; all 5 sandboxes ran concurrently on E2B and were cleaned up after the run. E2B API key stayed out of the generated worker configs (env passthrough), as designed.

Same score as the enroot validation on these rows (3/5, 60% — #1952, same Qwen3.6-27B family). Happy to follow up with a full 500-instance run if a Daytona-comparable number (#1513) would help.

[OndrejDrapalik]

@hemildesai benchmark is done: SWE-bench Verified through mini_swe_agent_2 on this PR's E2B provider with qwen/qwen3.6-27b over OpenRouter as you suggested, giving 3/5 resolved, pass@1 60%, mean reward 0.6 and 0% eval error rate on the committed smoke rows, with repro commands and per-instance results in my earlier comment (same score enroot posted on these rows in #1952, and both misses are genuine model failures since the patches applied in-sandbox but the FAIL_TO_PASS tests judged them insufficient). One thing before merge: the branch is 193 commits behind and now conflicts with main, so I rebased it locally to check, where only uv.lock conflicts (resolve by regenerating rather than hand-merging), 152 unit tests pass, ruff check and format are clean, and secrets-detector goes green on its own because main dropped the KeywordDetector plugin from .secrets.baseline that produced the Secret Keyword false positive, so no pragma is needed and I removed my suggestion on that line. Happy to run the full 500 alongside the Daytona number in #1513 if that would help.

[claude]

SHIP — no reliability concerns found.

Reviewed the full diff (new optional e2b sandbox provider, OCI→template builder, registry wiring, and mini_swe_agent_2 secret handling). This is a clean, cohesive, and unusually well-tested addition.

What I checked hardest, and why it holds:

• Async / connection-pool safety. E2B's httpx control-plane transport is rerouted through Gym's global aiohttp client (_configure_async_http in _sdk.py), mirroring the existing opensandbox adapter pattern; ConnectRPC command streams keep their SDK pyqwest transport. SOCKS proxies are rejected explicitly. No httpx.AsyncClient in async paths, no ray.get(). _with_retries uses bounded exponential backoff; create is deliberately not retried (avoids leaking a second billable sandbox on ambiguous transport failures) and 429s are surfaced without retry.
• Dependency hygiene. e2b>=2.36.0,<3.0.0 added to the optional sandbox extra; SDK import is lazy via require_e2b_sdk with a clear ImportError when absent, and a ModuleNotFoundError guard that preserves the real missing transitive module. Tests gate on pytest.mark.sandbox + importorskip, so the core library stays importable without the SDK.
• Secret handling (mini_swe_agent_2/app.py). api_key/headers/api_headers are stripped from the serialized worker config, passed through the worker env, and reconstructed only in worker memory; the key is read directly from E2B_API_KEY by the SDK. Covered by two focused tests asserting the dump omits the secret and the round-trip restores headers.
• Config conventions. The frozen-dataclass configs with Python defaults match the established OpenSandbox*Config pattern (the no-Python-defaults rule applies to TypedDicts, not these). YAML exemplar is consistent with the dataclass defaults.
• Template resolution correctness. An unmapped, non-direct image fails loudly rather than silently falling back to an unrelated template — the right call, since a wrong template would silently corrupt a benchmark run. Verified by test.
• Test coverage is strong (~1.4k lines): template-resolution failure modes, background-exec stream loss/reconnect with deadline math, retry classification, resource-request warn/strict paths, and a real-SDK signature-binding test that pins the 2.x call shapes.

Minor, non-blocking observations (author's call, no action required):

• _run_background mutates the SDK's returned CommandResult.stdout/stderr in place on the happy path. This relies on CommandResult staying a mutable object across the pinned 2.x range; it is, and the <3.0.0 cap + tests bound the risk. Fragile coupling, but acceptable.
• Attribution and transport patching mutate module-level SDK globals process-wide (guarded once-per-module by a lock). Fine for the single-SDK-version assumption the pin enforces.

No blockers, no correctness risks to the verifier/scoring path (this touches sandbox infra only). Good to merge.

[hemildesai]

/claude review

[hemildesai]

/ok to test 6177a87

[claude]

SHIP — LGTM, no reliability concerns.

Reviewed the full diff (E2B sandbox provider + template-builder CLI + docs + deps + mini_swe_agent_2 secret wiring). This is self-contained sandbox infrastructure; it does not touch verify(), scoring, or reward aggregation, so the highest-risk category is not in play.

Verified against the trusted base and CLAUDE.md conventions:

• Async HTTP — _sdk._configure_async_http() routes E2B's httpx control-plane transport through Gym's global aiohttp client (get_global_aiohttp_client), overriding aclose() to protect the shared session and rejecting SOCKS proxies. Mirrors the existing opensandbox adapter. No httpx.AsyncClient in async paths, no ray.get(), all coroutines awaited.
• Dependency hygiene — e2b>=2.36.0,<3.0.0 declared in the sandbox extra and reflected in uv.lock (+transitive); SDK imported lazily via require_e2b_sdk with a clear ImportError when absent; provider registered behind a lazy loader in registry.py.
• Config — frozen dataclasses with defaults mirrored by the exemplar e2b.yaml, matching the established OpenSandbox*Config pattern (the no-Python-defaults rule targets TypedDicts, not these).
• Secret handling — api_key/headers/api_headers stripped from the on-disk worker config, passed via worker env, reconstructed only in worker memory; covered by dedicated tests.
• Operability — create deliberately not retried (idempotency), rate-limits surfaced without retry, deterministic errors non-retryable, background-exec reconnect budget bounded, unmapped image fails loudly instead of silently picking a fallback template.
• Tests — ~1400 lines with a faked SDK plus a real-SDK importorskip test pinning version bounds, transport patching, connection-param scoping, reconnect semantics, and template resolution.