Feat: add enroot backend

[oyilmaz-nvidia]

Add enroot sandbox backend

Adds enroot as a built-in sandbox provider, giving NeMo Gym rootless, single-node container isolation via the local enroot CLI. This is well-suited to NVIDIA/Slurm HPC nodes where enroot (paired with pyxis) is the supported container runtime.

What's included

• EnrootProvider (nemo_gym/sandbox/providers/enroot/provider.py) — runs each sandbox as a long-lived enroot container by shelling out to the enroot binary. Owns the full container lifecycle (create/start/exec/cleanup) from a single process, supports .sqsh images, enroot import URIs, and Docker image references (e.g. ubuntu:22.04, nvcr.io/nvidia/pytorch:24.01), and implements the provider-neutral sandbox API (exec, upload, download, env/file injection, sync + async).
• Registry wiring — registers the enroot provider name in nemo_gym/sandbox/providers/registry.py via a lazy loader.
• Config — default nemo_gym/sandbox/providers/enroot/configs/enroot.yaml.
• Docs — nemo_gym/sandbox/providers/enroot/README.md covering requirements, quick start, and constraints (requires enroot on PATH, unprivileged user namespaces; does not auto-install).
• Tests — tests/unit_tests/test_enroot_provider.py (~900 lines).

Quick Test

In Gym folder

In the Gym folder, run;

1. source .venv/bin/activate

Then, to generate sqsh files, run;

2. python scripts/enroot_prefetch_sqsh.py \
    --sqsh-dir /path/to/enroot_sqshs \
    'docker.io/swebench/sweb.eval.x86_64.django_1776_django-10973:latest' \
    'docker.io/swebench/sweb.eval.x86_64.sphinx-doc_1776_sphinx-8595:latest' \
    'docker.io/swebench/sweb.eval.x86_64.scikit-learn_1776_scikit-learn-14141:latest' \
    'docker.io/swebench/sweb.eval.x86_64.sympy_1776_sympy-20916:latest' \
    'docker.io/swebench/sweb.eval.x86_64.pylint-dev_1776_pylint-4551:latest'

3. HF_LOGIN=... gym env start \
    --config responses_api_agents/mini_swe_agent_2/configs/mini_swe_agent_2.yaml \
    --config nemo_gym/sandbox/providers/enroot/configs/enroot.yaml \
    --model-type local_vllm_model \
    --model Qwen/Qwen3.6-27B-FP8 \
    '++policy_model.responses_api_models.local_vllm_model.vllm_serve_kwargs.tensor_parallel_size=2' \
    '++policy_model.responses_api_models.local_vllm_model.vllm_serve_env_vars.VLLM_RAY_DP_PACK_STRATEGY=strict' \
    '++policy_model.responses_api_models.local_vllm_model.vllm_serve_kwargs.enable_auto_tool_choice=true' \
    '++policy_model.responses_api_models.local_vllm_model.vllm_serve_kwargs.tool_call_parser=qwen3_coder' \
    '++policy_model.responses_api_models.local_vllm_model.vllm_serve_kwargs.reasoning_parser=qwen3' \
    '++policy_model.responses_api_models.local_vllm_model.uses_reasoning_parser=true' \
    '++policy_model.responses_api_models.local_vllm_model.vllm_serve_kwargs.quantization=fp8' \
    '++sandbox.enroot.create.sqsh_cache_dir=/path/to/enroot_sqshs' \
    '++sandbox.enroot.create.bypass_entrypoint=false' &

4. gym eval run --no-serve \
    --agent mini_swe_agent_2 \
    --input responses_api_agents/mini_swe_agent_2/data/example.jsonl \
    --output results/mini_swe_agent_2_v2.jsonl \
    --limit 5 \
    --num-repeats 1 \
    --temperature 0.5 \
    --max-output-tokens 2048

After running the above, the result is as follows;

Computing aggregate metrics
INFO: 127.0.0.1:36258 - "GET /global_config_dict_yaml HTTP/1.1" 200 OK

Key metrics for mini_swe_agent_2:
{
"pass@1/accuracy": 60.0,
"pass@1[avg-of-1]/accuracy": 60.0,
"mean/reward": 0.6,
"resolved_task_count": 3,
"task_count": 5,
"resolved_task_rate": 60.0,
"eval_error_rate": 0.0,
"tests_status_rate": 100.0
}
Finished rollout collection! View results at:
Fully materialized inputs: results/mini_swe_agent_2_v2_materialized_inputs.jsonl
Rollouts: results/mini_swe_agent_2_v2.jsonl
Aggregate metrics: results/mini_swe_agent_2_v2_aggregate_metrics.json

Notes

• Requires the enroot binary already installed; constructing the provider raises RuntimeError if it's missing.
• All enroot calls run as the same OS user (enroot exec re-enters namespaces owned by the launching user).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

Did you run this with any environments?

[oyilmaz-nvidia]

@cmunley1 Yes, I ran it using the following commands.

gym env start \
    --config responses_api_agents/mini_swe_agent_2/configs/mini_swe_agent_2.yaml \
    --config nemo_gym/sandbox/providers/enroot/configs/enroot.yaml \
    --model-type local_vllm_model \
    --model Qwen/Qwen3-1.7B \
    '++policy_model.responses_api_models.local_vllm_model.vllm_serve_kwargs.tensor_parallel_size=2' \
    '++policy_model.responses_api_models.local_vllm_model.vllm_serve_env_vars.VLLM_RAY_DP_PACK_STRATEGY=strict' \
    '++policy_model.responses_api_models.local_vllm_model.vllm_serve_kwargs.enable_auto_tool_choice=true' \
    '++policy_model.responses_api_models.local_vllm_model.vllm_serve_kwargs.tool_call_parser=hermes'

And then,

gym eval run --no-serve \
    --agent mini_swe_agent_2 \
    --input responses_api_agents/mini_swe_agent_2/data/example.jsonl \
    --output results/mini_swe_agent_2_pass8.jsonl \
    --limit 5 \
    --num-repeats 8 \
    --concurrency 3 \
    --temperature 0.6 \
    --top-p 0.95 \
    --max-output-tokens 2048

If you have any environment suggestion that I should test with, please let me know.

[oyilmaz-nvidia]

/claude review

[hemildesai]

High level looks good, left some specific comments based on my review session with Codex

[oyilmaz-nvidia]

@hemildesai Addressed the review and added instruction to run the mini swe bench with 5 environments. Got 0.6 average reward.

[hemildesai]

/claude review

[oyilmaz-nvidia]

/claude review

[claude]

SHIP — LGTM, no reliability concerns.

Reviewed all 12 files against the base. This is a clean addition of an enroot sandbox provider plus a shared-helper refactor and two small defensive fixes. Nothing blocks merge.

What I verified:

• utils.py extraction is safe. coerce_config / path_under_mount moved out of apptainer + opensandbox verbatim; both now import from nemo_gym.sandbox.providers.utils. Grepped all providers — no dangling local definitions in the refactored files, and the bodies match the originals. (Docker keeps its own private _coerce_config; not touched, not broken.)

• Async correctness in enroot/provider.py. All subprocess work goes through _run, which bounds concurrency with asyncio.Semaphore(concurrency), enforces timeouts via asyncio.wait_for, kills the whole process group with os.killpg on timeout, and decodes with errors="replace". The detached enroot start is launched with start_new_session=True and deliberately not awaited (correct — enroot's start does not daemonize). No blocking calls in async paths; /proc scan is offloaded via run_in_executor. No httpx, no ray.get().

• Correctness guards. PID-reuse is defended by re-checking the container's unique cmdline marker before every exec (_pid_has_container_marker), so a stale/reused PID returns a sandbox error instead of joining a wrong namespace. _is_runtime_failure anchors [error] at line-start and uses a narrow marker set, avoiding false sandbox-error classification from user command output — this matters because a misclassified exit code corrupts the signal returned to the agent, and it's explicitly tested. Import caching uses a per-image lock + atomic temp-then-rename. Symlink writes/reads into the staging mount are refused.

• Failure teardown. create() wraps PID-wait + probe in try/except that calls _cleanup_failed_create_handle (kill start group → enroot remove -f → rmtree staging) on any failure. Tests cover early-exit, start-timeout, and probe-failure cleanup paths.

• Test coverage is thorough (~1080 lines across two files): subprocess boundary mocked, pure-helper units, argv construction, user mapping, timeout/runtime-failure classification, upload/download fast+fallback+symlink-refusal, status mapping, close paths. The facade test skips gracefully when enroot is absent or can't launch containers, per repo policy.

• mini_swe_agent_2/app.py fixes are both strictly defensive: not propagating tool_choice: "auto" prevents clobbering the base config's tool_choice: required in swebench.yaml; default_model_kwargs.get("top_p", None) avoids a KeyError when the base config omits top_p. (Note: temperature on the line above still uses ["temperature"] — fine, it's always present in swebench.yaml.)

Config follows convention (YAML is source of truth; dataclass field defaults mirror the exemplar YAML, and the README documents them). Imports are all stdlib or existing internal — no new dependencies.