Add NeMo-Gym integration: mini_swe_agent_2 via the agent function

docs/docs.json

@@ -180,7 +180,8 @@
       "root": "user-guide/environments",
       "pages": [
        "user-guide/harbor",
-       "user-guide/openenv"
+       "user-guide/openenv",
+       "user-guide/nemo-gym"
       ]
      }
     ]

docs/user-guide/environments.md

@@ -20,6 +20,7 @@ where the environment itself comes from:
 |---|---|
 | [Harbor](/user-guide/harbor) | agent function |
 | [OpenEnv](/user-guide/openenv) | agent function |
+| [NeMo-Gym](/user-guide/nemo-gym) | agent function |
 | [Strands Agents](https://github.com/radixark/miles/tree/main/examples/experimental/strands_sglang) | generate function |
 | [τ-bench](https://github.com/radixark/miles/tree/main/examples/experimental/tau-bench) | generate function |
 
@@ -28,7 +29,7 @@ Sandbox providers are a different axis: they provision the task containers
 
 | Sandbox provider | Used within |
 |---|---|
-| [Daytona](https://www.daytona.io/) | OpenEnv, Harbor |
+| [Daytona](https://www.daytona.io/) | OpenEnv, Harbor, NeMo-Gym |
 
 All external ecosystem support is experimental.
 

docs/user-guide/nemo-gym.md

@@ -0,0 +1,57 @@
+---
+title: NeMo-Gym
+description: Train on NVIDIA NeMo-Gym environments through the agent-function extension point.
+---
+
+[NeMo-Gym](https://github.com/NVIDIA-NeMo/Gym) is NVIDIA's RL environment
+ecosystem: environments are HTTP *resources servers* (code execution, search,
+SWE tasks, ...) paired with *agents* that drive an episode end-to-end and
+grade it. Task containers run through NeMo-Gym's own sandbox provider API
+(`nemo_gym.sandbox`) — Docker locally, or Daytona / Apptainer / ECS Fargate /
+OpenSandbox — selected by config, no agent changes.
+
+Miles integrates NeMo-Gym as an
+[agent-function integration](/user-guide/environments): per sample, the agent
+function POSTs the task to a NeMo-Gym agent server's `/run` endpoint with
+`policy_base_url` set to the session's OpenAI-compatible URL. NeMo-Gym runs
+its agent harness (mini-swe-agent v2 in `mini_swe_agent_2`) against that URL,
+so Miles' session server records every turn losslessly (token ids, logprobs,
+loss masks — see [Rollout Endpoints](/user-guide/rollout-endpoints)); NeMo-Gym
+grades the episode itself and the grade enters training through a custom
+reward hook reading `sample.metadata["reward"]`.
+
+The per-request `policy_base_url` override is proposed upstream in
+[NVIDIA-NeMo/Gym#2166](https://github.com/NVIDIA-NeMo/Gym/pull/2166); until it
+merges, run the NeMo-Gym server from that PR's branch (upstream main plus one
+small commit pair).
+
+## Try it
+
+The maintained recipe is **SWE-bench GRPO with mini-swe-agent** in
+[`examples/experimental/nemo-gym`](https://github.com/radixark/miles/tree/main/examples/experimental/nemo-gym).
+In short:
+
+1. **Environment side** — on any docker-capable host, clone NeMo-Gym (the
+   PR branch above until #2166 merges) and start the `mini_swe_agent_2`
+   responses-API agent server with the docker sandbox provider config.
+2. **Data** — convert SWE-bench Verified to Miles prompt data with
+   `download_and_process_data.py`; the task instance rides in each sample's
+   `metadata`.
+3. **Training side** — point `NEMO_GYM_URL` at the agent server and launch
+   `run.py` (requires `MILES_EXPERIMENTAL_ROLLOUT_REFACTOR=1`, which the
+   launcher sets), wiring the chain:
+
+```bash
+--custom-generate-function-path miles.rollout.generate_hub.agentic_tool_call.generate
+--custom-agent-function-path nemogym_agent_function.run
+--custom-rm-path nemogym_generate.reward_func
+--use-session-server
+```
+
+The recipe is validated end-to-end: golden and API-policy scans on a real
+docker host, plus a 4-GPU GRPO training smoke whose episodes ran in real task
+containers with the SWE-bench harness grading them. Follow the
+[recipe README](https://github.com/radixark/miles/blob/main/examples/experimental/nemo-gym/README.md)
+for the NeMo-Gym server setup, no-GPU validation (golden scan / API-policy
+scan), the launch walkthrough, and known limitations (SWE-Gym eval specs,
+Qwen3 template soft-mismatch diagnostics).

examples/experimental/nemo-gym/README.md

@@ -0,0 +1,249 @@
+# SWE-agent training via NeMo-Gym
+
+## Introduction
+
+This example trains a SWE agent with Miles using NVIDIA's
+[NeMo-Gym](https://github.com/NVIDIA-NeMo/Gym) as the environment ecosystem:
+NeMo-Gym's sandbox-backed `mini_swe_agent_2` agent runs the
+[mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent) v2 harness inside
+per-task SWE-bench containers (via the `nemo_gym.sandbox` provider API) and
+grades every episode with the official SWE-bench harness; Miles owns training,
+batch orchestration, and lossless token recording.
+
+NeMo-Gym plugs in at the **agent function** layer (see the
+[Environments guide](../../../docs/user-guide/environments.md)), the same
+shape as the Harbor and OpenEnv connectors:
+
+```
+Miles trainer ── session server (records every chat-completions turn: token
+   │             ids + logprobs + loss masks, no re-tokenization)
+   │  per-sample POST /run { task fields, policy_base_url = session URL }
+   ▼
+NeMo-Gym responses-API agent server (mini_swe_agent_2)
+   │  runs mini-swe-agent v2 against policy_base_url,
+   │  per-task container via a nemo_gym.sandbox provider (docker here;
+   │  daytona / apptainer / ecs_fargate / opensandbox also exist)
+   ▼
+reward (official SWE-bench harness) ──► sample.metadata ──► reward hook
+```
+
+- `nemogym_agent_function.py` — the connector: one `/run` POST per sample.
+- `nemogym_generate.py` — reward hook (reads the NeMo-Gym grade from
+  `sample.metadata["reward"]`).
+- `eval_nemogym_via_api.py`, `tests/` — no-GPU validation tooling (below).
+
+The per-request `policy_base_url` override this example relies on is proposed
+upstream in [NVIDIA-NeMo/Gym#2166](https://github.com/NVIDIA-NeMo/Gym/pull/2166).
+Until it merges, run the NeMo-Gym server from the PR branch
+(`nblintao/Gym@mini-swe-agent-per-request-policy-url`, upstream main + that
+one commit pair); afterwards, use upstream directly.
+
+## Validation status
+
+Validated end-to-end (2026-07-28) — the commands in this README are the exact
+ones used:
+
+- offline contract tests: 7/7 pass;
+- golden scan: gold patch through the official
+  `swebench/sweb.eval.x86_64.*` container scored **reward 1.0**;
+- API-policy scan: DeepSeek drove a full episode through the
+  `policy_base_url` override — patch applied, FAIL_TO_PASS 4/5, a legitimate
+  reward 0.0;
+- **GPU training smoke** (`run.py` defaults, 4x H200,
+  Qwen3-4B-Instruct-2507, SWE-bench Verified prompts): 3 synchronous GRPO
+  steps completed twice; every episode ran mini-swe-agent v2 in a real task
+  container on the NeMo-Gym host, the SWE-bench harness executed the task's
+  full test suite (e.g. 175/175 PASS_TO_PASS on an unresolved attempt), and
+  the grade flowed back into `rollout/raw_reward`.
+
+Two known limitations from the smoke run:
+
+- A 4B policy solves none of these tasks, so rewards were uniformly 0 —
+  GRPO then has zero advantage (`rollout/zero_std` fires). That's a model
+  capability floor, not a pipeline defect; expect the same until you use a
+  stronger policy or an easier task pool.
+- `rollout/tito_session_mismatch_rate` reads 1.0 with this model: Qwen3
+  chat templates insert an empty `<think></think>` skeleton when re-rendering
+  assistant history, which the engine's actual output never contains. It is a
+  soft diagnostic — training tokens and loss masks come from the engine's
+  recorded token ids, which stay lossless — and is a property of the
+  model-family template, not of this connector.
+
+## Setting up the NeMo-Gym server
+
+Any docker-capable host works: a CPU box next to the cluster, or a container
+beside the trainer (mount `/var/run/docker.sock` and share a docker network
+with the trainer — that variant is not validated here). Set `NEMO_GYM_URL` to
+wherever the server listens.
+
+```bash
+# Until NVIDIA-NeMo/Gym#2166 merges; afterwards clone NVIDIA-NeMo/Gym instead.
+git clone -b mini-swe-agent-per-request-policy-url https://github.com/nblintao/Gym.git
+cd Gym
+
+curl -LsSf https://astral.sh/uv/install.sh | sh
+source $HOME/.local/bin/env
+uv venv --python 3.12 && source .venv/bin/activate
+uv sync --extra dev
+# Do NOT install the agent's requirements.txt into this venv. `gym env start`
+# builds a per-server venv from it automatically; installing it here bumps
+# shared pins (e.g. openai) past nemo-gym's caps, and the injected versions
+# then make the child venv unresolvable.
+
+# Global config. The model-server entry must boot but receives no policy
+# traffic in this setup — every /run carries its own policy_base_url override
+# pointing at a Miles session URL. policy_model_name is the model name the
+# harness sends on each request (any string Miles' router accepts).
+echo "policy_base_url: http://localhost:9/v1
+policy_api_key: dummy
+policy_model_name: model
+default_host: 0.0.0.0" > env.yaml
+```
+
+Start the server, composing the agent config, the docker sandbox provider
+config, and a model server config:
+
+```bash
+gym env start \
+    --config responses_api_agents/mini_swe_agent_2/configs/mini_swe_agent_2.yaml \
+    --config nemo_gym/sandbox/providers/docker/configs/docker.yaml \
+    --model-type vllm_model \
+    '++mini_swe_agent_2.responses_api_agents.mini_swe_agent_2.port=12000' \
+    '++mini_swe_agent_2.responses_api_agents.mini_swe_agent_2.concurrency=16'
+```
+
+The first spin-up resolves each server's own venv, which takes a few minutes;
+the server is ready when the startup table lists `mini_swe_agent_2` on
+port 12000 and uvicorn reports it running.
+
+## Preparing data
+
+On the trainer, download the task instances and convert them to Miles' prompt
+data format. The validated smoke run uses **SWE-bench Verified**:
+
+```bash
+cd miles/examples/experimental/nemo-gym
+python download_and_process_data.py --input princeton-nlp/SWE-bench_Verified \
+    --split test --subset verified --output /root/swe_verified.jsonl
+```
+
+Each row keeps the full SWE-bench-format instance (`instance_id`, `repo`,
+`base_commit`, `problem_statement`, ...) in `metadata`, plus `subset` /
+`split` — the agent function forwards all of it in the `/run` body, and
+NeMo-Gym selects the per-task image from it.
+
+**SWE-Gym caveat**: `--input SWE-Gym/SWE-Gym --subset gym` produces the
+training dataset this recipe ultimately targets (per-task images from
+`docker.io/xingyaoww/...`), and episodes run fine — but the official
+`swebench` package `mini_swe_agent_2` scores with does not carry eval specs
+for several SWE-Gym repos (`KeyError: 'getmoto/moto'` at
+`make_test_spec`), so those episodes error at grading and score 0. Until
+SWE-Gym eval specs are available in that path (upstream gap), train on
+SWE-bench-family instances or filter SWE-Gym to repos the `swebench` package
+knows.
+
+## Wiring it into training
+
+The launcher is [`run.py`](run.py) (4 GPUs, smoke-scale defaults — scale up
+--num-rollout / batch sizes for real training). Its prepare step downloads the
+HF checkpoint and converts it to torch_dist on first run (`--skip-prepare` to
+skip):
+
+```bash
+export NEMO_GYM_URL="http://<nemo-gym-host>:12000"
+# Only if the NeMo-Gym host cannot resolve the trainer's hostname (e.g. it
+# reaches the trainer over a tailnet):
+export MILES_ROUTER_EXTERNAL_HOST="<trainer host/IP reachable from that host>"
+python examples/experimental/nemo-gym/run.py
+```
+
+To wire the connector into a different launch script, the essential pieces
+are this example's directory on `PYTHONPATH`,
+`MILES_EXPERIMENTAL_ROLLOUT_REFACTOR=1` in the environment (it gates the
+dynamic registration of the agentic flags below — without it train.py fails
+with "unrecognized arguments"), and:
+
+```bash
+--prompt-data /root/swe_verified.jsonl
+--input-key prompt
+--metadata-key metadata
+--max-seq-len 16384
+
+--custom-generate-function-path miles.rollout.generate_hub.agentic_tool_call.generate
+--custom-agent-function-path nemogym_agent_function.run
+--custom-rm-path nemogym_generate.reward_func
+--dynamic-sampling-filter-path miles.rollout.filter_hub.dynamic_sampling_filters.check_no_aborted
+--use-session-server
+--session-server-ip 0.0.0.0  # listen on all interfaces for the dial-back
+--tito-model qwen3           # match your policy model's TITO family
+```
+
+Per sample, `agentic_tool_call` opens a session on Miles' session server and
+hands its OpenAI-compatible URL to `nemogym_agent_function.run`, which POSTs
+the task to the NeMo-Gym server's `/run` with `policy_base_url` set to that
+session URL and the sampling settings mapped onto `responses_create_params`
+(`temperature`, `top_p`, `max_output_tokens`). NeMo-Gym's mini-swe-agent v2
+then talks to the policy exclusively through the session URL (litellm chat
+completions), so Miles records every turn losslessly — token ids, logprobs,
+and loss masks come from the session server, not from re-tokenizing message
+text. The episode grade rides back in the `/run` response (`reward`, with the
+SWE-bench eval report in `metadata` → `sample.metadata["eval_report"]`) and
+enters training through `sample.metadata["reward"]`.
+
+Episodes that fail before the first model call produce no session records; the
+sample is marked aborted and `check_no_aborted` drops its group from training.
+
+## Validating without a GPU
+
+Everything except the session server and the training loop can be validated
+on CPU-only machines, in three independent layers (all three pass as of
+2026-07-28, see [Validation status](#validation-status)):
+
+1. **Offline unit tests** — the `/run` request contract, response mapping,
+   failure semantics, and the data conversion. No network, no docker:
+
+   ```bash
+   pytest examples/experimental/nemo-gym/tests/ -q
+   ```
+
+2. **Golden scan** — the sandbox + per-task image + SWE-bench harness chain,
+   with no model involved at all: start the server with
+   `'++mini_swe_agent_2.responses_api_agents.mini_swe_agent_2.run_golden=true'`
+   appended to the `gym env start` command, then
+
+   ```bash
+   python eval_nemogym_via_api.py --input /root/swe_verified.jsonl --golden --limit 5
+   ```
+
+   Every gold patch must score reward 1.0; the script exits non-zero
+   otherwise.
+
+3. **API-policy scan** — a real model drives full episodes through the same
+   `policy_base_url` override the trainer uses (so this also exercises the
+   NVIDIA-NeMo/Gym#2166 field end-to-end). Start the server *without* the
+   golden override and with `policy_model_name` set to the API model name
+   (e.g. `deepseek-chat`) in `env.yaml`, then:
+
+   ```bash
+   export DEEPSEEK_API_KEY=...   # or OPENAI_API_KEY
+   python eval_nemogym_via_api.py --input /root/swe_verified.jsonl --limit 2 \
+       --policy-base-url https://api.deepseek.com/v1
+   ```
+
+## Troubleshooting
+
+1. `train.py: error: unrecognized arguments: --max-seq-len
+   --custom-agent-function-path`: `MILES_EXPERIMENTAL_ROLLOUT_REFACTOR=1` is
+   missing from the training environment (it must reach the ray job — run.py
+   sets it via the ray runtime env).
+2. `mini_swe_agent_2` dies at spin-up with an unresolvable-dependency error
+   (`openai==X` vs `nemo-gym depends on openai<=Y`): the main venv has extra
+   packages installed. Recreate it with `uv sync --extra dev` only — see the
+   setup note above.
+3. Slow episodes are usually docker pulls (each task has its own image,
+   fetched on first use) or the in-container SWE-bench evaluation
+   (server-side `eval_timeout`, default 1800s). `NEMO_GYM_RUN_TIMEOUT`
+   (default 3600s) caps one episode end-to-end on the Miles side.
+4. A failed episode surfaces as `sample.metadata["eval_report"]["error"]` with
+   a traceback from the NeMo-Gym server — check there before digging into
+   server logs.