Add NeMo-Gym integration: mini_swe_agent_2 via the agent function

[nblintao]

Summary

Adds examples/experimental/nemo-gym: an agentic training recipe that integrates NVIDIA's NeMo-Gym as an external environment ecosystem at the agent function layer — the same connector shape as Harbor and OpenEnv. Miles' session server records every chat-completions turn losslessly (token ids + logprobs + loss masks), and a thin agent function POSTs each sample to NeMo-Gym's sandbox-backed mini_swe_agent_2 agent with policy_base_url set to the per-episode session URL. The environment runs mini-swe-agent v2 in a per-task container, grades the episode with the official SWE-bench harness, and the grade flows back through a reward hook.

The per-request policy_base_url override is proposed upstream in NVIDIA-NeMo/Gym#2166; until it merges, the README points at that PR's branch (upstream main + two small commits).

Why

docs/user-guide/environments.md positions Miles as agnostic about where environments come from, with prebuilt connectors for external ecosystems. This adds NVIDIA's ecosystem to that table — driven through upstream NeMo-Gym: no submodules, no forks, and lossless token recording. It supersedes the fork-based generate-function integration removed in #1918 (which re-tokenized message text and depended on two personal-fork submodules).

What is in the example

File	Purpose
nemogym_agent_function.py	The connector: one POST /run per sample; sampling mapped onto responses_create_params; no miles imports so it loads on CPU-only machines.
nemogym_generate.py	Reward hook reading the environment's grade from sample.metadata.
run-qwen3-4b-instruct.sh	The validated 4-GPU launcher (sets MILES_EXPERIMENTAL_ROLLOUT_REFACTOR=1, which gates the dynamic registration of the agentic flags).
eval_nemogym_via_api.py	No-GPU validation driver: golden scan (gold patch, no model) and API-policy scan (an external API model drives real episodes through the same override the trainer uses).
download_and_process_data.py	Converts SWE-bench-family datasets to Miles JSONL (--subset gym|verified).
tests/	Offline unit tests of the /run contract (no network, no docker).
README.md	Server setup, data prep, launch, no-GPU validation, known limitations.

Docs: a user-guide/nemo-gym page plus rows in the environments tables (NeMo-Gym at the agent-function layer; its own sandbox provider API covers Docker/Daytona/Apptainer/Fargate/OpenSandbox).

Validation

All four layers were run for real on 2026-07-28; the commands in the README are the exact ones used.

• Offline contract tests: 7/7 pass (pytest examples/experimental/nemo-gym/tests/).
• Golden scan (no model): gold patch through the official swebench/sweb.eval.x86_64.* container scored reward 1.0.
• API-policy scan: DeepSeek drove a full episode through the policy_base_url override — patch applied, FAIL_TO_PASS 4/5, a legitimate reward 0.0.
• GPU training smoke (4x H200, Qwen3-4B-Instruct-2507, SWE-bench Verified): 3 synchronous GRPO steps completed twice; every episode ran mini-swe-agent v2 in a real task container, the SWE-bench harness executed the full test suite (e.g. 175/175 PASS_TO_PASS on an unresolved attempt), and grades flowed back into rollout/raw_reward.

Known limitations (documented in the README)

• The official swebench package lacks eval specs for several SWE-Gym repos (KeyError: 'getmoto/moto' at make_test_spec), so SWE-Gym episodes run but error at grading — train on SWE-bench-family instances until that upstream gap closes.
• A 4B policy solves none of these tasks, so smoke rewards were uniformly 0 (zero GRPO advantage) — a capability floor, not a pipeline defect.
• rollout/tito_session_mismatch_rate reads 1.0 with Qwen3-family templates (they insert an empty <think></think> skeleton when re-rendering assistant history, which the engine's output never contains). Soft diagnostic only: training tokens and loss masks come from the engine's recorded token ids. The CPU chat-template verifier passes 14/14 for this model.

Test plan

• Offline unit tests (7/7)
• Golden scan on a real docker host (reward 1.0)
• API-policy scan with DeepSeek (legitimate grade round-trip)
• 4x H200 GRPO training smoke, 3 steps x 2 runs, rewards flowing end-to-end
• ruff / black / bash -n on the example

🤖 Generated with Claude Code

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[Shi-Dong]

Nit: perhaps remove the leading underscore in _build_responses_create_params and _post_json since they are not private functions?

[nblintao]

Fixed.

[Shi-Dong]

I think probably we'll want to move this example out of experimental in the future, so let's switch to .py launch scripts instead of .sh.

[nblintao]

Fixed.