Merge main into gdpval-devel - #2923
Open
agronskiy wants to merge 197 commits into
Open
Conversation
## Summary - Bumps \`mlflow\`, \`mlflow-skinny\`, and \`mlflow-tracing\` from \`3.14.0\` to \`3.15.1\` - Pins \`stirrup==0.1.12\` to block the coincidentally-released 0.2.0 which broke \`ChatCompletionsClient\` (unrelated to the mlflow bump, but fixed here to unblock CI) - Updates \`uv.lock\` accordingly ## Test plan - [ ] CI green --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com>
…r wiring. (#2297) ## Summary - Add a Configure Models page for the default `POST /v1/messages` dialect on every Gym model server, and how to wire `claude_code_agent` through `model_server` (or call Anthropic / another Messages host directly). - Link the page from the model-server index, Integrate Existing Agents, release notes, and the Claude Code agent README. Closes #1476 ## Test plan - [ ] `cd fern && npm run check` - [ ] Spot-check `/main/model-server/anthropic-messages` in Fern preview - [ ] Confirm cards/links from Configure Models and Integrate Existing Agents resolve Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
Follow-up to #2163, the unpaired call guard now makes one pass over the output items instead of two comprehensions. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: adil-a <adil.asif2000@hotmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
The kilocode agent called its model provider directly, so its requests
and responses never reached Gym. Model calls went uncaptured, the
provider block had to be rewritten per backend, and there was no path to
token IDs or logprobs.
This adds `model_server` to `KiloCodeAgentConfig`. When set, the agent
writes a `nemo` provider into `kilo.json` pointed at that server and
passes `-m nemo/<model>`, so one config runs against vLLM, OpenAI, or an
inference provider by swapping `--model-type`. Setting `model_server:
null` keeps the previous behaviour of calling a provider declared in
`kilo_config`.
## What changed
- `model_server` wiring: `_write_kilo_config` becomes build-then-write.
It used to return early when `kilo_config` was empty, which would have
left a model-server-only run with no provider at all. `_build_command`
and `_env` now prefer the resolved URL, so a config carrying both a
model server and `openai_base_url` does not point the subprocess
environment at the provider the `nemo` provider is meant to replace.
- Per-rollout capture: During `/run` the base URL carries the
`/ng-rollout/<id>` prefix, using the base class's `url_path_for_run` and
`resolve_model_base_url` rather than a local reimplementation. Captured
model calls are attributable to the rollout that made them.
## On the model limits
`context_window` and `max_output_tokens` set kilo's `limit`. The
defaults are sized for a 32k-window model server rather than copied from
`opencode_agent`.
Kilo's system prompt and tool definitions run to ~10k tokens, so a large
output budget pushes `prompt + max_tokens` past `max_model_len`. vLLM
rejects that with a 400, which the Gym model server converts into an
empty completion with `finish_reason: length` rather than an error
(`vllm_model/app.py`, `is_out_of_context_length`). The run then produces
no assistant message and scores zero, with nothing in the CLI's output
to say why.
Two related notes:
- There is no truncating default to guard against. An unlisted model
gets `limit.output: 0`, and kilo's `min(limit.output, 32000) || 32000`
falls back to 32000. The hazard is a budget that is too *large*, not too
small.
- `reasoning_field` sets `interleaved.field`. Kilo turns interleaved
reasoning off for custom openai-compatible providers unless the field is
named (its built-in default only applies it to model ids containing
"deepseek"), so without this the reasoning channel is dropped. It
defaults to `reasoning_content`, which is what Gym model servers write;
vLLM >= 0.16 also sends `reasoning`, which is why it is configurable
rather than hardcoded.
Kilo splits `-m` on the first `/` only (`let [A, ...L] = H.split("/")`),
so a slashed model name such as `nemo/Qwen/Qwen2.5-7B-Instruct` resolves
to the model `Qwen/Qwen2.5-7B-Instruct` under the provider `nemo`.
Verified against the shipped binary and live.
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…2317) `fern/versions/main.yml` puts `training-tutorials` and `evaluation-tutorials` in the same `section: Tutorials`, so both publish under `/tutorials/`. Evaluation tutorials are already linked that way. Training tutorials are not. Against the published site: ``` /nemo/gym/tutorials/training-tutorials/nemo-rl-grpo/ 200 /nemo/gym/training-tutorials/nemo-rl-grpo/ 308 -> /tutorials/training-tutorials/nemo-rl-grpo /nemo/gym/tutorials/evaluation-tutorials/evalplus/ 200 /nemo/gym/evaluation-tutorials/evalplus/ 404 ``` The old training form works, but only because Fern redirects it. The evaluation form has no such redirect. Having two sibling folders under one section linked two different ways makes the prefix look optional, and it isn't Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
## Standalone NeMo-Gym container
```
FROM cuda-dl-base:26.03 (ubuntu + CUDA, no NGC pytorch)
│
▼
[base stage]
Python 3.12, uv, git
│
▼
[hermetic stage]
uv sync --extra vllm --locked
├── nemo-gym core (ray, fastapi, aiohttp, mlflow …)
├── vllm == 0.26.0
└── flashinfer-python == 0.6.14 (pre-compiled kernels)
│
▼
[release stage]
install nemo-gym project
optional: NEMO_GYM_PREFETCH_SERVERS → per-server .venvs
entrypoint: gym
```
**Per-server venvs** (e.g. `local_vllm_model`, `genrm_model`) are
separate isolated environments created by `gym env start` at runtime via
`uv pip install -e .`. Pass `NEMO_GYM_PREFETCH_SERVERS` to bake them in
at build time instead.
## CI
`build-container.yml` runs on `pull_request` and `push` to main (when
`Dockerfile`/`pyproject.toml`/`uv.lock` change) using `ubuntu-latest`.
Validates the Docker build only — no push, no registry.
## Changes
- `docker/Dockerfile` — 4-stage build, vllm + flashinfer locked
- `pyproject.toml` — `[vllm]` optional extra (`vllm==0.26.0`,
`flashinfer-python==0.6.14`)
- `uv.lock` — locked packages
- `local_vllm_model/setup.py`, `genrm_model/setup.py` — flashinfer
pinned alongside vllm
- `.github/workflows/build-container.yml` — build-only CI, triggers on
PR and push to main
## Dispatch
https://github.com/NVIDIA-NeMo/Gym/actions/workflows/build-container.yml
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…a flag as its own correction (#2303) Related to #1434 Fixes DEFECT A — `-v` / `--verbose` is not global. Issue: `-v`/`--verbose` was registered only on leaf subparsers, so `gym -v list benchmarks` failed while the trailing form worked - it is now on the top-level parser too. The "did you mean?" hint also matched a misplaced flag against itself; the rejected token is now excluded from its own candidate set. Signed-off-by: Ewa Dobrowolska <edobrowolska@nvidia.com> Co-authored-by: Anwith Kiran <anwithk@nvidia.com>
…v resolve docs example (#2310) Fix defects E and G. `gym env validate` promised "a clean message, no traceback" but dumped omegaconf's stack trace when an interpolation couldn't be resolved; adds `ConfigInterpolationError` and translates the omegaconf exception at the parsing boundary. Also fixes the `gym env resolve` docs example, which needed three `++policy_*` keys it never mentioned - unlike `validate`, `resolve` substitutes no dummy model. --------- Signed-off-by: Ewa Dobrowolska <edobrowolska@nvidia.com>
## Summary - Bump Python 3.12 → 3.13.14 and uv 0.11.19 → 0.11.29 - Add `pyarrow>=23.0.1` transitive dep floor - Fix Python 3.13 compat in server deps (scipy, spacy, audioop-lts) - Regenerate `uv.lock` ## Test plan - [x] Unit tests pass - [x] Server suite passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
## `gym eval submit` — Slurm/Pyxis orchestration (initial
implementation)
Introduces `gym eval submit`, a new CLI command that takes a declarative
YAML config and submits one sbatch job per benchmark to a Pyxis-enabled
Slurm cluster, handling staging, SSH transfer, service startup, health
checks, and driver execution end-to-end.
### What's in this PR
**Config model (`nemo_gym/orchestration/api.py`)**
- `SubmitConfig` — top-level config with `services`, `compute`,
`driver`, `job` sections; all models use `extra="forbid"` so typos in
YAML surface immediately
- Discriminated unions for `ServiceConfig` (`vllm`, `ray`) and
`ComputeConfig` (`slurm`) — unknown types are rejected at parse time
- `BaseModelServiceConfig` — base for services that can be wired as the
policy model; currently `VllmServiceConfig`
- `NodePool` with structured allocation fields (`nodes`,
`ntasks_per_node`, `gpus_per_node`) the executor uses for deployment
decisions, plus `extra_args` for arbitrary `#SBATCH` directives
- `driver.policy_model` — syntactic sugar: naming a service auto-injects
`policy_base_url`, `policy_model_name`, `policy_api_key` into every
benchmark's `run` args, with conflict detection if those keys are
already set
- `driver.gym_install` — optional `{repo, ref}` to install gym from
source at runtime via `uv`
- `benchmarks` as a dict (name is the key); each benchmark has `prepare`
and `run` dicts of Hydra overrides
**Executor (`nemo_gym/orchestration/executors/`)**
- `BaseExecutor` — thin ABC; `SlurmExecutor` is the only implementation
- `SlurmExecutor` — stages job dirs locally, rsync's to remote, submits
all benchmarks in one SSH session, prints Slurm job IDs on success;
`--dry-run` prints generated scripts without touching the cluster
- `Connection` — `LocalConnection` (shutil + subprocess) vs
`SSHConnection` (ControlMaster socket, single session for copy + all
sbatch calls); dispatched based on whether hostname matches
`socket.gethostname()`
- `build_sbatch_script` — generates a Pyxis sbatch script per benchmark:
`#SBATCH` directives, backgrounded `srun` per service with `--overlap
--no-container-mount-home`, curl-based health checks with dead-process
detection, then a single driver `srun` that optionally runs `gym eval
prepare` before `gym eval run` in the same container step
- Hydra overrides use `+` (not `++`) so `config_paths` participates in
gym's `_merge_config_paths` coalescing and doesn't clobber model-type
configs
- Output artifacts land at
`<remote_bench_dir>/artifacts/rollouts.jsonl`; `logs/` and `artifacts/`
dirs pre-created during staging
**CLI**
- `gym eval submit --config <yaml> [--dry-run]`
- `@experimental` decorator warns users on invocation
### Known gaps / next steps
1. **Local submit (login node)** — `LocalConnection` is wired when
hostname matches, but untested end-to-end
2. **Env var injection** — per-service and driver env vars (literal
values and host env var references) not yet supported
3. **Rollout output path** — `artifacts/rollouts.jsonl` is set but
downstream profiling commands haven't been validated against it
4. **Multi-instance vLLM** — single vLLM instance per job;
multi-instance requires allocation shape reasoning (`gpus_per_node` is
modeled but not acted on yet)
5. **Mounts** — per-service and driver container mounts
(`--container-mounts`) not yet supported
6. **RayService** — placeholder only; needs a real Ray cluster spanning
the full allocation wired into Gym's Ray backend
7. **Multi-node vLLM** — `--distributed-executor-backend` (mp or ray)
not yet generated
8. **Broader benchmark coverage** — only gsm8k tested; more complex
benchmarks (e.g. MCP-Atlas) may surface config or timing issues
9. **Startup parallelism** — currently: start services → wait for health
→ prepare → run; prepare could overlap with service startup to reduce
wall time
---------
Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>
Signed-off-by: prokotg <19536019+prokotg@users.noreply.github.com>
…verage (#2293) ## Summary - add a versioned `ng_trajectory` projection for normalized model calls, token statistics, semantic turns, tool observations, and invocation-scoped model-visible history - preserve provider-reported cached, reasoning, and total token usage across capture, conversion, and multi-turn agent loops - preserve Responses lifecycle status as `response_status` without reinterpreting dialect-specific `finish_reason` - add an opt-in reference trajectory producer for Simple Agent and document current V/O/X coverage for C1-C7 across all 34 agents This PR establishes the shared schema and one reference producer. The capability matrix reports current support; it does not claim that every producer satisfies C1-C7. ## Correctness and compatibility - collector-derived task and rollout identities are canonical; producer mismatches are recorded and turn identities are normalized without dropping the producer trajectory - producer invocations remain authoritative on duplicate IDs, observation-only invocations are appended, and model and tool records are merged only by exact identifiers - projection failures retain an explicit gap and captured request and response payloads; successful projections remove the duplicate capture payloads - Simple Agent records the actual per-turn model input, cumulative tool-step count, and an explicit gap when resolution is unavailable - no existing fields are removed; response lifecycle status, provider token details and totals, and cached-token aggregation are intentional correctness changes; `ng_trajectory` is additive - Simple Agent retains self-dispatch when observability is disabled or `responses()` is overridden - trajectory collection remains gated by observability and independent of the token-ID capture correlation introduced by #2124 - captured request and response payloads are retained in persisted `ng_trajectory`; LabBench's `multimodal_history_redacted` gap omits those copies - W&B rollout tables omit `ng_trajectory` and model-call request and response payloads during collection and reverification - payload projection has no separate size cap or trajectory-specific opt-out; model-call capture remains opt-in ## NVBug alignment - NVBug 6535274 is addressed with its preferred lossless option: completed Responses calls preserve `response_status="completed"`; `finish_reason` remains unset when the dialect does not provide one. The `inference_provider` path now uses the shared Chat-to-Responses converter, so the fix applies to the backend named in the report. - This is the schema and reference-producer phase of NVBug 6555643 / #1867. The capability matrix marks unsupported and path-dependent producers as `X` or `O`; it is not a claim of full producer coverage. ## Follow-up pull requests PRs #2115 and #2117-#2120 are stacked; later PRs include earlier stack changes. Open PRs describe planned evidence and do not affect the current capability matrix. #2153 provides Claude Code tool timing and status; this PR joins tool results to those observations. No available PR from #2115 through #2122 adds canonical C3 turns. | PR | Producer or path | Evidence added | Remaining gap | |---|---|---|---| | #2115 | OpenClaw, PinchBench | Correlated model calls, conversations, parallel tool timing, and sandbox observations | No C3 turns | | #2116 | Claude Code | Superseded by #2153 | See #2153 and this PR | | #2117 | Hermes | Parallel tool timing and agent observations | No C3 turns | | #2118 | Pi | Correlated model calls, conversations, and parallel tool timing | No C3 turns | | #2119 | OpenCode | Rollout-level model calls, retained conversations, and parallel tool timing | No exact per-invocation model-call ownership; no C3 turns | | #2120 | SWE OpenCode, OpenHands | OpenCode model-call correlation plus retained conversations and sandbox observations for both paths | OpenHands calls, tool timing, and C3 turns remain unavailable | | #2121 | — | No PR exists | — | | #2122 | Stirrup, GDPVal | GDPVal judge-call correlation; Stirrup and Tau2 model-call correlation is already present | No agent turns or tool observations | ## Validation - 354 capture, conversion, trajectory, collector, reverification, Fern-link, and inference-provider tests passed, plus 5 subtests - 10 Simple Agent producer and dispatch tests passed - in-process endpoint-to-record test passed: prefixed Responses request → `inference_provider` → capture middleware → `CaptureStore` → `ModelCallRecord` - producer-to-collector-to-JSON trajectory round trip passed with both available and unavailable resolution status - 110 vLLM model and Responses conversion tests passed - 67 Claude Code observation tests and 4 LabBench redaction tests passed - the trajectory patch merges cleanly with #2124; combined-stack token-capture and trajectory tests passed - Ruff check, Ruff format check, and `git diff --check` passed Part of #1867. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
) ## What The OpenSandbox provider health-checks by default now: - `connect()` and the reconnect in `_connect_after_create()` no longer hardcode `skip_health_check=True`; both derive it from configuration, whose default is `False`. - Two shipped configs that turned the check off (`litmus_agent.yaml`, and the example in the `mini_swe_agent_2` README) are flipped back on. ## Why A sandbox id only proves the workload exists, not that its exec daemon is listening. The server reports a sandbox ready once its pod is Running with an IP, which happens before the daemon binds its port. A handle returned without a health check defers that gap to the first real call, where it surfaces as: ``` 502 {"code":"GENERAL::UNKNOWN_ERROR","message":"Could not connect to the backend sandbox endpoint=..."} ``` Because `connect()` skipped the check unconditionally, the setting was effectively opt-in rather than opt-out, and the two paths disagreed: `create()` honoured `skip_health_check` while `connect()` ignored it. `_verify_created_handle()` did not cover this either — it is a no-op unless `probe.command` is configured. This mirrors the reasoning already documented in the provider's own reference config (`configs/opensandbox.yaml`), which sets `skip_health_check: false`; this PR brings the code and the remaining configs in line with it. ## Behaviour change `connect()` now waits for the sandbox to answer instead of returning immediately, so it can raise where it previously returned an unusable handle. That is the intent: fail or wait at connect, rather than error on the first command. The opt-out is preserved via `skip_health_check` for callers that deliberately want an unchecked handle. Both updated configs also raise `create.timeout_s` above their `ready_timeout_s`, following the guidance in the reference config: that timeout bounds the whole create call, which now includes the readiness wait, so leaving the two equal would turn the wait into a timeout. ## Testing - Added `test_connect_health_checks_by_default` and `test_connect_honours_skip_health_check_opt_out`. - Updated two existing assertions that encoded the old hardcoded value. - `tests/unit_tests/test_opensandbox_provider.py` and `tests/unit_tests/test_sandbox.py`: all pass except `test_resolve_provider_config_named_reference`, which fails on `ModuleNotFoundError: No module named 'omegaconf'` and was confirmed pre-existing by reproducing it with this change stashed. Local runs used a minimal virtualenv: this repo pins a Python version the local toolchain could not fetch, and one dependency does not build on the newer interpreter available. Full-matrix verification is left to CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ch 2.1 config (#2296) ## What Integrates the `nemo_gym.sandbox` API into the harbor agent **non-intrusively**: a new `NemoGymSandboxEnvironment` (a Harbor `BaseEnvironment`) executes Harbor trials inside sandboxes managed by any NeMo Gym sandbox provider (opensandbox, docker, daytona, apptainer, ecs_fargate). Selected purely via config — no changes to the harbor_agent execution path. - `custom_envs/nemo_gym_sandbox/environment.py` — AsyncSandbox-backed backend: task.toml resources → `SandboxSpec`, tar-based `upload_dir`/`download_dir` with per-file fallback, `bash -ic` exec wrapping (parity with Harbor's docker/daytona backends), always-terminate on `stop()`, opt-in `allow_unenforced_internet_isolation`, `image_rewrites`. - `configs/harbor_agent_opensandbox.yaml` — Terminal-Bench 2.1 (89 tasks) with stock Terminus-2 through the opensandbox provider in server-proxy mode. Provider referenced via `sandbox_provider: ${sandbox}` (mini_swe_agent_2 style; include the shipped provider config with a second `--config`). Sampling/harness match the Qwen3.6-27B model card (temp 1.0 / top_p 0.95 / top_k 20, 3h agent timeout, 256K ctx / 80K max output). 30 GiB sandbox ephemeral storage to avoid kubelet evictions on disk-heavy tasks. - `prepare_terminal_bench_2_1.py` + checked-in 89-row benchmark input JSONL (pinned `harbor-framework/terminal-bench-2-1`). - `harbor_ray_task_num_cpus` config knob — Harbor jobs are I/O-bound Ray tasks; default 1 CPU/task caps concurrency at the driver CPU count, so a fractional value (0.25) enables true full-89 concurrency on a 24-CPU driver. - fix: `extract_usage` sets `input_tokens_details.cache_write_tokens` (required by openai ≥2.40; every live-model `/run` 500'd without it). - 18 unit tests (fake registered provider; tar round-trip verified at byte level). ## Result — Terminal-Bench 2.1, Qwen3.6-27B Ran the full 89-task benchmark end-to-end: vLLM-served Qwen3.6-27B on 2 GPU replicas, gym/Ray driver, and sandboxes on a live OpenSandbox deployment reached in server-proxy mode. Traces captured for all 89 tasks (rollouts + per-trial trajectory, asciinema recordings, verifier artifacts). | | Value | |---|---| | **This run** (TB 2.1, Harbor/Terminus-2, single run) | **pass@1 = 59.55%** (53/89) | | **Model card** (TB 2.0, Harbor/Terminus-2, avg of 5 runs) | **59.3** | **Reproduces the model-card Terminal-Bench number essentially exactly** (within ~0.25 points). Caveats: TB 2.1 vs 2.0 (26 tasks fixed; same count/harness), single run vs 5-run average, and sandbox resources set to 4 vCPU / 16 GiB / 30 GiB. Note: commits carry DCO sign-off; GPG signature to be added on amend (non-interactive session). 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Update: connection hardening + 5-run measurement (in progress) Follow-up commit hardens the opensandbox provider + harbor agent for sandboxes reached across a load-balanced network path, based on failure modes hit while running the benchmark end-to-end: - **`connection.disable_connection_pooling`** (provider option, off by default): a load balancer between client and OpenSandbox server can silently reap idle pooled connections; the SDK then reuses a dead socket and exec hangs (observed as dozens of CLOSE_WAIT sockets wedging tasks, while fresh-connection requests completed in ~1s). With pooling disabled every request opens a fresh connection — no reaped socket is ever reused. Verified A/B: identical run with pooling on collapsed (8/89 tasks in 40 min, CLOSE_WAIT≈80); with the option on, 57/89 in 27 min with CLOSE_WAIT=0. - **`operations.background_exec`** (provider option, off by default): runs each command as a background execution polled via short status/log requests instead of holding one SSE stream open for the command's whole duration — for load balancers that cap stream duration and would otherwise drop the stream and hang the client. Costs extra round trips per command. A window-matched A/B against the foreground SSE path scored within run-to-run noise, so this is an availability option rather than a quality one. - **Hard exec wall-clock cap** (provider): backstop around submit+poll+retries so a pathologically wedged exec raises `TimeoutError` (which Terminus-2 records as a command timeout) instead of hanging the task. - ~~**`default_exec_timeout_s` 14400 → 300**~~ — *superseded*: that reasoning accounted only for Terminus-2's tmux plumbing and missed Harbor's **verifier**, which also calls `exec()` without `timeout_sec`. The value is now **1800** in both the env default and the TB config — see the gold-patch section below for the measurement. - **`nemo_model_server_timeout_sec` 7200 → 2400**: lets a silently-dropped model call self-heal within 40 min instead of blocking a task for 2h. - **`cpu_pin_enabled`** (opt-in, default false): pins every Harbor exec (tmux session + verifier) to a random contiguous core block sized by the task's cpu count, so `nproc` inside reports the real budget and co-resident sandboxes spread across core blocks. Fail-open (no `taskset`/small host → unpinned). Measured on a full TB 2.1 run under card conditions: **no e2e-time or score delta vs baseline** (58.4% / 3h13m / 12.4% timeouts, all within the 5-run baseline spread) — task wall time on this benchmark is dominated by model turns, not sandbox CPU contention. Kept for CPU-bound workloads. **Serving guidance for Qwen3.6 (hybrid GDN attention) on vLLM**, learned the hard way: - Use stock `vllm/vllm-openai` (v0.25.1 used here; multi-arch incl. arm64). - ~~**Disable prefix caching** (`--no-enable-prefix-caching`)~~ — *superseded*: the engine-core freezes initially attributed to prefix caching turned out to be the default (FlashInfer) GDN prefill kernel; with `--gdn-prefill-backend triton`, prefix caching ON is stable (validated over a further 5-run cohort, zero freezes) and scores better — see the final card-parity section below. - Warm each replica before an eval burst (a short + a long generation crossing the ~8k boundary, plus a small concurrent batch) to pre-trigger kernel JIT. ### 5-run pass@1 (final) Two cohorts, differing only in the per-task agent timeout (the 4h budget compensates the per-turn re-prefill slowdown from running without prefix caching — see serving notes): | Run | Agent timeout | pass@1 | agent-timeout rate | |---|---|---|---| | 1 | 3h | 57.30% (51/89) | 16.9% | | 2 | 3h | 53.93% (48/89) | 20.2% | | 3 | 4h | 62.92% (56/89) | 10.1% | | 4 | 4h | 53.93% (48/89) | 11.2% | | 5 | 4h | **59.55% (53/89)** | 10.1% | - **3h cohort mean: 55.6** · **4h cohort mean: 58.8 ± 4.5** · all-5 mean: 57.5 ± 3.8 - References: earlier single clean run **59.55%**; model card **59.3** (TB 2.0, 5-run average) - The 4h cohort — whose timeout rate matches the card-harness conditions (~10%) — lands on the model-card number within noise; run-to-run spread of ±4-5 points on an 89-task benchmark is expected (1 task ≈ 1.1 points). ### Serving notes for Qwen3.6 (hybrid GDN) on vLLM Hard-won operational guidance from ~30h of serving under full-benchmark agentic load: - **`--gdn-prefill-backend triton`** — with the default (FlashInfer) GDN prefill kernel, the engine core froze roughly hourly under concurrent long-context load: requests parked, prompt/generation counters static, no errors, health endpoint green. Reproduced on two different vLLM builds, with prefix caching on and off, on fresh and long-lived engines. On the Triton/FLA kernel: **zero freezes across ~16h** of identical load. (An upstream issue with the full signature is worth filing.) - **Prefix caching: keep it ON once the Triton GDN kernel is in place.** It was temporarily disabled while isolating the freeze (it's flagged experimental for this hybrid-GDN family), but the final card-parity cohort ran 5 freeze-free runs with caching enabled — and it's worth ~3–4 points on this workload: per-turn re-prefill dominates deep agentic trajectories, so caching cuts turn latency, which cuts agent-timeout rate (~11% vs ~13–17% cacheless) even on a shorter 3h budget. - **`--compilation-config '{"cudagraph_mode": "PIECEWISE"}'`** — attention ops run eagerly, outside graph replay. - **Warm each replica before an eval burst** (one short + one long generation crossing the ~8k boundary + a small concurrent batch) to pre-trigger kernel JIT; and restart engines between long evals. - **Protect long-running eval driver and sandbox pods from autoscaler disruption** (a do-not-disrupt annotation or equivalent) — multi-hour drivers are otherwise casualties of routine node consolidation, and a mid-eval node roll takes in-flight sandboxes with it, surfacing as opaque proxy 500s. - **Sandbox resources 8 CPU / 16 GiB / 30 GiB** for TB 2.1's heavier tasks. ### Follow-up: 4-replica scaling + SGLang A/B (5 runs each, 4h timeout, 8C/16G sandboxes) | Serving stack | pass@1 (5 runs) | Mean ± std | Avg timeout rate | Engine freezes | |---|---|---|---|---| | vLLM v0.25.1, `--gdn-prefill-backend triton` | 61.80, 55.06, 56.18, 56.18, 57.30 | **57.3 ± 2.6** | ~13% | 0 | | SGLang v0.5.15.post1 (defaults + qwen3 reasoning parser) | 56.18, 56.18, 60.67, 51.69, 56.18 | **56.2 ± 3.2** | ~17% | 0 | - Scaling 2→4 replicas doubled fast-phase throughput (~50% more tokens/episode) but did not move pass@1 or the timeout rate: the benchmark's deep tail is bounded by the per-task wall-clock budget, not aggregate serving capacity. - Both stacks ran ~20h each under full-benchmark load with zero engine freezes — SGLang out of the box (after a readiness-probe adjustment: its `/health` generates a token and can exceed a 1s probe timeout, flapping healthy pods out of the Service; set probe `timeoutSeconds` ≥ 15), vLLM with the Triton GDN kernel noted above. Scores are statistically indistinguishable; vLLM sustained ~40% more tokens/episode at lower timeout rates on this workload. ### Harness validation: gold-patch (oracle) run — 89/89 Ran all 89 TB 2.1 tasks with Harbor's `OracleAgent`, which uploads each task's reference `solution/` and runs it instead of generating with a model. This exercises the sandbox backend, `upload_dir`, exec and the verifier end-to-end with the model removed from the equation, so the expected score is a clean sweep. **Result: 89/89 (reward 1.0 on every task), 22 min wall clock, zero agent timeouts.** Getting there surfaced one real bug in this integration, now fixed in this PR: **`default_exec_timeout_s` was silently truncating verification.** Harbor's verifier calls `environment.exec()` **without** `timeout_sec`, so this default — not the task's declared `[verifier] timeout_sec` — is what bounds verification. The old values (300 in the environment, 600 in the TB config) were justified by reasoning only about Terminus-2's short tmux plumbing, which missed the verifier entirely. **87 of the 89 TB 2.1 tasks declare a verifier budget above 600s** (median 900, max 12000), so verification was being killed mid-run and the task scored 0 instead of failing loudly. | exec cap | gold-patch score | what changed | |---|---|---| | 600s | 83/89 | `compile-compcert` killed at 602s; `query-optimize` verifier killed at 604s → `RewardFileNotFoundError` | | 1200s | 83/89 | `query-optimize` fixed; `compile-compcert` still killed | | **1800s** | **85/89** | both fixed | | 12000s | 85/89 | no further gain | Longest single command observed was 1384s, so **1800s is the smallest value that costs nothing** — now the default in both the environment and this config. Because this bug suppressed slow-verifying tasks, previously reported pass@1 numbers are conservative. The remaining 4 gold-patch failures were **not** harness issues and are deliberately not fixed here — three were upstream drift (a dataset host that refuses this network's egress; `RcppParallel 6.0.0` requiring `cmake`, published 4 days before the run; `planarity 1.0.0` renaming attributes `pyknotid` reads, published 4 weeks before) and one was macOS `._*` sidecar files in our local dataset copy. Patching those reference solutions would only change an oracle run — `solution/` is never uploaded for a model agent — so it cannot affect a scored eval, and fixing them for model runs would mean altering the task environment rather than the solution. Worth noting for anyone reproducing: sandbox resources were *not* implicated — the config's uniform 8 CPU / 16 GiB / 30 GiB is a strict superset of what all 89 task.toml files declare (max 4 CPU / 8 GiB / 10 GiB). ### Final: card-parity cohort (prefix cache ON + Triton GDN kernel, 3h agent timeout — model-card conditions) | Run | pass@1 | agent-timeout rate | |---|---|---| | 1 | 60.67% (54/89) | 10.1% | | 2 | 61.80% (55/89) | 9.0% | | 3 | 57.30% (51/89) | 13.5% | | 4 | **66.29% (59/89)** | 11.2% | | 5 | 59.55% (53/89) | 13.5% | | **Mean** | **61.1 ± 3.3** (model card: 59.3) | 11.5% | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Fixes #2217 ## Problem `gym eval run --no-serve --output results/mcqa_rollouts.jsonl` — the quickstart command in the README — raises `FileNotFoundError` when `results/` does not exist yet: ``` File "nemo_gym/rollout_collection.py", line 508, in run_from_config with config.materialized_jsonl_fpath.open("wb") as f: FileNotFoundError: [Errno 2] No such file or directory: 'results/mcqa_rollouts_materialized_inputs.jsonl' ``` The mkdir was already present in `run_from_config`, but it is called too late. It ran at the top of the dispatch section, while the first thing the run writes is the materialized inputs. ## Fix Move `output_fpath.parent.mkdir(parents=True, exist_ok=True)` to the top of `run_from_config`, above both the resume and fresh branches. The issue suggested adding a mkdir at each of the four output paths. One is enough: all four derive from `output_fpath` and keep its parent: materialized inputs and aggregate metrics via `with_stem`, the failures sidecar via `with_name`, rollouts is the path itself. Resume semantics are unchanged: creating an empty directory does not make `output_fpath.exists()` or `materialized_jsonl_fpath.exists()` true, so resume eligibility is decided the same way as before. ## Why this only showed up on the PyPI install path Two things were hiding it: - Without `--no-serve`, `gym eval run` dispatches to `e2e_rollout_collection`, which sets `output_dirpath = <output parent>/preprocessed_datasets` and lets `TrainDataProcessor` create it, incidentally creating the output parent. Only the `--no-serve` path goes straight to `collect_rollouts`. - The repo tracks a `results/.gitignore` placeholder, so `results/` exists in a git clone. A PyPI install running from an arbitrary cwd has no such directory. ## Scope This was the only affected entry point. The others already create their output parent first, so this brings rollout collection in line with the existing convention: - `gym eval reverify` — `rollout_reverification.py`, in `_prepare_output_fpaths` - `gym eval aggregate` — `rollout_collection.py`, in `RolloutAggregationHelper.run_from_config` - `gym dataset render` — `prompt.py`, in `materialize_prompts` - `gym eval profile` — writes next to an input rollouts file that must already exist ## Testing Added `test_run_from_config_creates_missing_output_dir`, parametrized over `resume_from_cache`, writing to a two-level-deep missing directory and asserting all four artifacts land there. Both parametrizations fail on `main` and pass with this change. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…oint override in /run (#2166) ## What does this PR do? Adds optional `policy_base_url` / `policy_api_key` fields to the `/run` request of both mini-SWE agents (`mini_swe_agent` and the sandbox-backed `mini_swe_agent_2`), letting a caller point that one episode at its own OpenAI-compatible policy endpoint. When the fields are absent, the configured `model_server` is used — **default behavior is unchanged**. On `mini_swe_agent_2`, the `ng-rollout` capture prefix is applied only on the built-in model-server path: a caller supplying its own endpoint owns routing (and any per-rollout correlation) for it. ## Motivation We are integrating NeMo Gym as an environment ecosystem for RL training in [miles](https://github.com/radixark/miles) (an open-source RL training framework), driving the mini-SWE agents with one `POST /run` per training episode. For lossless on-policy training (token-in/token-out), the trainer routes every policy call of an episode through a per-episode recording proxy: each episode gets its **own** OpenAI-compatible URL (e.g. `http://trainer:30000/sessions/<session_id>/v1`), and the proxy captures token ids / logprobs / loss masks for exactly that episode. This pattern isn't specific to our trainer — any external RL framework that records or routes rollouts per episode needs a per-request policy endpoint. Today `run()` builds the endpoint from the statically configured model server, fixed for the lifetime of the server process, so this integration currently requires carrying a fork of `app.py`. Everything downstream is already per-request capable — the swegym/sandbox runners accept `base_url` / `api_key` per invocation and inject them into `model_kwargs` — so this PR only wires the request fields through. The field names mirror the existing global-config keys (`policy_base_url`, `policy_api_key`). If this pattern proves useful beyond the mini-SWE agents, the fields could later be promoted to `BaseRunRequest`; a complementary general mechanism (per-rollout outbound routing at the model-server layer, building on the existing `ng-rollout` correlation prefix) could serve agents that call the policy through `server_client` — happy to open a separate issue to discuss that. ## Testing - `ruff check` / `ruff format --check` pass on the touched files. - No behavior change when the new fields are unset (they default to `None`; the existing paths are taken). - Happy to add unit tests for the override path if you can point me at the preferred test location for responses-API agents. --------- Signed-off-by: Tao Lin <tao.lin@radixark.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`model-recipes/nemotron-3-nano.mdx` tells users to provision ~110GB in the prerequisites and again when choosing a workspace, then checks for 200GB two lines later. Both figures now say 200GB, matching the existing success check. The prerequisite also states where the space goes, using numbers the page already verifies elsewhere: the container is ~15GB (step 1.4) and the model is ~59GB (step 1.6), with the rest covering data, the Hugging Face cache, and checkpoints and logs that accumulate across runs. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: Anwith Kiran <anwithk@nvidia.com>
`contribute/agent-skills.mdx` links to the Add a Benchmark page with a full `docs.nvidia.com` URL that carries no version slug. The URL resolves (200), so nothing is broken, but it drops readers on the default version regardless of which version they were reading. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…n output (#2305) Three small fixes (defects D, F, H): `gym list models --json` now includes a `name` key like every other component type (`model` kept, so `jq '.[].model'`still works); model inspection shows a usage example like benchmarks and environment already did, and `gym env packages` only prints its header in the default view so `--json` pipes cleanly. Agents still show no usage example - their names aren't valid `--agent` values, so there's nothing runnable to suggest until #1583 lands a real agent selector. --------- Signed-off-by: Ewa Dobrowolska <edobrowolska@nvidia.com> Co-authored-by: kajalj22 <kajalj@nvidia.com>
## Summary - Rewrite [Key Terminology](https://docs.nvidia.com/nemo/gym/main/about/concepts/key-terminology/) with Gym-aligned definitions (Environment, Agent, Rollout, Benchmark, etc.) - Add an Overview (run flow, Agent/Environment composition, Model server) plus a concept → Gym component map - Cross-link related how-tos (Data, Agent/Model Server, Verifiers, Evaluate, Training, Sandboxes) Fixes #1500 ## Test plan - [x] `make docs-check` (0 errors) - [x] `python3 tests/unit_tests/test_fern_docs_links.py` - [x] Confirm Fern docs preview comment on the PR - [x] Skim Overview + glossary sections for clarity on the preview URL --------- Signed-off-by: Seph Mard <smard@nvidia.com> Signed-off-by: Seph Mard <seph.mard@gmail.com> Co-authored-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
…#2353) ## Summary - **vllm** 0.20.0 → 0.24.0 (addresses security vulnerabilities; also updates flashinfer-python 0.6.8.post1 → 0.6.12) - **GitPython** ≥3.1.50 → ≥3.1.57 (addresses security vulnerability; lock resolves to 3.1.58) - **pyarrow** already at 25.0.0 in lock (existing `>=23.0.1` constraint sufficient; no change needed) - **diskcache** added to `exclude-dependencies` (no fix available; no longer pulled in by vllm 0.24.0 — blocked explicitly to prevent re-introduction) vllm 0.24.0 does not import `NamespaceTool` from `openai.types.responses`, so the existing `openai<=2.7.2` constraint is preserved and no openai SDK bump is required. ## Test plan - [x] CI passes (lint, unit tests, pre-commit hooks, server suite, build) - [x] `uv lock` resolves cleanly with no conflicts 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com>
## Summary - Document the Resources Server protocol landscape, including the seed/verify and reset/step base episode protocols and existing seed/verify extensions. - Clarify how Agent Server `run()` and `responses()` behavior maps to Resources Server lifecycle, verification, and transition APIs. - Explain direct HTTP, MCP, Gymnasium step-action, bespoke transition, and agent-embedded tool interactions. - Provide API contracts, implementation examples, state management, cleanup, YAML configuration, and matching built-in Agent Servers. - Add the guide to the Build Environments navigation. Closes #1363 ## Test plan - [x] Run `npm run check` from `fern/` --------- Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
## Summary - Closes [#1360](#1360) - Document execution-based verification for code, SQL, tool state, formal proofs, and sandboxed workloads. - Cover concurrency, timeouts, isolation, cleanup, and failure handling. - Add the guide to the Verification Patterns navigation. ## Test plan - [x] Run `cd fern && npm run check` Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
The Workplace Assistant docs quote four different figures for the same environment: 26 tools / 5 databases, 27 tools / 5 databases, 27 tools / 6 databases, and 27 tools split as five toolkits plus a company directory. `get_tools` in `resources_servers/workplace_assistant/utils.py` is the source of truth. It always registers the company directory lookup, then adds one toolkit per requested name. Running it with the five toolkits `app.py` seeds each session with: ``` functions: 27 schemas: 27 containers: analytics, calendar, company_directory, customer_relationship_manager, email, project_management per-toolkit: email 6, analytics 6, calendar 5, project_management 5, crm 4, company_directory 1 ``` So: 27 tools across five mutable databases plus a read-only company directory. `is_correct` (utils.py:166) compares final state for the five databases and not the company directory, which only exposes `find_email_address`. ## Changes - `evaluation/environment-list.mdx`: 26 tools to 27 tools - `generating-training-data.mdx`: "27 tools across 6 databases" to five databases plus a company directory lookup - `about-workplace-assistant.mdx`: the 27 tools are the five databases plus the company directory, not 27 tools spread across five databases - `environments/workplace_assistant/README.md` and `resources_servers/workplace_assistant/README.md`: 26 tools to 27 tools `mcp-resources-server.mdx` already matched the code and is unchanged. ## Test `resources_servers/workplace_assistant/tests/test_docs_counts.py` derives the count from `get_tools` and asserts every page and README that quotes it. Adding or removing a tool now fails a test instead of leaving the docs stale. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
#2332) ## Summary Adds two lightweight verifiers used by the blended Nemotron Super Omni RL dataset. `string_match` — exact/fuzzy answer comparison with boxed/latex extraction. Covers 14,988 of the 60,299 rows in that blend. `gui_coordinate` — 2D coordinate containment. Covers 16 rows. Both subclass `BaseVerifyResponse` and follow the existing simple-agent contract, so they need no special handling in the rollout path. Carrying them here removes the need for downstream launchers to copy verifier directories into the Gym checkout at submit time. --------- Signed-off-by: DanialTaheri <smohsenitahe@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…coordinate (#2377) `gym env test` counts every directory under `resources_servers/` etc. as a candidate module, but only tests the ones containing a `README.md` (`nemo_gym/cli/env.py:717`), then asserts the two counts match. #2332 added `string_match` and `gui_coordinate` without READMEs, so main fails with `Mismatch on the number of total modules found (144) and the number of actual modules tested (142)`. This adds a README to each, describing the extraction/scoring behavior, the input JSONL fields, and how to run them. Both servers' tests now run: 10 passed for `string_match`, 12 for `gui_coordinate`. Data validation preconditions hold for both (5 examples, 5 rollouts, `"Number of examples": 5`). Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Partially addresses #2216, and fixes the quickstart report behind it: `gym env start` prints "All 3 / 3 servers ready" against a `policy_base_url` that nothing is serving, and `gym eval run` then sits at `Collecting rollouts: 0%`. Gym reports ready because `wait_for_spinup` in the env CLI polls only Gym's own servers. The Gym-side model server is a proxy that answers as soon as it starts, whether or not an inference server is behind it, so the endpoint the run depends on is never checked. The existing troubleshooting note for this path blames a bad API key, which fails differently. This collects the model endpoints from the resolved config, waits for each to answer, and stops startup if one never does. Reading the endpoints has to handle every shape the model servers declare: - `base_url` is `Union[str, List[str]]` on vllm_model and both local vLLM servers, where the list spreads load across replicas, so a string-only reader would do nothing at all for a multi-replica training run. - The local servers default to an empty list and are filled in after Gym launches vLLM, so an empty list means "not yet" and is skipped, as is an unset value. - The config key travels with each URL, because the endpoint that fails may be a judge or user model and a message naming `policy_base_url` would then be wrong. Each probe result is sorted by whether waiting could change it. A hostname that does not resolve will not begin resolving because we waited, so it is reported once and not retried; this is what happens to a config left at a placeholder default, which costs one lookup instead of the whole timeout. A refused connection may be a server still coming up, so it is waited on. Any HTTP answer counts as listening, including 401 and 404, and so does a completed TLS handshake against a certificate this machine does not trust. `requests.exceptions.SSLError` subclasses `ConnectionError`, so deciding on the parent class would reject an HTTPS endpoint with an internal certificate authority even though its handshake proved something was there. On failure, the spawned servers are shut down before anything is raised. They hold ports and have neither a process group nor an atexit handler, and every caller reaches its own `shutdown()` only after `start()` returns. The failure is a `ConfigError` that the CLI entrypoints translate into an exit. `RunHelper` is imported and driven directly by NeMo-RL, so the caller decides what an unreachable endpoint means. Probing uses the synchronous `requests` library, matching `poll_for_status` in server utils, because this runs in the CLI process before any event loop exists. ## The change in flow Before: ```mermaid flowchart TD S["gym env start"] --> P["start mcqa, agent, policy_model"] P --> W["wait_for_spinup()"] W --> Q{"do Gym's own servers answer on /?"} Q -->|yes| OK["All 3 / 3 servers ready"] OK --> U["gym eval run"] U --> M["policy_model calls its configured endpoint"] M --> X["nothing listening<br>Collecting rollouts: 0% indefinitely"] ``` After: ```mermaid flowchart TD S["gym env start"] --> P["start mcqa, agent, policy_model"] P --> W["wait_for_spinup()"] W --> N["collect model endpoints from the resolved config<br>string and list base_url, skipping unset and empty"] N --> PR{"probe each endpoint"} PR -->|"any HTTP answer, or a completed TLS handshake"| OK["All 3 / 3 servers ready"] PR -->|"name does not resolve"| REP["report once, do not wait<br>waiting cannot create a DNS record"] REP --> OK PR -->|"connection refused"| WAIT["wait, a server may still be starting"] WAIT -->|answers in time| OK WAIT -->|"still refusing at the timeout"| SD["shut down the spawned servers"] SD --> CE["raise ConfigError naming each URL<br>and the config key it came from"] ``` Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
## Summary - Snapshots `fern/versions/latest/` → `fern/versions/v0.5.0/` (101 files) to freeze the docs at the 0.5.0 GA state - Creates `fern/versions/v0.5.0.yml` with nav config pointing to the frozen pages - Registers `v0.5.0` as `stable` in `fern/docs.yml` (between Main and 0.4.0) - Adds v0.5.0 release notes accordion to `release-notes.mdx` in both `latest/` and the snapshot - Drops pre-release tag: `PRE_RELEASE = "rc0"` → `""` in `nemo_gym/package_info.py` (version is now `0.5.0`) ## Test plan - [ ] Verify `docs.nvidia.com/nemo/gym` shows `0.5.0` in the version picker as stable - [ ] Verify the v0.5.0 docs snapshot renders correctly at `/v0.5.0/` - [ ] Verify release notes page shows v0.5.0 accordion open by default, v0.4.0 collapsed - [ ] Verify `nemo_gym.__version__` returns `"0.5.0"` (no `rc0` suffix) - [ ] Cherry-pick to `r0.5.0` after merging to main 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Kajal Jain <kajalj@nvidia.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
## Summary Fixes the `test_internal_pages_are_linked_by_path_not_by_absolute_url` test failure introduced by #2389. The v0.5.0 release notes sandboxing section contained a `docs.nvidia.com/nemo/gym/...` absolute link. The test requires all internal docs links in `latest/` to use relative paths (to keep readers within the version they're reading). Replaced with a relative path `/nemo-gym/nemo_gym/sandbox/providers` in both `latest/` and the `v0.5.0/` snapshot. ## Test plan - [ ] `test_internal_pages_are_linked_by_path_not_by_absolute_url` passes - [ ] Cherry-pick to `r0.5.0` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
## Summary - Remove the NeMo Evaluator row from the Evaluation & Training table on About → Ecosystem ## Test plan - [x] `make docs-check` - [x] Confirm Fern preview on the PR - [x] Verify Ecosystem page no longer lists NeMo Evaluator under Evaluation & Training Signed-off-by: Seph Mard <smard@nvidia.com>
Documents training on rollouts from an external agent harness that drives its own model calls. ## Summary - Explains run-level infrastructure enablement, static per-agent selection, `all_agents`, and the explicit `/training-token-capture` request path. - Distinguishes training capture from neutral rollout correlation and evaluation observability. - Documents async sink/source contracts, frozen snapshots, durable incomplete state, frozen retirement tombstones, and conditional retirement after downstream durability. - Shows how a training framework supplies paired transport-backed endpoints without adding framework dependencies to Gym. - Covers worker-local installation limits, startup validation, selected-agent finalization, reconstruction masking, rollout identity, and first-run metrics. - Provides the caller-owned `finalize_rollout_token_capture` and `retire_rollout_token_capture` sequence. The page is linked from the training tutorials index. Parent resolution and prefix supply are documented in #2349. --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? Declares `failure_reason` field on `BaseVerifyResponse` to standardize a way to report why a returned reward may not reflect the policy quality. `failure_reason` is meant to be human-readable; see #2750 for the whole design document. ## Checklist - [ ] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [ ] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [ ] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [ ] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [ ] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
## Summary - emit standardized `ng_agent_observations` for the legacy SWE OpenCode/OpenHands paths and the decoupled `opencode_sandboxed_agent` + SWE-bench path - correlate OpenCode calls through rollout-prefixed Gym Model Server routes when rollout observability or token capture is active - compose OpenCode invocation/tool evidence with separate agent- and verifier-sandbox observations - preserve direct response behavior, grading outputs, retained artifacts, and `subagent_trajectories` ## What changed ### Decoupled OpenCode / SWE-bench - parse OpenCode SQLite sessions into invocations, parent relationships, cumulative model-visible conversations, tool timing/outcomes, compaction events, and exact response IDs - keep the sandboxed agent independently installable by copying the minimal parser closure locally; it no longer imports the standalone `opencode_agent`, and a fresh-process test enforces that boundary - isolate OpenCode data per observed run, download and parse the database before teardown, and remove the local scratch database afterward - record the connected agent sandbox's real provider/ID and compose verifier-sandbox lifecycle evidence returned by `resources_servers/swebench` - emit observations only when a capture-derived rollout ID exists; direct `/v1/responses` behavior remains unchanged ### Legacy SWE harness - OpenCode records retained session invocations, parent relationships, exact response IDs, and the latest cumulative conversation; rollout-prefixed model calls provide capture and token accounting - OpenHands records its available cumulative root conversation and sandbox evidence, while explicitly reporting that exact model-call correlation is unavailable with the pinned fork - legacy Apptainer records leave `sandbox_id` unset and report `sandbox_identity_unavailable` because the runner exposes no real sandbox handle Observation construction fails open. Missing or malformed evidence becomes an explicit gap; provider sentinel values are not exposed as process exit codes, and unavailable resource or lifecycle measurements remain unset rather than inferred. ## Capability coverage | Producer | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `opencode_sandboxed_agent` | V | V | X | V | V | V | V | | `swe_agents` / OpenCode | V | V | X | V | X | X | V | | `swe_agents` / OpenHands | X | X | X | O | X | X | V | The capability matrix documents these evidence boundaries. Legacy artifacts do not provide standardized semantic turns, authoritative per-tool timing, or independent parallel-tool timing. ## Validation - 34 combined standalone and sandboxed OpenCode tests, including fresh-process import isolation - focused SWE-bench resource-server and legacy SWE-agent tests across disabled, observability-only, token-only, and combined capture states - 114 shared rollout-observability, correlation, and collection regressions rerun after the final rebase - SQLite artifact -> parser -> decoupled `/run` composition test covering invocation, tool, agent-sandbox, verifier-sandbox, and cleanup primitives - Ruff, formatting, Python compilation, `git diff --check`, and scoped pre-commit hooks A real artifact-compatibility smoke test used Docker Server 29.6.2 on Linux/aarch64 and the exact `swebench/sweb.eval.x86_64.astropy_1776_astropy-12907` image under x86_64 emulation. OpenCode 1.17.11 was installed only inside the temporary container and ran a real gpt-5.5-backed session whose bash tool executed `printf opencode-observability-smoke`; the actual tool result persisted and `opencode export` succeeded. The WAL-mode database contained 1 session, 3 messages, and 7 parts. After closing/exporting, only `opencode.db` was copied and parsed by the final sandbox-local parser, producing 1 completed invocation, 1 tool call, 0 compactions, and only the expected `model_call_ownership_unavailable` gap. The temporary container was stopped and auto-removed; nothing was installed on the host. ## Limitations - the live smoke used the direct NVIDIA gateway rather than Gym's rollout-prefixed model proxy, so it validates the real OpenCode artifact schema/parser but not model-call ownership or capture joining - OpenCode's final prose stream did not terminate after the tool result and was gracefully interrupted - the full decoupled `/run` + verifier flow was not run live because `DockerProvider` cannot reconnect across the resource-server and agent processes; that path requires OpenSandbox - the SQLite parser is intentionally duplicated to keep the two agent servers dependency-isolated and must remain synchronized ## Related rollout-observability work - #2114 shared observation and correlation contract - #2153 Claude Code producer - #2115 OpenClaw producer - #2117 Hermes producer - #2118 Pi producer - #2119 standalone OpenCode producer --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
## Summary
Adds multi-node vLLM service support to the Slurm orchestration layer
(`gym eval submit`), building on top of the existing single-node
multi-instance support (separate PR:
`onur/multi-instance-vllm-service`). A `vllm` service can now span
multiple physical Slurm nodes, either for a single replica's
tensor/pipeline-parallel footprint, or for multi-node data-parallel
replicas.
What's NOT included in this PR is that multi-node TP with multiple
instance which is a rare case. So, in order to use the features in this
PR, the model has to fit into a single node.
## What's new
- **`ray` distributed backend** — a new `distributed_backend` option
(alongside the existing single-node `mp` backend) using vLLM's Ray core
executor (`--distributed-executor-backend ray`, *not* the `ray.serve`
library) to span a service across nodes.
- **Automatic backend selection by node count** — if
`compute.node_pools` total more than one node, `distributed_backend` is
automatically forced to `ray`, overriding anything set/defaulted at the
service level. No need to write `distributed_backend: {type: ray}`
yourself.
- **Multi-node data-parallel validation** — `number_of_instances` must
divide evenly across the node count (each node hosts an equal share of
replicas).
- **Sbatch script generation** (`slurm_script.py`):
- injects a Ray head-node-IP prelude (`scontrol show hostnames`,
`HEAD_NODE_IP`/`RAY_HEAD_NODE_IP`) when any service uses the `ray`
backend,
- single-instance multi-node: wraps the vLLM command in `ray
symmetric-run` (falling back to manual `ray start --head`/`--block` for
older Ray), spanning TP/PP across nodes,
- multi-instance multi-node: uses vLLM's native multi-node data-parallel
pattern — head node serves the OpenAI API, worker nodes run `--headless`
with a `--data-parallel-start-rank` offset — no Ray involved for that
path,
- adds `--nodes=`/`--ntasks=` to each service's `srun` step on
multi-node compute (`--nodes=1 --ntasks=1` for the driver).
## New example config
- `examples/slurm_vllm_ray_multi_node.yaml` — 2-node compute, 8 replicas
(TP2) spanning both nodes, backend auto-selected as `ray` from node
count alone.
---------
Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>
Signed-off-by: prokotg <19536019+prokotg@users.noreply.github.com>
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com>
Co-authored-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>
Co-authored-by: prokotg <19536019+prokotg@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Judge models are commonly hosted independently of other jobs. This allows for their reuse. If a judge model must be rehosted, we cannot allow this to bring down the entire training cluster. Commonly, training infrastructure have job duration limits. If a judge model meets such a limit in the middle of a training run, it will change addresses, and it needs to communicate this to the rest of the Gym setup. This PR adds two optional arguments: - `max_connection_retries` prevents infinite retries (which cause hangs) - `endpoint_file` allows the judge model to communicate its address via a local filesystem; this flag comes with associated helper configs. --------- Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
## Summary - `tests/e2e/run_inference_provider_e2e.sh` now runs against a dedicated `tests/e2e/inference_provider_smoke.jsonl` prompt (unambiguous weather request) instead of the shared `example_single_tool_call` dataset, and pins `--temperature 0` for determinism. Mirrors the existing `tests/e2e/gpu_smoke.jsonl` pattern used by `gpu_e2e_test.sh`. - `provider_e2e_tests` in `cicd-main.yml` now uploads `rollouts.jsonl`/`gym.log` as a build artifact on every run (mirrors `gpu_e2e_tests`), for debugging future failures. ## Test plan - [ ] `fireworks-e2e` CI job passes on this PR --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## Summary - Bump Pillow from `~=11.0.0` to `==12.3.0` in `responses_api_agents/osworld_agent/requirements.txt`, matching the pin already used across every other agent/resource server in the repo. - `osworld==0.1.0` transitively pins `pillow>=11.0.0,<11.1.dev0`, which made the plain version bump unresolvable. Added an `overrides.txt` entry to bypass that stale pin, following the same pattern used in `stirrup_agent`. ## Test plan - [x] `uv pip install -r requirements.txt --override overrides.txt` resolves cleanly and installs `pillow==12.3.0` (verified locally) - [x] `uv lock --check` passes (unaffected — this server sits outside the uv workspace) - [ ] CI passes --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com>
moves gsm8k w calculator, hotpotqa w search, bixbench and bbh to environments --------- Signed-off-by: Christian Munley <cmunley@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? <!-- Briefly describe the change and the motivation. Link any related issue, e.g. "Closes #123". --> ## Checklist - [ ] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [ ] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [ ] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [ ] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [ ] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
…ext search (#2722) ## Summary Adds a local backend for `edgar_search` in `finance_sec_search`, answering full-text queries over SEC filings from a read-only SQLite FTS5 index. The tool name, description and parameter schema match the hosted equivalent, so an existing dataset runs against it unmodified. Opt-in and additive: - `local_edgar_index_path` enables it; unset (the default) and `edgar_search` reports itself unavailable. - `convert_questions.py` still defaults to `sec_filing_search`, so `data/example.jsonl` is unchanged. ## What's included - `local_edgar_search.py` — query translation, filtering, paging, ranking. - Optional metadata sidecar built by `scripts/build_local_edgar_metadata.py`, validated at startup against a fingerprint of its source index. On a ~570k document index it takes common queries from tens of seconds to under one. - `convert_questions.py --search-tool` to choose the filing-search tool. - `docs/local_edgar_index.md` — the index schema, the column formats it depends on, and how to obtain an index. No builder ships here; an index is a self-contained file plus its sidecar and can be copied between machines. ## Training Used for GRPO training runs against a held-out financial analysis benchmark (not included here), 3 seeds, mean ± std across seeds. Baseline is the untrained model with the same tool set, so the comparison is like for like. | | Baseline | Step 6 | Step 16 | Step 18 | Step 21 | |---|---|---|---|---|---| | Balanced accuracy | 19.58 ± 2.26 | 21.38 ± 1.53 | 24.44 ± 4.50 | 25.45 ± 2.08 | 26.23 ± 1.11 | | Accuracy | 24.50 ± 1.50 | 25.67 ± 2.02 | 29.17 ± 3.51 | 30.83 ± 2.52 | 30.83 ± 1.26 | | Turns / question | 19.4 ± 0.4 | 19.5 ± 1.2 | 18.1 ± 0.4 | 17.0 ± 0.5 | 17.0 ± 0.5 | | Output tokens / question | 21293 ± 203 | 20742 ± 756 | 13214 ± 429 | 11973 ± 295 | 11358 ± 132 | | `edgar_search` calls | 3.34 ± 0.10 | 3.33 ± 0.29 | 2.99 ± 0.29 | 3.02 ± 0.22 | 2.75 ± 0.08 | | `parse_html_page` calls | 3.30 ± 0.22 | 3.24 ± 0.23 | 2.85 ± 0.06 | 2.70 ± 0.11 | 2.66 ± 0.07 | | `retrieve_information` calls | 4.61 ± 0.28 | 4.57 ± 0.28 | 4.53 ± 0.07 | 4.33 ± 0.14 | 4.25 ± 0.09 | | `web_search` calls | 7.20 ± 0.14 | 7.42 ± 0.59 | 6.76 ± 0.26 | 6.00 ± 0.17 | 6.39 ± 0.29 | Accuracy rises from 24.50 to 30.83 while output tokens per question fall by 47%, with fewer calls to every tool. ## Testing `ng_test +entrypoint=resources_servers/finance_sec_search` — 84 passed. New coverage: query translation, request normalization, date/form/CIK filtering, paging, ranking order, filter-only browsing, sidecar build and fingerprint rejection, latency metrics, and the server route with and without an index configured. --------- Signed-off-by: Ushnish De <ude@nvidia.com> Co-authored-by: Christian Munley <cmunley@nvidia.com>
## Summary - replace the benchmark-local, user-wide cleanup helper with a standalone OpenSandbox cleanup CLI that requires an exact run and user scope - attach the Slurm run and user attribution to every sandbox created by the evaluation - submit one CPU-only `afterany` cleanup job immediately after the evaluation job, giving cleanup its own allocation and time budget after success, failure, timeout, or cancellation - paginate the full sandbox inventory, delete matches concurrently through one bounded connection pool, and repeat the sweep to handle teardown races - keep credentials in the existing connection config; the dependent job receives only its path and the exact cleanup scope ## Test plan - [x] focused cleanup and launcher tests: 37 passed locally, cleanup module at 100% coverage - [x] Bash 5 launcher behavior checks for workload failure, TERM cancellation, and server-first exit - [x] scoped pre-commit hooks, shell syntax check, and `git diff --check` - [x] live Slurm dependency probe: plain parent cancellation released the `afterany` CPU job and inherited GPU requests were overridden - [x] live cancellation E2E with real attributed rollouts plus controlled isolation fixtures: matching target deleted with HTTP 2xx, same-user wrong-run decoy preserved, cleanup completed successfully, and the final exact-scope audit found zero survivors - [x] fixture teardown verified no test fixtures remained --------- Signed-off-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…a connection (#2558) ## What `pty.exec(detach=True)` runs a command without holding a connection while it works: the command starts in a PTY session, the WebSocket is dropped, and the session is briefly re-attached every `poll_interval_s` to check for completion. A long-running command occupies a connection for milliseconds per poll instead of its whole runtime. ## Why At eval scale, session-mode PTY holds one WebSocket per rollout for the rollout's lifetime — thousands of standing connections held for hours. Detached exec replaces that with a handful of short-lived polls (~200 ms each), a ~100x reduction in standing connections, and structurally avoids the failure modes long-lived sockets are exposed to (load-balancer idle timeouts on quiet sessions, server deploys interrupting streams). ## How - `OpenSandboxPtySession.detach()` / `reattach()`: built on the existing resume machinery (`takeover=1`, `since=<bytes received>`). execd sessions run fine with no client attached — the socket is a view, not the session's lifeline. A detached session is not `closed`, so provider pruning leaves it alone; `close()` still releases and ends it. - `pty.exec(..., detach=True, poll_interval_s=...)`: same marker discipline as session-mode exec, plus file capture inside the sandbox (`>cap.out 2>cap.err`) because the server retains only ~1 MiB of terminal output across a detach. Output is `cat`-collected on completion, so stdout/stderr come back separated in both pty and pipe modes. A fast command that finishes within the first quiet window never detaches at all. - Without `session` a private session is opened (never registered as the default-shell session) and closed afterwards. An explicitly passed session is detached while the command works and comes back attached and reusable. ## Testing - Unit: wire-level detach/reattach (no DELETE on detach, `since`/`takeover` on re-dial, prune safety, close-after-detach) and facade-level detached exec (poll cycle, fast path, private-session lifecycle, timeout parity with `exec()`). 107 tests passing in `test_opensandbox_pty.py` + `test_sandbox.py`. - E2E SWE-bench eval with the agent command running detached, with connection-count telemetry: in progress, will post results here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? 1. Fix DeepSWE from @nv-mengxiwu : resources_servers/deepswe/app.py 2. Fix OpenCode performance regression: responses_api_agents/opencode_sandboxed_agent/app.py ## Checklist - [ ] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [ ] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [ ] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [ ] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [ ] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). Signed-off-by: Brian Yu <bxyu@nvidia.com>
## Summary - Add the required `type: "message"` discriminator to the tutorial's assistant output fixture. - Mark the message as completed so the copied verifier test matches the current Responses schema. ## Test plan - [x] Replayed the tutorial scaffold, app, config, data, and tests - [x] Ran `gym env test --resources-server my_weather_tool`. - [x] Ran `cd fern && npm run check`. Closes #2695. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…me (#2786) ## Problem In `benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh` the reported model name is derived from the serve path — one variable feeds both: ```bash vllm serve "$MODEL" ++policy_model_name=$MODEL ``` Under the current coupling, the only way to give a run a meaningful name is to rename the mount, and that is a trap: anything else addressing that mount by path (`--chat-template /checkpoint/...`, `--reasoning-parser-plugin /checkpoint/...`) silently points at a path that is no longer mounted, and vLLM fails to load it. We hit exactly that. ## Change `MODEL_NAME` names the model independently of where it is mounted: - vLLM serves under it (`--served-model-name`), so the endpoint answers to it. - The eval sends it (`++policy_model_name`), so the two cannot disagree. This lets the caller keep the mount fixed and still say which checkpoint a run used. Tested on a run --------- Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
`save_model_call_using_vllm_tokenize_endpoint` reads
`len(response["tokens"])`, which only exists on older vLLM builds;
mainstream >=0.19.1 returns `{"count": N, "max_model_len": ...}` and
omits the token list unless token ids are requested, so the pre-call
context-reset estimation dies with KeyError: 'tokens' on every sample.
Prefer the explicit count, fall back to the token list, and fail loudly
when neither is present.
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections
below. -->
## What does this PR do?
<!-- Briefly describe the change and the motivation. Link any related
issue, e.g. "Closes #123". -->
## Checklist
- [x] I have read the [contributing
guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup).
- [x] The change is focused; unrelated "drive-by" edits are tracked as
separate issues/PRs.
- [x] Tests added or updated and pass locally, or N/A for docs-only /
non-code changes (so CI unit/server checks pass when applicable).
- [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so
CI lint/format/copyright pass).
- [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check
passes).
---------
Signed-off-by: MJ Mikulski <mmikulski@nvidia.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## What does this PR do? Implements the [Rollout Quality Verification in NeMo Gym RFC](https://rfc.frontier-evals.nvidia.com/r/gym-rollout-verification) as an automatic, report-only health pass after evaluation. The implementation adds: - a typed check registry (`CheckSpec`) and structured evidence (`Finding`) - a focused internal package: `health/types.py` for contracts and `health/checks.py` for canonical evidence handling and checks, while `rollout_health.py` remains the public orchestration workflow - runner-derived `healthy`, `unhealthy`, and `unobserved` verdicts; checks never emit verdicts - a three-phase runner that indexes JSONL byte offsets, evaluates rollouts in a process pool, then deterministically reduces per-rollout digests into run/task reports; no Ray dependency - `quality_summary.json` and `rollout_verdicts.jsonl`, keyed by `_ng_task_index` and `_ng_rollout_index` - `gym eval health-check <run-dir>` for standalone verification - automatic health checks after `gym eval run` and `gym eval aggregate` - `--no-health-check`, worker-count controls, and explicit per-check exclusions for historical-corpus analysis - a short terminal summary without changing collection output, scores, or aggregate metrics Refs #2136. ## Canonical observability boundary Health reads exactly one observability representation: the final `TrajectoryRecord` persisted under each rollout record's `ng_trajectory` key. It never discovers or reads model-call sidecars, `ng_model_call_capture`, agent-specific transcripts, invocation conversations, or legacy `response.output` shapes. Gym's existing observability stack may use those sources internally during collection. Before the rollout is written, it normalizes them into: - `ng_trajectory.turns` for agent turns - `ng_trajectory.model_calls` for model-call evidence - `ng_trajectory.gaps` for evidence that could not be collected or joined exactly Health then applies one conservative rule: missing canonical evidence makes only the dependent checks unobserved, while an explicit contradiction in the canonical trajectory produces a finding. For example, `turns_unavailable` makes turn checks unobserved; incomplete model-call evidence makes binding-dependent checks unobserved; and explicit unmatched, ambiguous, or conflicting model-call references produce `trajectory_capture_mismatch` findings. Unreferenced calls do not fail because they may belong to a judge, user simulator, or another auxiliary model. ## Check registry The RFC's check families are represented as single-responsibility IDs. The correspondence family is split because missing token fields, reference contradictions, failed calls, and total-token disagreements have different subjects and missing-input behavior. `record_unreadable` is an explicit extension for artifact parse failures. `check_execution_error` separately reports an unexpected exception in check code, so it is not misattributed to a successfully parsed record or semantic check. `rollout_duplicate_identity` preserves and flags physical records that claim the same logical rollout identity while counting that identity only once for task-level repeat checks. | Check | Evaluation scope | Finding subject | |---|---|---| | `check_execution_error` | rollout | check execution | | `record_unreadable` | rollout | rollout record | | `rollout_duplicate_identity` | rollout | rollout | | `rollout_missing_agent_turns` | rollout | rollout | | `agent_turn_hollow` | rollout | agent turn | | `model_call_zero_completion_tokens` | rollout | model call | | `model_call_missing_token_counts` | rollout | model call | | `trajectory_capture_mismatch` | rollout | trajectory/model-call binding | | `model_call_failed` | rollout | model call | | `rollout_token_count_mismatch` | rollout | rollout | | `model_call_runaway_generation` | rollout | model call | | `task_consistently_unhealthy` | task | task | | `task_no_successful_model_calls` | task | task | ## Verdict semantics `healthy` is intentionally a strong claim: 1. any finding makes the rollout `unhealthy`; 2. otherwise, any enabled check lacking required evidence makes it `unobserved`; 3. only a fully evaluated rollout with no findings is `healthy`. Turn checks require canonical `TrajectoryTurn` evidence. Policy-model checks use only canonical model calls explicitly referenced by canonical turns. A turn without a model-call reference is not itself a failure; binding-dependent checks become unobserved when no usable references exist. ## CLI ```bash # Check an existing run directory gym eval health-check <run-dir> # Name a nonstandard rollout file explicitly gym eval health-check <run-dir> --rollouts-file evaluator_rollouts.jsonl # Limit worker processes or exclude known checks for one analysis gym eval health-check <run-dir> --workers 4 \ --ignore-checks model_call_missing_token_counts,rollout_token_count_mismatch # Opt out of the automatic post-run/post-aggregate pass gym eval run ... --no-health-check gym eval aggregate ... --no-health-check ``` Automatic paths also expose `--health-check-workers` and `--health-check-ignore`. Ignored checks remain registered but do not execute or affect findings, unobserved states, verdicts, or task flags. The summary records which checks were ignored and their coverage counts. ## Output guarantees - Reports are deterministic and sorted by task/repeat identity. - Every non-empty input line receives a verdict row, including malformed records. - Missing canonical observability degrades dependent checks to unobserved rather than false failure. - Duplicate task/repeat identities remain visible at run scope but are collapsed before task-level repeat reduction. - If a process pool is unavailable, the same work completes serially with a warning. - Health checks do not mutate rollout collection artifacts or aggregate metrics. ## Validation - `pytest tests/unit_tests/test_rollout_health.py --cov=nemo_gym.rollout_health --cov=nemo_gym.health --cov-fail-under=96 -q` — 42 passed; 98.43% coverage - focused CLI, collection, observability, and model-capture tests — 378 passed - `uv build` — sdist and wheel pass; the wheel contains `nemo_gym.health` - scoped pre-commit hooks pass for every changed file - full local unit suite — 2,779 passed, 41 skipped; three existing failures outside the touched paths remain on macOS (port binding, Linux `/proc`, and an AIME metadata fixture) Synthetic tests cover every registered semantic finding, each missing canonical evidence state, canonical bindings and gap translation, malformed records, duplicate identities, task reduction, serial fallback, CLI/config validation, and ignored checks. A golden test proves that collection artifacts and aggregate metrics are byte-identical with automatic health checks enabled and disabled. Historical VPR corpora used during development predate persisted `ng_trajectory`; under this final single-source contract they correctly report the trajectory-dependent checks as unobserved instead of being heuristically retrofitted. Positive real-corpus validation now requires runs produced by the current observability stack. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated drive-by edits are tracked separately. - [x] Tests added or updated and pass locally, subject to the documented pre-existing full-suite exceptions. - [x] Pre-commit checks pass for the changed files. - [x] All commits have DCO sign-off (`git commit -s`). --------- Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>
…2765) ## Summary Background-command **status polls** are sub-second idempotent GETs, but they inherit the general `request_timeout_s` budget (tuned for long submits/creates, often minutes). Against a sandbox whose pod has become unreachable (a blackholed IP that swallows SYNs), each poll then hangs for a full TCP connect timeout before failing — retries × minutes per dead sandbox before the typed unreachable error fires, tying up a rollout slot the whole time. ### feat: dedicated short timeout for background status polls - New `operations.status_poll_timeout_s: float | None = 10.0`: per-request budget for background-command **status polls only**. A healthy poll answers in milliseconds and even the slow legitimate path (endpoint re-resolution plus a connect retry) is a few seconds. `None` restores the previous behavior; submits keep their `timeout_s + 60` headroom and the logs fetch keeps the shared budget. - A per-call timeout surfaces as builtin `TimeoutError`, which the default retry classifier deliberately treats as terminal (a long timeout has already burned its budget — and `TimeoutError` subclasses `OSError`, which would otherwise be retried). With a 10s budget that would make a single slow poll rollout-fatal, so `_await_sdk_operation` gains a pluggable `is_retryable` predicate (default unchanged for all other callers) and the status-poll path passes one that retries timeouts — safe because a status poll is an idempotent GET, unlike a submit where a timeout must stay terminal. - With the retry budget, an unresponsive sandbox is detected in ~1 minute instead of ~25+. ### fix: stop relabeling inner timeouts as the exec hard cap Pre-existing bug the short poll budget makes frequent: since Python 3.11 `asyncio.TimeoutError` IS builtin `TimeoutError`, so the `wait_for`-based hard-cap wrapper around exec dispatch also caught any `TimeoutError` raised *inside* the dispatch (e.g. a status poll exhausting its budget and retries) and relabeled that minutes-scale failure as `exceeded hard cap of <hours>s; the command wedged` — factually wrong, corrupting failure taxonomy. Now uses `asyncio.timeout` and attaches the wedged message only when the cap itself expired (`Timeout.expired()`); inner timeouts propagate with their original message. ## Note on the branch's original scope This PR originally implemented duplicate-submission protection for background commands (adopting an already-scheduled execution after a lost submit response). Per-request tracking deployed at production scale (>115k requests across multiple 1024-way-parallel runs, with body-hash-level duplicate detection) observed **zero** duplicate submissions in practice, so that machinery was withdrawn to keep the provider simple; this PR now ships only the status-poll timeout work above. ## Tests 51 passed on the provider suite: parametrized per-operation timeout routing (poll gets the short budget; submit and logs keep theirs), `None` fallback, poll-timeout retryability, config validation, inner-timeout propagation with original message (fails against the old `wait_for` code), and a genuinely wedged dispatch still receiving the hard-cap label. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…2791) ## Problem `benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh` locates the run-scoped cleanup job's script under `submit_dir=$(pwd -P)`, which assumes the launcher is invoked from a Gym checkout. A caller that launches from a scheduler run directory has no such tree there, so the submission fails outright: ``` sbatch: error: Unable to open file <rundir>/nemo_gym/sandbox/providers/opensandbox/cleanup_sandboxes.py Failed to submit cleanup job for batch job 6540264; the batch job is still active ``` No cleanup job is scheduled and sandboxes are never reaped. This is how NVIDIA's eval-factory benchmarking runner drives the launcher: it stages a copy of the script into a per-run directory and `cd`s there, because that is where `slurm-logs/` and `results/` live. ## Changes 1. `NEMO_GYM_REPO_ROOT` names the checkout, alongside the launcher's other declared inputs at the top of the file. It defaults to the working directory, so a launch from a checkout is unchanged. 2. The connection comes from `OPENSANDBOX_DOMAIN` and `OPENSANDBOX_API_KEY` — the same two variables the run itself resolves `opensandbox.yaml` from. The launcher no longer looks for an `env.yaml` and no longer forwards `--connection-config`; the flag stays on the script for a manual run. 3. A failed cleanup submission no longer strands the main job. It printed no job ID and exited 1, so a caller read a live 4-node job as a failed submission — and resubmitted it. It now prints `Submitted batch job <id>`, warns that its sandboxes need reaping by hand, and exits 0. Dropping the `env.yaml` route is deliberate: that file is merged over every config a run names, so a connection read from it can silently override the evaluated config, and no hash covers it. ## Testing `pytest tests/unit_tests/test_opensandbox_cleanup.py` — 40 passed, 3 skipped. `ruff check` and `ruff format --check` clean on both changed files. ## Supersedes Replaces #2788, rebased onto current `main` and squashed. Same change, opened from a branch whose commits carry a sign-off. --------- Signed-off-by: Piotr Laszkiewicz <plaszkiewicz@nvidia.com> Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com> Co-authored-by: Grzegorz Chlebus <gchlebus@nvidia.com>
…g to Exa (+other cfg changes to align w NEL) (#2779) ## What Replaces the `bash -c` subprocess behind the BrowseComp harness's `bash_command` tool with `pocketshell` — an in-process, workspace-confined read-only shell, vendored into the server directory so the harness stays self-contained. Also aligns `benchmarks/browsecomp/config.yaml` with how the benchmark is actually run. ## Why The previous guard was a command-**name** deny/allow list plus `ulimit -f 0`. It never inspected path arguments, so `cat /etc/passwd` passed it, and its own comment described it as "NOT a security boundary". `pocketshell` cannot execute anything it does not implement, and confines every path to the sample workspace, so the protection is structural rather than advisory. ## Compatibility No accuracy-affecting change is intended: - the legacy deny/allow guard is kept **in front**, unchanged, so anything it used to reject still returns the byte-identical `[blocked: ...]` string; - the output shape (`stdout` / `--- stderr ---` / `[exit_code=N]`) is preserved; - grammar and command coverage were sized against 1.65M real agent-written `bash_command` calls; the unsupported syntax (command substitution, heredocs, process substitution, backgrounding) is exactly what the old guard already rejected. ## Config changes `benchmarks/browsecomp/config.yaml` now reflects the run recipe: - `search_provider: exa` + `max_results: 10` - `progress: true` — the durable progress board becomes the default - `max_run_retries: 3 -> 1` (per-sample retries) - judge moves off `policy_model` onto a dedicated `judge_model`, so a weak policy can no longer grade itself - agent `model_server`: `policy_model_no_interleaved_reasoning` -> `policy_model`; interleaved reasoning is now always on - drops `keep_rounds` (identical to the field default) and `context_reset_pct` (dead whenever `context_reset_tokens > 0`) — both inert - drops the unreferenced `Qwen3-235B-A22B-Instruct-2507-FP8` node API keys use `${oc.select:...,null}` so an unset key resolves to null instead of failing interpolation. ## Testing 215 tests pass (73 harness + 68 pocketshell + 74 agent) against current `main`. `ruff check` and `ruff format --check` clean. All commits DCO signed. --------- Signed-off-by: Ritu Gala <rgala@nvidia.com> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com>
## Summary
This PR adds model-specific serving configurations for Qwen3.5-122B-A10B
and Inkling Small, configurable decode routing and evaluation
concurrency, resumable rollouts, fail-fast multi-node serving, and
benchmark-level failure containment.
## Shared infrastructure and benchmark reliability
- **Configurable decode routing:** allow tuned runs to override the
default `cache_aware` decode-routing policy while preserving that
default for callers that do not set an override.
```bash
VLLM_DECODE_POLICY=power_of_two \
NUM_DECODE_NODES=4 \
bash benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh ...
```
- **Opt-in evaluation concurrency:** pass the number of parallel Gym
samples only when explicitly configured. This enables throughput tuning
without changing existing model runs by default.
```bash
NUM_SAMPLES_IN_PARALLEL=64 \
bash benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh ...
```
- **Resumable evaluations:** use a stable output path and reuse
completed rollouts after a Slurm requeue or manual resubmission with the
same compatible experiment configuration. Note: reuses compatible
completed rollouts when a command is restarted with the same experiment
name. It does not itself request a Slurm requeue.
```bash
RESUME_EVAL_ON_REQUEUE=1 \
EXPERIMENT_NAME="<stable-experiment-name>" \
bash benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh ...
```
- **Fail-fast multi-node serving:** add Slurm's `--kill-on-bad-exit=1`
so one failed vLLM worker terminates the server step and propagates
through the launcher's existing lifecycle monitoring instead of leaving
a partial deployment running. The Gym readiness timeout remains
configurable.
```bash
bash benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh \
--config <config.yaml> \
++model_endpoint_readiness_timeout_seconds=1200
```
- **Tau2 setup and rollout containment:** serialize Tau2 data
initialization; validate model-emitted tool arguments; retry malformed
JSON up to five generations; and, if a rollout still fails, emit a
structured zero-reward failure record instead of an HTTP 500 that aborts
the full suite.
- **Container prerequisite:** install the locked profiling dependency
required by `gym eval prepare` and verify it while building the
evaluation image.
```bash
INPUT_CONTAINER="<base.sqsh>" \
OUTPUT_CONTAINER="<with-gym.sqsh>" \
MOUNTS="<mounts>" \
GYM_CONFIG="<config.yaml>" \
sbatch benchmarks/nemotron_3.5_super/build_eval_container.sh
```
- **OpenCode export paths:** strip the sandbox `pwd` newline before
building the export path, preventing valid `/testbed/export.json`
artifacts from being requested with a trailing newline.
## Qwen3.5-122B-A10B
### Model-specific changes
- Configure Qwen's tool-call and reasoning parsers, multimodal encoder
parallelism, and TP4/EP4 execution.
- Split NIXL KV-cache transfer into prefill-producer and decode-consumer
settings.
- Use one DP1-per-node TP4 prefill server and four DP1-per-node TP4/EP4
decode servers (P1/D4), matching main's launcher topology.
- Use fixed-scale FP8 KV cache, CUDA graphs, synchronous scheduling, and
a 512-sequence engine limit.
- Enable two-token multi-token prediction (MTP, speculative decoding
from the model's prediction heads) on both serving roles when
`QWEN_ENABLE_MTP=1`; matching settings preserve compatible transferred
KV-cache layouts.
- **Implementation:**
`benchmarks/nemotron_3.5_super/vllm_configs/qwen3.5-122b-a10b.sh`.
### How to run
```bash
MODEL=/lustre/fsw/portfolios/llmservice/users/igitman/hf_models/Qwen3.5-122B-A10B \
VLLM_CONFIG=benchmarks/nemotron_3.5_super/vllm_configs/qwen3.5-122b-a10b.sh \
QWEN_ENABLE_MTP=1 \
RESUME_EVAL_ON_REQUEUE=1 \
EXPERIMENT_NAME=super3.5-e2e/qwen3.5-122b-a10b-p1d4-dp1-c64-p2-ep4x4-seqs512-cg256-fp8kv-scale1-sync-nopc-mtp2 \
NUM_SAMPLES_IN_PARALLEL=64 \
NUM_PREFILL_NODES=1 \
NUM_DECODE_NODES=4 \
VLLM_DECODE_POLICY=power_of_two \
EXPORT_TO_CSV=1 \
SBATCH_TIME=<time-limit> \
SBATCH_ACCOUNT=<slurm-account> \
SBATCH_PARTITION=<slurm-partition> \
SBATCH_QOS=normal \
SBATCH_GRES=gpu:4 \
CONTAINER=$(pwd)/results/vllm/vllm-openai:v0.25.1___tomer_with_gym.sqsh \
MOUNTS=/lustre:/lustre,$(pwd):/opt/Gym \
bash benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh \
--config benchmarks/nemotron_3.5_super/eval_container_config.yaml \
++model_endpoint_readiness_timeout_seconds=1200
```
### Validation
The full 11,213-rollout suite completed on HSG GB200 with P1/D4,
concurrency 64, TP4/EP4, MTP2, and zero failed rollouts.
| Serving metric | Average | Minimum | Maximum |
|---|---:|---:|---:|
| Output tokens/s/request | 149.6 | 127.2 | 177.4 |
| Waiting requests | 0 | 0 | 0 |
| GPU KV-cache usage | 10.7% | 3.0% | 14.5% |
| Aggregate output throughput | 8,701 tok/s | 6,400 tok/s | 9,492 tok/s
|
| Benchmark | Score |
|---|---:|
| Tau3-Banking | 14.2% |
| Tau3-Average | 60.1% |
| SciCode — whole problem | 8.1% |
| SciCode — subtask | 38.2% |
| AA-LCR | 65.9% |
| AA-Omniscience (OmniIndex) | 25.4% |
| GPQA Diamond | 85.9% |
## Inkling Small
### Model-specific changes
- Add Inkling's V2 runner, tokenizer mode, FlashAttention CuTe DSL
cache, tool-call parser, reasoning parser, and FlashInfer-autotune
setting from the upstream recipe while intentionally serving the local
BF16 checkpoint.
- Use TP4/EP4, prefix caching, chunked prefill, synchronous scheduling,
and full-decode CUDA graphs.
- Split NIXL transfer roles and use two DP1-per-node prefill servers
plus four DP1-per-node decode servers (P2/D4), matching main's launcher
topology. The second prefill node removes the sustained queue observed
with P1/D4; a larger token budget alone did not remove it.
- Give each prefill engine a 16,384-token scheduling budget and each
decode replica an 8,192-token budget.
- Make MTP draft width configurable and apply it symmetrically to
prefill and decode. MTP2 was selected because its approximately 63%
acceptance avoided the throughput collapse observed when drafting all
eight tokens.
- **Implementation:**
`benchmarks/nemotron_3.5_super/vllm_configs/inkling_small.sh`.
### Container
Inkling required a separate recipe-compatible image layered with Gym and
`vllm-router`. The validated image is
`results/vllm/vllm-openai:nightly-inkling-small-20260815_with_gym.sqsh`,
built from vLLM `0.27.2rc1.dev77+gac7509e2b`. The recipe's compatibility
floor is vLLM 0.26; this records the exact newer build tested.
### How to run
```bash
MODEL=/lustre/fsw/portfolios/llmservice/users/lvega/models/Inkling-Small \
VLLM_CONFIG=benchmarks/nemotron_3.5_super/vllm_configs/inkling_small.sh \
INKLING_ENABLE_MTP=1 \
INKLING_MTP_NUM_SPECULATIVE_TOKENS=2 \
RESUME_EVAL_ON_REQUEUE=1 \
EXPERIMENT_NAME=super3.5-e2e/inkling-small-bf16-p2d4-dp1-c64-p2-ep4x4-seqs256-cg256-sync-pc-mtp2-pbt16384 \
NUM_SAMPLES_IN_PARALLEL=64 \
NUM_PREFILL_NODES=2 \
NUM_DECODE_NODES=4 \
VLLM_DECODE_POLICY=power_of_two \
EXPORT_TO_CSV=1 \
SBATCH_ACCOUNT=<slurm-account> \
SBATCH_PARTITION=<slurm-partition> \
SBATCH_QOS=normal \
SBATCH_TIME=<time-limit> \
SBATCH_GRES=gpu:4 \
CONTAINER=$(pwd)/results/vllm/vllm-openai:nightly-inkling-small-20260815_with_gym.sqsh \
MOUNTS=/lustre:/lustre,$(pwd):/opt/Gym \
bash benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh \
--config benchmarks/nemotron_3.5_super/eval_container_config.yaml \
++model_endpoint_readiness_timeout_seconds=1200
```
Note: The full suite also requires the externally staged, gitignored
`benchmarks/scicode/data/test_data.h5` fixture.
### Validation
The validation run completed all 11,213 rows on HSG GB200 in 2:30:13. It
produced 11,212 normal rows and one structured failure caused by an
external Omniscience judge HTTP 500, not by Inkling or vLLM.
| Serving metric | Result |
|---|---:|
| Output tokens/s/request | 107.06 mean; 96.77 p5; 89.74 minimum;
826/827 loaded windows at least 90 |
| Derived local decode queue | 0 in all 3,432 samples |
| Literal decode waiting | NIXL-transfer deferrals only; per-replica
maxima 2, 3, 4, and 6 |
| Prefill waiting | Nonzero in 14/1,718 samples; brief maxima 8 and 5;
no sustained buildup |
| GPU KV-cache usage | 3.74-3.97% mean; 8.04-9.45% p95; 14.2-18.9%
maximum |
| Benchmark | Score |
|---|---:|
| Tau3-Banking | 11.3% |
| Tau3-Average | 59.5% |
| SciCode — whole problem | 16.9% |
| SciCode — subtask | 44.5% |
| AA-LCR | 66.9% |
| AA-Omniscience (OmniIndex) | 33.8% |
| GPQA Diamond | 87.9% |
- **Note on CRITPT batching:** Main commit `54cee4f1` (merge commit
`ef1dd91b`) enabled CRITPT in the Super suite. CRITPT's public scorer
waits for 70 distinct problems before scoring a batch. The suite
interleaves 5x70-item batches. At concurrency 64 (our tuned setting for
optimal output tokens/s/request) every Gym slot can become occupied by a
rollout waiting for an incomplete batch and block eval progress.
To clear CRITPT, you can cancel and resubmit with
`RESUME_EVAL_ON_REQUEUE=1` and `NUM_SAMPLES_IN_PARALLEL=350`. This will
lower output tokens/s/request but provide enough slots for all 350
CRITPT rollouts to complete their batches. Once all 350 CRITPT rows have
been scored and written to `resumable.jsonl`, either cancel and resume
at concurrency 64 or allow the run to finish at concurrency 350.
---------
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Co-authored-by: Brian Yu <bxyu@nvidia.com>
Contributor
Merges `main` at b793621 ("Super 3.5 vllm model tuning (Qwen3.5-122B-A10B, Inkling-Small)", #2599) rather than the current `main` tip. WHY THIS TARGET b793621 is the commit immediately before 65129dd ("chore(deps): pin openai to 2.44.0", #2456). That PR added class NeMoGymChatCompletionCreateParamsNonStreaming(BaseModel): model_config = ConfigDict(extra="forbid") which makes the proxy reject every agentic chat-completions request that carries `chat_template_kwargs` — the field agentic GDPVal runs use to drive thinking budgets on self-hosted models. Each such request comes back as 422 Unprocessable Entity {"detail":[{"type":"extra_forbidden","loc":["body","chat_template_kwargs"], "msg":"Extra inputs are not permitted","input":{"thinking":true}}]} so the rollout fails before it reaches the model. Merging `main` past 65129dd would take that regression into the GDPVal branch and break the production runs this branch exists to serve. b793621 was chosen because it is the last commit that is regression-free while still giving the branch everything else it needs: * `requires-python = ">=3.13.14"`, so the py313 runtime container stays correct; * `extra="forbid"` appears exactly once in nemo_gym/openai_utils.py, on `NeMoGymResponseCreateParamsNonStreaming` (the pre-existing 2025 Responses-class validation) and NOT on the ChatCompletion class; * the rollout-observability work (bc521f7) is an ancestor. CONFLICTS AND RESOLUTIONS Eight files conflicted; all GDPVal files auto-merged. Every conflict was resolved to keep both sides' behaviour rather than pick a winner. * nemo_gym/cli/env.py, cli/setup_command.py and their tests: `get_venv_path` and `resolve_server_venv_path` are the same function under two names (byte-identical bodies). Canonicalised on devel's name and docstring, kept main's `Path(...)` normalisation of `root_venv_path`, renamed main's call sites and the `monkeypatch.setattr` in test_cli.py, and dropped the now-unused `ROOT_DIR` import. * nemo_gym/openai_utils.py: took main's content-part types — its `video_url` part accepts `Union[str, Dict[str, Any]]` where devel's was dict-only, and it adds `NeMoGymChatCompletionContentPartFileParam` — while keeping devel's comment recording the 422 that motivated adding a video part at all. * responses_api_models/local_vllm_model/setup.py: both sides pin vllm==0.24.0 at this merge target, so the resolution is main's file (which additionally pins flashinfer-python==0.6.12) plus devel's note that 0.24.0 is the first release with MiniMax-M3 support. * nemo_gym/rollout_reverification.py: plain union of both import lists from nemo_gym.rollout_collection; dropped devel's `NG_TERMINAL_KEY as NG_TERMINAL_KEY` re-export idiom, nothing needs it. * nemo_gym/rollout_collection.py: kept both sides' features — main's exporters (`upload_rollouts`/`export_rollouts`/`get_exporters`, replacing the W&B-specific path) and token-capture retirement, devel's dispatch budget, drain margin and `DispatchLatencyTracker`, `kill_shaped` no-persist rows, `_validate_dispatch_concurrency` and `ordered_tasks`. Three resolutions here are semantic rather than textual: - main's `from time import time` SHADOWS the `time` module that devel's `time.monotonic()` calls need, so the module import was kept and main's two bare `time()` calls rewritten as `time.time()`; taking main's side verbatim compiles and then fails at runtime; - the persistence branch takes main's flat `if no_persist / elif failure_class is not None / else` chain with its token-capture retirement, dropping devel's `result_strs.append` (it fed the removed W&B table) and the stale "not the W&B table either" wording in the kill_shaped comment; - devel's `ordered_tasks = [asyncio.ensure_future(...)]` scheduling was kept, because main still passes bare coroutines to `as_completed`, which makes `dispatch_longest_first` a no-op, and the shared code after the conflict references `ordered_tasks`. * tests/unit_tests/test_rollout_collection.py: import union, and devel's version of the fresh-run cleanup test, which is a superset of main's and additionally asserts the aggregate-metrics file is cleared. The fresh-run cleanup in `run_from_config` now unlinks the failures sidecar and the aggregate-metrics file so a re-run cannot inherit retry attempts or published metrics from an older run at the same output path. VERIFICATION `ruff check` and `ruff format --check` with the CI-pinned ruff 0.9.9 are clean across nemo_gym, tests, responses_api_models and resources_servers. resources_servers/gdpval/tests: 386 passed, 11 skipped — the `reference_missing` path (#2796) and `strict_comparison_trials` (#2807) are both intact. tests/unit_tests against an `origin/main` baseline built in a scratch worktree on the same venv: 79 failures on the baseline, 101 here. The 44 new failures are all in test_openai_utils.py (43) plus test_responses_api_model_streaming.py::test_prunes_nested_extra_fields, and are an environment artefact, not a merge defect: b793621 predates #2456 and still declares `openai<=2.7.2`, while the venv has openai 2.44.0 installed. Those tests enumerate the *installed* SDK's item tags (`shell_call`, `apply_patch_call`, `compaction`, `tool_search_call`, ...) and require a Gym union member for each; the pre-2.44 schemas in this tree have none. Re-running the same suite with an openai inside the declared pin (2.7.2 shadowed onto PYTHONPATH, venv untouched) gives 1 failed / 2976 passed — the single failure, test_opensandbox_cleanup.py::test_script_help_runs_by_ direct_path, is also present in the origin/main baseline. So the new-failure set against baseline is EMPTY once the SDK matches the pin, and no failure touches rollout collection, reverification, the CLI or GDPVal. Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>
agronskiy
force-pushed
the
agronskiy/gdpval-devel-with-main
branch
from
September 1, 2026 16:07
8cba113 to
21b0408
Compare
github-actions
Bot
added
the
sla:triage-overdue
This was referenced Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
gdpval-develup to date withmain— but deliberately not withmain's tip.What this now merges, and why
This branch merges
mainat b793621 ("Super 3.5 vllm model tuning (Qwen3.5-122B-A10B, Inkling-Small)", #2599), the commit immediately before 65129dd ("chore(deps): pin openai to 2.44.0", #2456). An earlier revision of this PR merged the tip ofmain; that has been rebuilt.#2456 added
which makes the proxy reject every agentic chat-completions request carrying
chat_template_kwargs— the field agentic GDPVal runs use to drive thinking budgets on self-hosted models. Every such request comes back asso the rollout dies before it reaches the model. Merging past 65129dd would import that regression into the branch the production GDPVal runs are pinned to.
b793621 is the last commit that is free of the regression while still giving the branch everything else it needs:
requires-python = ">=3.13.14", so the py313 runtime container stays correct;extra="forbid"appears exactly once innemo_gym/openai_utils.py— onNeMoGymResponseCreateParamsNonStreaming(the pre-existing 2025 Responses-class server-side validation) and not on the ChatCompletion class;Conflicts
Eight files conflicted; every GDPVal file auto-merged. Each conflict was resolved to keep both sides' behaviour rather than pick a winner.
get_venv_pathandresolve_server_venv_pathturned out to be the same function under two names (byte-identical bodies), canonicalised on devel's name and docstring with main'sPath(...)normalisation, main's call sites and themonkeypatch.setattrintest_cli.pyrenamed, and the now-unusedROOT_DIRimport dropped.openai_utils.pytook main's content-part types — itsvideo_urlpart acceptsUnion[str, Dict[str, Any]]where devel's was dict-only, and it addsFile— while keeping devel's comment recording the 422 that motivated adding a video part at all.rollout_collection.pynow carries main's exporters (upload_rollouts/export_rollouts/get_exporters, replacing the W&B-specific path) and token-capture retirement alongside devel's dispatch budget, drain margin,DispatchLatencyTracker,kill_shapedno-persist rows,_validate_dispatch_concurrencyandordered_tasks.rollout_reverification.pyis a plain union of both import lists, dropping devel'sNG_TERMINAL_KEY as NG_TERMINAL_KEYre-export idiom.Three resolutions are semantic rather than textual and are worth a reviewer's eye:
from time import timeshadows thetimemodule that devel'stime.monotonic()calls need. The module import was kept and main's baretime()calls rewritten astime.time()— taking main's side verbatim compiles and then fails at runtime.if no_persist / elif failure_class is not None / elsechain with its token-capture retirement, dropping devel'sresult_strs.append(it fed the removed W&B table) and the stale "not the W&B table either" wording in thekill_shapedcomment.ordered_tasks = [asyncio.ensure_future(...)]scheduling was kept, becausemainstill passes bare coroutines toas_completed, which makesdispatch_longest_firsta no-op, and the shared code after the conflict referencesordered_tasks.Fresh-run cleanup in
run_from_confignow unlinks both the failures sidecar and the aggregate-metrics file, so a re-run cannot inherit retry attempts or published metrics from an older run at the same output path;test_rollout_collection.pykeeps devel's version of that test, which is a superset of main's.Two resolutions changed relative to the previous (main-tip) revision of this PR, because the older merge target predates the code they applied to.
local_vllm_model/setup.py: both sides pinvllm==0.24.0here, so there is no 0.24.0-vs-0.25.1 decision to make — the file is main's (which also pinsflashinfer-python==0.6.12) plus devel's note that 0.24.0 is the first release with MiniMax-M3 support. And the guard on main's "None of the N dispatched rollouts produced a result"RuntimeError, plus the two test fixes for main'srun_examplesagent-name validation and its_ng_rollout_latency_msattachment, are all unnecessary: none of that code exists at b793621.Verification
ruff checkandruff format --checkwith the CI-pinned ruff 0.9.9 are clean acrossnemo_gym,tests,responses_api_modelsandresources_servers. (Newer local ruff builds report drift in files inherited unchanged frommain; under the pinned version there is none, so this revision carries no ruff-cleanup commit.)resources_servers/gdpval/tests: 386 passed, 11 skipped — thereference_missingpath (#2796) andstrict_comparison_trials(#2807) are both intact.tests/unit_testsagainst anorigin/mainbaseline built in a scratch worktree on the same environment: 79 failures on the baseline, 101 here. The 44 new ones are all intest_openai_utils.py(43) plustest_responses_api_model_streaming.py::test_prunes_nested_extra_fields, and are an environment artefact rather than a merge defect — b793621 predates #2456 and still declaresopenai<=2.7.2, while the venv hasopenai==2.44.0installed. Those tests enumerate the installed SDK's item tags (shell_call,apply_patch_call,compaction,tool_search_call, …) and require a Gym union member for each; the pre-2.44 schemas in this tree have none. Re-running the same suite with an SDK inside the declared pin (2.7.2 shadowed ontoPYTHONPATH, venv untouched) gives 1 failed / 2976 passed, and that one failure —test_opensandbox_cleanup.py::test_script_help_runs_by_direct_path— is also present in theorigin/mainbaseline. The new-failure set against baseline is therefore empty, and no failure touches rollout collection, reverification, the CLI or GDPVal.🤖 Generated with Claude Code