feat(evaluator): add NeMo Gym runner for agent-eval

packages/nemo_evaluator_sdk/examples/gym/README.md

@@ -0,0 +1,86 @@
+# Run a NeMo Gym environment through NeMo Evaluator
+
+`run_gym_eval.py` runs an **existing** NeMo Gym environment (the `mcqa` benchmark by default) through the Evaluator's `GymAgentTaskRunner` and scores it with `AgentEvaluator`. Use it when you already have a Gym environment and want to run and score it through NeMo Evaluator without migrating it. Gym owns execution *and* scoring; the runner shells out to the `gym` CLI and adapts the rollout bundle into trials, and `GymRewardMetric` surfaces Gym's per-attempt reward.
+
+Mapping: one Gym dataset → one run; each distinct row → one `AgentEvalTask` (id = content hash of the row); each attempt (`--num-repeats`) → one `AgentEvalTrial`. So `--num-repeats 2` over a 5-row dataset yields 5 tasks × 2 = 10 trials.
+
+## Prerequisites
+
+A working **NeMo Gym checkout** is required — Gym resolves its environments from the repo, not from a package install. From the Gym checkout:
+
+```bash
+# 1. Gym venv + framework (Gym pins uv >= 0.9.30; the workspace floor must satisfy that)
+uv venv --python 3.12 .venv
+uv pip install --no-config --python .venv/bin/python -e ".[dev]" "ray[default]>=2.55.1"
+
+# 2. The target env's own deps (each resources_server ships a requirements.txt)
+uv pip install --no-config --python .venv/bin/python tiktoken        # mcqa needs tiktoken
+
+# 3. Model credentials for the collector — a gitignored env.yaml at the Gym repo root:
+cat > env.yaml <<'YAML'
+policy_base_url: https://<your-openai-compatible-endpoint>/v1
+policy_api_key: <key>
+policy_model_name: <model, e.g. nvidia/meta/llama-3.3-70b-instruct>
+YAML
+```
+
+> `env.yaml` is gitignored by the Gym repo — the credentials stay local.
+
+## Run
+
+From the nemo-platform repo root (any Python with `nemo_evaluator_sdk` importable — the runner shells out to Gym's own venv):
+
+```bash
+python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval --gym-root /path/to/Gym
+```
+
+Useful flags: `--resources-server`, `--agent`, `--model-type` (`inference_provider` for OpenAI-compatible **chat** endpoints; `openai_model` uses the OpenAI **Responses API** and 500s against chat-only endpoints), `--num-repeats`, `--output-dir`.
+
+For the full set of knobs the underlying `gym env start` / `gym eval run` commands accept, see the [NeMo Gym documentation](https://github.com/NVIDIA-NeMo/Gym). Anything `GymRuntimeConfig` does not expose as a field can be passed through with its `env_overrides` escape hatch (Hydra `+key=value` overrides applied to `gym env start`).
+
+Each run writes its bundle to a fresh temporary directory by default. Pass `--output-dir` to choose one, but give every run its own: the runner refuses to reuse a directory that already holds Gym rollout output (Gym appends to its failures sidecar, so reusing one would mix runs) and raises rather than clearing a prior run's results.
+
+Expected output:
+
+```text
+discovered 5 tasks from .../resources_servers/mcqa/data/example.jsonl
+=== RESULT ===
+tasks: 5  trials: 10
+aggregate scores:
+  gym_reward.reward: mean=0.6
+
+Run bundle (run.json, trials.jsonl, scores.jsonl, report.html): /var/folders/.../gym-eval-ab12cd34
+```
+
+## How it runs Gym
+
+The runner uses Gym's **two-step** flow, which reads a dataset file directly (no split-driven data-prep, no HuggingFace downloads):
+
+1. `gym env start …` — brings up the resources-server + agent + model servers.
+2. `gym eval run --no-serve --input <dataset> …` — collects rollouts against them.
+
+It does **not** import `nemo_gym` and does **not** handle secrets — it invokes the `gym` executable in your checkout, and Gym reads credentials from that checkout's `env.yaml`.
+
+The dataset handed to step 2 is not your source file. The runner **materializes** a normalized one into the run's work directory: one row per requested task, with `_ng_task_index` stamped explicitly. Gym honors a caller-supplied `_ng_task_index` (it only assigns one when a row lacks it) and echoes it back on every rollout record, so rollouts join back to tasks through a map the runner owns rather than a guess about Gym's internal row ordering. It also means running a *subset* of tasks only rolls out that subset.
+
+### Logs
+
+Gym's subprocess output is streamed to files in the run's work directory — `gym_env.log` for `gym env start`, and `gym_eval.stdout.log` / `gym_eval.stderr.log` for the collection — and mirrored to the `nemo_evaluator_sdk.agent_eval.runtimes.gym_runtime` logger at `DEBUG`. Startup and collection failures name the relevant files and inline the last lines. To watch Gym's output in your own terminal, turn that logger up:
+
+```python
+logging.getLogger("nemo_evaluator_sdk.agent_eval.runtimes.gym_runtime").setLevel(logging.DEBUG)
+```
+
+## Notes & caveats
+
+- **One distinct row is one task.** Task identity is the row's content hash, so duplicate rows collapse into a single task and the runner warns. Repeating a row is *not* how you ask for repeated attempts — `num_repeats` is, since attempts are a run-level concern. Duplicates usually mean a data problem.
+- **Per-env deps are heterogeneous.** mcqa needs only `tiktoken`; other Gym envs pull `torch`/COMET/GPU or docker. The caller is responsible for a Gym runtime whose deps are installed.
+- **`--no-serve --input` bypasses Gym's data-prep** (prompt templating / dataset materialization). This example's `example.jsonl` rows are already complete, so it's faithful; an env whose rows need templating would need that step first.
+- Service-side execution (docker/k8s, Ray provisioning) is out of scope for this SDK path — that's the evaluator plugin's job.
+
+## Next steps
+
+- [`../harbor/`](../harbor/) — the same `AgentEvaluator` seam driven by the Harbor runtime.
+- [`../fabric_harness_runtimes.py`](../fabric_harness_runtimes.py) — the Fabric runtime equivalent.
+- [`gym_runtime.py`](../../src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py) — `GymRuntimeConfig` field reference plus the attribution and teardown rationale.
+- [NeMo Gym documentation](https://github.com/NVIDIA-NeMo/Gym) — authoring environments, agents, and datasets.

packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py

@@ -0,0 +1,113 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Live end-to-end: run an existing NeMo Gym environment through NeMo Evaluator.
+
+Points the Gym runner (:class:`GymAgentTaskRunner`) at an installed Gym checkout and runs a dataset
+(mcqa's bundled ``example.jsonl`` by default) via the two-step Gym flow — ``gym env start`` brings up
+the resources-server + agent + model servers, then ``gym eval run --no-serve --input`` collects
+rollouts against them — and scores the result with :class:`AgentEvaluator`. Gym owns execution *and*
+scoring; :class:`GymRewardMetric` surfaces the per-attempt reward (Evaluator does not re-derive it).
+
+One Gym dataset → one run; each distinct row → one task (id = content hash of the row); each attempt
+(``num_repeats``) → one trial. The runner materializes the tasks into a normalized dataset with
+``_ng_task_index`` stamped per row and hands *that* to Gym, which honors the stamp — so rollouts join
+back to tasks by an explicit map rather than by position (see the README).
+
+Prerequisites (see README.md):
+
+* A **NeMo Gym checkout** whose venv has the target env's deps installed — each env ships its own
+  ``requirements.txt`` (mcqa needs ``tiktoken``; Gym itself needs ``ray`` and ``uv >= 0.9.30``).
+* A **gitignored ``<gym_root>/env.yaml``** holding the model credentials the collector calls
+  (``policy_base_url`` / ``policy_api_key`` / ``policy_model_name``). This SDK never handles secrets —
+  the ``gym`` subprocess reads them from that file.
+
+Run from the repository root::
+
+    python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval --gym-root /path/to/Gym
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import tempfile
+from pathlib import Path
+
+from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
+from nemo_evaluator_sdk.agent_eval.runtimes.gym_runtime import (
+    GymAgentTaskRunner,
+    GymRuntimeConfig,
+    discover_gym_tasks,
+)
+from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig
+
+
+def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument(
+        "--gym-root", required=True, type=Path, help="NeMo Gym checkout; its venv + env.yaml provide deps/creds."
+    )
+    parser.add_argument(
+        "--dataset",
+        type=Path,
+        default=None,
+        help="Dataset jsonl (default: <gym-root>/resources_servers/<resources-server>/data/example.jsonl).",
+    )
+    parser.add_argument("--resources-server", default="mcqa", help="Gym resources-server (environment) name.")
+    parser.add_argument("--agent", default="simple_agent", help="Agent to collect rollouts with.")
+    parser.add_argument(
+        "--agent-config",
+        default="responses_api_agents/simple_agent/configs/simple_agent.yaml",
+        help="Repo-relative agent config passed to `gym env start`.",
+    )
+    parser.add_argument(
+        "--model-type",
+        default="inference_provider",
+        help="`inference_provider` speaks OpenAI-compatible chat; `openai_model` uses the Responses API.",
+    )
+    parser.add_argument("--num-repeats", type=int, default=2, help="Attempts per task (each becomes a trial).")
+    parser.add_argument(
+        "--output-dir",
+        type=Path,
+        default=None,
+        help="Where the run bundle is written (default: a fresh temporary directory). Each run needs its own "
+        "directory — the runner refuses to reuse one that already holds Gym rollout output.",
+    )
+    return parser.parse_args(argv)
+
+
+async def _main(args: argparse.Namespace) -> int:
+    output_dir = args.output_dir or Path(tempfile.mkdtemp(prefix="gym-eval-"))
+    dataset = args.dataset or (args.gym_root / "resources_servers" / args.resources_server / "data" / "example.jsonl")
+    tasks = discover_gym_tasks(dataset)
+    print(f"discovered {len(tasks)} tasks from {dataset}")
+
+    runner = GymAgentTaskRunner(
+        config=GymRuntimeConfig(
+            gym_root=args.gym_root,
+            agent=args.agent,
+            agent_config=args.agent_config,
+            resources_server=args.resources_server,
+            model_type=args.model_type,
+            num_repeats=args.num_repeats,
+        )
+    )
+
+    result = await AgentEvaluator().run(
+        tasks=tasks,
+        target=runner,
+        config=AgentEvalRunConfig(output_dir=output_dir, parallelism=1),
+    )
+
+    print("=== RESULT ===")
+    print(f"tasks: {result.summary.task_count}  trials: {result.summary.trial_count}")
+    print("aggregate scores:")
+    for aggregate in result.summary.scores.scores:
+        print(f"  {aggregate.name}: mean={aggregate.mean}")
+    print(f"\nRun bundle (run.json, trials.jsonl, scores.jsonl, report.html): {output_dir}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(asyncio.run(_main(_parse_args())))