feat: environment registry + 'gym list environments' (#1205 friction #8 / M2) by wprazuch · Pull Request #1635 · NVIDIA-NeMo/Gym

wprazuch · 2026-06-17T10:44:57Z

What

The M2 keystone — discover environments by name from the CLI:

1. nemo_gym/registry.py (new) — discovery only

discover_environments() → name → EnvironmentEntry for every environments/<name>/config.yaml (name = dir), with description/domain read without resolving interpolations/secrets (a config referencing an unset key is still discoverable — no crash).

2. gym list environments (CLI wiring)

list_environments() in cli/env.py renders discovered environments (name / domain / description) as a table, or --json for programmatic use — matching gym list benchmarks / gym list agents (feat: agent registry — name-based agent discovery + composability (M3 core) #1671). Registered in the gym router; the list group is no longer "benchmarks only".

$ gym list environments
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Environment         ┃ Domain ┃ Description                                   ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ workplace_assistant │ agent  │ Workplace assistant multi-step tool-using env │
└─────────────────────┴────────┴───────────────────────────────────────────────┘
$ gym list environments --json
[{"name": "workplace_assistant", "domain": "agent", "description": "..."}, ...]

Why

Epic #1205 friction 8. Directly answers the reviewer asks for "gym list by environment name". The environments/ layout was live but unconsumed in code; this surfaces it in the CLI.

Alignment with the unified CLI (#1434)

Run-by-name for environments is already provided by the unified CLI's generic --environment asset selector (on gym env start / gym eval run, with --search-dir and name/flavor support). So this registry is intentionally discovery-only — an earlier resolve_environment_config_paths was removed as a duplicate of that selector. gym list environments is the piece the asset selectors don't provide.

Tests

test_registry.py (discovery: name/metadata, missing dir, malformed YAML tolerance, unset-secret tolerance, real-repo discovery) + test_cli.py / test_cli_main.py (list_environments table / empty / --json output + router dispatch incl. --json). Smoke-tested gym list environments [--json] end-to-end. ruff + pre-commit clean.

copy-pr-bot · 2026-06-17T10:45:01Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

cmunley1

i feel benchmarks/ should be registered as environments too maybe, not sure.

github-actions · 2026-06-24T07:05:23Z

🌿 Preview your docs: https://nvidia-preview-wprazuch-environment-registry.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

wprazuch · 2026-06-24T14:03:56Z

@cmunley1 I aligned with CLI changes for this PR in d2986b8

The base branch was changed.

…s/ by name Epic #1205 friction #8 (FEP-1022) / RFC milestone M2 keystone. New nemo_gym/registry.py: - discover_environments(): map name -> EnvironmentEntry for every environments/<name>/config.yaml (name = dir), with best-effort description/ domain read WITHOUT resolving interpolations/secrets. - resolve_environment_config_paths(name): the run-by-name primitive — returns the env's config_paths, with a difflib 'did you mean?' on unknown names. Self-contained config-layer module (no CLI/router changes); the foundation for 'gym list' enumerating environments and run-by-name. CLI wiring is a follow-up to coordinate with the unified-CLI epic (#1434). 100% covered. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Consumes nemo_gym.registry in the unified gym CLI: list_environments() in cli/env.py renders discovered environments (name / domain / description) as a table, registered as 'gym list environments' in the router; updates the 'list' group summary (no longer benchmarks-only). Delivers the discoverability the RFC's M2 calls for (reviewers asked for 'gym list' by environment name). Tests: list_environments() output + empty case; router dispatch gym list environments -> nemo_gym.cli.env:list_environments. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…te resolver Align the environment registry with the unified CLI: - Add --json to `gym list environments` (flags=(JSON,) + the get_global_config_dict / model_validate / json.dumps pattern), matching `gym list benchmarks` and `gym list agents`. - Drop resolve_environment_config_paths and EnvironmentNotFoundError: name->config resolution for running is already provided by the CLI`s generic --environment asset selector (with --search-dir and name/flavor support); the registry is now discovery-only (powers the listing). Remove the now- dead resolver tests. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…duplicates the asset selector Per a duplication audit against main: resolve_agent_config_path was dead code in this PR (no CLI wires it) and reinvented the unified CLI`s generic name->config resolution (cli/main.py _asset_config_path) — the same pattern #1635 deliberately removed on the environment side. Drop it (and its AgentNotFound/Variant/NotComposable errors and the duplicate _did_you_mean). This PR is now purely discovery + Pattern A/B classification + `gym list agents`. Run-by-name resolution (with the require_composable guard) will land with its real consumer, the config composer (#1673), as an `agent` asset row rather than a parallel resolver. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

@cmunley1

… core) (#1671) ## What Adds `nemo_gym.agent_registry` + the **`gym list agents`** command — the foundation for RFC **M3 / friction 6** ("every agent×environment combination requires a separate config"). Maps an agent's short `<name>` (directory under `responses_api_agents/`) to its config variant(s) and classifies how each harness composes. ## `gym list agents` (CLI) Lists every discovered agent and its composition pattern — the one thing the unified CLI's generic `--<asset>` selectors can't surface. Table + `--json`, mirroring `gym list benchmarks` / `gym list environments` (#1635). ``` $ gym list agents agent composition variants description simple_agent composable (A) simple_agent swe_agents self-contained (B) swebench_openhands, ... Software engineering tasks ... ... ``` Across the repo: **23 agents — 15 Pattern A (composable), 8 Pattern B (self-contained).** ## API - `discover_agents() -> Dict[str, AgentEntry]` — one entry per agent dir; `AgentEntry(name, path, config_paths, self_contained, description)` with a `.variants` map (variant stem → config). - `resolve_agent_config_path(name, variant=None, *, require_composable=False) -> str` — explicit variant wins; a single config is used directly; a variant named after the agent is the default when several exist; otherwise raises. The run-by-name primitive consumed by the composer (#1673). - Errors: `AgentNotFoundError` (did-you-mean), `AgentVariantError` (zero-config / ambiguous), `AgentNotComposableError` (for the composer's `require_composable` guard). ## Composition classification (`self_contained`) - **Pattern A (`self_contained=False`):** references a *separate* `resources_server` → wireable into a compatible environment (e.g. `simple_agent`, `langgraph_agent`). Which harness↔resources-server pairings are actually compatible is the **composer's** concern, not the registry's. - **Pattern B (`self_contained=True`):** bundles its own environment/framework or external LLM loop (`agent_framework`; e.g. `swe_agents`, `harbor_agent`, `verifiers_agent`, `claude_code_agent`) and runs with its own config. `require_composable=True` rejects these with a clear error. - Non-agent YAML in a `configs/` dir is filtered out; zero-config agents (e.g. `aviary_agent`) are still discovered (their wiring lives in a paired benchmark/resources config). Resolution-safe (`resolve=False, throw_on_missing=False`; never starts servers), so it works with unset secrets. > Reflects @cmunley1's review: classification reframed around Pattern A vs self-contained B, field renamed `composable`→`self_contained`, and cross-pattern compatibility documented as the composer's job (separate commit, history preserved). ## Tests `test_agent_registry.py` (21 cases — A/B classification, external-harness carve-out, zero-config, variant selection, did-you-mean, `require_composable`) + `test_cli_agents.py` / `test_cli_main.py` (`gym list agents` table / empty / `--json` / router dispatch). Part of epic #1205 (M3 / friction 6). --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…tion #12) (#1599) ## What Adds **`gym env validate`** (+ `ng_validate` / `nemo_gym_validate` deprecated shims) — runs the full config parse with **no Ray and no server subprocesses**, then exits **0 (valid) / 1 (invalid)** with a clean, rich-escaped message (**no traceback**). Returns in well under a second instead of after a ~30–60s Ray bootstrap. ```bash gym env validate --config resources_servers/<env>/configs/<env>.yaml --config responses_api_models/<model>/configs/<model>.yaml gym env validate --benchmark gsm8k --model-type openai_model ``` ## How `validate()` lives in `cli/env.py` and is registered as `env validate` in the `gym` router (`cli/main.py` COMMANDS) with the same config-selection flags as `env start` (`--config`, `--benchmark`, `--environment`, `--resources-server`, `--model-type`, `--search-dir`, `--model*`). It reuses the same `get_global_config_dict()` parse path the other commands use, so the validation checks stay in sync: - **config_paths** resolution — missing/typo'd ([#1488](#1488)) and malformed ([#1490](#1490)) - **server cross-references** — unknown `name:` refs ([#1561](#1561)) - **mandatory `???`** values ([#1575](#1575)) - **schema** (`BaseNeMoGymCLIConfig`) Wrapped in `exit_cleanly_on_config_error` (from #1609) so any `ConfigError` becomes a clean message + `exit 1`. A dummy `policy_model` is injected (the `NO_MODEL` parser config, as in `gym list` / `env compose`) so model interpolations like `${policy_base_url}` resolve without real creds — validation is about config **well-formedness**; the real model is supplied by the `--model*` flags at run time. ## Targets `main` Originally drafted on the unified-CLI epic branch; rebuilt directly on `main` now that [#1630](#1630) (and #1637/#1609/#1635/#1671) have merged. The old branch contents (a snapshot of the CLI refactor + unrelated CI commits) were superseded and replaced. ## Scope note The zero-server check ([#1489](#1489), "nothing configured to run") is intentionally **not** part of `validate`: `NO_MODEL` injects a dummy model server (which would defeat the check), and "is anything configured to run" is a *start*-time concern already enforced by `gym env start` before Ray init. `validate` focuses on config well-formedness. ## Why Epic [#1205](#1205) friction #12 (no config validation tooling) — the M1 "fast failure triage" deliverable. Config errors otherwise only surface after Ray starts (~30–60s). ## Tests - `test_cli_main.py`: `gym env validate --config X` routes to `nemo_gym.cli.env:validate` with `+config_paths=[X]` (added to the parametrized config-command matrix). - `test_cli.py`: `validate()` prints OK on a valid config; a raised `ConfigError` becomes `exit 1` (no traceback). - All `test_cli` + `test_cli_main` + `test_cli_legacy` pass (the only failures are the pre-existing Python-3.12 `TestDidYouMean` argparse issue on `main`); ruff + pre-commit clean. Smoke-tested end-to-end: `✓ Config is valid.` on a real benchmark, clean error + `exit 1` on a bad path, and the `ng_validate` deprecation shim. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

… / M2) (#1635) ## What The **M2 keystone** — discover environments by name from the CLI: **1. `nemo_gym/registry.py` (new) — discovery only** - `discover_environments()` → name → `EnvironmentEntry` for every [`environments/<name>/config.yaml`](https://github.com/NVIDIA-NeMo/Gym/tree/main/environments) (name = dir), with `description`/`domain` read **without resolving interpolations/secrets** (a config referencing an unset key is still discoverable — no crash). **2. `gym list environments` (CLI wiring)** - `list_environments()` in `cli/env.py` renders discovered environments (name / domain / description) as a table, or `--json` for programmatic use — matching `gym list benchmarks` / `gym list agents` (#1671). Registered in the `gym` router; the `list` group is no longer "benchmarks only". ``` $ gym list environments ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Environment ┃ Domain ┃ Description ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ workplace_assistant │ agent │ Workplace assistant multi-step tool-using env │ └─────────────────────┴────────┴───────────────────────────────────────────────┘ $ gym list environments --json [{"name": "workplace_assistant", "domain": "agent", "description": "..."}, ...] ``` ## Why Epic [#1205](#1205) friction 8. Directly answers the reviewer asks for "`gym list` by environment name". The `environments/` layout was live but unconsumed in code; this surfaces it in the CLI. ## Alignment with the unified CLI (#1434) Run-by-**name** for environments is already provided by the unified CLI's generic **`--environment`** asset selector (on `gym env start` / `gym eval run`, with `--search-dir` and `name/flavor` support). So this registry is intentionally **discovery-only** — an earlier `resolve_environment_config_paths` was removed as a duplicate of that selector. `gym list environments` is the piece the asset selectors don't provide. ## Tests `test_registry.py` (discovery: name/metadata, missing dir, malformed YAML tolerance, unset-secret tolerance, real-repo discovery) + `test_cli.py` / `test_cli_main.py` (`list_environments` table / empty / `--json` output + router dispatch incl. `--json`). Smoke-tested `gym list environments [--json]` end-to-end. ruff + pre-commit clean. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>

@cmunley1

… core) (#1671) ## What Adds `nemo_gym.agent_registry` + the **`gym list agents`** command — the foundation for RFC **M3 / friction 6** ("every agent×environment combination requires a separate config"). Maps an agent's short `<name>` (directory under `responses_api_agents/`) to its config variant(s) and classifies how each harness composes. ## `gym list agents` (CLI) Lists every discovered agent and its composition pattern — the one thing the unified CLI's generic `--<asset>` selectors can't surface. Table + `--json`, mirroring `gym list benchmarks` / `gym list environments` (#1635). ``` $ gym list agents agent composition variants description simple_agent composable (A) simple_agent swe_agents self-contained (B) swebench_openhands, ... Software engineering tasks ... ... ``` Across the repo: **23 agents — 15 Pattern A (composable), 8 Pattern B (self-contained).** ## API - `discover_agents() -> Dict[str, AgentEntry]` — one entry per agent dir; `AgentEntry(name, path, config_paths, self_contained, description)` with a `.variants` map (variant stem → config). - `resolve_agent_config_path(name, variant=None, *, require_composable=False) -> str` — explicit variant wins; a single config is used directly; a variant named after the agent is the default when several exist; otherwise raises. The run-by-name primitive consumed by the composer (#1673). - Errors: `AgentNotFoundError` (did-you-mean), `AgentVariantError` (zero-config / ambiguous), `AgentNotComposableError` (for the composer's `require_composable` guard). ## Composition classification (`self_contained`) - **Pattern A (`self_contained=False`):** references a *separate* `resources_server` → wireable into a compatible environment (e.g. `simple_agent`, `langgraph_agent`). Which harness↔resources-server pairings are actually compatible is the **composer's** concern, not the registry's. - **Pattern B (`self_contained=True`):** bundles its own environment/framework or external LLM loop (`agent_framework`; e.g. `swe_agents`, `harbor_agent`, `verifiers_agent`, `claude_code_agent`) and runs with its own config. `require_composable=True` rejects these with a clear error. - Non-agent YAML in a `configs/` dir is filtered out; zero-config agents (e.g. `aviary_agent`) are still discovered (their wiring lives in a paired benchmark/resources config). Resolution-safe (`resolve=False, throw_on_missing=False`; never starts servers), so it works with unset secrets. > Reflects @cmunley1's review: classification reframed around Pattern A vs self-contained B, field renamed `composable`→`self_contained`, and cross-pattern compatibility documented as the composer's job (separate commit, history preserved). ## Tests `test_agent_registry.py` (21 cases — A/B classification, external-harness carve-out, zero-config, variant selection, did-you-mean, `require_composable`) + `test_cli_agents.py` / `test_cli_main.py` (`gym list agents` table / empty / `--json` / router dispatch). Part of epic #1205 (M3 / friction 6). --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>

…CLI commands (#1754) Documents the config/CLI features merged for epic #1205 that were not yet reflected in the docs. Requested follow-up: a single doc PR covering the recently merged config/CLI work. ## What's documented | Area | Change | Merged in | |---|---|---| | `reference/cli-commands.mdx` | New `gym list environments`, `gym list agents`, and `gym env validate` sections (+ Quick-Reference entries) | #1635, #1671, #1599 | | `troubleshooting/configuration.mdx` | Three `config_paths` startup errors (Config Path Not Found, Malformed `config_paths`, No Server Instances) + a `gym env validate` pre-flight tip; corrected the stale `ServerRefNotFound` message | #1609, #1599 | | `data/index.mdx` | Unified dataset `source:` block + dedicated "Dataset `source`" subsection + legacy-deprecation note | #1637 | | `data/download-huggingface.mdx` | Auto-download example migrated to `source: { type: huggingface }` | #1637 | | `reference/faq.mdx` | Dataset example and field bullets rewritten around `source:` | #1637 | | `environment-tutorials/single-step-environment.mdx` | Scaffold comment updated to reference `source:` | #1637 | --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

wprazuch changed the title ~~feat(registry): environment registry — discover + resolve environments/ by name (#1205 friction #8 / M2)~~ feat: environment registry + 'gym list environments' (#1205 friction #8 / M2) Jun 17, 2026

This was referenced Jun 17, 2026

feat(cli): run an environment by name — 'gym env run --env <name>' (#1205 friction #8 / FEP-1022) #1636

Closed

feat: environment suite manager — named multi-env sweeps (M4 core) #1643

Draft

wprazuch force-pushed the wprazuch/ng-test-concurrency branch from 2fdee46 to eebfcf8 Compare June 22, 2026 13:45

wprazuch requested a review from a team as a code owner June 22, 2026 13:45

wprazuch force-pushed the wprazuch/environment-registry branch from df7a13d to 6784a52 Compare June 22, 2026 13:48

wprazuch mentioned this pull request Jun 23, 2026

feat: agent registry — name-based agent discovery + composability (M3 core) #1671

Merged

cmunley1 previously approved these changes Jun 23, 2026

View reviewed changes

wprazuch force-pushed the wprazuch/environment-registry branch from eb06d7b to c237f13 Compare June 24, 2026 07:04

wprazuch changed the base branch from wprazuch/ng-test-concurrency to wprazuch/cli-clean-base June 24, 2026 07:04

wprazuch closed this Jun 24, 2026

wprazuch reopened this Jun 24, 2026

wprazuch force-pushed the wprazuch/environment-registry branch 2 times, most recently from e7c8af4 to 42d6da2 Compare June 24, 2026 07:18

wprazuch changed the base branch from wprazuch/cli-clean-base to martas/1434 June 24, 2026 07:18

wprazuch force-pushed the wprazuch/environment-registry branch from 42d6da2 to fe103a2 Compare June 24, 2026 07:21

anwithk linked an issue Jun 25, 2026 that may be closed by this pull request

epic: reduce configuration friction for environment authors and users #1205

Closed

5 tasks

Base automatically changed from martas/1434 to main June 25, 2026 10:50

ritaneves mentioned this pull request Jun 25, 2026

[Epic] Continuation of configuration works #1726

Closed

wprazuch added 3 commits June 25, 2026 13:57

wprazuch force-pushed the wprazuch/environment-registry branch from d2986b8 to 3165789 Compare June 25, 2026 12:00

marta-sd approved these changes Jun 25, 2026

View reviewed changes

wprazuch merged commit afa664a into main Jun 25, 2026
16 checks passed

wprazuch deleted the wprazuch/environment-registry branch June 25, 2026 12:18

wprazuch mentioned this pull request Jun 25, 2026

feat(cli): add 'gym env validate' pre-flight config check (#1205 friction #12) #1599

Merged

ritaneves mentioned this pull request Jun 25, 2026

epic: reduce configuration friction for environment authors and users #1205

Closed

5 tasks

ritaneves linked an issue Jun 25, 2026 that may be closed by this pull request

8 - Configuration Friction #1731

Closed

1 task

wprazuch mentioned this pull request Jun 26, 2026

docs: source: union, clean config errors, and new discovery/validate CLI commands #1754

Merged

sephmard mentioned this pull request Jun 29, 2026

docs: move env list to docs #918

Closed

Navigation Menu

Uh oh!

feat: environment registry + 'gym list environments' (#1205 friction #8 / M2) - #1635

feat: environment registry + 'gym list environments' (#1205 friction #8 / M2)#1635
wprazuch merged 3 commits into
mainfrom
wprazuch/environment-registry

wprazuch commented Jun 17, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jun 17, 2026

Uh oh!

cmunley1 left a comment

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

wprazuch commented Jun 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

wprazuch commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Alignment with the unified CLI (#1434)

Tests

Uh oh!

copy-pr-bot Bot commented Jun 17, 2026

Uh oh!

cmunley1 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

wprazuch commented Jun 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wprazuch commented Jun 17, 2026 •

edited

Loading