feat(config): unified clean errors for bad/malformed/empty config_paths (#1205 #8/#12; #1488/#1489/#1490) by wprazuch · Pull Request #1609 · NVIDIA-NeMo/Gym

wprazuch · 2026-06-16T11:57:32Z

What

One coherent fix for the three config_paths failure modes, all failing fast with a clean message and no traceback:

Issue	Bad input	Before	After
#1488	typo'd / missing path	7-frame `FileNotFoundError`	`Error: config_paths entry '...' was not found. Looked in: ...`
#1490	scalar instead of list	raw Pydantic `ValidationError` (+pydantic.dev URL)	`Error: 'config_paths' must be a list of paths ...`
#1489	empty / omitted (zero servers)	Ray starts, hangs until SIGTERM	`Error: No server instances are configured ...` before Ray

How

New ConfigError base in config_types.py; ConfigPathNotFoundError, MalformedConfigPathsError, NoServerInstancesError (and the existing ServerRefNotFoundError) inherit it. They stay ordinary exceptions, so ng_validate (feat(cli): add 'gym env validate' pre-flight config check (#1205 friction #12) #1599, except Exception) can still catch and format them.
load_extra_config_paths wraps OmegaConf.load (bug: nonexistent config path dumps raw FileNotFoundError traceback #1488); parse() wraps ta.validate_python (Bug: Malformed +config_paths surfaces raw Pydantic ValidationError stack #1490).
raise_on_no_server_instances (validated server-instance check, not a raw key count) runs in RunHelper.start() before initialize_ray() — covers both run and e2e_rollout_collection (bug: ng_run with no config spawns Ray and hangs instead of failing fast #1489).
A CLI decorator exit_cleanly_on_config_error converts any ConfigError to a rich-escaped message + exit 1, no traceback. Unexpected errors still propagate unchanged.

Supersedes #1510

Consolidates #1510: typed exceptions ng_validate can catch/format (a bare SystemExit would escape it), a validated zero-server check, the guard in start() so e2e is covered, and deterministic tests.

Tests

test_global_config.py: path-not-found (both-locations / dedup / absolute), malformed config_paths, zero-server (raise + pass). test_cli.py: the decorator (ConfigError → clean exit; non-ConfigError propagates unchanged; success passes through). 67/67 test_global_config+test_cli pass; ruff + pre-commit clean.

copy-pr-bot · 2026-06-16T11:58:01Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

wprazuch · 2026-06-16T12:31:50Z

#1510 is similar to this PR.

Ports #1609 onto the unified gym CLI (#1434). Covers epic #1205 friction #8 + #12 and issues #1488/#1489/#1490: - #1488: missing config_paths entry -> ConfigPathNotFoundError - #1490: malformed (non-list) config_paths -> MalformedConfigPathsError - #1489: zero configured servers -> NoServerInstancesError, in RunHelper.start() before Ray (covers env run + eval run / e2e) All subclass ConfigError; a cli/env.py decorator (exit_cleanly_on_config_error, also applied to e2e_rollout_collection in cli/eval.py) turns them into a clean, rich-escaped message + exit 1, no traceback. Targets martas/1434. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Shared CI fixes for the martas/1434-stacked CLI work: pin uv (0.11.20 drops pinned deps -> 7 servers fail; = #1576) and pull main's graphwalks example_rollouts.jsonl (fixes its data validation). This branch is the base for the ng_validate (#1599) and config-error (#1609) PRs so the fixes live in one place. Drop when martas/1434 rebases on main. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Ports #1609 onto the shared CLI base (martas/1434 + uv pin + concurrency). Covers epic #1205 friction #8 + #12 and issues #1488/#1489/#1490: ConfigPathNotFoundError, MalformedConfigPathsError, NoServerInstancesError (all ConfigError); fail-fast guard in RunHelper.start(); exit_cleanly_on_config_error decorator on run()/e2e_rollout_collection() -> clean message, no traceback. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Shared CI fixes for the martas/1434-stacked CLI work: pin uv (0.11.20 drops pinned deps -> 7 servers fail; = #1576) and pull main's graphwalks example_rollouts.jsonl (fixes its data validation). This branch is the base for the ng_validate (#1599) and config-error (#1609) PRs so the fixes live in one place. Drop when martas/1434 rebases on main. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Ports #1609 onto the shared CLI base (martas/1434 + uv pin + concurrency). Covers epic #1205 friction #8 + #12 and issues #1488/#1489/#1490: ConfigPathNotFoundError, MalformedConfigPathsError, NoServerInstancesError (all ConfigError); fail-fast guard in RunHelper.start(); exit_cleanly_on_config_error decorator on run()/e2e_rollout_collection() -> clean message, no traceback. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Shared CI fixes for the martas/1434-stacked CLI work: pin uv (0.11.20 drops pinned deps -> 7 servers fail; = #1576) and pull main's graphwalks example_rollouts.jsonl (fixes its data validation). This branch is the base for the ng_validate (#1599) and config-error (#1609) PRs so the fixes live in one place. Drop when martas/1434 rebases on main. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Ports #1609 onto the shared CLI base (martas/1434 + uv pin + concurrency). Covers epic #1205 friction #8 + #12 and issues #1488/#1489/#1490: ConfigPathNotFoundError, MalformedConfigPathsError, NoServerInstancesError (all ConfigError); fail-fast guard in RunHelper.start(); exit_cleanly_on_config_error decorator on run()/e2e_rollout_collection() -> clean message, no traceback. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

github-actions · 2026-06-24T07:00:22Z

🌿 Preview your docs: https://nvidia-preview-wprazuch-config-load-errors.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

Ports #1609 onto the shared CLI base (martas/1434 + uv pin + concurrency). Covers epic #1205 friction #8 + #12 and issues #1488/#1489/#1490: ConfigPathNotFoundError, MalformedConfigPathsError, NoServerInstancesError (all ConfigError); fail-fast guard in RunHelper.start(); exit_cleanly_on_config_error decorator on run()/e2e_rollout_collection() -> clean message, no traceback. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

The base branch was changed.

Ports #1609 onto the shared CLI base (martas/1434 + uv pin + concurrency). Covers epic #1205 friction #8 + #12 and issues #1488/#1489/#1490: ConfigPathNotFoundError, MalformedConfigPathsError, NoServerInstancesError (all ConfigError); fail-fast guard in RunHelper.start(); exit_cleanly_on_config_error decorator on run()/e2e_rollout_collection() -> clean message, no traceback. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Marta review fixes: - Drop the convoluted dict.fromkeys dedup in load_extra_config_paths: build searched_locations with cwd/install listed once at construction (they coincide when run from the repo root). - Update the NoServerInstancesError and MalformedConfigPathsError hints to the new CLI (`gym env start --config ...`) instead of the deprecated `ng_run "+config_paths=[...]"`. - Clarify test_non_config_error_propagates: assert the non-ConfigError propagates unchanged (same type + message via match=, not converted to SystemExit), with a comment on the intended behavior. - Retarget the empty-config assertion to the new `gym env start` guidance. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…tion #12) (#1599) ## What Adds **`gym env validate`** (+ `ng_validate` / `nemo_gym_validate` deprecated shims) — runs the full config parse with **no Ray and no server subprocesses**, then exits **0 (valid) / 1 (invalid)** with a clean, rich-escaped message (**no traceback**). Returns in well under a second instead of after a ~30–60s Ray bootstrap. ```bash gym env validate --config resources_servers/<env>/configs/<env>.yaml --config responses_api_models/<model>/configs/<model>.yaml gym env validate --benchmark gsm8k --model-type openai_model ``` ## How `validate()` lives in `cli/env.py` and is registered as `env validate` in the `gym` router (`cli/main.py` COMMANDS) with the same config-selection flags as `env start` (`--config`, `--benchmark`, `--environment`, `--resources-server`, `--model-type`, `--search-dir`, `--model*`). It reuses the same `get_global_config_dict()` parse path the other commands use, so the validation checks stay in sync: - **config_paths** resolution — missing/typo'd ([#1488](#1488)) and malformed ([#1490](#1490)) - **server cross-references** — unknown `name:` refs ([#1561](#1561)) - **mandatory `???`** values ([#1575](#1575)) - **schema** (`BaseNeMoGymCLIConfig`) Wrapped in `exit_cleanly_on_config_error` (from #1609) so any `ConfigError` becomes a clean message + `exit 1`. A dummy `policy_model` is injected (the `NO_MODEL` parser config, as in `gym list` / `env compose`) so model interpolations like `${policy_base_url}` resolve without real creds — validation is about config **well-formedness**; the real model is supplied by the `--model*` flags at run time. ## Targets `main` Originally drafted on the unified-CLI epic branch; rebuilt directly on `main` now that [#1630](#1630) (and #1637/#1609/#1635/#1671) have merged. The old branch contents (a snapshot of the CLI refactor + unrelated CI commits) were superseded and replaced. ## Scope note The zero-server check ([#1489](#1489), "nothing configured to run") is intentionally **not** part of `validate`: `NO_MODEL` injects a dummy model server (which would defeat the check), and "is anything configured to run" is a *start*-time concern already enforced by `gym env start` before Ray init. `validate` focuses on config well-formedness. ## Why Epic [#1205](#1205) friction #12 (no config validation tooling) — the M1 "fast failure triage" deliverable. Config errors otherwise only surface after Ray starts (~30–60s). ## Tests - `test_cli_main.py`: `gym env validate --config X` routes to `nemo_gym.cli.env:validate` with `+config_paths=[X]` (added to the parametrized config-command matrix). - `test_cli.py`: `validate()` prints OK on a valid config; a raised `ConfigError` becomes `exit 1` (no traceback). - All `test_cli` + `test_cli_main` + `test_cli_legacy` pass (the only failures are the pre-existing Python-3.12 `TestDidYouMean` argparse issue on `main`); ruff + pre-commit clean. Smoke-tested end-to-end: `✓ Config is valid.` on a real benchmark, clean error + `exit 1` on a bad path, and the `ng_validate` deprecation shim. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…hs (#1205 #8/#12; #1488/#1489/#1490) (#1609) ## What One coherent fix for the three `config_paths` failure modes, all failing fast with a **clean message and no traceback**: | Issue | Bad input | Before | After | |-------|-----------|--------|-------| | [#1488](#1488) | typo'd / missing path | 7-frame `FileNotFoundError` | `Error: config_paths entry '...' was not found. Looked in: ...` | | [#1490](#1490) | scalar instead of list | raw Pydantic `ValidationError` (+pydantic.dev URL) | `Error: 'config_paths' must be a list of paths ...` | | [#1489](#1489) | empty / omitted (zero servers) | Ray starts, hangs until SIGTERM | `Error: No server instances are configured ...` before Ray | ## How - New `ConfigError` base in `config_types.py`; `ConfigPathNotFoundError`, `MalformedConfigPathsError`, `NoServerInstancesError` (and the existing `ServerRefNotFoundError`) inherit it. They stay ordinary exceptions, so `ng_validate` (#1599, `except Exception`) can still catch and format them. - `load_extra_config_paths` wraps `OmegaConf.load` (#1488); `parse()` wraps `ta.validate_python` (#1490). - `raise_on_no_server_instances` (validated server-instance check, not a raw key count) runs in **`RunHelper.start()`** before `initialize_ray()` — covers both run and `e2e_rollout_collection` (#1489). - A CLI decorator `exit_cleanly_on_config_error` converts any `ConfigError` to a rich-escaped message + `exit 1`, **no traceback**. Unexpected errors still propagate unchanged. ## Supersedes #1510 Consolidates [#1510](#1510): typed exceptions `ng_validate` can catch/format (a bare `SystemExit` would escape it), a validated zero-server check, the guard in `start()` so `e2e` is covered, and deterministic tests. ## Tests `test_global_config.py`: path-not-found (both-locations / dedup / absolute), malformed config_paths, zero-server (raise + pass). `test_cli.py`: the decorator (ConfigError → clean exit; non-ConfigError propagates unchanged; success passes through). 67/67 `test_global_config`+`test_cli` pass; ruff + pre-commit clean. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>

…CLI commands (#1754) Documents the config/CLI features merged for epic #1205 that were not yet reflected in the docs. Requested follow-up: a single doc PR covering the recently merged config/CLI work. ## What's documented | Area | Change | Merged in | |---|---|---| | `reference/cli-commands.mdx` | New `gym list environments`, `gym list agents`, and `gym env validate` sections (+ Quick-Reference entries) | #1635, #1671, #1599 | | `troubleshooting/configuration.mdx` | Three `config_paths` startup errors (Config Path Not Found, Malformed `config_paths`, No Server Instances) + a `gym env validate` pre-flight tip; corrected the stale `ServerRefNotFound` message | #1609, #1599 | | `data/index.mdx` | Unified dataset `source:` block + dedicated "Dataset `source`" subsection + legacy-deprecation note | #1637 | | `data/download-huggingface.mdx` | Auto-download example migrated to `source: { type: huggingface }` | #1637 | | `reference/faq.mdx` | Dataset example and field bullets rewritten around `source:` | #1637 | | `environment-tutorials/single-step-environment.mdx` | Scaffold comment updated to reference `source:` | #1637 | --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…1807) ## Summary C4 validation hardening plus the `gym env test`-from-an-external-cwd plumbing deferred from #1806. Part of epic #1205 (criteria **C4** pre-start validation, **C5** external-cwd). ## Clean config errors (C4) - Two raw `ValueError`s become `ConfigError` subclasses so they print cleanly (no traceback) via `exit_cleanly_on_config_error` and `gym env validate`, consistent with the #1609 error family: - `InheritPathNotFoundError` — a `_inherit_from` / swap / copy directive points at a missing config path (`global_config.py`). - `AlmostServerError` — almost-servers detected with `error_on_almost_servers` set. - Docs: `release-notes.mdx` referenced `+dryrun=true`; the actual key is `dry_run`. ## Run / test built-in servers from any cwd (C5) - New `_resolve_server_dir` (cwd first, then the Gym install root) and `TestConfig.resolved_dir_path`. `gym env test` (single + all), data validation, venv cleanup, and `gym env packages` now resolve built-in servers under the install root instead of assuming a repo checkout. - `gym env test` (all) discovers modules under **both** the cwd and the install root (deduped, cwd shadows install). - `setup_command` PYTHONPATH now includes the project root (the dir holding the server-type dirs) so generated `resources_servers.<name>` imports resolve from outside a checkout. ## Tests - `ConfigError` assertion on the almost-server path. - `_resolve_server_dir` (cwd-preferred / install-fallback) and `TestConfig.resolved_dir_path`. - Updated `run_command` PYTHONPATH expectations (now include the project root). ruff clean; `test_cli` / `test_cli_setup_command` / `test_global_config` pass (115). The `gym env test` execution body is `# pragma: no cover` (spawns per-server venvs); the resolution helpers it relies on are unit-tested. ## Out of scope (documented in the tracking issue) - **`extra="allow"` "typo field" detection** is **not** a simple `forbid`: `BaseRunServerTypeConfig` intentionally allows arbitrary fields because server authors add custom config their server reads, so forbidding would break configs. A fuzzy "did you mean `host`?" *warning* is the only non-breaking option — deferred. - Turning `gym env validate` into a full dataset/prepare/runtime **preflight** is a separate enhancement. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

wprazuch mentioned this pull request Jun 16, 2026

fix: fail fast with clear errors for invalid +config_paths #1510

Closed

wprazuch changed the title ~~feat(config): actionable errors for bad config_paths and zero-server runs (#1205 friction #8/#12)~~ feat(config): unified clean errors for bad/malformed/empty config_paths (#1205 #8/#12; #1488/#1489/#1490) Jun 16, 2026

wprazuch force-pushed the wprazuch/config-load-errors branch from f22a409 to 205816b Compare June 17, 2026 08:46

wprazuch requested a review from a team as a code owner June 17, 2026 08:46

wprazuch changed the base branch from main to martas/1434 June 17, 2026 08:46

wprazuch force-pushed the wprazuch/config-load-errors branch from 205816b to 95b37ea Compare June 17, 2026 09:33

wprazuch changed the base branch from martas/1434 to wprazuch/ng-test-concurrency June 17, 2026 09:33

wprazuch force-pushed the wprazuch/config-load-errors branch from 95b37ea to 6f3d915 Compare June 22, 2026 13:45

wprazuch force-pushed the wprazuch/ng-test-concurrency branch from 2fdee46 to eebfcf8 Compare June 22, 2026 13:45

wprazuch force-pushed the wprazuch/config-load-errors branch from 68b756f to 7f1624b Compare June 23, 2026 16:07

wprazuch changed the base branch from wprazuch/ng-test-concurrency to martas/1434 June 23, 2026 16:07

wprazuch closed this Jun 23, 2026

wprazuch reopened this Jun 23, 2026

wprazuch force-pushed the wprazuch/config-load-errors branch from 7f1624b to 439ea7a Compare June 24, 2026 06:59

wprazuch changed the base branch from martas/1434 to wprazuch/cli-clean-base June 24, 2026 06:59

wprazuch force-pushed the wprazuch/config-load-errors branch from 439ea7a to 8d54c74 Compare June 24, 2026 07:18

wprazuch changed the base branch from wprazuch/cli-clean-base to martas/1434 June 24, 2026 07:18

wprazuch force-pushed the wprazuch/config-load-errors branch from 8d54c74 to 26e20eb Compare June 24, 2026 07:21

marta-sd reviewed Jun 24, 2026

View reviewed changes

Comment thread nemo_gym/global_config.py Outdated

Comment thread nemo_gym/global_config.py Outdated

Comment thread tests/unit_tests/test_cli.py Outdated

marta-sd previously approved these changes Jun 24, 2026

View reviewed changes

anwithk linked an issue Jun 25, 2026 that may be closed by this pull request

epic: reduce configuration friction for environment authors and users #1205

Closed

5 tasks

Base automatically changed from martas/1434 to main June 25, 2026 10:50

ritaneves mentioned this pull request Jun 25, 2026

[Epic] Continuation of configuration works #1726

Closed

wprazuch added 2 commits June 25, 2026 13:47

marta-sd previously approved these changes Jun 25, 2026

View reviewed changes

wprazuch dismissed marta-sd’s stale review via feaa3d6 June 25, 2026 11:48

wprazuch force-pushed the wprazuch/config-load-errors branch from c091be0 to feaa3d6 Compare June 25, 2026 11:48

wprazuch merged commit bcffd48 into main Jun 25, 2026
16 checks passed

wprazuch deleted the wprazuch/config-load-errors branch June 25, 2026 11:54

wprazuch mentioned this pull request Jun 25, 2026

feat(cli): add 'gym env validate' pre-flight config check (#1205 friction #12) #1599

Merged

ritaneves mentioned this pull request Jun 25, 2026

epic: reduce configuration friction for environment authors and users #1205

Closed

5 tasks

ritaneves linked an issue Jun 25, 2026 that may be closed by this pull request

12 - Configuration Friction #1735

Closed

1 task

wprazuch mentioned this pull request Jun 26, 2026

docs: source: union, clean config errors, and new discovery/validate CLI commands #1754

Merged

wprazuch mentioned this pull request Jun 29, 2026

feat: clean config errors + run/test built-in servers from any cwd #1807

Merged

Navigation Menu

Uh oh!

feat(config): unified clean errors for bad/malformed/empty config_paths (#1205 #8/#12; #1488/#1489/#1490) - #1609

feat(config): unified clean errors for bad/malformed/empty config_paths (#1205 #8/#12; #1488/#1489/#1490)#1609
wprazuch merged 2 commits into
mainfrom
wprazuch/config-load-errors

wprazuch commented Jun 16, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jun 16, 2026

Uh oh!

wprazuch commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

wprazuch commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Supersedes #1510

Tests

Uh oh!

copy-pr-bot Bot commented Jun 16, 2026

Uh oh!

wprazuch commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wprazuch commented Jun 16, 2026 •

edited

Loading