fix(config): aggregated error for unset '???' config values by wprazuch · Pull Request #1575 · NVIDIA-NeMo/Gym

wprazuch · 2026-06-11T09:16:24Z

What

A required config value left unset (OmegaConf MISSING, ???) currently surfaces as an opaque
omegaconf.errors.MissingMandatoryValue — one field at a time, raised deep in parsing
(while resolving inheritance), with no guidance on how to set it.

This adds a fast-fail check in parse() that raises a single ConfigMissingValuesError listing
every unset value with a ready-to-use override example.

Before / after

# before
omegaconf.errors.MissingMandatoryValue: Missing mandatory value:
swe_agents.responses_api_agents.swe_agents.container_formatter

# after
2 required config value(s) are unset (still '???') after merging:
  - swe_agents.responses_api_agents.swe_agents.container_formatter
  - swe_agents.responses_api_agents.swe_agents.dataset_path

Provide each value via a CLI override, in env.yaml, or in a config you pass via config_paths.
For example, on the command line:
  ++swe_agents.responses_api_agents.swe_agents.container_formatter=<value>
  ++swe_agents.responses_api_agents.swe_agents.dataset_path=<value>

How

New ConfigMissingValuesError(ValueError) in config_types.py.
collect_missing_value_paths() / raise_on_missing_values() in global_config.py. The scan
runs after all sources are merged (CLI + env.yaml + config_paths) and after
_recursively_swap_keys — so that the _delete_key / _inherit_from / _copy directives have
been applied first. By that point any remaining ??? is genuinely unset (not a value that is
about to be deleted, or moved/filled by a swap), so it's reported with no false positives.
_recursively_swap_keys itself is made missing-tolerant (items_ex(resolve=False)) so a real
??? doesn't trip the opaque MissingMandatoryValue before the aggregated scan reports it.
The walk uses OmegaConf.to_container(resolve=False, throw_on_missing=False), so it never
raises on MISSING values or unresolved ${...} interpolations.

Scope / safety

Base configs that intentionally ship ??? (api keys, container paths, model names — ~33 files)
are unaffected: they're filled at run time via CLI/env, and no test parses a bare ??? config.
This only changes the error a user sees when they forget to supply one.

Testing

test_collect_missing_value_paths — nested dict + list, asserts ["a", "b.d", "e[1]"].
test_get_global_config_dict_raises_on_missing_values — asserts the dotted path and the
++...=<value> hint appear in the message.
pytest tests/unit_tests/test_global_config.py 28/28; related test_train_data_utils /
test_config_types_help / test_rollout_collection 54/54. ruff clean. No new dependencies.

Part of #1205 (friction point 10). Companion to #1561 (friction 3).

copy-pr-bot · 2026-06-11T09:16:28Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…ite) (#1576) ## Problem The **Full test suite** (`Test`) job has been red on every PR, failing 7 servers (`math_with_code`, `newton_bench`, `arena_judge`, `reasoning_gym`, `ether0`, `aviary`, `stirrup_agent`) with `ModuleNotFoundError` at test collection — `scipy`, `scikit-learn`, `matplotlib`, `PIL`. These deps **are** declared (and pinned) in each server's `requirements.txt`. The real cause: **`uv 0.11.20` (released 2026-06-10) has a resolver regression** — it silently drops pinned direct dependencies from `uv pip install -r requirements.txt` when the requirements also include an editable `-e` install (as every server's `-e nemo-gym[dev] @ ../../` does). No error, no conflict — the package is just omitted. CI installs uv **unpinned** (`curl -LsSf https://astral.sh/uv/install.sh | sh`), so it picked up 0.11.20 the day it released — which is exactly when the suite started failing. The first (passing) run used an earlier uv. ## Evidence (reproduced locally with the exact CI install command) For `resources_servers/reasoning_gym` (`source .venv/bin/activate && uv pip install -r requirements.txt openai==2.7.2`): | uv version | Resolved | `matplotlib==3.10.6` | |---|---|---| | 0.11.19 | 154 packages | ✅ installed → `import matplotlib` OK | | **0.11.20** | 150 packages | ❌ dropped → `ModuleNotFoundError` | Bisected 0.10.2 → 0.11.20: every version **through 0.11.19 works**; only **0.11.20** is broken. ## Fix Pin the uv installer to **0.11.19** (latest known-good) in `full-test-suite.yml` (Test + wheel jobs) and `unit-tests.yml`, with a comment explaining why. Once uv ships a fix, the pin can be bumped. Not caused by — and unblocks — any PR that triggers the full matrix (e.g. #1561, #1575). Worth also reporting the regression upstream to astral-sh/uv. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

wprazuch · 2026-06-15T13:45:28Z

/ok to test edc9b02

wprazuch · 2026-06-15T14:16:38Z

/ok to test 713dffe

wprazuch · 2026-06-16T07:19:31Z

/ok to test 1168c9f

ananthsub

mostly LGTM, but requesting changes to add more test coverage to ensure we dont have regressions from these changes

…erage Address review on #1575: - _recursive_index_dict_using_path now navigates with _get_node and propagates MISSING for an unset referenced value, so '${copy:source.model}' where source.model is '???' reports the missing value instead of an opaque 'path does not exist' error. - Tests: _copy/_inherit_from carry '???' into the target (reported); copy of a missing leaf is reported not opaque; plain '${a.b.c}' interpolation still resolves through full parse (no regression); e2e get_global_config_dict() aggregates multiple missing values. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…ite) (#1576) ## Problem The **Full test suite** (`Test`) job has been red on every PR, failing 7 servers (`math_with_code`, `newton_bench`, `arena_judge`, `reasoning_gym`, `ether0`, `aviary`, `stirrup_agent`) with `ModuleNotFoundError` at test collection — `scipy`, `scikit-learn`, `matplotlib`, `PIL`. These deps **are** declared (and pinned) in each server's `requirements.txt`. The real cause: **`uv 0.11.20` (released 2026-06-10) has a resolver regression** — it silently drops pinned direct dependencies from `uv pip install -r requirements.txt` when the requirements also include an editable `-e` install (as every server's `-e nemo-gym[dev] @ ../../` does). No error, no conflict — the package is just omitted. CI installs uv **unpinned** (`curl -LsSf https://astral.sh/uv/install.sh | sh`), so it picked up 0.11.20 the day it released — which is exactly when the suite started failing. The first (passing) run used an earlier uv. ## Evidence (reproduced locally with the exact CI install command) For `resources_servers/reasoning_gym` (`source .venv/bin/activate && uv pip install -r requirements.txt openai==2.7.2`): | uv version | Resolved | `matplotlib==3.10.6` | |---|---|---| | 0.11.19 | 154 packages | ✅ installed → `import matplotlib` OK | | **0.11.20** | 150 packages | ❌ dropped → `ModuleNotFoundError` | Bisected 0.10.2 → 0.11.20: every version **through 0.11.19 works**; only **0.11.20** is broken. ## Fix Pin the uv installer to **0.11.19** (latest known-good) in `full-test-suite.yml` (Test + wheel jobs) and `unit-tests.yml`, with a comment explaining why. Once uv ships a fix, the pin can be bumped. Not caused by — and unblocks — any PR that triggers the full matrix (e.g. #1561, #1575). Worth also reporting the regression upstream to astral-sh/uv. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…erage Address review on #1575: - _recursive_index_dict_using_path now navigates with _get_node and propagates MISSING for an unset referenced value, so '${copy:source.model}' where source.model is '???' reports the missing value instead of an opaque 'path does not exist' error. - Tests: _copy/_inherit_from carry '???' into the target (reported); copy of a missing leaf is reported not opaque; plain '${a.b.c}' interpolation still resolves through full parse (no regression); e2e get_global_config_dict() aggregates multiple missing values. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

wprazuch · 2026-06-24T07:02:49Z

/ok to test 3cbe609

github-actions · 2026-06-24T09:03:47Z

🌿 Preview your docs: https://nvidia-preview-wprazuch-config-missing-values.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

nemo/gym/troubleshooting/configuration

…erage Address review on #1575: - _recursive_index_dict_using_path now navigates with _get_node and propagates MISSING for an unset referenced value, so '${copy:source.model}' where source.model is '???' reports the missing value instead of an opaque 'path does not exist' error. - Tests: _copy/_inherit_from carry '???' into the target (reported); copy of a missing leaf is reported not opaque; plain '${a.b.c}' interpolation still resolves through full parse (no regression); e2e get_global_config_dict() aggregates multiple missing values. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

A required value left as OmegaConf MISSING ('???') currently surfaces as an opaque MissingMandatoryValue deep in parsing — one field at a time, with no guidance on how to set it. Add a fast-fail check that runs after all sources are merged (CLI + env.yaml + config_paths) and before inheritance resolution, raising a single ConfigMissingValuesError listing every unset value with a ++path=<value> override example. The scan uses OmegaConf.to_container(resolve=False, throw_on_missing=False) so it never trips on MISSING values or unresolved interpolations. Part of #1205 (friction point #10). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…anches Codex flagged a regression: raise_on_missing_values ran before _recursively_swap_keys, so a '???' inside a branch removed by _delete_key (or overwritten by _inherit_from/_copy) was wrongly reported as missing. Moving the check after swap is not sufficient on its own: swap_keys iterated with .items(), which resolves MISSING and raises MissingMandatoryValue mid-swap on any genuine '???'. Fix is two parts: - _recursively_swap_keys_helper: iterate with items_ex(resolve=False) so directive strings stay raw (swap detection still matches) and '???' leaves are returned as-is instead of crashing. - parse(): run raise_on_missing_values after _recursively_swap_keys, so deleted/overwritten branches are excluded from the missing-value report. Tests: unit (swap + collect ignores deleted branch) and e2e (full get_global_config_dict parse with _delete_key removing the only '???'). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…erage Address review on #1575: - _recursive_index_dict_using_path now navigates with _get_node and propagates MISSING for an unset referenced value, so '${copy:source.model}' where source.model is '???' reports the missing value instead of an opaque 'path does not exist' error. - Tests: _copy/_inherit_from carry '???' into the target (reported); copy of a missing leaf is reported not opaque; plain '${a.b.c}' interpolation still resolves through full parse (no regression); e2e get_global_config_dict() aggregates multiple missing values. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Swap iteration over ListConfig now skips missing ('???') elements without resolving, so a '???' in a list (or in a dict nested in a list) is reported by raise_on_missing_values instead of crashing mid-swap with MissingMandatoryValue. Completes the missing-tolerance started for dict leaves. Found by review. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…ue crash Address review: _recursive_index_dict_using_path returned a bare '???' string for a missing referenced path, which broke the swap callers — .pop() on a str (AttributeError) and OmegaConf.merge('???', v) (ValueError) — so ${inherit_from:}/${copy:} and the {_inherit_from}/ {_copy} property forms of an unset value hit a *new* opaque error. Return a _MISSING_REF sentinel instead (and widen the inaccurate -> DictConfig annotation); the swap caller then makes the target '???' and skips pop/merge/delete, so raise_on_missing_values reports it in the aggregated ConfigMissingValuesError. Add tests across all four quadrants {string, property} x {missing leaf, missing parent} for copy and inherit, plus an end-to-end swap -> raise check. Also refresh fern/versions/latest troubleshooting/configuration.mdx to show the new ConfigMissingValuesError (aggregated) and ServerRefNotFoundError ('Did you mean') messages instead of the superseded MissingMandatoryValue / AssertionError. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Reconcile the config-error work with #1575 (aggregated error for unset "???" values), which landed on main. Both PRs add user-facing config-error types; fold ConfigMissingValuesError into the ConfigError family (ConfigError, ValueError) so the CLI`s `except ConfigError` handler prints unset- value errors cleanly (no traceback), consistent with the other config errors. Union the imports in global_config.py and test_global_config.py. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…ction #12 / FEP-1016) Add `gym env validate` (+ ng_validate/nemo_gym_validate deprecated shims): run the full config parse with no Ray and no server subprocesses, exit 0 (valid) / 1 (invalid) with a clean, traceback-free message — config errors otherwise only surface ~30-60s later after Ray bootstrap. It reuses get_global_config_dict so the checks stay in sync: config_paths resolution (#1488/#1490), server cross-references (#1561), mandatory ??? values (#1575), and schema. A dummy policy_model (NO_MODEL) is injected so model interpolations resolve without real creds — the model is supplied by --model* at run time. Registered in the env group with the same config-selection flags as env start; wrapped in exit_cleanly_on_config_error. Tests: env validate routing + validate() valid/invalid behavior. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…tion #12) (#1599) ## What Adds **`gym env validate`** (+ `ng_validate` / `nemo_gym_validate` deprecated shims) — runs the full config parse with **no Ray and no server subprocesses**, then exits **0 (valid) / 1 (invalid)** with a clean, rich-escaped message (**no traceback**). Returns in well under a second instead of after a ~30–60s Ray bootstrap. ```bash gym env validate --config resources_servers/<env>/configs/<env>.yaml --config responses_api_models/<model>/configs/<model>.yaml gym env validate --benchmark gsm8k --model-type openai_model ``` ## How `validate()` lives in `cli/env.py` and is registered as `env validate` in the `gym` router (`cli/main.py` COMMANDS) with the same config-selection flags as `env start` (`--config`, `--benchmark`, `--environment`, `--resources-server`, `--model-type`, `--search-dir`, `--model*`). It reuses the same `get_global_config_dict()` parse path the other commands use, so the validation checks stay in sync: - **config_paths** resolution — missing/typo'd ([#1488](#1488)) and malformed ([#1490](#1490)) - **server cross-references** — unknown `name:` refs ([#1561](#1561)) - **mandatory `???`** values ([#1575](#1575)) - **schema** (`BaseNeMoGymCLIConfig`) Wrapped in `exit_cleanly_on_config_error` (from #1609) so any `ConfigError` becomes a clean message + `exit 1`. A dummy `policy_model` is injected (the `NO_MODEL` parser config, as in `gym list` / `env compose`) so model interpolations like `${policy_base_url}` resolve without real creds — validation is about config **well-formedness**; the real model is supplied by the `--model*` flags at run time. ## Targets `main` Originally drafted on the unified-CLI epic branch; rebuilt directly on `main` now that [#1630](#1630) (and #1637/#1609/#1635/#1671) have merged. The old branch contents (a snapshot of the CLI refactor + unrelated CI commits) were superseded and replaced. ## Scope note The zero-server check ([#1489](#1489), "nothing configured to run") is intentionally **not** part of `validate`: `NO_MODEL` injects a dummy model server (which would defeat the check), and "is anything configured to run" is a *start*-time concern already enforced by `gym env start` before Ray init. `validate` focuses on config well-formedness. ## Why Epic [#1205](#1205) friction #12 (no config validation tooling) — the M1 "fast failure triage" deliverable. Config errors otherwise only surface after Ray starts (~30–60s). ## Tests - `test_cli_main.py`: `gym env validate --config X` routes to `nemo_gym.cli.env:validate` with `+config_paths=[X]` (added to the parametrized config-command matrix). - `test_cli.py`: `validate()` prints OK on a valid config; a raised `ConfigError` becomes `exit 1` (no traceback). - All `test_cli` + `test_cli_main` + `test_cli_legacy` pass (the only failures are the pre-existing Python-3.12 `TestDidYouMean` argparse issue on `main`); ruff + pre-commit clean. Smoke-tested end-to-end: `✓ Config is valid.` on a real benchmark, clean error + `exit 1` on a bad path, and the `ng_validate` deprecation shim. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

## What A required config value left unset (OmegaConf MISSING, `???`) currently surfaces as an opaque `omegaconf.errors.MissingMandatoryValue` — **one field at a time**, raised deep in parsing (while resolving inheritance), with no guidance on how to set it. This adds a fast-fail check in `parse()` that raises a single `ConfigMissingValuesError` listing **every** unset value with a ready-to-use override example. ### Before / after ``` # before omegaconf.errors.MissingMandatoryValue: Missing mandatory value: swe_agents.responses_api_agents.swe_agents.container_formatter # after 2 required config value(s) are unset (still '???') after merging: - swe_agents.responses_api_agents.swe_agents.container_formatter - swe_agents.responses_api_agents.swe_agents.dataset_path Provide each value via a CLI override, in env.yaml, or in a config you pass via config_paths. For example, on the command line: ++swe_agents.responses_api_agents.swe_agents.container_formatter=<value> ++swe_agents.responses_api_agents.swe_agents.dataset_path=<value> ``` ## How - New `ConfigMissingValuesError(ValueError)` in `config_types.py`. - `collect_missing_value_paths()` / `raise_on_missing_values()` in `global_config.py`. The scan runs **after** all sources are merged (CLI + env.yaml + config_paths) **and after** `_recursively_swap_keys` — so that the `_delete_key` / `_inherit_from` / `_copy` directives have been applied first. By that point any remaining `???` is genuinely unset (not a value that is about to be deleted, or moved/filled by a swap), so it's reported with no false positives. `_recursively_swap_keys` itself is made missing-tolerant (`items_ex(resolve=False)`) so a real `???` doesn't trip the opaque `MissingMandatoryValue` before the aggregated scan reports it. - The walk uses `OmegaConf.to_container(resolve=False, throw_on_missing=False)`, so it never raises on MISSING values or unresolved `${...}` interpolations. ## Scope / safety Base configs that intentionally ship `???` (api keys, container paths, model names — ~33 files) are unaffected: they're filled at run time via CLI/env, and no test parses a bare `???` config. This only changes the *error* a user sees when they forget to supply one. ## Testing - `test_collect_missing_value_paths` — nested dict + list, asserts `["a", "b.d", "e[1]"]`. - `test_get_global_config_dict_raises_on_missing_values` — asserts the dotted path and the `++...=<value>` hint appear in the message. - `pytest tests/unit_tests/test_global_config.py` 28/28; related `test_train_data_utils` / `test_config_types_help` / `test_rollout_collection` 54/54. ruff clean. No new dependencies. Part of #1205 (friction point 10). Companion to #1561 (friction 3). --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>

wprazuch mentioned this pull request Jun 11, 2026

ci: pin uv to 0.11.19 (0.11.20 resolver regression breaks the test suite) #1576

Merged

wprazuch force-pushed the wprazuch/config-missing-values branch from 0ccb3cf to edc9b02 Compare June 15, 2026 13:07

copy-pr-bot Bot temporarily deployed to public June 15, 2026 13:46 Inactive

copy-pr-bot Bot temporarily deployed to public June 15, 2026 13:48 Inactive

copy-pr-bot Bot temporarily deployed to public June 15, 2026 13:49 Inactive

wprazuch force-pushed the wprazuch/config-missing-values branch from edc9b02 to 713dffe Compare June 15, 2026 14:15

cmunley1 reviewed Jun 15, 2026

View reviewed changes

Comment thread nemo_gym/global_config.py Outdated

wprazuch mentioned this pull request Jun 15, 2026

feat(cli): add 'gym env validate' pre-flight config check (#1205 friction #12) #1599

Merged

wprazuch force-pushed the wprazuch/config-missing-values branch from 1168c9f to f49c6c2 Compare June 16, 2026 11:32

ananthsub reviewed Jun 16, 2026

View reviewed changes

Comment thread tests/unit_tests/test_global_config.py

Comment thread nemo_gym/global_config.py

Comment thread tests/unit_tests/test_global_config.py

Comment thread nemo_gym/global_config.py

ananthsub requested changes Jun 16, 2026

View reviewed changes

wprazuch force-pushed the wprazuch/config-missing-values branch from b73795a to b52ac3c Compare June 22, 2026 13:50

wprazuch requested a review from a team as a code owner June 22, 2026 13:50

wprazuch force-pushed the wprazuch/config-missing-values branch from b52ac3c to 6a4dd8c Compare June 23, 2026 09:11

github-project-automation Bot moved this to Dev Todo in NeMo Gym 0.4.0 - July 1 Jun 23, 2026

anwithk added this to NeMo Gym 0.4.0 - July 1 Jun 23, 2026

wprazuch requested a review from ananthsub June 23, 2026 15:20

wprazuch force-pushed the wprazuch/config-missing-values branch from 6a4dd8c to 3cbe609 Compare June 24, 2026 07:02

ananthsub reviewed Jun 24, 2026

View reviewed changes

Comment thread nemo_gym/global_config.py Outdated

Comment thread nemo_gym/config_types.py

wprazuch force-pushed the wprazuch/config-missing-values branch from 0de4e9f to 03dba7e Compare June 24, 2026 09:04

wprazuch requested a review from ananthsub June 24, 2026 09:55

ananthsub approved these changes Jun 24, 2026

View reviewed changes

wprazuch added 5 commits June 24, 2026 13:00

wprazuch force-pushed the wprazuch/config-missing-values branch from 03dba7e to c64f332 Compare June 24, 2026 11:00

wprazuch merged commit 797db29 into main Jun 24, 2026
17 checks passed

wprazuch deleted the wprazuch/config-missing-values branch June 24, 2026 11:11

github-project-automation Bot moved this from Dev Todo to Done in NeMo Gym 0.4.0 - July 1 Jun 24, 2026

wprazuch mentioned this pull request Jun 24, 2026

feat(config): unified clean errors for bad/malformed/empty config_paths (#1205 #8/#12; #1488/#1489/#1490) #1609

Merged

ritaneves mentioned this pull request Jun 25, 2026

[Epic] Continuation of configuration works #1726

Closed

ritaneves mentioned this pull request Jun 25, 2026

epic: reduce configuration friction for environment authors and users #1205

Closed

5 tasks

ritaneves linked an issue Jun 25, 2026 that may be closed by this pull request

10 - Configuration Friction #1733

Closed

1 task

Navigation Menu

Uh oh!

fix(config): aggregated error for unset '???' config values - #1575

fix(config): aggregated error for unset '???' config values#1575
wprazuch merged 5 commits into
mainfrom
wprazuch/config-missing-values

wprazuch commented Jun 11, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jun 11, 2026

Uh oh!

wprazuch commented Jun 15, 2026

Uh oh!

wprazuch commented Jun 15, 2026

Uh oh!

Uh oh!

wprazuch commented Jun 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ananthsub left a comment

Uh oh!

wprazuch commented Jun 24, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

wprazuch commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Before / after

How

Scope / safety

Testing

Uh oh!

copy-pr-bot Bot commented Jun 11, 2026

Uh oh!

wprazuch commented Jun 15, 2026

Uh oh!

wprazuch commented Jun 15, 2026

Uh oh!

Uh oh!

wprazuch commented Jun 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ananthsub left a comment

Choose a reason for hiding this comment

Uh oh!

wprazuch commented Jun 24, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wprazuch commented Jun 11, 2026 •

edited

Loading