feat: clean config errors + run/test built-in servers from any cwd - #1807
Merged
Conversation
Contributor
|
🌿 Preview your docs: https://nvidia-preview-wprazuch-gym-validation-hardening.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
wprazuch
force-pushed
the
wprazuch/gym-validation-hardening
branch
from
June 29, 2026 11:51
cd1f3e7 to
544634a
Compare
prokotg
approved these changes
Jun 29, 2026
C4 validation hardening and the deferred gym-env-test install-root plumbing for epic #1205 (criteria C4 pre-start validation, C5 external-cwd) / nvbug 6383277. Clean errors (C4): - convert two raw ValueErrors to ConfigError subclasses so they print cleanly (no traceback) via exit_cleanly_on_config_error and gym env validate: InheritPathNotFoundError (bad _inherit_from/swap/copy path) and AlmostServerError (almost-servers present + error_on_almost_servers) - docs: release-notes referenced `+dryrun=true`; the key is `dry_run` Run/test built-in servers from any cwd (C5): - add `_resolve_server_dir` (cwd first, then the Gym install root) and a TestConfig.resolved_dir_path; gym env test (single + all), data validation, venv cleanup, and gym env packages now resolve built-in servers under the install root instead of assuming a repo checkout - gym env test (all) discovers modules under both the cwd and the install root - setup_command PYTHONPATH now includes the project root (the dir holding the server-type dirs) so generated `resources_servers.<name>` imports resolve Tests: ConfigError assertion on the almost-server path; _resolve_server_dir and TestConfig.resolved_dir_path resolution; updated run_command PYTHONPATH expectations. Out of scope (documented in nvbug): the extra="allow" "typo field" item is not a simple forbid — server authors add arbitrary custom fields their server reads, so forbidding would break configs (a fuzzy "did you mean" warning is the only non-breaking option, deferred). Turning gym env validate into a full dataset/ prepare/runtime preflight is a separate enhancement. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
The latest release notes were corrected, but the same invalid `+dryrun=true` (the key is `dry_run`) remained in the frozen v0.3.0 copy, which is still served. This is a factual key typo (never valid), not CLI-version history, so fix it too. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
…erver_dir Pre-review cleanups: - run_command no longer unconditionally injects work_dir.parent.parent into PYTHONPATH (a silent contract change on a publicly re-exported helper with a baked-in layout assumption). Add an opt-in `project_root` param; only `gym env test` (whose generated tests import `resources_servers.<name>...`) passes it. `gym env start`/`gym env packages` keep the original work_dir-only PYTHONPATH. - RunHelper.start now reuses _resolve_server_dir instead of an inline duplicate of the cwd-then-install-root heuristic. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
wprazuch
force-pushed
the
wprazuch/gym-validation-hardening
branch
from
June 29, 2026 13:36
94b6006 to
e9eea33
Compare
wprazuch
added a commit
that referenced
this pull request
Jun 29, 2026
…ition (RFC M3) Compose a benchmark with a swapped agent harness and/or edited benchmark-dataset params, without writing a bespoke per-combination config (epic #1205, criterion C3): - nemo_gym/config_composer.py: compose(), substitute_agent() (wires the env's resources_server/model_server/datasets into the swapped agent, filling a generic agent's mandatory resources_server.name), find_agent_block_key, substitute_dataset_params, _validate_no_mandatory_placeholders (scans for remaining ??? WITHOUT resolving interpolations), and a composability guard (self-contained Pattern B agents rejected). - gym env compose command + an agent asset selector in the unified CLI. - docs: document gym env compose in the CLI reference. Rebased onto current main (composes with #1805/#1806/#1807): #1807's env.py install-root resolution + validate are preserved; gym env compose and gym env validate coexist as separate commands. Verified: compose --benchmark gsm8k --agent simple_agent wires resources_server; validate works; env-test resolution intact. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
wprazuch
added a commit
that referenced
this pull request
Jun 29, 2026
…ition (RFC M3) Compose a benchmark with a swapped agent harness and/or edited benchmark-dataset params, without writing a bespoke per-combination config (epic #1205, criterion C3): - nemo_gym/config_composer.py: compose(), substitute_agent() (wires the env's resources_server/model_server/datasets into the swapped agent, filling a generic agent's mandatory resources_server.name), find_agent_block_key, substitute_dataset_params, _validate_no_mandatory_placeholders (scans for remaining ??? WITHOUT resolving interpolations), and a composability guard (Pattern B rejected). - gym env compose command + an agent asset selector in the unified CLI. - docs: document gym env compose in the CLI reference. Rebased onto current main (composes with #1805/#1806/#1807): #1807's env.py install-root resolution + validate preserved; gym env compose and gym env validate coexist. Verified: compose --benchmark gsm8k --agent simple_agent wires resources_server; validate works; env-test resolution intact. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
wprazuch
added a commit
that referenced
this pull request
Jun 29, 2026
…ition (RFC M3) Compose a benchmark with a swapped agent harness and/or edited benchmark-dataset params, without writing a bespoke per-combination config (epic #1205, criterion C3): - nemo_gym/config_composer.py: compose(), substitute_agent() (wires the env's resources_server/model_server/datasets into the swapped agent, filling a generic agent's mandatory resources_server.name), find_agent_block_key, substitute_dataset_params, _validate_no_mandatory_placeholders (scans for remaining ??? WITHOUT resolving interpolations), and a composability guard (Pattern B rejected). - gym env compose command + an agent asset selector in the unified CLI. - docs: document gym env compose in the CLI reference. Rebased onto current main (composes with #1805/#1806/#1807): #1807's env.py install-root resolution + validate preserved; gym env compose and gym env validate coexist. Verified: compose --benchmark gsm8k --agent simple_agent wires resources_server; validate works; env-test resolution intact. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
C4 validation hardening plus the
gym env test-from-an-external-cwd plumbing deferred from #1806. Part of epic #1205 (criteria C4 pre-start validation, C5 external-cwd).Clean config errors (C4)
ValueErrors becomeConfigErrorsubclasses so they print cleanly (no traceback) viaexit_cleanly_on_config_errorandgym env validate, consistent with the feat(config): unified clean errors for bad/malformed/empty config_paths (#1205 #8/#12; #1488/#1489/#1490) #1609 error family:InheritPathNotFoundError— a_inherit_from/ swap / copy directive points at a missing config path (global_config.py).AlmostServerError— almost-servers detected witherror_on_almost_serversset.release-notes.mdxreferenced+dryrun=true; the actual key isdry_run.Run / test built-in servers from any cwd (C5)
_resolve_server_dir(cwd first, then the Gym install root) andTestConfig.resolved_dir_path.gym env test(single + all), data validation, venv cleanup, andgym env packagesnow resolve built-in servers under the install root instead of assuming a repo checkout.gym env test(all) discovers modules under both the cwd and the install root (deduped, cwd shadows install).setup_commandPYTHONPATH now includes the project root (the dir holding the server-type dirs) so generatedresources_servers.<name>imports resolve from outside a checkout.Tests
ConfigErrorassertion on the almost-server path._resolve_server_dir(cwd-preferred / install-fallback) andTestConfig.resolved_dir_path.run_commandPYTHONPATH expectations (now include the project root).ruff clean;
test_cli/test_cli_setup_command/test_global_configpass (115). Thegym env testexecution body is# pragma: no cover(spawns per-server venvs); the resolution helpers it relies on are unit-tested.Out of scope (documented in the tracking issue)
extra="allow""typo field" detection is not a simpleforbid:BaseRunServerTypeConfigintentionally allows arbitrary fields because server authors add custom config their server reads, so forbidding would break configs. A fuzzy "did you meanhost?" warning is the only non-breaking option — deferred.gym env validateinto a full dataset/prepare/runtime preflight is a separate enhancement.