feat: environment suite manager — named multi-env sweeps (M4 core) - #1643
Draft
wprazuch wants to merge 31 commits into
Draft
feat: environment suite manager — named multi-env sweeps (M4 core)#1643wprazuch wants to merge 31 commits into
wprazuch wants to merge 31 commits into
Conversation
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…nt deprecation note Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
… to stderr) Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…a single --model flag Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
wprazuch
force-pushed
the
wprazuch/environment-registry
branch
from
June 22, 2026 13:48
df7a13d to
6784a52
Compare
wprazuch
force-pushed
the
wprazuch/suite-manager
branch
from
June 22, 2026 13:49
dd9a1e9 to
e1c3361
Compare
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…or benchmark preparation Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
wprazuch
force-pushed
the
wprazuch/environment-registry
branch
from
June 24, 2026 07:04
eb06d7b to
c237f13
Compare
wprazuch
force-pushed
the
wprazuch/suite-manager
branch
from
June 24, 2026 07:05
14b0b9b to
1572ca4
Compare
wprazuch
force-pushed
the
wprazuch/environment-registry
branch
from
June 24, 2026 07:13
c237f13 to
e7c8af4
Compare
…s/ by name Epic #1205 friction #8 (FEP-1022) / RFC milestone M2 keystone. New nemo_gym/registry.py: - discover_environments(): map name -> EnvironmentEntry for every environments/<name>/config.yaml (name = dir), with best-effort description/ domain read WITHOUT resolving interpolations/secrets. - resolve_environment_config_paths(name): the run-by-name primitive — returns the env's config_paths, with a difflib 'did you mean?' on unknown names. Self-contained config-layer module (no CLI/router changes); the foundation for 'gym list' enumerating environments and run-by-name. CLI wiring is a follow-up to coordinate with the unified-CLI epic (#1434). 100% covered. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
wprazuch
force-pushed
the
wprazuch/environment-registry
branch
from
June 24, 2026 07:18
e7c8af4 to
42d6da2
Compare
wprazuch
force-pushed
the
wprazuch/suite-manager
branch
from
June 24, 2026 07:19
1572ca4 to
d28475e
Compare
Consumes nemo_gym.registry in the unified gym CLI: list_environments() in cli/env.py renders discovered environments (name / domain / description) as a table, registered as 'gym list environments' in the router; updates the 'list' group summary (no longer benchmarks-only). Delivers the discoverability the RFC's M2 calls for (reviewers asked for 'gym list' by environment name). Tests: list_environments() output + empty case; router dispatch gym list environments -> nemo_gym.cli.env:list_environments. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
wprazuch
force-pushed
the
wprazuch/environment-registry
branch
from
June 24, 2026 07:21
42d6da2 to
fe103a2
Compare
Add nemo_gym.suite_manager: a suite is a named list of environment names that expands to the config_paths needed to run them together, so a multi-environment sweep becomes a single --suite <name> instead of an 80+ entry config_paths. - discover_suites(): built-in suites under nemo_gym/suites/ + user suites under ~/.config/nemo_gym/suites/ (user shadows built-in of the same name). - resolve_suite_config_paths(): expands a suite to ordered, de-duplicated config_paths via the environment registry, with fast-fail on unknown suite (did-you-mean), empty suite, and unknown environment (annotated with the suite). Resolution-safe (only reads configs, no interpolation/servers), mirroring the registry. Built-in suites (ultra_v3/reasoning/coding) ship as the corresponding environments are migrated into environments/; the CLI wiring (gym list suites, gym run --suite) is deferred to align with the unified CLI. Implements the M4 core (epic #1205, friction #8). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
wprazuch
force-pushed
the
wprazuch/suite-manager
branch
from
June 24, 2026 07:22
d28475e to
ef2ada2
Compare
wprazuch
marked this pull request as draft
June 24, 2026 08:03
wprazuch
force-pushed
the
wprazuch/environment-registry
branch
from
June 25, 2026 12:00
d2986b8 to
3165789
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
nemo_gym.suite_manager: the M4 core. A suite is a named list of environment namesthat expands to the
config_pathsneeded to run them together — so a production sweep overdozens of environments becomes
gym run --suite ultra_v3instead of a hand-maintained 80+ entryconfig_paths.API
discover_suites()— built-in suites undernemo_gym/suites/+ user suites under~/.config/nemo_gym/suites/; a user suite shadows a built-in of the same name.resolve_suite_config_paths(name)— expands a suite into ordered, de-duplicatedconfig_pathsby resolving each environment name through the registry (feat: environment registry + 'gym list environments' (#1205 friction #8 / M2) #1635). Fast-fails on:
SuiteNotFoundErrorwith a "did you mean?" hint,EmptySuiteError,EnvironmentNotFoundErrorannotated with the suite name.Like the registry, resolution only reads config files — no interpolation, no servers — so it's
safe even when a referenced config needs unset secrets.
Scope / deferred
ultra_v3,reasoning,coding) are intentionally not shipped yet:a suite only references environments that exist, and
environments/currently holds one migratedenv (
workplace_assistant). They land here as the environment migration proceeds — seenemo_gym/suites/README.md. (No premature suite that would fast-fail on an unmigrated env.)gym list suites,gym run --suite,gym suite add/remove) is deferred to alignwith the unified CLI — same split that worked for the registry (feat: environment registry + 'gym list environments' (#1205 friction #8 / M2) #1635) → run-by-name (feat(cli): run an environment by name — 'gym env run --env <name>' (#1205 friction #8 / FEP-1022) #1636).
Tests
tests/unit_tests/test_suite_manager.py— 13 cases, 100% module coverage: discovery(built-in/user/shadowing, non-suite & malformed-file skipping, non-string filtering), resolution
(ordering, dedup, all three fast-fail paths), and an end-to-end resolve of the real
workplace_assistantenv through a user suite.Part of the configuration-friction epic (#1205), milestone M4 (friction #8). Targets
wprazuch/environment-registrysince it builds on the registry'sresolve_environment_config_paths.