Skip to content

feat: environment suite manager — named multi-env sweeps (M4 core) - #1643

Draft
wprazuch wants to merge 31 commits into
mainfrom
wprazuch/suite-manager
Draft

feat: environment suite manager — named multi-env sweeps (M4 core)#1643
wprazuch wants to merge 31 commits into
mainfrom
wprazuch/suite-manager

Conversation

@wprazuch

Copy link
Copy Markdown
Contributor

What

Adds nemo_gym.suite_manager: the M4 core. A suite is a named list of environment names
that expands to the config_paths needed to run them together — so a production sweep over
dozens of environments becomes gym run --suite ultra_v3 instead of a hand-maintained 80+ entry
config_paths.

# nemo_gym/suites/ultra_v3.yaml  (or ~/.config/nemo_gym/suites/<name>.yaml)
description: Ultra V3 evaluation suite
environments:
  - gpqa
  - workplace_assistant

API

  • discover_suites() — built-in suites under nemo_gym/suites/ + user suites under
    ~/.config/nemo_gym/suites/; a user suite shadows a built-in of the same name.
  • resolve_suite_config_paths(name) — expands a suite into ordered, de-duplicated config_paths
    by resolving each environment name through the registry (feat: environment registry + 'gym list environments' (#1205 friction #8 / M2) #1635). Fast-fails on:
    • unknown suite → SuiteNotFoundError with a "did you mean?" hint,
    • empty suite → EmptySuiteError,
    • unknown environment in a suite → EnvironmentNotFoundError annotated with the suite name.

Like the registry, resolution only reads config files — no interpolation, no servers — so it's
safe even when a referenced config needs unset secrets.

Scope / deferred

Tests

tests/unit_tests/test_suite_manager.py — 13 cases, 100% module coverage: discovery
(built-in/user/shadowing, non-suite & malformed-file skipping, non-string filtering), resolution
(ordering, dedup, all three fast-fail paths), and an end-to-end resolve of the real
workplace_assistant env through a user suite.

Part of the configuration-friction epic (#1205), milestone M4 (friction #8). Targets
wprazuch/environment-registry since it builds on the registry's resolve_environment_config_paths.

marta-sd added 20 commits June 12, 2026 13:40
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…nt deprecation note

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
… to stderr)

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…a single --model flag

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jun 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
@wprazuch
wprazuch force-pushed the wprazuch/environment-registry branch from df7a13d to 6784a52 Compare June 22, 2026 13:48
@wprazuch
wprazuch requested a review from a team as a code owner June 22, 2026 13:48
@wprazuch
wprazuch force-pushed the wprazuch/suite-manager branch from dd9a1e9 to e1c3361 Compare June 22, 2026 13:49
marta-sd added 5 commits June 23, 2026 13:08
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
…or benchmark preparation

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
@wprazuch
wprazuch force-pushed the wprazuch/environment-registry branch from eb06d7b to c237f13 Compare June 24, 2026 07:04
@wprazuch
wprazuch force-pushed the wprazuch/suite-manager branch from 14b0b9b to 1572ca4 Compare June 24, 2026 07:05
@wprazuch
wprazuch force-pushed the wprazuch/environment-registry branch from c237f13 to e7c8af4 Compare June 24, 2026 07:13
…s/ by name

Epic #1205 friction #8 (FEP-1022) / RFC milestone M2 keystone. New
nemo_gym/registry.py:
- discover_environments(): map name -> EnvironmentEntry for every
  environments/<name>/config.yaml (name = dir), with best-effort description/
  domain read WITHOUT resolving interpolations/secrets.
- resolve_environment_config_paths(name): the run-by-name primitive — returns
  the env's config_paths, with a difflib 'did you mean?' on unknown names.

Self-contained config-layer module (no CLI/router changes); the foundation for
'gym list' enumerating environments and run-by-name. CLI wiring is a follow-up
to coordinate with the unified-CLI epic (#1434). 100% covered.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
@wprazuch
wprazuch force-pushed the wprazuch/environment-registry branch from e7c8af4 to 42d6da2 Compare June 24, 2026 07:18
@wprazuch
wprazuch force-pushed the wprazuch/suite-manager branch from 1572ca4 to d28475e Compare June 24, 2026 07:19
Consumes nemo_gym.registry in the unified gym CLI: list_environments() in
cli/env.py renders discovered environments (name / domain / description) as a
table, registered as 'gym list environments' in the router; updates the 'list'
group summary (no longer benchmarks-only). Delivers the discoverability the
RFC's M2 calls for (reviewers asked for 'gym list' by environment name).

Tests: list_environments() output + empty case; router dispatch
gym list environments -> nemo_gym.cli.env:list_environments.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
@wprazuch
wprazuch force-pushed the wprazuch/environment-registry branch from 42d6da2 to fe103a2 Compare June 24, 2026 07:21
Add nemo_gym.suite_manager: a suite is a named list of environment names that
expands to the config_paths needed to run them together, so a multi-environment
sweep becomes a single --suite <name> instead of an 80+ entry config_paths.

- discover_suites(): built-in suites under nemo_gym/suites/ + user suites under
  ~/.config/nemo_gym/suites/ (user shadows built-in of the same name).
- resolve_suite_config_paths(): expands a suite to ordered, de-duplicated
  config_paths via the environment registry, with fast-fail on unknown suite
  (did-you-mean), empty suite, and unknown environment (annotated with the suite).

Resolution-safe (only reads configs, no interpolation/servers), mirroring the
registry. Built-in suites (ultra_v3/reasoning/coding) ship as the corresponding
environments are migrated into environments/; the CLI wiring (gym list suites,
gym run --suite) is deferred to align with the unified CLI.

Implements the M4 core (epic #1205, friction #8).

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
@wprazuch
wprazuch force-pushed the wprazuch/suite-manager branch from d28475e to ef2ada2 Compare June 24, 2026 07:22
@wprazuch
wprazuch marked this pull request as draft June 24, 2026 08:03
@wprazuch
wprazuch force-pushed the wprazuch/environment-registry branch from d2986b8 to 3165789 Compare June 25, 2026 12:00
Base automatically changed from wprazuch/environment-registry to main June 25, 2026 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants