Skip to content

fix(global_config): fail fast when the parent openai version violates nemo-gym's constraint; explicit opt-in for version skew - #2447

Merged
ananthsub merged 4 commits into
NVIDIA-NeMo:mainfrom
michal2409:mfutrega/fix-head-server-openai-clamp
Aug 13, 2026
Merged

fix(global_config): fail fast when the parent openai version violates nemo-gym's constraint; explicit opt-in for version skew#2447
ananthsub merged 4 commits into
NVIDIA-NeMo:mainfrom
michal2409:mfutrega/fix-head-server-openai-clamp

Conversation

@michal2409

@michal2409 michal2409 commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Problem

head_server_deps pins the parent process's openai version into every server venv. When the parent environment ships an openai release outside nemo-gym's own constraint (e.g. openai 2.52.x preinstalled in a base image while nemo-gym caps openai<=2.7.2), that pin makes every server venv resolution unsatisfiable. This does not fail cleanly: the dry-run prefetch bakes venvs that contain nothing but pip, and the problem surfaces only at runtime as import errors far from the cause.

Change (after review)

  • Fail fast, loudly: the parser now raises ConfigError at parse time when the parent openai violates nemo-gym's constraint, naming both versions and the remedy. This preserves the install-time guarantee the previous behavior intended to give, minus the empty-venv mystery.
  • Explicit, scoped skew mode: a new reserved top-level key allow_openai_version_skew: true opts into letting server venvs resolve openai from nemo-gym's own constraint, with a logging.warning naming both versions. Parent and servers exchange requests across the HTTP/JSON boundary, so the skew surface is the JSON contract, not Python type identity — we run this topology in production (parent 2.52.x, servers on the nemo-gym pin) on 64-node RL trainings.
  • Requirement-name comparison canonicalized per PEP 503.
  • The constraint is read from the installed distribution metadata at runtime (never hardcoded), so pin bumps like chore(deps): pin openai to 2.44.0 #2456 need no change here.

Tests

Parser-level (through get_global_config_dict): compatible parent → pin present in head_server_deps; incompatible parent → ConfigError naming the opt-in key; incompatible parent + opt-in → pin omitted. Plus unit coverage of the constraint matcher (case-insensitive names, marker'd requirements skipped, missing metadata → conservative pin-preserving fallback).

@copy-pr-bot

copy-pr-bot Bot commented Aug 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@michal2409
michal2409 force-pushed the mfutrega/fix-head-server-openai-clamp branch 2 times, most recently from 2645685 to 6f65c8f Compare August 10, 2026 18:32
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Aug 11, 2026

@ananthsub ananthsub left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i've added one blocking comment inline on the risk of parent vs child process openAI version differences. previously this would be caught at install time, but now we have a risk of version skew

Comment thread nemo_gym/global_config.py Outdated
Comment thread nemo_gym/global_config.py Outdated
Comment thread nemo_gym/global_config.py Outdated
Comment thread tests/unit_tests/test_global_config.py
@michal2409 michal2409 changed the title fix(global_config): only pin the parent openai version into sub-venvs when nemo-gym's own constraint accepts it fix(global_config): fail fast when the parent openai version violates nemo-gym's constraint; explicit opt-in for version skew Aug 11, 2026
@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Aug 11, 2026
@michal2409
michal2409 requested a review from ananthsub August 11, 2026 19:47
@cmunley1

Copy link
Copy Markdown
Contributor

/claude review

@claude

claude Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

SHIP — LGTM, no reliability concerns.

Reviewed the openai version-skew gate in global_config.py. It replaces the unconditional openai==<parent> pin with a constraint-aware gate: pin when the parent satisfies nemo-gym's own openai<=2.7.2, hard-fail with a clear ConfigError when it doesn't, or fall back to nemo-gym's resolution behind the explicit allow_openai_version_skew opt-in.

Correctness:

  • Distribution name (nemo-gym) and constraint (openai<=2.7.2) match pyproject.toml.
  • All failure modes in _nemo_gym_openai_requirement/_openai_version_matches_nemo_gym_constraint (missing packaging, unparseable req, None from requires(), marker'd reqs, PackageNotFoundError) conservatively return True, preserving the original pin-the-parent behavior. packaging is correctly treated as best-effort via lazy import + broad except.
  • prereleases=True avoids false-negative rejections on pre-release parents.
  • New allow_openai_version_skew key uses runtime setdefault, consistent with dry_run/skip_venv_if_present — no TypedDict/YAML default divergence.

The one behavioral change — a new default hard-fail on version skew — replaces a silently broken state (dry-run prefetch baking empty venvs) with fail-fast plus a documented escape hatch. Net operability win.

Test coverage is thorough: all three pin/skew/opt-in branches plus the helper's edge cases are exercised. No async, verifier, scorer, reward-aggregation, or public-API surface touched.

marta-sd added a commit that referenced this pull request Aug 12, 2026
…2476)

`wait_for_dry_run_spinup` polled each process until it exited, then
dropped the exit code. A dry run builds each server's venv and exits, so
a finished process is the expected outcome here rather than the failure
`poll()` treats it as.

We need to use the exit code to distinguish success from failure. `gym
env start` with `dry_run=true` reported the run as complete either way.

The code has to be checked because uv creates the venv before installing
into it. A failed install still leaves an interpreter and an activate
script behind. That is enough to satisfy `should_skip_venv_setup` in
`setup_env_command`, so with `skip_venv_if_present` a later run skips
setup and reuses the venv. The first visible symptom is then an
ImportError from a server, long after the install that caused it, with
nothing pointing back at it.

This covers any dependency failure in a dry run rather than one
particular package:

- a typo in a server's `requirements.txt`
- a yanked or unavailable release
- a network failure part-way through resolution
- a dependency conflict between a server and `nemo-gym`

All of these reported success before.

`gym env prefetch` already checks the return code of each setup process
and raises `SystemExit`. This brings the dry-run path in line with it.

The failure in #2447 is the last case in that list: a parent openai
version that `nemo-gym`'s own constraint rejects makes every sub-venv
resolution unsatisfiable, and this is the reason it surfaced as empty
venvs rather than as a build error. That PR is still worth having on its
own terms, since a resolvable install beats a clear error about an
unresolvable one. This is about the silence, not the conflict.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>
@michal2409
michal2409 force-pushed the mfutrega/fix-head-server-openai-clamp branch from 5d60b77 to 71b5398 Compare August 12, 2026 11:38
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Aug 12, 2026
Comment thread nemo_gym/global_config.py Outdated
Comment thread nemo_gym/global_config.py
Comment thread tests/unit_tests/test_global_config.py Outdated
ananthsub
ananthsub previously approved these changes Aug 13, 2026
@github-actions github-actions Bot removed the sla:review-overdue Review response is over the one-business-day SLA label Aug 13, 2026
michal2409 and others added 4 commits August 13, 2026 00:56
… when nemo-gym's own constraint accepts it

head_server_deps pins the parent process's openai version into every
sub-venv for consistency. When the parent environment ships an openai
release outside nemo-gym's own requirement (e.g. openai 2.52.x
preinstalled in the base image while nemo-gym caps openai<=2.7.2), that
pin makes every sub-venv resolution unsatisfiable. uv/pip then fail the
resolve, and the dry-run venv prefetch silently bakes venvs that contain
nothing but pip - the failure only surfaces much later at server startup
as missing imports.

Fix: before appending 'openai==<parent version>' to head_server_deps,
check the parent's version against nemo-gym's own openai requirement (via
importlib.metadata). If it does not satisfy the constraint, omit the pin
and let each sub-venv resolve openai from nemo-gym's requirement instead
(nemo-gym is installed into the sub-venvs, so its constraint still
applies there).

Behavior is unchanged whenever the parent's openai is compatible, and the
check conservatively falls back to the original pin-the-parent behavior
when the constraint cannot be determined (packaging missing, marker'd or
absent openai requirement, metadata lookup failure). Unit tests cover the
compatible/incompatible/fallback paths.

Signed-off-by: Michal Futrega <mfutrega@nvidia.com>
Silently dropping the pin would leave no breadcrumb for anyone debugging
a parent/sub-venv openai version mismatch. Print which version was not
pinned and why, mirroring the parser's other configuration notices.

Signed-off-by: Michal Futrega <michal.futrega@gmail.com>
…sion skew

Address review: instead of silently omitting the openai pin from
head_server_deps, raise ConfigError at parse time when the parent
process's openai violates nemo-gym's own constraint, with a message
naming both versions and the remedy. A new reserved top-level key
allow_openai_version_skew=true opts into the previous fallback (server
venvs resolve openai from nemo-gym's constraint) with a warning log.
Canonicalize the requirement name per PEP 503. Add parser-level tests
for all three modes (compatible pin, incompatible raise, opt-in skew).

Signed-off-by: Michal Futrega <michal.futrega@gmail.com>
…lean for skew opt-in, message-content assertions

- Drop concrete openai versions from docstrings (they drift and trip grep-based tooling)
- Validate allow_openai_version_skew as a boolean: a string like "false" is truthy and must not silently enable skew; raise ConfigError instead
- Assert the fail-fast error and the opt-in warning both name the parent version and nemo-gym's requirement; cover the non-boolean rejection

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Michal Futrega <michal.futrega@gmail.com>
@michal2409
michal2409 force-pushed the mfutrega/fix-head-server-openai-clamp branch from 960dc83 to 20a41a4 Compare August 13, 2026 08:02
@ananthsub
ananthsub merged commit 93aeccd into NVIDIA-NeMo:main Aug 13, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants