Skip to content

test: [2/3] add L2 Gym-to-RL rollout acceptance - #4014

Merged
terrykong merged 18 commits into
mainfrom
codex/gym-rollout-acceptance-tests
Sep 9, 2026
Merged

test: [2/3] add L2 Gym-to-RL rollout acceptance#4014
terrykong merged 18 commits into
mainfrom
codex/gym-rollout-acceptance-tests

Conversation

@yaoyu-33

@yaoyu-33 yaoyu-33 commented Sep 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Purpose

Add the L2 acceptance gate for bumping NeMo Gym inside NeMo RL: representative Gym data must complete a rollout through real Gym services and the RL consumer boundary.

Coverage

Seven representative environments are covered, each with accepted and rejected behavior (14 rollout paths):

  • math_with_judge
  • code_gen
  • mcqa
  • equivalence_llm_judge
  • single_step_tool_use_with_argument_comparison
  • structured_outputs_v4
  • workplace_assistant

Each case uses one SHA256-pinned Gym example and a deterministic CPU OpenAI-compatible model endpoint. The real path is NemoGymDataset -> nemo_gym_data_processor/collate -> Gym services -> run_nemo_gym_rollout_sync -> RL postprocess.

Assertions cover positive/zero reward goldens, verifier-derived outputs, preservation of source request parameters plus RL generation overrides, exact prompt/generation token IDs and logprobs, policy-versus-judge request contracts, agent routing, tool schemas and outputs, and multi-turn token/history continuity.

The PR also hardens actor lifecycle behavior: failed Gym startup or tokenizer installation attempts graceful shutdown with a bounded 120-second wait, always attempts ray.kill, and preserves the original startup exception even if cleanup fails. Dedicated unit tests cover _spinup, set_tokenizer, shutdown.remote, shutdown ray.get, and ray.kill failures.

L0 in file and job names is the explicitly selected RL CI tier because the rollout acceptance test is CPU-only. Conceptually this is L2 in the three-PR series because it verifies the Gym-to-RL integration boundary. No smoke-specific abstraction is added to production code.

Validation

Current head: b897b7a3eae1826e8c269876007820e9f27b0a03

Local macOS pytest is unavailable because the lockfile supports Linux x86_64/aarch64; Linux CI is authoritative. EOS launchers and clean detached worktrees are prepared, but exact-head local execution currently requires pull access or an approved mirror for the private CI ECR image; older NGC images were rejected before tests because they use Python 3.13.13 while current main requires 3.13.14.

Three-PR series

  1. test: [1/3] add L1 environment verifier acceptance coverage Gym#3127 - L1 standalone Gym environment/verifier acceptance
  2. this PR - L2 Gym-to-RL rollout acceptance
  3. test: [3/3] add L3 Nano 4B Gym training E2E #4015 - L3 Nano 4B Gym-backed training E2E

Draft: the author will review before this is ready for merge.

@yaoyu-33
test: add Gym rollout acceptance matrix
0341f3b 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added the CI:L0 Run doctests and unit tests label Sep 5, 2026
@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 0341f3b

@yaoyu-33
test: allowlist acceptance row hashes
e4a9d93 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e4a9d932dc447d48f6c88c725405c8c4d393d9dd

@copy-pr-bot

copy-pr-bot Bot commented Sep 5, 2026

Copy link
Copy Markdown

/ok to test e4a9d932dc447d48f6c88c725405c8c4d393d9dd

@yaoyu-33, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e4a9d93

@yaoyu-33 yaoyu-33 mentioned this pull request Sep 5, 2026
Closed
10 tasks
@yaoyu-33
test: expand L0 Gym rollout acceptance
d5251d9 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 changed the title test: add Gym rollout acceptance matrix test: add L0 Gym rollout acceptance matrix Sep 5, 2026
@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test d5251d9

@yaoyu-33 yaoyu-33 mentioned this pull request Sep 5, 2026
Open
@yaoyu-33 yaoyu-33 changed the title test: add L0 Gym rollout acceptance matrix test: [2/3] add L2 Gym-to-RL rollout acceptance Sep 5, 2026
@yaoyu-33 yaoyu-33 mentioned this pull request Sep 5, 2026
Open
@yaoyu-33
test: tighten Gym rollout transport assertions
2beac64 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
test: distinguish rollout and judge contracts
85e137e 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 85e137e

@yaoyu-33
test: use Ray-serializable tokenizer for Gym acceptance
bce2879 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test bce2879

@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Targeted L2 CI for the reviewed Ray-serializable-tokenizer head: https://github.com/NVIDIA-NeMo/RL/actions/runs/33984301832

@yaoyu-33
test: allow long Gym tool continuation prompts
5533d83 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 5533d83

@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

The previous targeted run passed the first six acceptance environments and exposed a test-harness limit on workplace_assistant's 90,056-token continuation. Reviewed fix raises the CI-only byte-token limit to 131,072. Rerun: https://github.com/NVIDIA-NeMo/RL/actions/runs/33989903365

@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Optimized L2 rerun reuses the compatible image from the prior successful build while mounting head 5533d83; the redundant rebuild run was cancelled. Active run: https://github.com/NVIDIA-NeMo/RL/actions/runs/33989961341

@yaoyu-33
test: allow Gym subprocesses to shut down
4368454 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 4368454

@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

All 7 L2 environment rollouts passed in the prior run; its only error was the shared actor's strict 10-second teardown. Reviewed fix allows 60 seconds (15 subprocesses have a ~45-second internal worst-case bound), still fails on a real shutdown hang, and always releases the Ray actor. Rerun: https://github.com/NVIDIA-NeMo/RL/actions/runs/33991595090

@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Latest targeted CI is green: https://github.com/NVIDIA-NeMo/RL/actions/runs/33991595090

  • L0_Unit_Tests_Nemo_Gym passed (25m01s)
  • All 7 Gym-to-RL acceptance cases passed: math_with_judge, code_gen, single_step_tool_use_with_argument_comparison, mcqa, equivalence_llm_judge, structured_outputs_v4, and workplace_assistant
  • Full shard result: 19 passed, 13 skipped
  • Strict Gym subprocess cleanup completed successfully with the 60s graceful wait
  • Lint, script coverage, quality, and coverage jobs passed

Keeping this PR in Draft for review.

@yaoyu-33
test: cover rejected Gym rollout paths
8954ff2 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 8954ff2

@yaoyu-33
style: format L2 rollout acceptance test
b77e034 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b77e034

@yaoyu-33
test: preserve L2 verifier metadata contracts
220faac 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 220faac

@yaoyu-33
test: allow Gym response schema defaults
3545d7a 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@copy-pr-bot

copy-pr-bot Bot commented Sep 6, 2026

Copy link
Copy Markdown

/ok to test

@yaoyu-33, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

@yaoyu-33
test: assert Gym verifier outputs
6b9e450 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
fix: harden Gym acceptance lifecycle
5232839 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Merge origin/main into L2 Gym acceptance
7f59503 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
test: isolate Gym actor cleanup failures
b897b7a 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b897b7a

@yaoyu-33

yaoyu-33 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 94a595d

@yaoyu-33
yaoyu-33 marked this pull request as ready for review September 8, 2026 18:38
@yaoyu-33
yaoyu-33 requested review from a team as code owners September 8, 2026 18:38
@yaoyu-33
test: use tier-neutral NeMo Gym rollout module name
b8588ae 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
yaoyu-33 marked this pull request as draft September 8, 2026 18:44
@yaoyu-33
yaoyu-33 marked this pull request as ready for review September 8, 2026 18:55
@yaoyu-33

yaoyu-33 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b8588ae

@terrykong
terrykong merged commit 74b857c into main Sep 9, 2026
95 checks passed
@terrykong
terrykong deleted the codex/gym-rollout-acceptance-tests branch September 9, 2026 04:01
asolergi-nv added a commit that referenced this pull request Sep 9, 2026
@asolergi-nv @claude
Merge upstream/main into feat/sc-resiliency-04-restart-readmission
31bcc1f 
Two conflicts, both pure adjacency -- independent additions landing at the same
file position, so both sides are kept:

- nemo_rl/algorithms/single_controller.py: this branch's
  _SUPERVISOR_DRAIN_TIMEOUT_S constant sits where #3924 (periodic rollout
  checkpointing) added _MAX_CONSECUTIVE_ROLLOUT_CHECKPOINT_FAILURES and the
  _RolloutCheckpointCut dataclass. Neither refers to the other.

- tests/unit/environments/test_nemo_gym_utils.py: #4014 (L2 Gym-to-RL rollout
  acceptance) appended a spinup-cleanup test to the same end-of-file this
  branch appended TestUnresolvedAgentRefsAreDiagnosable to. main's function
  first, this branch's class after it.

The other three commits (#3730 colocated MInf, #4002 actor venv list from a
Python leaf module, #4048 partial rerun routing) merged clean.

Verified after the merge: the restart keys are still in both SC exemplars,
on_dead_shard is still absent from the config and both yamls, and 1287 tests
pass across single_controller, fleet_health, weight_sync and the gym utils.

Signed-off-by: asolergibert <asolergibert@nvidia.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@terrykong terrykong terrykong approved these changes
@bxyu-nvidia bxyu-nvidia bxyu-nvidia approved these changes

Assignees

No one assigned

Labels

CI:L0 Run doctests and unit tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yaoyu-33 @terrykong @bxyu-nvidia