feat: ultra recipes for gb200 by macandro96 · Pull Request #3276 · NVIDIA-NeMo/RL

macandro96 · 2026-07-17T22:52:02Z

What does this PR do ?

Upstreaming ultra recipes for main

Issues

List issues that this PR closes (syntax):

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

...

copy-pr-bot · 2026-07-17T22:52:05Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

macandro96 · 2026-07-24T23:29:13Z

/ok to test 18d19b0

macandro96 · 2026-07-24T23:43:38Z

/ok to test 784f8e4

yfw · 2026-07-28T20:40:54Z

Let's mirror the super config location: examples/nemo_gym/nemotron-3-ultra

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

…erences Address review feedback on the Ultra recipes PR: - Replace `sif_dir: ???` / `_teachers.general: ???` with concrete placeholders so example configs resolve under `OmegaConf.to_container(resolve=True)`, fixing the blocking CI regression in test_config_validation.py. Fail-fast is preserved at the launch boundary: ultra_launch.sh now hard-requires SIF_DIR whenever the selected config interpolates `${sif_dir}` (mopd already guards NRL_GENERAL_TEACHER_PATH). - Correct nonexistent `student_rlvr.yaml` references to `student_rlvr1.yaml` in the launch guide and ultra_launch.sh. - Drop the tautological `save_optimizer is True` assert in the Ultra config contract test, keeping the meaningful schema + CheckpointManager checks. - Use a non-"interactive" reaper reason for the batch training submit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

…ron-3-ultra Move the 7 Ultra recipe YAMLs and ultra_launch.sh from examples/configs/ultra/ and the repo root into examples/nemo_gym/nemotron-3-ultra/, mirroring the existing nemotron-3-super layout (recipe configs + launcher co-located in one nemo_gym recipe dir). Update all path references so everything resolves: - tests/unit/utils/test_config.py: repoint ULTRA_CONFIG_PATHS to the new dir (otherwise test_ultra_configs_satisfy_current_grpo_contract errors on missing files). - ultra_launch.sh: update usage comments, the CONFIG_PATH example, and the invocation hints to the new script path (run from the repo root, matching super_launch.sh). - docs/guides/nemotron-3-ultra.md: update config paths and the launch command. The configs now sit outside examples/configs/**, so test_config_validation.py no longer globs them (same as the super recipe); the Ultra contract test still covers them via ULTRA_CONFIG_PATHS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

Runtime tuning for the GB200 Ultra recipes: - Point judge models (genrm/nl2bash) at the async vLLM worker venv via ray_worker_py_executable, matching the generation path. - Add max_num_seqs=256 and max_num_batched_tokens=8192 to vLLM serve kwargs. - Enable NCCL_MNNVL_ENABLE and the trtllm FlashInfer allreduce backend (VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm, VLLM_ALLREDUCE_USE_SYMM_MEM=0); route DP through Ray (distributed_executor_backend/data_parallel_backend=ray). - Switch load_format to auto and replace cudagraph_capture_sizes with pass_config.fuse_allreduce_rms=false in compilation_config. - student_rlvr2: lower gpu_memory_utilization 0.85 -> 0.80 to leave headroom for the packed buffer during Megatron->vLLM refits, and document the NL2BASH_BASE_URL / GENRM_MODEL launcher overrides. - ultra_launch.sh: set the batch reaper reason to "other" (was "batch"). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

Ultra runs on the container's stock vLLM, so remove the USE_CUSTOM_VLLM switch and everything that only served the fork: the case block that sourced 3rdparty/vllm/nemo-rl.env, the ${VLLM_ENV_SOURCE} prefix in TRAIN_CMD, the 3rdparty/vllm code-snapshot symlink, the status echo, and the related header/mount comments. The vLLM compile-cache and runtime env (VLLM_CACHE_ROOT, FlashInfer MoE, etc.) are untouched — they serve the stock vLLM too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

…rams - Container section: Ultra runs on the stock NeMo RL container (upstream aarch64 vLLM wheel); document pulling a prebuilt nightly from NGC and drop the BUILD_CUSTOM_VLLM build args and the USE_CUSTOM_VLLM note. - Prepare the code: clone -b main instead of the ultra-v3 branch. - Sync recipe parameters to the configs: student RLVR phase-1 group_answer_length_penalty_coeff 0.15 -> 0.25, MOPD CP 8 -> 32, and the RLHF/Reasoning teacher default cluster shapes (Gym node counts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

- mopd.yaml: remove the "Disabled on main" header note. OPD/MOPD runtime support (OPDAdvantageEstimator, non-colocated teachers) is already on main and the recipe ships as an active .yaml, so the note told readers not to use a shipped recipe. - nemotron-3-ultra.md: GENRM_MODEL is served at TP=4 with DP varying by stage (DP=16 in the RLHF and student RLVR2 stages), not a fixed DP=4 / 16 GPUs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

macandro96 · 2026-07-30T00:04:20Z

/ok to test 31c3bf7

macandro96 requested review from a team as code owners July 17, 2026 22:52

macandro96 marked this pull request as draft July 17, 2026 22:52

github-actions Bot added the Documentation Improvements or additions to documentation label Jul 17, 2026

macandro96 force-pushed the amahishi/ultra-gb200 branch from 19e368b to b48bb44 Compare July 22, 2026 02:28

macandro96 marked this pull request as ready for review July 22, 2026 23:39

macandro96 requested a review from yfw July 22, 2026 23:40

yfw reviewed Jul 23, 2026

View reviewed changes

Comment thread examples/configs/ultra/swe_teacher.yaml Outdated

Comment thread examples/configs/ultra/mopd.yaml Outdated

Comment thread docs/guides/nemotron-3-ultra.md Outdated

Comment thread tests/unit/utils/test_config.py Outdated

Comment thread ultra_launch.sh Outdated

macandro96 force-pushed the amahishi/ultra-gb200 branch from b48bb44 to 18d19b0 Compare July 23, 2026 03:53

macandro96 requested a review from yfw July 23, 2026 04:07

yfw previously approved these changes Jul 24, 2026

View reviewed changes

macandro96 added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Jul 24, 2026

macandro96 force-pushed the amahishi/ultra-gb200 branch from 18d19b0 to 784f8e4 Compare July 24, 2026 23:43

macandro96 dismissed yfw’s stale review via 1bce23f July 25, 2026 04:09

macandro96 force-pushed the amahishi/ultra-gb200 branch from 784f8e4 to 1bce23f Compare July 25, 2026 04:09

macandro96 requested review from a team as code owners July 25, 2026 04:09

github-actions Bot added the CI Relating to CI label Jul 25, 2026

yfw reviewed Jul 29, 2026

View reviewed changes

Comment thread ultra_launch.sh Outdated

macandro96 added 3 commits July 29, 2026 15:44

docs: add Nemotron 3 Ultra launch guide

de7c519

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

add original ultra-v3 recipes

0720271

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

WIP: port ultra configs to main

242e795

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

macandro96 and others added 10 commits July 29, 2026 15:44

fix(ultra): update runtime config compatibility

0afe4af

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

fix(ultra): update Ultra recipe config compatibility

4b35401

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

test(ultra): cover Ultra config composition

fdcc717

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

fix: ultra recipe fixes for main

222406c

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

fix: fix genrm related configs

d337d77

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

macandro96 force-pushed the amahishi/ultra-gb200 branch from 1bce23f to a99f247 Compare July 29, 2026 21:32

github-actions Bot removed the CI Relating to CI label Jul 29, 2026

docs: note about external genrm service

b65d3fd

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

yfw reviewed Jul 29, 2026

View reviewed changes

Comment thread examples/nemo_gym/nemotron-3-ultra/mopd.yaml Outdated

Comment thread docs/guides/nemotron-3-ultra.md Outdated

yfw approved these changes Jul 30, 2026

View reviewed changes

yfw enabled auto-merge (squash) July 30, 2026 00:06

yfw merged commit 64cb9f9 into main Jul 30, 2026
81 checks passed

yfw deleted the amahishi/ultra-gb200 branch July 30, 2026 02:42

yfw mentioned this pull request Aug 10, 2026

feat(ultra): serve GenRM and NL2Bash as external vLLM pools #3529

Merged

4 tasks

Navigation Menu

Uh oh!

feat: ultra recipes for gb200 - #3276

feat: ultra recipes for gb200#3276
yfw merged 15 commits into
mainfrom
amahishi/ultra-gb200

macandro96 commented Jul 17, 2026

Uh oh!

copy-pr-bot Bot commented Jul 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

macandro96 commented Jul 24, 2026

Uh oh!

macandro96 commented Jul 24, 2026

Uh oh!

yfw commented Jul 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

macandro96 commented Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

macandro96 commented Jul 17, 2026

What does this PR do ?

Issues

Usage

Before your PR is "Ready for review"

Additional Information

Uh oh!

copy-pr-bot Bot commented Jul 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

macandro96 commented Jul 24, 2026

Uh oh!

macandro96 commented Jul 24, 2026

Uh oh!

yfw commented Jul 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

macandro96 commented Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants