feat(mopd): add Super Omni multimodal distillation by nvilia · Pull Request #3430 · NVIDIA-NeMo/RL

nvilia · 2026-07-30T18:41:12Z

What does this PR do?

Adds image MOPD for Nemotron 3 Super Omni on top of the Super GRPO support merged in #3494.

Two runtime changes are still needed for MOPD:

Forward row-aligned multimodal inputs to non-colocated teachers. Without this, the teacher receives the rollout tokens but not the corresponding images.
Optionally clear vLLM encoder outputs after a successful weight refit. This prevents cached image features from surviving an update to the vision weights. The MOPD recipe disables in-flight refits so the reset happens while generation is paused.

The PR also adds:

a 10-node Super Omni MOPD recipe;
a four-node, three-step smoke recipe;
a thin wrapper around the existing Super Omni launcher;
deterministic circle-count data preparation;
focused tests and documentation.

Relationship to #3494

This branch is based on current main and uses #3494's Super recipe, launcher, Gym integration, provider setup, and dynamic-image handling unchanged.

It does not restore placeholder sanitization. That work remains separate in #3631 and NVIDIA-NeMo/Megatron-Bridge#5573. The circle-count data used here has one structured image per row and no literal media tokens, so it does not depend on that work.

MTP is disabled for this MOPD recipe (mtp_num_layers: 0). The MTP recipes added by #3494 are unaffected.

Validation

Four-node, three-step smoke:

Slurm job 1687836, completed with exit code 0
W&B: https://wandb.ai/adlr/mopd-nemotron-super-omni/runs/f40m6xtz
three optimizer/refit steps completed;
eight multimodal teacher-logprob batches completed;
zero sequences masked for logprob error;
rollout requests contained real input_image payloads;
step-3 advantage mean: 0.00173;
step-3 advantage range: [-0.639, 0.491];
step-3 loss: 0.00313;
step-3 grad norm: 0.643.

Local checks:

30 OPD tests
7 async vLLM refit/cache tests
recipe schema and test-suite accounting
Ruff formatting/lint and shell syntax checks

Test plan

Multimodal rows are routed to the correct teacher
Teacher DP padding repeats the matching multimodal row
Text-only teachers do not receive empty modality arguments
Encoder-cache reset runs only when enabled and after successful refit
Three distributed Super Omni MOPD steps complete

copy-pr-bot · 2026-07-30T18:41:15Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Route row-aligned multimodal inputs to non-colocated teachers and optionally clear vLLM encoder outputs after quiesced weight updates. Signed-off-by: Ilia Karmanov <ikarmanov@nvidia.com>

Add production and smoke overlays, a thin launcher, deterministic circle-count data preparation, and focused usage documentation. Signed-off-by: Ilia Karmanov <ikarmanov@nvidia.com>

Install each Gym service into its newly created venv when a managed Python interpreter is selected globally. Signed-off-by: Ilia Karmanov <ikarmanov@nvidia.com>

yfw

Multi-agent review of the Super Omni image-MOPD change (7 agents: RL / Gym / design / bug / test experts plus an adversarial challenge round).

Overall this is solid work. I verified the parts easiest to get wrong and they hold up: AsyncLLM.reset_encoder_cache() is real and correctly used on the pinned vLLM 0.25.1; the staleness it fixes is genuine (vLLM's EncoderCacheManager keys on mm_hash, which is weight-independent, and retains finished entries until capacity pressure); the Gym contract matches exactly (make_example signature, return shape, agent_ref discriminator, both config_paths) and the data-prep script runs clean; node accounting (1 gen + 1 teacher + 8 policy = 10) and teacher TP8xEP8 are valid; and the row-alignment logic including DP-pad duplication is correct.

One blocking item: both new tests/test_suites/vlm/mopd-*.sh omit the # ===== BEGIN CONFIG ===== block, which fails an existing non-GPU unit test. Suggestions are inline; I verified the fix takes that suite from 1 failed, 14 passed to 15 passed.

The rest are suggestions. Two are worth reading carefully because a partial fix makes things worse: the config-conventions item is a paired edit (exemplar + reference config, or CI goes red), and the encoder-cache flag is silently ignored on transports other than the three you instrumented.

A few things this review deliberately does not ask for, having considered and rejected them: routing the reset through invalidate_kv_cache (would couple a correctness requirement to an opt-in staleness policy — your placement is better), renaming the recipes to vlm_mopd- (would fail test_all_recipes_start_with_algo_hyphen), and a before/after benchmark for the cache reset.

Generated by Claude Code

tools/launch extracts NUM_NODES/GPUS_PER_NODE/STEPS_PER_RUN/MAX_STEPS/ NUM_RUNS/NUM_MINUTES from a delimited CONFIG block and exits 1 when the markers are absent, which failed test_dry_run_does_not_fail_and_prints_total_gpu_hours (disabled.txt does not exempt drivers from that glob). Wrap the existing vars in the 10n8g driver and add a mirroring block to the smoke wrapper, keeping NUM_NODES/NUM_MINUTES exported for the exec'd child. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

Document the new flag's default in the exemplar YAML and its paired reference config (test_reference_configs_up_to_date requires both), and make normalize_vllm_refit_config raise when the flag is combined with a refit transport whose weight-landing path never resets the encoder cache (nixl, sparse-delta, custom checkpoint engines). Only the collective/IPC and nccl_reshard async paths implement the reset; on any other transport the flag was a silent no-op that kept stale vision embeddings across refits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

The existing tests cover all-image and all-text groups; a single teacher receiving both takes the filter branch that keeps a PackedTensor with a None segment, and nothing asserted the surviving None stays row-aligned with its token row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

The guide was added to the docs/index.md toctree but not to the Nemotron landing page that lists every sibling guide. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

…her_alias Replace the SPDX two-liner with the standard Apache block (and fix the ruff isort blank-line failure it sat next to), and remove default_teacher_alias from the recipe: resolve_reference_aliases checks strict_agent_name_match before the fallback branch, so with strict matching enabled the alias could never be used. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

prefetch_omni_envs.yaml mirrored only the Super Omni GRPO recipe's five Gym servers, so images built with NEMO_GYM_PREFETCH_CONFIGS=examples/nemo_gym/prefetch_omni_envs.yaml had no baked venv for the circle-click GRPO recipe or the new circle-count MOPD recipe, forcing an on-node uv venv build (network egress plus a writable /opt/gym_venvs) at first startup. Add both servers; these are the only Gym-using omni recipes not already covered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

yfw · 2026-08-14T21:15:12Z

/ok to test 870aa0d

DanialTaheri · 2026-08-14T21:45:54Z

LGTM. Just some failures in CI.

The lint workflow's ratchet requires every file with zero pyrefly errors to be listed in project-includes; the new examples/nemo_gym/nemotron-3-super-omni/prepare_circle_count_mopd_data.py type-checks clean, so CI failed until it was whitelisted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

yfw · 2026-08-14T22:00:18Z

/ok to test eec4023

normalize_vllm_refit_config is also called by worker-side NIXL setup with partial generation configs that carry no vllm_cfg key, so the new guard's direct subscript raised KeyError (test_configure_nixl_worker_ignores_other_configs). Use a presence check instead; the guard still raises when the flag is actually set on an unsupported transport. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

yfw · 2026-08-14T23:57:54Z

/ok to test ec99431

nvilia requested a review from a team as a code owner July 30, 2026 18:41

rohitrango force-pushed the rohit/gymv-mm-integration-v2 branch from d60004f to d3e02b4 Compare August 5, 2026 16:55

rohitrango requested review from a team as code owners August 5, 2026 16:55

Base automatically changed from rohit/gymv-mm-integration-v2 to main August 7, 2026 03:51

nvilia force-pushed the ikarmanov/f9-mopd-super-multimodal branch from 768375e to cc54510 Compare August 7, 2026 09:20

nvilia requested a review from a team as a code owner August 7, 2026 09:20

github-actions Bot added the Documentation Improvements or additions to documentation label Aug 7, 2026

nvilia added 3 commits August 14, 2026 04:24

fix(mopd): support image-aware async teacher refits

1f58f5d

Route row-aligned multimodal inputs to non-colocated teachers and optionally clear vLLM encoder outputs after quiesced weight updates. Signed-off-by: Ilia Karmanov <ikarmanov@nvidia.com>

feat(mopd): add Super Omni image distillation recipe

940f40c

Add production and smoke overlays, a thin launcher, deterministic circle-count data preparation, and focused usage documentation. Signed-off-by: Ilia Karmanov <ikarmanov@nvidia.com>

fix(mopd): target Gym service virtual environments

441229e

Install each Gym service into its newly created venv when a managed Python interpreter is selected globally. Signed-off-by: Ilia Karmanov <ikarmanov@nvidia.com>

nvilia force-pushed the ikarmanov/f9-mopd-super-multimodal branch from 5d75673 to 441229e Compare August 14, 2026 11:26

nvilia changed the title ~~chore(examples): add Super Omni multi-turn MOPD recipe with image tools~~ feat(mopd): add Super Omni multimodal distillation Aug 14, 2026

yfw reviewed Aug 14, 2026

View reviewed changes

yfw and others added 7 commits August 14, 2026 12:13

Merge branch 'main' into ikarmanov/f9-mopd-super-multimodal

870aa0d

yfw added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Aug 14, 2026

yfw approved these changes Aug 17, 2026

View reviewed changes

DanialTaheri approved these changes Aug 17, 2026

View reviewed changes

yfw merged commit 4a1454b into main Aug 17, 2026
83 checks passed

yfw deleted the ikarmanov/f9-mopd-super-multimodal branch August 17, 2026 17:19

Navigation Menu

Uh oh!

feat(mopd): add Super Omni multimodal distillation - #3430

feat(mopd): add Super Omni multimodal distillation#3430
yfw merged 12 commits into
mainfrom
ikarmanov/f9-mopd-super-multimodal

nvilia commented Jul 30, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 30, 2026

Uh oh!

yfw left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yfw commented Aug 14, 2026

Uh oh!

DanialTaheri commented Aug 14, 2026 •

edited

Loading

Uh oh!

yfw commented Aug 14, 2026

Uh oh!

yfw commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

nvilia commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Relationship to #3494

Validation

Test plan

Uh oh!

copy-pr-bot Bot commented Jul 30, 2026

Uh oh!

yfw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yfw commented Aug 14, 2026

Uh oh!

DanialTaheri commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yfw commented Aug 14, 2026

Uh oh!

yfw commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nvilia commented Jul 30, 2026 •

edited

Loading

DanialTaheri commented Aug 14, 2026 •

edited

Loading