feat(vlm): Nemotron Super Omni GRPO with MTP under context parallelism - #3494

Merged

yfw merged 58 commits into

mainfrom

smohsenitahe/super-omni-mtp-cp

Aug 14, 2026

Commits on Aug 13, 2026

feat(nemo-gym): let structured images drive Super media placement
DanialTaheri
committed
7bc1112
feat(vlm): apply Nemotron Omni tower controls and keep dynamic modules importable
DanialTaheri
committed
352bb3b
fix(vllm): load the embedded chat template and honor the module cache
DanialTaheri
committed
8871c9a
feat(vlm): add Nemotron Super Omni async Gym recipe and launcher (MTP disabled)
DanialTaheri
committed
ac66b32
feat(vlm): add MTP variant of the Super Omni recipe
DanialTaheri
committed
3c977ff
feat(megatron): support MTP on the caller-packed THD context-parallel path
DanialTaheri
committed
e0b7345
fix(vlm): align Super Omni MTP settings with the validated CP=2 reference
DanialTaheri
committed
9c6ba4f
fix(megatron): unpack the sliced loss mask from self-CP-slicing models
DanialTaheri
committed
900ed26
feat(vlm): Super Omni recipe with trained MTP and speculative decoding
DanialTaheri
committed
429216f
fix(launcher): make the MTP cudagraph-capture-sizes delete opt-out
DanialTaheri
committed
81ce0d6
docs(patches): carry the Megatron-LM MTP SP-scatter fix as a patch
DanialTaheri
committed
b66515a
chore(vlm): use placeholder paths in the Super Omni launcher defaults
DanialTaheri
committed
0e77143
chore(vlm): drop the Super Omni launcher and test-suite scripts
DanialTaheri
committed
ee44f1d
chore(vlm): drop the vendored Gym verifier staging copies
DanialTaheri
committed
cda02c3
feat(vlm): standalone Super Omni launcher; leave super_launch.sh untouched
DanialTaheri
committed
2880278
docs(patches): note that submodule sync discards the MTP patch
DanialTaheri
committed
ec4bce1
fix(vlm): carry MTP speculative decoding in the recipe
DanialTaheri
committed
04e6145
fix(vlm): verify checkpoint and data are inside the container mounts
DanialTaheri
committed
20f4857
fix(vlm): mount-check GYM_VENV_DIR when it points at host storage
DanialTaheri
committed
c183031
fix(nemo-gym): stop fabricating content on items that carry none

DanialTaheri
and
claude
committed
682641c
test(megatron): pin the MTP mask layout for model-CP-slicing models

DanialTaheri
and
claude
committed
7aaef4f
test(suites): add Super Omni recipe drivers and disable them

DanialTaheri
and
claude
committed
a58844e
fix(vlm): select the flashinfer_cutlass MoE backend for Super Omni

DanialTaheri
and
claude
committed
907918d
refactor(vlm): make the MTP+specdec recipe self-contained

DanialTaheri
and
claude
committed
ecdb0ca
fix(nemo_gym): point the Super Omni launcher at the surviving entrypoint

DanialTaheri
and
claude
committed
3cbace9
feat(vlm): train the MTP head in the Super Omni MTP recipe

DanialTaheri
and
claude
committed
e592727
Fix tests and linking to correct py script for grpo_nemo_gym
DanialTaheri
committed
6d619c8
docs(super-omni): name the override variable the launcher actually reads

DanialTaheri
and
claude
committed
5274d11
docs(vlm): correct the inverted MTP gradient-clipping warning

DanialTaheri
and
claude
committed
4a055c2
fix(vlm): stop keying checkpoint selection on an uncollected val metric

DanialTaheri
and
claude
committed
868da54
fix(vlm): cap retained Super Omni checkpoints at 5

DanialTaheri
and
claude
committed
a3f7a9c
refactor(vllm): use vLLM's own default_chat_template_kwargs

DanialTaheri
and
claude
committed
df64881
test(megatron): make the full-MTP-mask assertions load-bearing

DanialTaheri
and
claude
committed
802c302
style(vllm): drop the doubled blank line left by the renderer refactor

DanialTaheri
and
claude
committed
4acd5bb
refactor(vlm): restore recipe inheritance and resolve cluster in the launcher

DanialTaheri
and
claude
committed
6b34de8
docs(super-omni): register the guide in the toctree and document the MTP variants

DanialTaheri
and
claude
committed
1e3e89b
docs(super-omni): describe the launcher's placeholders, not defaults it lacks

DanialTaheri
and
claude
committed
ddc1b6f
fix(nemo_gym): populate the multimodal config fields in the builder

DanialTaheri
and
claude
committed
26e4471
Update nemo_rl/__init__.py

DanialTaheri
and
yfw
committed
c129d0f
Update nemo_rl/models/megatron/setup.py

DanialTaheri
and
yfw
committed
69e55a8
Update nemo_rl/__init__.py

DanialTaheri
and
yfw
committed
d3764bc
chore: drop patches/ now that the fix is in the pinned Megatron-LM

DanialTaheri
and
claude
committed
a50ac7a
fix(nemo_rl): restore the module-cache lookup in patch_transformers_module_dir

DanialTaheri
and
claude
committed
4bf9ef7
test(nemo_gym): cover image-placeholder sanitising and the ragged image path

DanialTaheri
and
claude
committed
ac1b129
fix(nemo_rl): thread apply_to_current_interpreter through as a keyword

DanialTaheri
and
claude
committed
36346f7
fix(megatron): repair the VLM override loop and align both call sites

DanialTaheri
and
claude
committed
a3477b7
fix(vllm): give the chat-template defaults to all three serving consumers

DanialTaheri
and
claude
committed
7e480d6
docs(super-omni): export the W&B opt-out so it reaches the launcher

DanialTaheri
and
claude
committed
0095e6f
fix(refit): keep the refit broadcast stream on one device
DanialTaheri
committed
771f5ad
fix(nemo-gym): restore processor tensors on the ragged image path
DanialTaheri
committed
7006208
refactor(nemo-gym)!: remove image-placeholder sanitization
DanialTaheri
committed
a4ac696
fix(nemo-gym): make the launcher's cluster read survive a missing uv env
DanialTaheri
committed
4ef7f7f
docs(super-omni): drop the standalone guide from this PR
DanialTaheri
committed
d71b1e2
test(nemo_rl): stop an ambient HF_MODULES_CACHE from deciding the result
DanialTaheri
committed
9a5f3d8
style(nemo-gym): drop the stray blank line left by removing the sanitize tests
DanialTaheri
committed
5be93ae
test(vllm): stub the chat-template loader the async server now imports
DanialTaheri
committed
dfc4e56

Commits on Aug 14, 2026