[dev] [5/5] Qwen3.5 support: Qwen3.5-VL training example by wplf · Pull Request #4751 · NVIDIA/Megatron-LM

wplf · 2026-05-12T06:56:56Z

Qwen3.5 support series

This is part of a 5-PR series adding Qwen3.5-VL support, split for review clarity.

Dev PRs (this series):

[1/5] MTP packed-seq CP+THD fix — [Dev] fix(mtp): use padded cu_seqlens in MTP roll for THD with CP #4494
[2/5] FSDP DTensor Bridge checkpoint compatibility — [dev] [2/5] Qwen3.5 support: FSDP DTensor Bridge checkpoint compatibility #4748
[3/5] SharedExpertMLP meta init — [dev] [3/5] Qwen3.5 support: SharedExpertMLP meta init #4749
[4/5] Interleaved MRoPE layout — [dev] [4/5] Qwen3.5 support: Interleaved MRoPE layout #4750
[5/5] Qwen3.5-VL training example — [dev] [5/5] Qwen3.5 support: Qwen3.5-VL training example #4751 ← this PR

Main PRs (corresponding mirrors):

Summary

Adds a standalone VLM training playground under examples/multimodal_dev/ with Qwen3.5-VL end-to-end.

Model-agnostic harness

pretrain_multimodal.py entry point and MODEL_REGISTRY so a new architecture is just a registry entry + backing module.
models/base.py, forward_step.py, arguments.py, data/ (mock + CORD-V2 dataset with THD pack/pad in collate).

Qwen3.5-VL

Full model: models/qwen35_vl/ — vision encoder, MRoPE (pre-computed for THD), decoder, factory, specs, configurations for proxy / 9B / 397B-A17B variants.
Run script + README, plus tests: tests/test_mrope_parity.py, test_cp_correctness.py, test_cp_support.py, test_cp_thd_correctness.py, test_thd_correctness.py, test_thd_e2e.py.

One-line training infra change

megatron/training/datasets/data_samplers.py: enable the vanilla-collate torch DataLoader path when the new arg use_vanilla_collate_fn is set (needed for CORD-V2 under BSHD).

Dependency

This example sets mrope_interleaved=True in its TransformerConfig and relies on the core MRoPE interleaved layout introduced in #4750. The diff here is self-contained (only examples/ + the 1-line data_samplers.py change), but the example won't run end-to-end until #4750 merges.

Functionality support

Resume training loss curve

CP / THD correctness verification

tests/test_cp_thd_correctness.py runs CP=1 and CP=4 in a single torchrun --nproc-per-node 4 invocation (in-process destroy_model_parallel + initialize_model_parallel between phases, weights pinned by a state_dict snapshot, identical inputs via a seeded torch.Generator). Loss aggregated via AllReduce(SUM) on (num, den); grad_norm aggregated via AllReduce(SUM) of gradients on the CP group then divided by cp_size, so each rank holds the CP-mean gradient that matches CP=1's backward on the full-batch mean loss.

Default config (B=2, S=64, H=256, L=2, bf16):

Test	CP=1	CP=4	abs diff	rel diff
BSHD loss	7.03250265	7.03217983	3.23e-04	4.59e-05
BSHD grad_norm	4.84910854	4.84744710	1.66e-03	3.43e-04
THD loss	7.03250265	7.03241825	8.44e-05	1.20e-05
THD grad_norm	4.84910839	4.84912564	1.73e-05	3.56e-06

Cross-check: BSHD CP=1 loss ≡ THD CP=1 loss = 7.03250265, and BSHD CP=1 grad_norm ≈ THD CP=1 grad_norm to 7 decimals — equal-length sequences make the two attention paths mathematically identical, so the CP=1 grad_norm match confirms BSHD/THD parity at the gradient level as well.

Checkpoint conversion (HF → Megatron-FSDP DTensor)

The example consumes a Megatron-FSDP DTensor checkpoint, converted from the HuggingFace release via Megatron-Bridge.

Setup — clone Bridge and pin its 3rdparty/Megatron-LM submodule to this branch:

git clone --recurse-submodules https://github.com/NVIDIA-NeMo/Megatron-Bridge.git
cd Megatron-Bridge/3rdparty/Megatron-LM
git remote add wplf https://github.com/wplf/Megatron-LM.git
git fetch wplf feat/qwen35-vl-example
git checkout feat/qwen35-vl-example
cd ../..

Convert (single 8×H100 node, EP=8 / TP=CP=1; --hf-model can be any Qwen3.5 variant, e.g. Qwen/Qwen3.5-35B-A3B):

PYTHONPATH=./src:./3rdparty/Megatron-LM/ \
  torchrun --nproc_per_node=8 \
  examples/conversion/mfsdp/convert_checkpoints_fsdp.py import \
  --hf-model Qwen/Qwen3.5-35B-A3B \
  --megatron-path ${WORKSPACE}/models/Qwen/Qwen3.5-35B-A3B-fsdp \
  --ckpt-format fsdp_dtensor \
  --ep 8

HF weights are auto-fetched on first run via huggingface_hub. Adjust --tp / --cp / --ep to match the training topology (must satisfy WORLD_SIZE % (TP*CP*EP) == 0).

Output

${WORKSPACE}/models/Qwen/Qwen3.5-35B-A3B-fsdp/
├── iter_0000000/
│   ├── __0_0.distcp .. __7_0.distcp   # FSDP DTensor shards, one per rank (~18 GB each for 35B-A3B)
│   ├── .metadata
│   ├── run_config.yaml
│   └── train_state.pt
├── latest_checkpointed_iteration.txt
└── latest_train_state.pt

Bridge dependency — requires NVIDIA-NeMo/Megatron-Bridge#3987 (skip tokenizer save in convert_checkpoints_fsdp.py). Without that fix the checkpoint is still written correctly but the script exits non-zero after save with AttributeError: 'TokenizerConfig' object has no attribute 'make_vocab_size_divisible_by' against this branch's megatron.core.tokenizers.utils.build_tokenizer.

Risk

All new files under examples/multimodal_dev/.
data_samplers.py change is fully backwards-compatible: behavior is unchanged unless use_vanilla_collate_fn is explicitly set.

Test plan

pytest examples/multimodal_dev/tests/ passes.
scripts/run_qwen35_vl.sh proxy variant trains a few steps on mock data.
CORD-V2 dataset loads with --use-vanilla-collate-fn and trains a few steps.
torchrun --nproc-per-node 4 examples/multimodal_dev/tests/test_cp_thd_correctness.py — CP=1 vs CP=4 BSHD/THD loss + grad_norm within tolerance.

🤖 Generated with Claude Code

copy-pr-bot · 2026-05-12T06:57:00Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Adds a standalone VLM training playground under ``examples/multimodal_dev/`` with Qwen3.5-VL end-to-end. Highlights - Model-agnostic entry point (``pretrain_multimodal.py``) with a ``MODEL_REGISTRY`` so adding a new architecture is just a registry entry plus a backing module. - Qwen3.5-VL model: vision encoder, MRoPE, decoder, factory, specs, configurations covering proxy / 9B / 397B-A17B variants. - Datasets: mock data and CORD-V2 VLM dataset, with THD pack/pad in the collate function. - THD + CP support consolidated in ``forward_step.py`` and the model layer (uses MRoPE THD pre-computation and ``cu_seqlens_q_padded`` CP partitioning). - Run script + README, plus tests for MRoPE parity, CP correctness, CP support, and THD correctness / e2e. Also gates the torch DataLoader vanilla-collate path on the new ``use_vanilla_collate_fn`` arg (one-line change to ``megatron/training/datasets/data_samplers.py``) so CORD-V2 works under BSHD. Functional dependency: the new model arch sets ``mrope_interleaved=True`` in its config and relies on the core MRoPE interleaved layout introduced in a separate PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: BestJuly <19769279+BestJuly@users.noreply.github.com>

… preprocessing Fixes 8 issues in vlm_dataset.py found by review against Megatron-Bridge's qwen2_5_collate_fn reference implementation. - loss_mask off-by-one (Bug 1): the previous mask was built on input_ids while labels were shifted, dropping the image->text supervision signal at the boundary. Now masks structural tokens on the shifted labels and also shifts loss_mask itself left by 1. - missing SFT prompt masking (Bug 2): user-turn and chat-template tokens were trained on. Now uses backward substring token search (mirroring create_multiturn_loss_mask_by_search) to unmask only the assistant answer span. - seq_length not enforced (Bug 3): long CORD-V2 samples could overflow. Now end-truncates input_ids in __getitem__ with a warning. - unsafe pad_token_id fallback (Bug 4): falling back to 0 silently masked a real vocab token. Now falls back to EOS and raises if neither is set. - silent image_token_id miss (Bug 6): fallback could return None, causing dataset / model disagreement. Now raises ValueError. - stale docstrings (Bug 8): updated Qwen2.5-VL / --image-size references to Qwen3.5-VL / --total-seq-length. - narrow skipped_tokens set (Bug 14): vision_start/end, im_start/end, video_pad, endoftext were not masked on labels. Now uses tok.all_special_ids union {pad_id, image_token_id}. - lost Qwen-VL dynamic resolution (Bugs 15/17/19): fixed-square resize removed; conversation content carries the image object; qwen_vl_utils.process_vision_info extracts images; processor is called with min_pixels / max_pixels. - pixel_values bf16 conversion (Bug 18): moved from forward_step into the dataset so per-step dtype checks become no-ops. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- raise --manual-gc-interval 5 → 50 to cut GC pause frequency on long runs. - enable --moe-permute-fusion and --moe-router-fusion in the MoE branch (no-op for dense variants since MOE_ARGS is gated on NUM_EXPERTS>0). - enable grad-accumulation fusion under FSDP by dropping --no-gradient-accumulation-fusion from FSDP_ARGS. - add --log-timers-to-tensorboard and --log-params-norm to surface timer breakdown and parameter L2 norm in TB/wandb. - drop the hardcoded CKPT_LOAD path from the in-script example invocations so the comment reflects from-scratch CP correctness runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Update the 'Copyright (c) 2025, NVIDIA CORPORATION' line to 2026 across all newly-added Python files under examples/multimodal_dev/ for the Qwen3.5-VL training example. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Victarry · 2026-05-24T06:54:57Z

While running the PR's examples/multimodal_dev/tests tests, I hit a collection failure on the original branch: test_thd_e2e.py and test_thd_correctness.py import _pack_batch from examples.multimodal_dev.forward_step, but forward_step.py does not define/export _pack_batch.

Could you add that helper or update the tests to use the intended packing API?

BestJuly · 2026-05-25T04:02:00Z

Please add the latest checkpoint conversion guide so users to resume from HF checkpoint and run. Previously we record the steps in this issue and there should have some updates now.

wplf · 2026-05-25T04:06:20Z

OK, I'm working on the UT of CP now. Checkpoint conversion guide may be finished this afternoon.

wplf · 2026-05-27T06:23:33Z

/ok to test 334d4e1

cryoco · 2026-05-27T07:30:13Z

May I learn about the corresponding TE version in qwen3.5 support?

wplf · 2026-05-27T07:40:27Z

May I learn about the corresponding TE version in qwen3.5 support?
FYI, TE dependency is fairly loose.

I'm using the latest TE with a cherry-pick from TE pr 2932 on gb.
If you're using Hopper, please ensure your cuDNN version is greater than 9.19.0.
Using Hopper with fused attention + thd + cuDNN below 9.19.0 will cause NaN issues during THD training.

cryoco · 2026-05-27T11:21:53Z

May I learn about the corresponding TE version in qwen3.5 support?
FYI, TE dependency is fairly loose.

I'm using the latest TE with a cherry-pick from TE pr 2932 on gb. If you're using Hopper, please ensure your cuDNN version is greater than 9.19.0. Using Hopper with fused attention + thd + cuDNN below 9.19.0 will cause NaN issues during THD training.

Thanks. BTW, is there any current performance benchmark on hopper or blackwell?

wplf · 2026-05-27T12:22:03Z

May I learn about the corresponding TE version in qwen3.5 support?
FYI, TE dependency is fairly loose.

I'm using the latest TE with a cherry-pick from TE pr 2932 on gb. If you're using Hopper, please ensure your cuDNN version is greater than 9.19.0. Using Hopper with fused attention + thd + cuDNN below 9.19.0 will cause NaN issues during THD training.

Thanks. BTW, is there any current performance benchmark on hopper or blackwell?

We are still working on it.

Victarry

Thanks for this PR!
Reproduced the E2E pipeline from weight conversion to model training and worked well. Left a few comments.

@Victarry

Resolves the inline comments from @Victarry's PR review on NVIDIA#4751. * vision_encoder.py — patch merger GELU was `approximate='tanh'` while the in-code NOTE acknowledged HF uses `approximate='none'`. Switched to `approximate='none'` to match the official Qwen3VLVisionPatchMerger numerics for HF -> Megatron checkpoint parity. * pretrain_multimodal.py — added an explicit guard against `--pipeline-model-parallel-size > 1`. The model_provider builds the full model on every rank and ignores pre_process / post_process stage flags, so PP>1 would silently break Megatron's pipeline-parallel contract. Fail fast instead. * scripts/run_qwen35_vl.sh — three fixes: 1. `EP` now defaults to 1 (was 2). MoE variants must opt in via the environment override. 2. After the variant case block, fail fast if `NUM_EXPERTS=0 && EP>1` so a dense run such as `MODEL_VARIANT=9b ./run_qwen35_vl.sh` no longer trips Megatron's arg validation downstream. 3. `--moe-router-force-load-balancing` was unconditionally added to GPT_MODEL_ARGS (and therefore enabled even when no MoE args were emitted). It is now gated behind `FORCE_LOAD_BALANCING=1`, defaults off, and is appended to MOE_ARGS only when MoE is active. Real finetuning runs no longer freeze router routing decisions by default. * data/{vlm_dataset.py -> cord_v2.py} + models/__init__.py — renamed the CORD-V2-specific module from the generic-sounding `vlm_dataset.py` to `cord_v2.py`, updated the model registry path string accordingly, and added an "Adding another VLM dataset" section to the module docstring documenting the per-dataset module + `MODEL_REGISTRY["..."]["dataset_providers"]` registration pattern. * models/qwen35_vl/mrope.py — added a performance note on the `_build_sample_mrope_positions` helper documenting the `.tolist()` / `.item()` GPU<->CPU sync points and CUDA-graph incompatibility, and the precompute-in-collate / cache-by-shape follow-up plan. Behavior preserved here pending a follow-up data pipeline change. The other tests-import comment (test_thd_*.py importing `_pack_batch`) is already addressed on this branch: the helper is now named `pack_or_pad_batch` and the tests import that symbol.

@Victarry

Resolves the inline comments from @Victarry's PR review on NVIDIA#4751. * vision_encoder.py — patch merger GELU was `approximate='tanh'` while the in-code NOTE acknowledged HF uses `approximate='none'`. Switched to `approximate='none'` to match the official Qwen3VLVisionPatchMerger numerics for HF -> Megatron checkpoint parity. * pretrain_multimodal.py — added an explicit guard against `--pipeline-model-parallel-size > 1`. The model_provider builds the full model on every rank and ignores pre_process / post_process stage flags, so PP>1 would silently break Megatron's pipeline-parallel contract. Fail fast instead. * scripts/run_qwen35_vl.sh — three fixes: 1. `EP` now defaults to 1 (was 2). MoE variants must opt in via the environment override. 2. After the variant case block, fail fast if `NUM_EXPERTS=0 && EP>1` so a dense run such as `MODEL_VARIANT=9b ./run_qwen35_vl.sh` no longer trips Megatron's arg validation downstream. 3. `--moe-router-force-load-balancing` was unconditionally added to GPT_MODEL_ARGS (and therefore enabled even when no MoE args were emitted). It is now gated behind `FORCE_LOAD_BALANCING=1`, defaults off, and is appended to MOE_ARGS only when MoE is active. Real finetuning runs no longer freeze router routing decisions by default. * data/{vlm_dataset.py -> cord_v2.py} + models/__init__.py — renamed the CORD-V2-specific module from the generic-sounding `vlm_dataset.py` to `cord_v2.py`, updated the model registry path string accordingly, and added an "Adding another VLM dataset" section to the module docstring documenting the per-dataset module + `MODEL_REGISTRY["..."]["dataset_providers"]` registration pattern. * models/qwen35_vl/mrope.py — added a performance note on the `_build_sample_mrope_positions` helper documenting the `.tolist()` / `.item()` GPU<->CPU sync points and CUDA-graph incompatibility, and the precompute-in-collate / cache-by-shape follow-up plan. Behavior preserved here pending a follow-up data pipeline change. The other tests-import comment (test_thd_*.py importing `_pack_batch`) is already addressed on this branch: the helper is now named `pack_or_pad_batch` and the tests import that symbol.

New test ``tests/test_vision_patch_merger_parity.py`` verifies the Megatron patch merger against an inlined verbatim copy of HuggingFace ``Qwen3VLVisionPatchMerger`` (``use_postshuffle_norm=False`` branch from ``transformers/src/transformers/models/qwen3_vl/modeling_qwen3_vl.py``). The HF reference is inlined so the test has no runtime dependency on the ``transformers`` package. The test copies HF state-dict tensors into the Megatron module (TP=1, 1:1 mapping), runs both on the same random input, and asserts ``torch.testing.assert_close`` on the logits in fp32 and bf16: [torch.float32] shape=(16, 3584) max_abs_diff=2.551e-05 (atol=1e-4) [torch.bfloat16] shape=(16, 3584) max_abs_diff=3.906e-03 (atol=5e-2) The fp32 residual is structural (TE LayerNorm vs nn.LayerNorm use different fused reduction orders) and the bf16 figure is at the arithmetic floor for a two-layer MLP. This pins the GELU ``approximate='none'`` fix (commit 8aace7b) against future regressions. Run with:: torchrun --nproc_per_node=1 \\ examples/multimodal_dev/tests/test_vision_patch_merger_parity.py

@Victarry

Resolves the inline comments from @Victarry's PR review on NVIDIA#4751. * vision_encoder.py — patch merger GELU was `approximate='tanh'` while the in-code NOTE acknowledged HF uses `approximate='none'`. Switched to `approximate='none'` to match the official Qwen3VLVisionPatchMerger numerics for HF -> Megatron checkpoint parity. * pretrain_multimodal.py — added an explicit guard against `--pipeline-model-parallel-size > 1`. The model_provider builds the full model on every rank and ignores pre_process / post_process stage flags, so PP>1 would silently break Megatron's pipeline-parallel contract. Fail fast instead. * scripts/run_qwen35_vl.sh — three fixes: 1. `EP` now defaults to 1 (was 2). MoE variants must opt in via the environment override. 2. After the variant case block, fail fast if `NUM_EXPERTS=0 && EP>1` so a dense run such as `MODEL_VARIANT=9b ./run_qwen35_vl.sh` no longer trips Megatron's arg validation downstream. 3. `--moe-router-force-load-balancing` was unconditionally added to GPT_MODEL_ARGS (and therefore enabled even when no MoE args were emitted). It is now gated behind `FORCE_LOAD_BALANCING=1`, defaults off, and is appended to MOE_ARGS only when MoE is active. Real finetuning runs no longer freeze router routing decisions by default. * data/{vlm_dataset.py -> cord_v2.py} + models/__init__.py — renamed the CORD-V2-specific module from the generic-sounding `vlm_dataset.py` to `cord_v2.py`, updated the model registry path string accordingly, and added an "Adding another VLM dataset" section to the module docstring documenting the per-dataset module + `MODEL_REGISTRY["..."]["dataset_providers"]` registration pattern. * models/qwen35_vl/mrope.py — added a performance note on the `_build_sample_mrope_positions` helper documenting the `.tolist()` / `.item()` GPU<->CPU sync points and CUDA-graph incompatibility, and the precompute-in-collate / cache-by-shape follow-up plan. Behavior preserved here pending a follow-up data pipeline change. The other tests-import comment (test_thd_*.py importing `_pack_batch`) is already addressed on this branch: the helper is now named `pack_or_pad_batch` and the tests import that symbol.

wplf · 2026-05-28T03:41:32Z

/ok to test df54ae7

copy-pr-bot · 2026-05-28T03:41:36Z

/ok to test df54ae7

@wplf, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

wplf · 2026-05-28T03:42:35Z

/ok to test 4dfdd06

svcnvidia-nemo-ci · 2026-05-29T09:30:18Z

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/26629578247

…shadow PR Adopt the merged dev [5/5] shadow PR NVIDIA#4751 (commit 58f3e67) verbatim for examples/multimodal_dev/ — it carries newer bug fixes: - replace data/vlm_dataset.py with data/cord_v2.py - add tests/_helpers.py, tests/test_cp_thd_correctness.py, tests/test_vision_patch_merger_parity.py - sync the remaining 23 example files to NVIDIA#4751's content data_samplers.py is intentionally NOT changed to match NVIDIA#4751: main uses args.hybrid_context_parallel whereas dev uses args.dynamic_context_parallel (the arg was renamed across branches), so NVIDIA#4756's existing line is the correct main adaptation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Squash of the fused-mRoPE work (Add Qwen3.5 MRoPE fusion benchmark support; Fix THD mRoPE CP fallback consistency; mRoPE THD review cleanup; enforce per-sequence CP divisibility on the fused THD launch path; unit-test coverage for real Qwen3.5-VL shapes). Adds a fused mRoPE kernel (megatron/core/fusions/fused_mrope.py) with an is_fused_mrope_available() gate, raw-mrope-freqs plumbing through rope_utils / rotary_pos_embedding / gpt_model / attention, the transformer_config + arguments toggles, and tests/unit_tests/fusions/ test_fused_mrope.py. Core only: the examples/multimodal_dev integration is intentionally dropped because that example is already upstream (NVIDIA#4751) and has diverged from this branch's copy. Co-Authored-By: Li Tao <litao@nvidia.com>

Squash of the fused-mRoPE work (Add Qwen3.5 MRoPE fusion benchmark support; Fix THD mRoPE CP fallback consistency; mRoPE THD review cleanup; enforce per-sequence CP divisibility on the fused THD launch path; unit-test coverage for real Qwen3.5-VL shapes). Adds a fused mRoPE kernel (megatron/core/fusions/fused_mrope.py) with an is_fused_mrope_available() gate, raw-mrope-freqs plumbing through rope_utils / rotary_pos_embedding / gpt_model / attention, the transformer_config + arguments toggles, and tests/unit_tests/fusions/ test_fused_mrope.py. Core only: the examples/multimodal_dev integration is intentionally dropped because that example is already upstream (NVIDIA#4751) and has diverged from this branch's copy. Co-Authored-By: Li Tao <litao@nvidia.com> Signed-off-by: jinliangl <975761915@qq.com>

* test: re-enable test_pp2_create_cudagraphs_first_stage on TE 2.15+ (NVIDIA#4985) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tests): initialize num_microbatches calculator in vision cudagraph tests (NVIDIA#4986) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Add allow_failure flag to gpt and moe recipes that are failing in nightlies (NVIDIA#4905) * Drain predecessor reduce-scatter at dispatch time (NVIDIA#4940) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * nightly(ci): Update golden values for functional t5 tests (NVIDIA#4995) * chore: rotate oncall schedule * [main] Refactor and Improve MoE Logginginit commit (NVIDIA#3431) Co-authored-by: Robin Zhang <robinz@nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> Co-authored-by: Shifang Xu <shifangx@nvidia.com> * ci: validate release branch-rules (NVIDIA#4929) Signed-off-by: oliver könig <okoenig@nvidia.com> * [Megatron-FSDP] Add conditional param.grad dereferencing logic to support full-iteration (FWD-BWD) CUDA graphability. (NVIDIA#4663) Signed-off-by: Cory Ye <cye@nvidia.com> * test: restrict iter-time comparison to steady-state window (NVIDIA#5010) Signed-off-by: oliver könig <okoenig@nvidia.com> * Add mHC support for HybridModel on dev (NVIDIA#4949) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> * fix(test): pin eval-global-batch-size on 15b gb200 release configs (NVIDIA#5022) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [fix] Release MTP assertion when EP overlap with PP=1 (NVIDIA#4796) * fix(test): widen iter-time steady-state window for short tests (NVIDIA#5023) Signed-off-by: oliver könig <okoenig@nvidia.com> * Perf fix (NVIDIA#4996) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add dev-feature preservation gate and change schedule (NVIDIA#4773) * chore(test): remove orphan nemotron3_super_release_g200 dir (NVIDIA#5024) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ignore Claude worktree directory (NVIDIA#5020) * Update copy-pr-bot.yaml [skip ci] * ci: update CI workflow conditions for integration tests (NVIDIA#4658) * Add NVSkills CI request workflow (NVIDIA#5033) * DDP wrap pg size fixes (NVIDIA#5006) Signed-off-by: Maanu Grover <maanug@nvidia.com> * fix(layer_wise): tag MTP-stage word_embeddings as is_embedding_or_output_parameter (NVIDIA#5034) * [Dev] Add Qwen3 30B MoE recipes (NVIDIA#5012) * [DEV] fix(megatron-fsdp): reduce padding for grouped expert weights (NVIDIA#5013) Co-authored-by: Xin Yao <xiny@nvidia.com> * [Dev] fix(combined-1f1b): release loss-node input storage after combined backward (NVIDIA#4908) Co-authored-by: Xin Yao <xiny@nvidia.com> * [dev] [fix] [DeepSeek-v4] fix dense loss and rope type in DSv4 Hybrid Attention (NVIDIA#5018) Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com> * Move LTS dependencies from pyproject.toml to Dockerfile.ci.lts (NVIDIA#4877) * Use shared ModelOpt calibration loop on 0.45+ with 0.44 fallback fix (NVIDIA#4881) Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * test(release): skip golden comparison on intermediate resume windows (NVIDIA#5040) Signed-off-by: oliver könig <okoenig@nvidia.com> * [mimo] Thread position_ids through MimoModel for multimodal RoPE (NVIDIA#4938) Signed-off-by: Li Ding <liding@nvidia.com> * build: Switch DSv3 on H100 to HybridEP (NVIDIA#5039) Signed-off-by: oliver könig <okoenig@nvidia.com> * Fix: Import unwrap_model from megatron.core.utils in modelopt examples (NVIDIA#5045) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Simple and stable Inference APIs (NVIDIA#4697) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci: Add notification step for MBridge downstream test results (NVIDIA#5028) * [dev] [5/5] Qwen3.5 support: Qwen3.5-VL training example (NVIDIA#4751) Co-authored-by: BestJuly <19769279+BestJuly@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: Update Docker image version to 26.04-py3 on dev (NVIDIA#5051) Signed-off-by: oliver könig <okoenig@nvidia.com> * Delete output tensor early (NVIDIA#4742) * Support ScaledSReLU in TE grouped MLP fuser (NVIDIA#4859) Co-authored-by: sraman <sraman@users.noreply.github.com> Co-authored-by: Siddhartha Raman S <sraman@login-lyris01.lyris.clusters.nvidia.com> Co-authored-by: a <a> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: sraman-rgb <270218152+sraman-rgb@users.noreply.github.com> * Skip gradient updates when grad norm exceeds threshold (NVIDIA#3460) Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Gerald Shen <geshen@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add 9 user skills (NVIDIA#5066) Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com> Co-authored-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com> * test(nemotron): align nemotron3 super GB200 goldens with exit-interval 4768 (NVIDIA#5069) Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Update transformer-engine dependency to version 2.16.0 (NVIDIA#4992) * Update energon version requirement (NVIDIA#4572) Signed-off-by: Maanu Grover <maanug@nvidia.com> * Fix test failures for new inference APIs (NVIDIA#5068) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): set PYTHONUNBUFFERED=1 in JET workload env (NVIDIA#5072) Signed-off-by: oliver könig <okoenig@nvidia.com> * Preserve non-FSDP-unit buckets across AllGatherPipeline reset (NVIDIA#4717) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add opt-in MXFP8 LM-head output projection (NVIDIA#4825) Co-authored-by: a <a> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> * chore(beep boop 🤖): Bump (main) (2026-06-01) * fix(ci): bound JET pipeline polling with a watchdog to prevent indefinite hangs (NVIDIA#5076) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: prune old artifacts on cluster lustre during weekly/release runs (NVIDIA#5084) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(test): isolate ckpt-resume tensorboard per phase (NVIDIA#5074) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: unmark EP A2A activation offload test flaky (NVIDIA#5009) * Change ownership groups (NVIDIA#5021) * test: skip mfsdp_fully_shard cases when world_size < mesh size (NVIDIA#4487) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix mimo optimizer checkpoint metadata restore (NVIDIA#4791) Signed-off-by: Li Ding <liding@nvidia.com> * [mimo] Support bridge fan-out for variable modality tokens (NVIDIA#5062) Signed-off-by: Li Ding <liding@nvidia.com> * Add separate mtp_grad_scale_func for MTP loss scaling (NVIDIA#3459) Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [training migration] Migrate GPT builder (NVIDIA#4741) Signed-off-by: Maanu Grover <maanug@nvidia.com> * Make Mamba conv params direct mixer params (NVIDIA#4899) Signed-off-by: Jingyue Wu <wujingyue@gmail.com> * Update copy-pr-bot.yaml [skip ci] * Update oncall reviewer assignment (NVIDIA#5093) * Pass explicit process groups to hybrid logging (NVIDIA#4781) * Clean up top-level repository files (NVIDIA#5097) * [main] fix(moe): Fix several bugs for DSA rope and spec. (NVIDIA#3026) Co-authored-by: Yuzhong Wang <yuzhongw@computelab-frontend-3.nvidia.com> * [Dev] Skip identity alltoall chunk sort (NVIDIA#5102) * Move MIMO unit tests into models/mimo (NVIDIA#5063) * test: update DeepSeek FSDP2 GB200 memory golden (NVIDIA#5094) * Remove DeepEP hardware limit check (NVIDIA#4846) * Update transformer-engine dependency to revision 4220403 (NVIDIA#5112) * ci: make CI resilient to pip/uv network timeouts (NVIDIA#5118) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: treat docker container-removal conflict as flaky (NVIDIA#5120) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix GDN DTensor splitting for FSDP checkpointing (NVIDIA#4843) Signed-off-by: conver334 <conver334@gmail.com> * Fix MoE aux_loss / z_loss gradient scaling with TP > 1 (NVIDIA#5047) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update copy-pr-bot.yaml [skip ci] * Update Claude copy workflow to enforce user restrictions and improve error messages (NVIDIA#5117) * Add advisory process group guidance to Claude reviews (NVIDIA#5111) * build: cap pydantic<2.14 in transformer-engine dependency metadata (NVIDIA#5125) Signed-off-by: Chen Cui <chcui@nvidia.com> * fix(test): skip scalar-less tensorboard event files in resume checks (NVIDIA#5121) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: rotate oncall schedule * docs: fix contributor guide typo (NVIDIA#4858) Signed-off-by: LeSingh1 <sshaurya914@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> * ci(unit-tests): split slow unit-test buckets over 15min SLA (NVIDIA#5133) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix DSA indexer loss not averaged across micro-batches (NVIDIA#4070) Co-authored-by: mokai <mokai@baidu.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> * Update MINOR version to 19 (NVIDIA#5096) * Fix Muon QKV split for gated attention (NVIDIA#4728) Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> * Roll input IDs for MTP labels (NVIDIA#3457) Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Refactor: Move paged stashing Triton kernels (NVIDIA#5003) * Adding blackwell tests (NVIDIA#5113) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Relax atol for test_router_gating_linear router_dtype=torch.float32 (NVIDIA#4915) Signed-off-by: Aditya Singh <adisin650@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> * Fix incorrect inference metadata tensor dtypes (NVIDIA#4855) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Signed-off-by: qiyuw <qiyuw@nvidia.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com> Co-authored-by: Ajay <abalasa@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> Co-authored-by: sraman-rgb <sraman@nvidia.com> Co-authored-by: Siddhartha Raman S <sraman@login-lyris02.lyris.clusters.nvidia.com> Co-authored-by: Xin Yao <xiny@nvidia.com> Co-authored-by: gautham-kollu <gkollu@nvidia.com> Co-authored-by: Siddhartha Raman S <sraman@login-lyris01.lyris.clusters.nvidia.com> Co-authored-by: Qiyu Wan <39144338+WanZzzzzz@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Devil1716 <149754374+Devil1716@users.noreply.github.com> * Disable TE cross entropy loss fusion (NVIDIA#5115) Co-authored-by: Mike Chrzanowski <mchrzanowski@gcp-nrt-cs-001-login-001.cm.cluster> * fix(optimizer): gate ChainedOptimizer MXFP8 defer-sync on DDP-level overlap_param_gather (NVIDIA#4982) Signed-off-by: oliver könig <okoenig@nvidia.com> * Update copy-pr-bot.yaml [skip ci] * Pass TP group to unfused cross entropy (NVIDIA#5128) * fix: correct dsv4_hybrid Q-up FLOPs by using args.v_head_dim (NVIDIA#5142) * [dev] [follow-up] Qwen3.5 support: MoE aux loss padding_mask (NVIDIA#4776) Co-authored-by: BestJuly <19769279+BestJuly@users.noreply.github.com> * [Dev] Support isolated MTP loss (NVIDIA#5080) Co-authored-by: liuzhenhai93 <liuzhenhai93@outlook.com> * test(elastification): quarantine flaky test_gumbel_determinism as flaky_in_dev (NVIDIA#5156) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(notify): mention mcore-oncall and philipp on critical CI events (NVIDIA#5152) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Change the cudagraph distribution from linearly to exponentially-decreasing + grid for mixed prefill (NVIDIA#3509) * ci: Disable a few gb200 test cases to support 2 branches. (NVIDIA#5151) * build: Switch DSv3 on H100 to HybridEP (NVIDIA#5164) * Add MTP acceptance rate metrics (NVIDIA#3458) Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Nemotron Ultra config for ModelOpt examples (NVIDIA#5159) Signed-off-by: Jennifer Chen <jennifchen@nvidia.com> * Make MTP / prefix cache stats persist for engine lifetime (NVIDIA#4101) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Restore Greptile configuration (NVIDIA#5166) * chore: bump `_code_freeze` workflow to `v1.4.2` (NVIDIA#5132) Signed-off-by: oliver könig <okoenig@nvidia.com> * [dev] moe(fix): Avoid TE cuda graph dummy attention masks (NVIDIA#5131) Co-authored-by: Yuzhong Wang <yuzhongw@computelab-frontend-3.nvidia.com> * ci: Remove docs build test in favor of release test (NVIDIA#5182) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Move TE cross entropy guard to training args (NVIDIA#5162) Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix error in deepseek parser (NVIDIA#5136) * Fix logprob slicing for 0 generated token case (NVIDIA#5167) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> * [Perf] Fold frozen linear dgrad matmul (NVIDIA#5092) Signed-off-by: Chen Cui <chcui@nvidia.com> * Clamp `max_new_tokens` in MInf to mirror vllm (NVIDIA#5181) * build: add managed = true to [tool.uv] (NVIDIA#5190) Signed-off-by: Kajal Jain <kajalj@nvidia.com> * Stabilize GB200 inference perf tests against cold-start noise (NVIDIA#5171) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: post-CI corrections (docs dup fields, nvrx version gate, utils re-export) - Remove merge-introduced duplicate definitions of use_transformer_engine_op_fuser and moe_expert_rank_capacity_factor (transformer_config.py) and the duplicate _post_param_sync method (param_and_grad_buffer.py) — tripped the docs build. - Relax is_nvrx_min_version() to compare release segments so dev's pinned nvidia-resiliency-ext pre-release (0.6.0.dev33) satisfies main's '>= 0.6.0' assertion (the required-symbol check remains the authoritative guard). - Re-export unwrap_model from megatron/training/utils/__init__.py. main refactored the monolithic training/utils.py into a package whose __init__ dropped this re-export; dev's training.py (and others) import it via 'from .utils import', which broke tests/unit_tests/conftest.py's import chain (cascaded to all suites). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [dev] [DeepSeek-v4] Add ClampedSwiGLU to MoE mlp_op_fuser and add force balance to hash routing (NVIDIA#5130) * nvidia style guide audit for getting started folder (NVIDIA#5168) Signed-off-by: meg miranda <mmiranda@nvidia.com> * AI aided audit for Nvidia Style guidance (NVIDIA#5141) Signed-off-by: meg miranda <mmiranda@nvidia.com> Co-authored-by: Philip Petrakian <pgpetrak@gmail.com> * Enable selective recompute for `norm_out` in GDN layers (NVIDIA#4715) * fix(elastification): align with get_batch + utils refactors (NVIDIA#5194) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(combined-1f1b): release loss-node input storage after combined backward (NVIDIA#4909) * Minor improvements for Dynamic-cp (NVIDIA#4226) Signed-off-by: xiaoyao0115 <1804647152@qq.com> Signed-off-by: tailaim <tailaim@nvidia.com> * fix: post-CI corrections * chore(beep boop 🤖): Bump (main) (2026-06-08) * Avoid stat syscall in rerun result validation (NVIDIA#5107) Signed-off-by: dimapihtar <dpykhtar@nvidia.com> * docs: Update Latest News in README.md (NVIDIA#3790) * Fix bug with Megatron-FSDP zero counter not working with decoupled gradients. (NVIDIA#4802) Signed-off-by: Cory Ye <cye@nvidia.com> * varlendataset for thd e2e and benchmark (NVIDIA#4832) Signed-off-by: tailaim <tailaim@nvidia.com> Signed-off-by: xiaoyao0115 <1804647152@qq.com> * ci: add smoke tests (NVIDIA#5143) * Add mtp_detach_heads config to detach MTP head inputs (NVIDIA#3456) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: wrap mtp logging comment * docs: fix install guide NGC container anchor (NVIDIA#5224) * [Dev] Add separate toggle for varlen input padding for HybridEP in THD training (NVIDIA#5048) Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> Co-authored-by: Xin Yao <xiny@nvidia.com> * Fuse per-sequence AlltoAll into a unified one in GDN forward (NVIDIA#4913) * Apply MIMO SP/CP sharding with explicit groups and enable THD in non-colocated path (NVIDIA#5150) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix CUDA IMA in fsdp_double_buffer when an FSDP unit's bucket doesn't fit the pool (NVIDIA#4810) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add named layouts to HyperCommGrid for heterogeneous parallelism (NVIDIA#5148) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix wgrad race condition when using double buffers. (NVIDIA#5222) Signed-off-by: Cory Ye <cye@nvidia.com> * Move uneven DTensor distributed fixture to conftest (NVIDIA#5237) * Route bridge communicator cross-grid P2P through a dedicated process group (NVIDIA#5234) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix test_split_tensor_along_last_dim to actually assert correctness (NVIDIA#4710) Co-authored-by: peibli <lipeibao@126.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> * chore: rotate oncall schedule * Add optional group= to common_utils model/data-parallel reduction helpers (NVIDIA#5251) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: Allow DCO check in merge queue and add DCO requirement to Contribution guide (NVIDIA#5278) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Update copy-pr-bot.yaml [skip ci] * Add MIMO hetero topology + distributed bootstrap (examples/mimo training-loop folder) (NVIDIA#5260) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Remove checkpoint-time GPU cache reclaim workaround (NVIDIA#5170) Signed-off-by: Skand Hurkat <shurkat@nvidia.com> Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * [Dev] DeepSeek-V4-Flash recipe 20260610 (NVIDIA#5266) * [Dev] Generalized fix for mxfp8 param gather (NVIDIA#4994) Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> Co-authored-by: Xin Yao <xiny@nvidia.com> * [Dev] Cherry-pick MTP detach heads (NVIDIA#5223) * [TE] Restore original CP group after dynamic CP forward in TEDotProductAttention (NVIDIA#5215) Co-authored-by: rionawang <rionawang@tencent.com> * [examples] Add dynamic context parallel benchmark example (NVIDIA#5123) * Remove duplicate nccl_allocator import (NVIDIA#5057) * [dev] Add experimental Megatron Lite as agentic exploration (NVIDIA#4885) Co-authored-by: Deyu Fu <deyuf@nvidia.com> * fix(ci): resolve t5 dataloader stall + GRPO cudagraph-memory regression (CI-validated) (NVIDIA#5280) Signed-off-by: Yan Xu <yxu1@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix Dockerfile warnings (NVIDIA#4856) * Fix fused MLA delayed weight grad hooks (NVIDIA#5273) Signed-off-by: Siddhartha Raman Sundara Raman <270218152+sraman-rgb@users.noreply.github.com> Co-authored-by: Siddhartha Raman Sundara Raman <270218152+sraman-rgb@users.noreply.github.com> * ci: limit retries on unsuccessful test launches (NVIDIA#5275) Signed-off-by: Ajay Balasa <abalasa@nvidia.com> * Thread pg_collection into get_model DDP bucket sizing (NVIDIA#5250) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Enable non-deterministic results in model configuration for nemotron tests (NVIDIA#5239) * Stabilize hybrid nanov3 gb200 perf (NVIDIA#5295) * Clip mtp grads separately when mtp_detach_heads=True (NVIDIA#4116) Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com> Co-authored-by: Anish Mahishi <amahishi@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Thread pg_collection into train_step reductions (NVIDIA#5259) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: Allow DCO check in merge queue and add DCO requirement (NVIDIA#5305) Signed-off-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> * fix: restore dev-only training.py features dropped by the main override The merge takes main's training.py wholesale (per the sync skill's override list), which silently reverted four dev-only features whose supporting core code the merge keeps at dev's version. Each surfaced as a CI failure: 1. Dynamic context-parallel API rename. Dev renamed get_hybrid_data_context_parallel_groups -> get_dynamic_data_context_parallel_groups (identical signature) and args.hybrid_context_parallel -> dynamic_context_parallel. Point training.py at dev's names. Fixes the conftest.py ImportError that cascaded to every unit-test bucket. 2. Dynamic-CP / sequence-packing data loading. Replace main's setup-time HybridCPDataLoaderWrapper wrap with dev's per-step wrap_data_iterator (gated on config.sequence_packing_scheduler), which returns a RerunDataIterator-compatible iterator. Fixes the *_cp4_dcp RerunDataIterator assertion. 3. MTP loss logging scale. Dev's MTPLossLoggingHelper stores raw loss sums and token counts and computes the per-token loss after reduction, so the log scale must be 1.0; main's 1/get_num_microbatches() divided the reported mtp_N loss by num_microbatches. Fixes moe gpt3_..._scoped_cudagraph mtp_1 loss (was 16x too small: 0.682 vs golden 10.915). 4. DSA indexer loss cross-PP reduction. dsa.py requires num_layers (and csa_compress_ratios) so first-pipeline-stage ranks lazily initialize the tracker and join the cross-PP all_reduce in reduce_loss_in_tracker; main's call omitted them, so stage 0 returned early and the last stage hung on an unmatched all_reduce. Fixes the gpt3_..._dsv4_hybrid_mhc_mtp NCCL timeout. Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@users.noreply.github.com> * [dev]: faster implementation of mHC fused kernels (NVIDIA#4624) Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> * Allow for pre-bound socket to be passed in server (NVIDIA#5301) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> * Offline Logits-Based Knowledge Distillation (NVIDIA#5019) Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> * Handle None values in sampling parameters (NVIDIA#5300) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com> * Add moe loss normalization for RL SFT (NVIDIA#3956) Signed-off-by: Pranav Prashant Thombre <pthombre@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Add code owners for optimizer-related files (NVIDIA#5297) Signed-off-by: janEbert <janpabloe@nvidia.com> Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> * Fix EP=1 inference by allocating buffers anyway (NVIDIA#5233) Signed-off-by: Helen Ngo <helenn@nvidia.com> * Fix crash due to tool call at sequence length (NVIDIA#5302) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com> * Inference: Cudagraph-aware admission gating in prefill scheduler (NVIDIA#4870) Signed-off-by: Helen Ngo <helenn@nvidia.com> * Account for reasoning token stripping (NVIDIA#5313) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> * Thread pg_collection through wrap_model_chunks_with_ddp (NVIDIA#5328) Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Dev] Add MoE recipe performance summary (NVIDIA#5289) Signed-off-by: Dennis Liu <denliu@denliu.nvidia.com> Co-authored-by: Dennis Liu <denliu@denliu.nvidia.com> * [Dev] Add DeepEP v2 flex dispatcher backend (NVIDIA#4793) Signed-off-by: tongliu <tongliu@nvidia.com> Co-authored-by: Dennis(Zhenhuan) Liu <denliu@nvidia.com> * fix tflops calculation when sequence_packing_scheduler is not none (NVIDIA#5342) Signed-off-by: xiaoyao0115 <1804647152@qq.com> * chore(beep boop 🤖): Bump (main) (2026-06-15) * [dev] bump emerging optimizers to v0.3.0 (NVIDIA#5320) Signed-off-by: Deyu Fu <deyuf@nvidia.com> * Fix LatentMoE theoretical memory estimate (NVIDIA#5145) Signed-off-by: Shijie Wang <jaywan@nvidia.com> * Add zstandard package to Docker LTS requirements. Fix nightly failures (NVIDIA#5347) Signed-off-by: Ajay Balasa <abalasa@nvidia.com> * fix: preserve seqlen stats in train_step Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com> * Thread MIMO support through the stock training loop (schedule + optimizer) (NVIDIA#5333) Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Enable Deepseek-v4 hybrid_model in dev branch Part (1/N) (NVIDIA#5042) Signed-off-by: guihong-nv <guihongl@nvidia.com> Signed-off-by: Guihong Li <guihongl@oci-hsg-cs-001-vscode-02.cm.cluster> Signed-off-by: Yan Xu <yxu1@nvidia.com> Signed-off-by: Guihong Li <guihongl@nvidia.com> Co-authored-by: Guihong Li <guihongl@oci-hsg-cs-001-vscode-02.cm.cluster> Co-authored-by: Yan Xu <yxu1@nvidia.com> Co-authored-by: hx <hongxiaob@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ssm): whole-module 'gdn' selective recompute for GatedDeltaNet (NVIDIA#5296) Signed-off-by: jinliangl <975761915@qq.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Xin Yao <xiny@nvidia.com> * [Fix] Fix optimizer parameter override bugs. (NVIDIA#5213) Signed-off-by: yangfan.bai <yangfan.bai@shopee.com> Co-authored-by: yangfan.bai <yangfan.bai@shopee.com> Co-authored-by: Xin Yao <xiny@nvidia.com> * [Dev] Add Megatron-FSDP weight prefetch for full recompute (NVIDIA#5175) Signed-off-by: hongbinl <hongbinl@nvidia.com> * ci: default functional test time limit to 4h for release/weekly scopes (NVIDIA#5360) Signed-off-by: oliver könig <okoenig@nvidia.com> * [Dev] add cuda graph support for thd format training. (NVIDIA#4359) Signed-off-by: HaochenYuan <haocheny@nvidia.com> Co-authored-by: Haochen Yuan <haocheny@login-eos01.eos.clusters.nvidia.com> * Fix memory leak with log_max_attention_logit (NVIDIA#4699) (NVIDIA#5067) Signed-off-by: Antoni-Joan Solergibert <asolergibert@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com> * Clean up pretrain_gpt.py and pretrain_hybrid.py formatting and remove module globals (NVIDIA#5351) Signed-off-by: ilml <tolong@nvidia.com> * Add full model cuda graph support for MTP inference (NVIDIA#4950) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> * Expand the Mamba prefix caching memory safety check to include scratch space buffers (NVIDIA#5348) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> * Make Megatron RL only materialize last token logit (NVIDIA#4551) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> * Profiling (NVIDIA#3110) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> * Support fused MLA QKV checkpoint reload (NVIDIA#5310) Signed-off-by: sraman <sraman@nvidia.com> * Add minimal DBuffer implementation (NVIDIA#4835) Signed-off-by: Jingyue Wu <wujingyue@gmail.com> * [Dev] restore DSv4 tflops calc in training and fix the packed seq case (NVIDIA#5358) Signed-off-by: Hongxiao Bai <hongxiaob@nvidia.com> * [split 1/5] Fix packed THD RoPE under CP (NVIDIA#5243) Signed-off-by: Hollow Man <hollowman@opensuse.org> * Update copy-pr-bot.yaml [skip ci] * Document agent PR commit sign-off and signing (NVIDIA#5381) Signed-off-by: Jingyue Wu <wujingyue@gmail.com> * Remove unused distributed pytest markers (NVIDIA#5380) Signed-off-by: Jingyue Wu <wujingyue@gmail.com> * [feat] Support fine-grained activation offloading in fused group mlp (NVIDIA#5082) Signed-off-by: hongbinl <hongbinl@nvidia.com> * Thread tensor-parallel group into the RADIO patch embedder (NVIDIA#5371) Signed-off-by: ykarnati <ykarnati@nvidia.com> * Add MimoModel.zero_grad_buffer delegating to active DDP submodules (NVIDIA#5372) Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * [split 3/5] Refactor absorbed MLA projection handling (NVIDIA#5245) Signed-off-by: Hollow Man <hollowman@opensuse.org> * [dev] moe(perf): Restore fused GDN THD all-to-all on dev (NVIDIA#5389) Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com> * chore: rotate oncall schedule * ci: Remove sync skills workflow (NVIDIA#5091) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Add flaky marker to fine-grained activation offloading test (NVIDIA#5350) (NVIDIA#5368) Signed-off-by: Ajay Balasa <abalasa@nvidia.com> * Revert "Remove checkpoint-time GPU cache reclaim workaround (NVIDIA#5170)" (NVIDIA#5366) Signed-off-by: Ajay Balasa <abalasa@nvidia.com> * Update goldens for weekly tests after pytorch and TE bumps. (NVIDIA#5399) Signed-off-by: Ajay Balasa <abalasa@nvidia.com> * Add MIMO runtime setup: per-role RNG seeding and DDP wrapping (NVIDIA#5285) Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Add --mamba-training-ssm-states-dtype argument (NVIDIA#5309) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com> * chore(beep boop 🤖): Bump (main) (2026-06-22) * Fix Mamba prefix match for chunked prefill (NVIDIA#4758) Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com> * Disag MR2: Refit into multiple destination pools and tied-embedding + UVM fixes (NVIDIA#5187) Signed-off-by: wdykas <wdykas@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Disag MR1: Add inference shard specs and pg-collection building (NVIDIA#5186) Signed-off-by: wdykas <wdykas@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * [Fix] Fix MoE router z-loss compatibility with TE CUDA Graph capture. (NVIDIA#5401) Signed-off-by: yangfan.bai <yangfan.bai@shopee.com> Co-authored-by: yangfan.bai <yangfan.bai@shopee.com> * test: mark ep_a2a_overlap activation-offloading test flaky_in_dev (NVIDIA#5450) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: mark TestParallelTransformerBlockCudagraphs::test_gpu_cudagraph flaky_in_dev (NVIDIA#5475) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: mark gated_delta_net selective-recompute test flaky_in_dev (NVIDIA#5476) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: post-CI corrections for sync test/impl reconciliation 1) test_optimizer.py: reverted to dev. The sync kept dev's multi_latent_attention.py (split q/kv down-proj, no _synthesize_fused_qkv_down_weight), but auto-merged main's test asserting the fused linear_qkv_down_proj.weight key. 2) training.py: guard the dev-only sequence_packing_scheduler config access with getattr (lines in train_step and train()). main's new MIMO schedule-plumbing test (NVIDIA#5333) passes an empty SimpleNamespace config; the reconciled training.py keeps dev's packing path, so the access must tolerate a config lacking the attribute. Real configs are unaffected (getattr returns the same value). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> * fix sequence packing wrapper for eval (NVIDIA#5483) Signed-off-by: xiaoyao0115 <1804647152@qq.com> * [dev] Megatron Lite (4/4) shared attention (NVIDIA#5427) Signed-off-by: Yan Bai <bayan@nvidia.com> * fix: restore fused group MLP offload in main2dev sync (NVIDIA#5493) Signed-off-by: hongbinl <hongbinl@nvidia.com> Signed-off-by: svcnvidia-nemo-ci <svc-nvidia-nemo-ci@nvidia.com> * [dev] [DeepSeek-v4] Packed Sequence (THD) support for DSv4 Hybrid Attention (NVIDIA#5011) Signed-off-by: Hongxiao Bai <hongxiaob@nvidia.com> * Improve default dynamic CP packing scheduler (NVIDIA#5154) Signed-off-by: tailaim <tailaim@nvidia.com> * [dev] moe(perf): Pre-GDR kernel fusion (NVIDIA#5361) Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com> * Preserve DSA output across fused inverse RoPE (NVIDIA#5526) Signed-off-by: kunlunl <kunlunl@nvidia.com> Co-authored-by: Kaixiang Lei <5780122+shyoshyo@users.noreply.github.com> * Enable Deepseek-v4 hybrid_model in dev branch Part (2/N) (NVIDIA#5485) Signed-off-by: guihong-nv <guihongl@nvidia.com> * [dev] Add experimental decoupled compact LayerWise DDP layout for Muon (NVIDIA#5388) Signed-off-by: pingtianl <pingtianl@nvidia.com> Signed-off-by: Pingtian Li <pingtianl@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [dev] Sync Megatron Lite with the latest implementation (NVIDIA#5577) Signed-off-by: Yan Bai <bayan@nvidia.com> * [Dev] fix padding mask docstring (NVIDIA#5598) Signed-off-by: HaochenYuan <haocheny@nvidia.com> * handle split_dtensor --------- Signed-off-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Cory Ye <cye@nvidia.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Signed-off-by: Li Ding <liding@nvidia.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com> Signed-off-by: Jingyue Wu <wujingyue@gmail.com> Signed-off-by: conver334 <conver334@gmail.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: LeSingh1 <sshaurya914@gmail.com> Signed-off-by: Aditya Singh <adisin650@gmail.com> Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Signed-off-by: qiyuw <qiyuw@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Jennifer Chen <jennifchen@nvidia.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com> Signed-off-by: meg miranda <mmiranda@nvidia.com> Signed-off-by: xiaoyao0115 <1804647152@qq.com> Signed-off-by: tailaim <tailaim@nvidia.com> Signed-off-by: dimapihtar <dpykhtar@nvidia.com> Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> Signed-off-by: Shivanjan Chakravorty <shivanjanc@nvidia.com> Signed-off-by: Ajay Balasa <abalasa@nvidia.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com> Signed-off-by: yuzhongw <yuzhongw@nvidia.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by: ksivamani <ksivamani@nvidia.com> Signed-off-by: Xin Yao <xiny@nvidia.com> Signed-off-by: Pavel Gein <pavel.gein@gmail.com> Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com> Signed-off-by: jinliangl <jinliangl@nvidia.com> Signed-off-by: Skand Hurkat <shurkat@nvidia.com> Signed-off-by: Yan Xu <yxu1@nvidia.com> Signed-off-by: Siddhartha Raman Sundara Raman <270218152+sraman-rgb@users.noreply.github.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com> Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@users.noreply.github.com> Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> Signed-off-by: Pranav Prashant Thombre <pthombre@nvidia.com> Signed-off-by: janEbert <janpabloe@nvidia.com> Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com> Signed-off-by: Helen Ngo <helenn@nvidia.com> Signed-off-by: ykarnati <ykarnati@nvidia.com> Signed-off-by: Dennis Liu <denliu@denliu.nvidia.com> Signed-off-by: tongliu <tongliu@nvidia.com> Signed-off-by: Deyu Fu <deyuf@nvidia.com> Signed-off-by: Shijie Wang <jaywan@nvidia.com> Signed-off-by: guihong-nv <guihongl@nvidia.com> Signed-off-by: Guihong Li <guihongl@oci-hsg-cs-001-vscode-02.cm.cluster> Signed-off-by: Guihong Li <guihongl@nvidia.com> Signed-off-by: jinliangl <975761915@qq.com> Signed-off-by: yangfan.bai <yangfan.bai@shopee.com> Signed-off-by: hongbinl <hongbinl@nvidia.com> Signed-off-by: HaochenYuan <haocheny@nvidia.com> Signed-off-by: Antoni-Joan Solergibert <asolergibert@nvidia.com> Signed-off-by: ilml <tolong@nvidia.com> Signed-off-by: sraman <sraman@nvidia.com> Signed-off-by: Hongxiao Bai <hongxiaob@nvidia.com> Signed-off-by: Hollow Man <hollowman@opensuse.org> Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com> Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com> Signed-off-by: wdykas <wdykas@nvidia.com> Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Signed-off-by: Yan Bai <bayan@nvidia.com> Signed-off-by: svcnvidia-nemo-ci <svc-nvidia-nemo-ci@nvidia.com> Signed-off-by: kunlunl <kunlunl@nvidia.com> Signed-off-by: pingtianl <pingtianl@nvidia.com> Signed-off-by: Pingtian Li <pingtianl@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Ajay <abalasa@nvidia.com> Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Zijie Yan <zijiey@nvidia.com> Co-authored-by: Robin Zhang <robinz@nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> Co-authored-by: Shifang Xu <shifangx@nvidia.com> Co-authored-by: Cory Ye <44509866+cspades@users.noreply.github.com> Co-authored-by: Yan Xu <45385219+Connor-XY@users.noreply.github.com> Co-authored-by: Pingtian Li <158665726+Wohox@users.noreply.github.com> Co-authored-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: xuwchen <xuwenc@nvidia.com> Co-authored-by: Xin Yao <xiny@nvidia.com> Co-authored-by: hx <hongxiaob@nvidia.com> Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com> Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Li Ding <liding@nvidia.com> Co-authored-by: Fei Wu <33940270+YangFei1990@users.noreply.github.com> Co-authored-by: Li Jinliang <jinliangl@nvidia.com> Co-authored-by: BestJuly <19769279+BestJuly@users.noreply.github.com> Co-authored-by: Siddhartha Raman Sundara Raman <sraman@nvidia.com> Co-authored-by: sraman <sraman@users.noreply.github.com> Co-authored-by: Siddhartha Raman S <sraman@login-lyris01.lyris.clusters.nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: sraman-rgb <270218152+sraman-rgb@users.noreply.github.com> Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Gerald Shen <geshen@nvidia.com> Co-authored-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com> Co-authored-by: Jingyue Wu <wujingyue@gmail.com> Co-authored-by: Gao Deng <160076886+gdengk@users.noreply.github.com> Co-authored-by: Hongbin Liu <lhb8125@users.noreply.github.com> Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com> Co-authored-by: Yuzhong Wang <yuzhongw@computelab-frontend-3.nvidia.com> Co-authored-by: janEbert <janpabloe@nvidia.com> Co-authored-by: Simiao Zhang <56124251+conver334@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Shaurya Singh <sshaurya914@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Kevin <23258141+kaimo455@users.noreply.github.com> Co-authored-by: mokai <mokai@baidu.com> Co-authored-by: Moozy <108285604+Moozy23232@users.noreply.github.com> Co-authored-by: Aditya Singh <60082699+adityasingh2400@users.noreply.github.com> Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Siddhartha Raman S <sraman@login-lyris02.lyris.clusters.nvidia.com> Co-authored-by: gautham-kollu <gkollu@nvidia.com> Co-authored-by: Qiyu Wan <39144338+WanZzzzzz@users.noreply.github.com> Co-authored-by: Devil1716 <149754374+Devil1716@users.noreply.github.com> Co-authored-by: Mike Chrzanowski <mchrzanowski@nvidia.com> Co-authored-by: Mike Chrzanowski <mchrzanowski@gcp-nrt-cs-001-login-001.cm.cluster> Co-authored-by: Dingqing Yang <dingqingy@nvidia.com> Co-authored-by: liuzhenhai93 <liuzhenhai93@outlook.com> Co-authored-by: mathemakitten <helenn@nvidia.com> Co-authored-by: Jenny Chen <jennifchen@nvidia.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> Co-authored-by: kajalj22 <kajalj@nvidia.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: megnvidia <mmiranda@nvidia.com> Co-authored-by: Philip Petrakian <pgpetrak@gmail.com> Co-authored-by: Xuanteng Huang <44627253+xuantengh@users.noreply.github.com> Co-authored-by: Tailai Ma <58548582+xiaoyao0115@users.noreply.github.com> Co-authored-by: Santosh Bhavani <santosh.bhavani@live.com> Co-authored-by: Zhongbo Zhu <42691305+zhongbozhu@users.noreply.github.com> Co-authored-by: Nick Schank <nick@reflection.ai> Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com> Co-authored-by: Shivanjan Chakravorty <schakravorty846@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Tuomas Rintamaki <trintamaki@nvidia.com> Co-authored-by: Tyler Poon <tylerpoon@gmail.com> Co-authored-by: Collin McCarthy <cmccarthy@nvidia.com> Co-authored-by: Matthieu Le <matthieul@nvidia.com> Co-authored-by: Piotr Zelasko <pzelasko@nvidia.com> Co-authored-by: Ehsan Hosseini Asl <ehosseiniasl@nvidia.com> Co-authored-by: Siddharth Singh <sidsingh@nvidia.com> Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com> Co-authored-by: Xiaowei Ren <103958965+xrennvidia@users.noreply.github.com> Co-authored-by: wdykas <73254672+wdykas@users.noreply.github.com> Co-authored-by: William Dykas <wdykas@oci-hsg-cs-001-vscode-03.cm.cluster> Co-authored-by: kunlunl <kunlunl@nvidia.com> Co-authored-by: Xuesong Ye <xuesongyey@gmail.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: Daisy Gao <daisyg@nvidia.com> Co-authored-by: lichenlu <lichenlu_8618@163.com> Co-authored-by: peibli <lipeibao@126.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Co-authored-by: Gao Deng <gdeng@login-lyris02.lyris.clusters.nvidia.com> Co-authored-by: Pavel Gein <pavelgejn@yandex.ru> Co-authored-by: Gao Deng <gdeng@login-lyris01.lyris.clusters.nvidia.com> Co-authored-by: Eric Harper <eharper@nvidia.com> Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> Co-authored-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-01.cm.cluster> Co-authored-by: root <root@pool0-01849.cm.cluster> Co-authored-by: Kamran Jafari <kjafarisadeg@nvidia.com> Co-authored-by: Nan Zheng <80790206+nanz-nv@users.noreply.github.com> Co-authored-by: Li Tao <lit@nvidia.com> Co-authored-by: Zhongbo Zhu <zhongboz@nvidia.com> Co-authored-by: Robert Kirby <ArEsKay3@users.noreply.github.com> Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Co-authored-by: Cole Hawkins <chawkins@nvidia.com> Co-authored-by: Qi Zhang <qizhang@nvidia.com> Co-authored-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> Co-authored-by: tongliu <tongliu@nvidia.com> Co-authored-by: shurkat-nvidia <shurkat@nvidia.com> Co-authored-by: Rui WANG <51357798+rui23@users.noreply.github.com> Co-authored-by: rionawang <rionawang@tencent.com> Co-authored-by: Tom Long <tolong@nvidia.com> Co-authored-by: Lintch <44701395+returnL@users.noreply.github.com> Co-authored-by: Yan Bai <bayan@nvidia.com> Co-authored-by: Deyu Fu <deyuf@nvidia.com> Co-authored-by: Anish Mahishi <amahishi@nvidia.com> Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@users.noreply.github.com> Co-authored-by: jingqiny-99 <jingqiny@nvidia.com> Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com> Co-authored-by: Pranav Thombre <pthombre@nvidia.com> Co-authored-by: Dennis Liu <denliu@denliu.nvidia.com> Co-authored-by: Shijie <505749828@qq.com> Co-authored-by: Guihong Li <guihongl@nvidia.com> Co-authored-by: Guihong Li <guihongl@oci-hsg-cs-001-vscode-02.cm.cluster> Co-authored-by: Yan Xu <yxu1@nvidia.com> Co-authored-by: Li Jinliang <975761915@qq.com> Co-authored-by: Baibaifan <39549453+Baibaifan@users.noreply.github.com> Co-authored-by: yangfan.bai <yangfan.bai@shopee.com> Co-authored-by: HaochenYuan <106647990+HaochenYuan@users.noreply.github.com> Co-authored-by: Haochen Yuan <haocheny@login-eos01.eos.clusters.nvidia.com> Co-authored-by: ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟 <hollowman@opensuse.org> Co-authored-by: Lawrence McAfee <85179052+lmcafee-nvidia@users.noreply.github.com> Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com> Co-authored-by: Kaixiang Lei <5780122+shyoshyo@users.noreply.github.com>

* test: re-enable test_pp2_create_cudagraphs_first_stage on TE 2.15+ (NVIDIA#4985) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tests): initialize num_microbatches calculator in vision cudagraph tests (NVIDIA#4986) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Add allow_failure flag to gpt and moe recipes that are failing in nightlies (NVIDIA#4905) * Drain predecessor reduce-scatter at dispatch time (NVIDIA#4940) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * nightly(ci): Update golden values for functional t5 tests (NVIDIA#4995) * chore: rotate oncall schedule * [main] Refactor and Improve MoE Logginginit commit (NVIDIA#3431) Co-authored-by: Robin Zhang <robinz@nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> Co-authored-by: Shifang Xu <shifangx@nvidia.com> * ci: validate release branch-rules (NVIDIA#4929) Signed-off-by: oliver könig <okoenig@nvidia.com> * [Megatron-FSDP] Add conditional param.grad dereferencing logic to support full-iteration (FWD-BWD) CUDA graphability. (NVIDIA#4663) Signed-off-by: Cory Ye <cye@nvidia.com> * test: restrict iter-time comparison to steady-state window (NVIDIA#5010) Signed-off-by: oliver könig <okoenig@nvidia.com> * Add mHC support for HybridModel on dev (NVIDIA#4949) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> * fix(test): pin eval-global-batch-size on 15b gb200 release configs (NVIDIA#5022) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [fix] Release MTP assertion when EP overlap with PP=1 (NVIDIA#4796) * fix(test): widen iter-time steady-state window for short tests (NVIDIA#5023) Signed-off-by: oliver könig <okoenig@nvidia.com> * Perf fix (NVIDIA#4996) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add dev-feature preservation gate and change schedule (NVIDIA#4773) * chore(test): remove orphan nemotron3_super_release_g200 dir (NVIDIA#5024) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ignore Claude worktree directory (NVIDIA#5020) * Update copy-pr-bot.yaml [skip ci] * ci: update CI workflow conditions for integration tests (NVIDIA#4658) * Add NVSkills CI request workflow (NVIDIA#5033) * DDP wrap pg size fixes (NVIDIA#5006) Signed-off-by: Maanu Grover <maanug@nvidia.com> * fix(layer_wise): tag MTP-stage word_embeddings as is_embedding_or_output_parameter (NVIDIA#5034) * [Dev] Add Qwen3 30B MoE recipes (NVIDIA#5012) * [DEV] fix(megatron-fsdp): reduce padding for grouped expert weights (NVIDIA#5013) Co-authored-by: Xin Yao <xiny@nvidia.com> * [Dev] fix(combined-1f1b): release loss-node input storage after combined backward (NVIDIA#4908) Co-authored-by: Xin Yao <xiny@nvidia.com> * [dev] [fix] [DeepSeek-v4] fix dense loss and rope type in DSv4 Hybrid Attention (NVIDIA#5018) Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com> * Move LTS dependencies from pyproject.toml to Dockerfile.ci.lts (NVIDIA#4877) * Use shared ModelOpt calibration loop on 0.45+ with 0.44 fallback fix (NVIDIA#4881) Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * test(release): skip golden comparison on intermediate resume windows (NVIDIA#5040) Signed-off-by: oliver könig <okoenig@nvidia.com> * [mimo] Thread position_ids through MimoModel for multimodal RoPE (NVIDIA#4938) Signed-off-by: Li Ding <liding@nvidia.com> * build: Switch DSv3 on H100 to HybridEP (NVIDIA#5039) Signed-off-by: oliver könig <okoenig@nvidia.com> * Fix: Import unwrap_model from megatron.core.utils in modelopt examples (NVIDIA#5045) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Simple and stable Inference APIs (NVIDIA#4697) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci: Add notification step for MBridge downstream test results (NVIDIA#5028) * [dev] [5/5] Qwen3.5 support: Qwen3.5-VL training example (NVIDIA#4751) Co-authored-by: BestJuly <19769279+BestJuly@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: Update Docker image version to 26.04-py3 on dev (NVIDIA#5051) Signed-off-by: oliver könig <okoenig@nvidia.com> * Delete output tensor early (NVIDIA#4742) * Support ScaledSReLU in TE grouped MLP fuser (NVIDIA#4859) Co-authored-by: sraman <sraman@users.noreply.github.com> Co-authored-by: Siddhartha Raman S <sraman@login-lyris01.lyris.clusters.nvidia.com> Co-authored-by: a <a> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: sraman-rgb <270218152+sraman-rgb@users.noreply.github.com> * Skip gradient updates when grad norm exceeds threshold (NVIDIA#3460) Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Gerald Shen <geshen@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add 9 user skills (NVIDIA#5066) Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com> Co-authored-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com> * test(nemotron): align nemotron3 super GB200 goldens with exit-interval 4768 (NVIDIA#5069) Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Update transformer-engine dependency to version 2.16.0 (NVIDIA#4992) * Update energon version requirement (NVIDIA#4572) Signed-off-by: Maanu Grover <maanug@nvidia.com> * Fix test failures for new inference APIs (NVIDIA#5068) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): set PYTHONUNBUFFERED=1 in JET workload env (NVIDIA#5072) Signed-off-by: oliver könig <okoenig@nvidia.com> * Preserve non-FSDP-unit buckets across AllGatherPipeline reset (NVIDIA#4717) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add opt-in MXFP8 LM-head output projection (NVIDIA#4825) Co-authored-by: a <a> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> * chore(beep boop 🤖): Bump (main) (2026-06-01) * fix(ci): bound JET pipeline polling with a watchdog to prevent indefinite hangs (NVIDIA#5076) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: prune old artifacts on cluster lustre during weekly/release runs (NVIDIA#5084) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(test): isolate ckpt-resume tensorboard per phase (NVIDIA#5074) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: unmark EP A2A activation offload test flaky (NVIDIA#5009) * Change ownership groups (NVIDIA#5021) * test: skip mfsdp_fully_shard cases when world_size < mesh size (NVIDIA#4487) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix mimo optimizer checkpoint metadata restore (NVIDIA#4791) Signed-off-by: Li Ding <liding@nvidia.com> * [mimo] Support bridge fan-out for variable modality tokens (NVIDIA#5062) Signed-off-by: Li Ding <liding@nvidia.com> * Add separate mtp_grad_scale_func for MTP loss scaling (NVIDIA#3459) Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [training migration] Migrate GPT builder (NVIDIA#4741) Signed-off-by: Maanu Grover <maanug@nvidia.com> * Make Mamba conv params direct mixer params (NVIDIA#4899) Signed-off-by: Jingyue Wu <wujingyue@gmail.com> * Update copy-pr-bot.yaml [skip ci] * Update oncall reviewer assignment (NVIDIA#5093) * Pass explicit process groups to hybrid logging (NVIDIA#4781) * Clean up top-level repository files (NVIDIA#5097) * [main] fix(moe): Fix several bugs for DSA rope and spec. (NVIDIA#3026) Co-authored-by: Yuzhong Wang <yuzhongw@computelab-frontend-3.nvidia.com> * [Dev] Skip identity alltoall chunk sort (NVIDIA#5102) * Move MIMO unit tests into models/mimo (NVIDIA#5063) * test: update DeepSeek FSDP2 GB200 memory golden (NVIDIA#5094) * Remove DeepEP hardware limit check (NVIDIA#4846) * Update transformer-engine dependency to revision 4220403 (NVIDIA#5112) * ci: make CI resilient to pip/uv network timeouts (NVIDIA#5118) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: treat docker container-removal conflict as flaky (NVIDIA#5120) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix GDN DTensor splitting for FSDP checkpointing (NVIDIA#4843) Signed-off-by: conver334 <conver334@gmail.com> * Fix MoE aux_loss / z_loss gradient scaling with TP > 1 (NVIDIA#5047) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update copy-pr-bot.yaml [skip ci] * Update Claude copy workflow to enforce user restrictions and improve error messages (NVIDIA#5117) * Add advisory process group guidance to Claude reviews (NVIDIA#5111) * build: cap pydantic<2.14 in transformer-engine dependency metadata (NVIDIA#5125) Signed-off-by: Chen Cui <chcui@nvidia.com> * fix(test): skip scalar-less tensorboard event files in resume checks (NVIDIA#5121) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: rotate oncall schedule * docs: fix contributor guide typo (NVIDIA#4858) Signed-off-by: LeSingh1 <sshaurya914@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> * ci(unit-tests): split slow unit-test buckets over 15min SLA (NVIDIA#5133) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix DSA indexer loss not averaged across micro-batches (NVIDIA#4070) Co-authored-by: mokai <mokai@baidu.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> * Update MINOR version to 19 (NVIDIA#5096) * Fix Muon QKV split for gated attention (NVIDIA#4728) Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> * Roll input IDs for MTP labels (NVIDIA#3457) Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Refactor: Move paged stashing Triton kernels (NVIDIA#5003) * Adding blackwell tests (NVIDIA#5113) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Relax atol for test_router_gating_linear router_dtype=torch.float32 (NVIDIA#4915) Signed-off-by: Aditya Singh <adisin650@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> * Fix incorrect inference metadata tensor dtypes (NVIDIA#4855) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Signed-off-by: qiyuw <qiyuw@nvidia.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com> Co-authored-by: Ajay <abalasa@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> Co-authored-by: sraman-rgb <sraman@nvidia.com> Co-authored-by: Siddhartha Raman S <sraman@login-lyris02.lyris.clusters.nvidia.com> Co-authored-by: Xin Yao <xiny@nvidia.com> Co-authored-by: gautham-kollu <gkollu@nvidia.com> Co-authored-by: Siddhartha Raman S <sraman@login-lyris01.lyris.clusters.nvidia.com> Co-authored-by: Qiyu Wan <39144338+WanZzzzzz@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Devil1716 <149754374+Devil1716@users.noreply.github.com> * Disable TE cross entropy loss fusion (NVIDIA#5115) Co-authored-by: Mike Chrzanowski <mchrzanowski@gcp-nrt-cs-001-login-001.cm.cluster> * fix(optimizer): gate ChainedOptimizer MXFP8 defer-sync on DDP-level overlap_param_gather (NVIDIA#4982) Signed-off-by: oliver könig <okoenig@nvidia.com> * Update copy-pr-bot.yaml [skip ci] * Pass TP group to unfused cross entropy (NVIDIA#5128) * fix: correct dsv4_hybrid Q-up FLOPs by using args.v_head_dim (NVIDIA#5142) * [dev] [follow-up] Qwen3.5 support: MoE aux loss padding_mask (NVIDIA#4776) Co-authored-by: BestJuly <19769279+BestJuly@users.noreply.github.com> * [Dev] Support isolated MTP loss (NVIDIA#5080) Co-authored-by: liuzhenhai93 <liuzhenhai93@outlook.com> * test(elastification): quarantine flaky test_gumbel_determinism as flaky_in_dev (NVIDIA#5156) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(notify): mention mcore-oncall and philipp on critical CI events (NVIDIA#5152) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Change the cudagraph distribution from linearly to exponentially-decreasing + grid for mixed prefill (NVIDIA#3509) * ci: Disable a few gb200 test cases to support 2 branches. (NVIDIA#5151) * build: Switch DSv3 on H100 to HybridEP (NVIDIA#5164) * Add MTP acceptance rate metrics (NVIDIA#3458) Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Nemotron Ultra config for ModelOpt examples (NVIDIA#5159) Signed-off-by: Jennifer Chen <jennifchen@nvidia.com> * Make MTP / prefix cache stats persist for engine lifetime (NVIDIA#4101) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Restore Greptile configuration (NVIDIA#5166) * chore: bump `_code_freeze` workflow to `v1.4.2` (NVIDIA#5132) Signed-off-by: oliver könig <okoenig@nvidia.com> * [dev] moe(fix): Avoid TE cuda graph dummy attention masks (NVIDIA#5131) Co-authored-by: Yuzhong Wang <yuzhongw@computelab-frontend-3.nvidia.com> * ci: Remove docs build test in favor of release test (NVIDIA#5182) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Move TE cross entropy guard to training args (NVIDIA#5162) Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix error in deepseek parser (NVIDIA#5136) * Fix logprob slicing for 0 generated token case (NVIDIA#5167) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> * [Perf] Fold frozen linear dgrad matmul (NVIDIA#5092) Signed-off-by: Chen Cui <chcui@nvidia.com> * Clamp `max_new_tokens` in MInf to mirror vllm (NVIDIA#5181) * build: add managed = true to [tool.uv] (NVIDIA#5190) Signed-off-by: Kajal Jain <kajalj@nvidia.com> * Stabilize GB200 inference perf tests against cold-start noise (NVIDIA#5171) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: post-CI corrections (docs dup fields, nvrx version gate, utils re-export) - Remove merge-introduced duplicate definitions of use_transformer_engine_op_fuser and moe_expert_rank_capacity_factor (transformer_config.py) and the duplicate _post_param_sync method (param_and_grad_buffer.py) — tripped the docs build. - Relax is_nvrx_min_version() to compare release segments so dev's pinned nvidia-resiliency-ext pre-release (0.6.0.dev33) satisfies main's '>= 0.6.0' assertion (the required-symbol check remains the authoritative guard). - Re-export unwrap_model from megatron/training/utils/__init__.py. main refactored the monolithic training/utils.py into a package whose __init__ dropped this re-export; dev's training.py (and others) import it via 'from .utils import', which broke tests/unit_tests/conftest.py's import chain (cascaded to all suites). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [dev] [DeepSeek-v4] Add ClampedSwiGLU to MoE mlp_op_fuser and add force balance to hash routing (NVIDIA#5130) * nvidia style guide audit for getting started folder (NVIDIA#5168) Signed-off-by: meg miranda <mmiranda@nvidia.com> * AI aided audit for Nvidia Style guidance (NVIDIA#5141) Signed-off-by: meg miranda <mmiranda@nvidia.com> Co-authored-by: Philip Petrakian <pgpetrak@gmail.com> * Enable selective recompute for `norm_out` in GDN layers (NVIDIA#4715) * fix(elastification): align with get_batch + utils refactors (NVIDIA#5194) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(combined-1f1b): release loss-node input storage after combined backward (NVIDIA#4909) * Minor improvements for Dynamic-cp (NVIDIA#4226) Signed-off-by: xiaoyao0115 <1804647152@qq.com> Signed-off-by: tailaim <tailaim@nvidia.com> * fix: post-CI corrections * chore(beep boop 🤖): Bump (main) (2026-06-08) * Avoid stat syscall in rerun result validation (NVIDIA#5107) Signed-off-by: dimapihtar <dpykhtar@nvidia.com> * docs: Update Latest News in README.md (NVIDIA#3790) * Fix bug with Megatron-FSDP zero counter not working with decoupled gradients. (NVIDIA#4802) Signed-off-by: Cory Ye <cye@nvidia.com> * varlendataset for thd e2e and benchmark (NVIDIA#4832) Signed-off-by: tailaim <tailaim@nvidia.com> Signed-off-by: xiaoyao0115 <1804647152@qq.com> * ci: add smoke tests (NVIDIA#5143) * Add mtp_detach_heads config to detach MTP head inputs (NVIDIA#3456) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: wrap mtp logging comment * docs: fix install guide NGC container anchor (NVIDIA#5224) * [Dev] Add separate toggle for varlen input padding for HybridEP in THD training (NVIDIA#5048) Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> Co-authored-by: Xin Yao <xiny@nvidia.com> * Fuse per-sequence AlltoAll into a unified one in GDN forward (NVIDIA#4913) * Apply MIMO SP/CP sharding with explicit groups and enable THD in non-colocated path (NVIDIA#5150) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix CUDA IMA in fsdp_double_buffer when an FSDP unit's bucket doesn't fit the pool (NVIDIA#4810) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add named layouts to HyperCommGrid for heterogeneous parallelism (NVIDIA#5148) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix wgrad race condition when using double buffers. (NVIDIA#5222) Signed-off-by: Cory Ye <cye@nvidia.com> * Move uneven DTensor distributed fixture to conftest (NVIDIA#5237) * Route bridge communicator cross-grid P2P through a dedicated process group (NVIDIA#5234) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix test_split_tensor_along_last_dim to actually assert correctness (NVIDIA#4710) Co-authored-by: peibli <lipeibao@126.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> * chore: rotate oncall schedule * Add optional group= to common_utils model/data-parallel reduction helpers (NVIDIA#5251) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: Allow DCO check in merge queue and add DCO requirement to Contribution guide (NVIDIA#5278) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Update copy-pr-bot.yaml [skip ci] * Add MIMO hetero topology + distributed bootstrap (examples/mimo training-loop folder) (NVIDIA#5260) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Remove checkpoint-time GPU cache reclaim workaround (NVIDIA#5170) Signed-off-by: Skand Hurkat <shurkat@nvidia.com> Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * [Dev] DeepSeek-V4-Flash recipe 20260610 (NVIDIA#5266) * [Dev] Generalized fix for mxfp8 param gather (NVIDIA#4994) Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> Co-authored-by: Xin Yao <xiny@nvidia.com> * [Dev] Cherry-pick MTP detach heads (NVIDIA#5223) * [TE] Restore original CP group after dynamic CP forward in TEDotProductAttention (NVIDIA#5215) Co-authored-by: rionawang <rionawang@tencent.com> * [examples] Add dynamic context parallel benchmark example (NVIDIA#5123) * Remove duplicate nccl_allocator import (NVIDIA#5057) * [dev] Add experimental Megatron Lite as agentic exploration (NVIDIA#4885) Co-authored-by: Deyu Fu <deyuf@nvidia.com> * fix(ci): resolve t5 dataloader stall + GRPO cudagraph-memory regression (CI-validated) (NVIDIA#5280) Signed-off-by: Yan Xu <yxu1@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix Dockerfile warnings (NVIDIA#4856) * Fix fused MLA delayed weight grad hooks (NVIDIA#5273) Signed-off-by: Siddhartha Raman Sundara Raman <270218152+sraman-rgb@users.noreply.github.com> Co-authored-by: Siddhartha Raman Sundara Raman <270218152+sraman-rgb@users.noreply.github.com> * ci: limit retries on unsuccessful test launches (NVIDIA#5275) Signed-off-by: Ajay Balasa <abalasa@nvidia.com> * Thread pg_collection into get_model DDP bucket sizing (NVIDIA#5250) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Enable non-deterministic results in model configuration for nemotron tests (NVIDIA#5239) * Stabilize hybrid nanov3 gb200 perf (NVIDIA#5295) * Clip mtp grads separately when mtp_detach_heads=True (NVIDIA#4116) Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com> Co-authored-by: Anish Mahishi <amahishi@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Thread pg_collection into train_step reductions (NVIDIA#5259) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: Allow DCO check in merge queue and add DCO requirement (NVIDIA#5305) Signed-off-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> * fix: restore dev-only training.py features dropped by the main override The merge takes main's training.py wholesale (per the sync skill's override list), which silently reverted four dev-only features whose supporting core code the merge keeps at dev's version. Each surfaced as a CI failure: 1. Dynamic context-parallel API rename. Dev renamed get_hybrid_data_context_parallel_groups -> get_dynamic_data_context_parallel_groups (identical signature) and args.hybrid_context_parallel -> dynamic_context_parallel. Point training.py at dev's names. Fixes the conftest.py ImportError that cascaded to every unit-test bucket. 2. Dynamic-CP / sequence-packing data loading. Replace main's setup-time HybridCPDataLoaderWrapper wrap with dev's per-step wrap_data_iterator (gated on config.sequence_packing_scheduler), which returns a RerunDataIterator-compatible iterator. Fixes the *_cp4_dcp RerunDataIterator assertion. 3. MTP loss logging scale. Dev's MTPLossLoggingHelper stores raw loss sums and token counts and computes the per-token loss after reduction, so the log scale must be 1.0; main's 1/get_num_microbatches() divided the reported mtp_N loss by num_microbatches. Fixes moe gpt3_..._scoped_cudagraph mtp_1 loss (was 16x too small: 0.682 vs golden 10.915). 4. DSA indexer loss cross-PP reduction. dsa.py requires num_layers (and csa_compress_ratios) so first-pipeline-stage ranks lazily initialize the tracker and join the cross-PP all_reduce in reduce_loss_in_tracker; main's call omitted them, so stage 0 returned early and the last stage hung on an unmatched all_reduce. Fixes the gpt3_..._dsv4_hybrid_mhc_mtp NCCL timeout. Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@users.noreply.github.com> * [dev]: faster implementation of mHC fused kernels (NVIDIA#4624) Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> * Allow for pre-bound socket to be passed in server (NVIDIA#5301) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> * Offline Logits-Based Knowledge Distillation (NVIDIA#5019) Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> * Handle None values in sampling parameters (NVIDIA#5300) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com> * Add moe loss normalization for RL SFT (NVIDIA#3956) Signed-off-by: Pranav Prashant Thombre <pthombre@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Add code owners for optimizer-related files (NVIDIA#5297) Signed-off-by: janEbert <janpabloe@nvidia.com> Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> * Fix EP=1 inference by allocating buffers anyway (NVIDIA#5233) Signed-off-by: Helen Ngo <helenn@nvidia.com> * Fix crash due to tool call at sequence length (NVIDIA#5302) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com> * Inference: Cudagraph-aware admission gating in prefill scheduler (NVIDIA#4870) Signed-off-by: Helen Ngo <helenn@nvidia.com> * Account for reasoning token stripping (NVIDIA#5313) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> * Thread pg_collection through wrap_model_chunks_with_ddp (NVIDIA#5328) Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Dev] Add MoE recipe performance summary (NVIDIA#5289) Signed-off-by: Dennis Liu <denliu@denliu.nvidia.com> Co-authored-by: Dennis Liu <denliu@denliu.nvidia.com> * [Dev] Add DeepEP v2 flex dispatcher backend (NVIDIA#4793) Signed-off-by: tongliu <tongliu@nvidia.com> Co-authored-by: Dennis(Zhenhuan) Liu <denliu@nvidia.com> * fix tflops calculation when sequence_packing_scheduler is not none (NVIDIA#5342) Signed-off-by: xiaoyao0115 <1804647152@qq.com> * chore(beep boop 🤖): Bump (main) (2026-06-15) * [dev] bump emerging optimizers to v0.3.0 (NVIDIA#5320) Signed-off-by: Deyu Fu <deyuf@nvidia.com> * Fix LatentMoE theoretical memory estimate (NVIDIA#5145) Signed-off-by: Shijie Wang <jaywan@nvidia.com> * Add zstandard package to Docker LTS requirements. Fix nightly failures (NVIDIA#5347) Signed-off-by: Ajay Balasa <abalasa@nvidia.com> * fix: preserve seqlen stats in train_step Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com> * Thread MIMO support through the stock training loop (schedule + optimizer) (NVIDIA#5333) Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Enable Deepseek-v4 hybrid_model in dev branch Part (1/N) (NVIDIA#5042) Signed-off-by: guihong-nv <guihongl@nvidia.com> Signed-off-by: Guihong Li <guihongl@oci-hsg-cs-001-vscode-02.cm.cluster> Signed-off-by: Yan Xu <yxu1@nvidia.com> Signed-off-by: Guihong Li <guihongl@nvidia.com> Co-authored-by: Guihong Li <guihongl@oci-hsg-cs-001-vscode-02.cm.cluster> Co-authored-by: Yan Xu <yxu1@nvidia.com> Co-authored-by: hx <hongxiaob@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ssm): whole-module 'gdn' selective recompute for GatedDeltaNet (NVIDIA#5296) Signed-off-by: jinliangl <975761915@qq.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Xin Yao <xiny@nvidia.com> * [Fix] Fix optimizer parameter override bugs. (NVIDIA#5213) Signed-off-by: yangfan.bai <yangfan.bai@shopee.com> Co-authored-by: yangfan.bai <yangfan.bai@shopee.com> Co-authored-by: Xin Yao <xiny@nvidia.com> * [Dev] Add Megatron-FSDP weight prefetch for full recompute (NVIDIA#5175) Signed-off-by: hongbinl <hongbinl@nvidia.com> * ci: default functional test time limit to 4h for release/weekly scopes (NVIDIA#5360) Signed-off-by: oliver könig <okoenig@nvidia.com> * [Dev] add cuda graph support for thd format training. (NVIDIA#4359) Signed-off-by: HaochenYuan <haocheny@nvidia.com> Co-authored-by: Haochen Yuan <haocheny@login-eos01.eos.clusters.nvidia.com> * Fix memory leak with log_max_attention_logit (NVIDIA#4699) (NVIDIA#5067) Signed-off-by: Antoni-Joan Solergibert <asolergibert@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com> * Clean up pretrain_gpt.py and pretrain_hybrid.py formatting and remove module globals (NVIDIA#5351) Signed-off-by: ilml <tolong@nvidia.com> * Add full model cuda graph support for MTP inference (NVIDIA#4950) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> * Expand the Mamba prefix caching memory safety check to include scratch space buffers (NVIDIA#5348) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> * Make Megatron RL only materialize last token logit (NVIDIA#4551) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> * Profiling (NVIDIA#3110) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> * Support fused MLA QKV checkpoint reload (NVIDIA#5310) Signed-off-by: sraman <sraman@nvidia.com> * Add minimal DBuffer implementation (NVIDIA#4835) Signed-off-by: Jingyue Wu <wujingyue@gmail.com> * [Dev] restore DSv4 tflops calc in training and fix the packed seq case (NVIDIA#5358) Signed-off-by: Hongxiao Bai <hongxiaob@nvidia.com> * [split 1/5] Fix packed THD RoPE under CP (NVIDIA#5243) Signed-off-by: Hollow Man <hollowman@opensuse.org> * Update copy-pr-bot.yaml [skip ci] * Document agent PR commit sign-off and signing (NVIDIA#5381) Signed-off-by: Jingyue Wu <wujingyue@gmail.com> * Remove unused distributed pytest markers (NVIDIA#5380) Signed-off-by: Jingyue Wu <wujingyue@gmail.com> * [feat] Support fine-grained activation offloading in fused group mlp (NVIDIA#5082) Signed-off-by: hongbinl <hongbinl@nvidia.com> * Thread tensor-parallel group into the RADIO patch embedder (NVIDIA#5371) Signed-off-by: ykarnati <ykarnati@nvidia.com> * Add MimoModel.zero_grad_buffer delegating to active DDP submodules (NVIDIA#5372) Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * [split 3/5] Refactor absorbed MLA projection handling (NVIDIA#5245) Signed-off-by: Hollow Man <hollowman@opensuse.org> * [dev] moe(perf): Restore fused GDN THD all-to-all on dev (NVIDIA#5389) Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com> * chore: rotate oncall schedule * ci: Remove sync skills workflow (NVIDIA#5091) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Add flaky marker to fine-grained activation offloading test (NVIDIA#5350) (NVIDIA#5368) Signed-off-by: Ajay Balasa <abalasa@nvidia.com> * Revert "Remove checkpoint-time GPU cache reclaim workaround (NVIDIA#5170)" (NVIDIA#5366) Signed-off-by: Ajay Balasa <abalasa@nvidia.com> * Update goldens for weekly tests after pytorch and TE bumps. (NVIDIA#5399) Signed-off-by: Ajay Balasa <abalasa@nvidia.com> * Add MIMO runtime setup: per-role RNG seeding and DDP wrapping (NVIDIA#5285) Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Add --mamba-training-ssm-states-dtype argument (NVIDIA#5309) Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com> * chore(beep boop 🤖): Bump (main) (2026-06-22) * Fix Mamba prefix match for chunked prefill (NVIDIA#4758) Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com> * Disag MR2: Refit into multiple destination pools and tied-embedding + UVM fixes (NVIDIA#5187) Signed-off-by: wdykas <wdykas@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Disag MR1: Add inference shard specs and pg-collection building (NVIDIA#5186) Signed-off-by: wdykas <wdykas@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * [Fix] Fix MoE router z-loss compatibility with TE CUDA Graph capture. (NVIDIA#5401) Signed-off-by: yangfan.bai <yangfan.bai@shopee.com> Co-authored-by: yangfan.bai <yangfan.bai@shopee.com> * test: mark ep_a2a_overlap activation-offloading test flaky_in_dev (NVIDIA#5450) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: mark TestParallelTransformerBlockCudagraphs::test_gpu_cudagraph flaky_in_dev (NVIDIA#5475) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: mark gated_delta_net selective-recompute test flaky_in_dev (NVIDIA#5476) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: post-CI corrections for sync test/impl reconciliation 1) test_optimizer.py: reverted to dev. The sync kept dev's multi_latent_attention.py (split q/kv down-proj, no _synthesize_fused_qkv_down_weight), but auto-merged main's test asserting the fused linear_qkv_down_proj.weight key. 2) training.py: guard the dev-only sequence_packing_scheduler config access with getattr (lines in train_step and train()). main's new MIMO schedule-plumbing test (NVIDIA#5333) passes an empty SimpleNamespace config; the reconciled training.py keeps dev's packing path, so the access must tolerate a config lacking the attribute. Real configs are unaffected (getattr returns the same value). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> * fix sequence packing wrapper for eval (NVIDIA#5483) Signed-off-by: xiaoyao0115 <1804647152@qq.com> * [dev] Megatron Lite (4/4) shared attention (NVIDIA#5427) Signed-off-by: Yan Bai <bayan@nvidia.com> * fix: restore fused group MLP offload in main2dev sync (NVIDIA#5493) Signed-off-by: hongbinl <hongbinl@nvidia.com> Signed-off-by: svcnvidia-nemo-ci <svc-nvidia-nemo-ci@nvidia.com> * [dev] [DeepSeek-v4] Packed Sequence (THD) support for DSv4 Hybrid Attention (NVIDIA#5011) Signed-off-by: Hongxiao Bai <hongxiaob@nvidia.com> * Improve default dynamic CP packing scheduler (NVIDIA#5154) Signed-off-by: tailaim <tailaim@nvidia.com> * [dev] moe(perf): Pre-GDR kernel fusion (NVIDIA#5361) Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com> * Preserve DSA output across fused inverse RoPE (NVIDIA#5526) Signed-off-by: kunlunl <kunlunl@nvidia.com> Co-authored-by: Kaixiang Lei <5780122+shyoshyo@users.noreply.github.com> * Enable Deepseek-v4 hybrid_model in dev branch Part (2/N) (NVIDIA#5485) Signed-off-by: guihong-nv <guihongl@nvidia.com> * [dev] Add experimental decoupled compact LayerWise DDP layout for Muon (NVIDIA#5388) Signed-off-by: pingtianl <pingtianl@nvidia.com> Signed-off-by: Pingtian Li <pingtianl@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [dev] Sync Megatron Lite with the latest implementation (NVIDIA#5577) Signed-off-by: Yan Bai <bayan@nvidia.com> * [Dev] fix padding mask docstring (NVIDIA#5598) Signed-off-by: HaochenYuan <haocheny@nvidia.com> * handle split_dtensor --------- Signed-off-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Cory Ye <cye@nvidia.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Signed-off-by: Li Ding <liding@nvidia.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com> Signed-off-by: Jingyue Wu <wujingyue@gmail.com> Signed-off-by: conver334 <conver334@gmail.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: LeSingh1 <sshaurya914@gmail.com> Signed-off-by: Aditya Singh <adisin650@gmail.com> Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Signed-off-by: qiyuw <qiyuw@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Jennifer Chen <jennifchen@nvidia.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com> Signed-off-by: meg miranda <mmiranda@nvidia.com> Signed-off-by: xiaoyao0115 <1804647152@qq.com> Signed-off-by: tailaim <tailaim@nvidia.com> Signed-off-by: dimapihtar <dpykhtar@nvidia.com> Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> Signed-off-by: Shivanjan Chakravorty <shivanjanc@nvidia.com> Signed-off-by: Ajay Balasa <abalasa@nvidia.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com> Signed-off-by: yuzhongw <yuzhongw@nvidia.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by: ksivamani <ksivamani@nvidia.com> Signed-off-by: Xin Yao <xiny@nvidia.com> Signed-off-by: Pavel Gein <pavel.gein@gmail.com> Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com> Signed-off-by: jinliangl <jinliangl@nvidia.com> Signed-off-by: Skand Hurkat <shurkat@nvidia.com> Signed-off-by: Yan Xu <yxu1@nvidia.com> Signed-off-by: Siddhartha Raman Sundara Raman <270218152+sraman-rgb@users.noreply.github.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com> Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@users.noreply.github.com> Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> Signed-off-by: Pranav Prashant Thombre <pthombre@nvidia.com> Signed-off-by: janEbert <janpabloe@nvidia.com> Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com> Signed-off-by: Helen Ngo <helenn@nvidia.com> Signed-off-by: ykarnati <ykarnati@nvidia.com> Signed-off-by: Dennis Liu <denliu@denliu.nvidia.com> Signed-off-by: tongliu <tongliu@nvidia.com> Signed-off-by: Deyu Fu <deyuf@nvidia.com> Signed-off-by: Shijie Wang <jaywan@nvidia.com> Signed-off-by: guihong-nv <guihongl@nvidia.com> Signed-off-by: Guihong Li <guihongl@oci-hsg-cs-001-vscode-02.cm.cluster> Signed-off-by: Guihong Li <guihongl@nvidia.com> Signed-off-by: jinliangl <975761915@qq.com> Signed-off-by: yangfan.bai <yangfan.bai@shopee.com> Signed-off-by: hongbinl <hongbinl@nvidia.com> Signed-off-by: HaochenYuan <haocheny@nvidia.com> Signed-off-by: Antoni-Joan Solergibert <asolergibert@nvidia.com> Signed-off-by: ilml <tolong@nvidia.com> Signed-off-by: sraman <sraman@nvidia.com> Signed-off-by: Hongxiao Bai <hongxiaob@nvidia.com> Signed-off-by: Hollow Man <hollowman@opensuse.org> Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com> Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com> Signed-off-by: wdykas <wdykas@nvidia.com> Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Signed-off-by: Yan Bai <bayan@nvidia.com> Signed-off-by: svcnvidia-nemo-ci <svc-nvidia-nemo-ci@nvidia.com> Signed-off-by: kunlunl <kunlunl@nvidia.com> Signed-off-by: pingtianl <pingtianl@nvidia.com> Signed-off-by: Pingtian Li <pingtianl@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Ajay <abalasa@nvidia.com> Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Zijie Yan <zijiey@nvidia.com> Co-authored-by: Robin Zhang <robinz@nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com> Co-authored-by: Shifang Xu <shifangx@nvidia.com> Co-authored-by: Cory Ye <44509866+cspades@users.noreply.github.com> Co-authored-by: Yan Xu <45385219+Connor-XY@users.noreply.github.com> Co-authored-by: Pingtian Li <158665726+Wohox@users.noreply.github.com> Co-authored-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: xuwchen <xuwenc@nvidia.com> Co-authored-by: Xin Yao <xiny@nvidia.com> Co-authored-by: hx <hongxiaob@nvidia.com> Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com> Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Li Ding <liding@nvidia.com> Co-authored-by: Fei Wu <33940270+YangFei1990@users.noreply.github.com> Co-authored-by: Li Jinliang <jinliangl@nvidia.com> Co-authored-by: BestJuly <19769279+BestJuly@users.noreply.github.com> Co-authored-by: Siddhartha Raman Sundara Raman <sraman@nvidia.com> Co-authored-by: sraman <sraman@users.noreply.github.com> Co-authored-by: Siddhartha Raman S <sraman@login-lyris01.lyris.clusters.nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: sraman-rgb <270218152+sraman-rgb@users.noreply.github.com> Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Gerald Shen <geshen@nvidia.com> Co-authored-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com> Co-authored-by: Jingyue Wu <wujingyue@gmail.com> Co-authored-by: Gao Deng <160076886+gdengk@users.noreply.github.com> Co-authored-by: Hongbin Liu <lhb8125@users.noreply.github.com> Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com> Co-authored-by: Yuzhong Wang <yuzhongw@computelab-frontend-3.nvidia.com> Co-authored-by: janEbert <janpabloe@nvidia.com> Co-authored-by: Simiao Zhang <56124251+conver334@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Shaurya Singh <sshaurya914@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Kevin <23258141+kaimo455@users.noreply.github.com> Co-authored-by: mokai <mokai@baidu.com> Co-authored-by: Moozy <108285604+Moozy23232@users.noreply.github.com> Co-authored-by: Aditya Singh <60082699+adityasingh2400@users.noreply.github.com> Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Siddhartha Raman S <sraman@login-lyris02.lyris.clusters.nvidia.com> Co-authored-by: gautham-kollu <gkollu@nvidia.com> Co-authored-by: Qiyu Wan <39144338+WanZzzzzz@users.noreply.github.com> Co-authored-by: Devil1716 <149754374+Devil1716@users.noreply.github.com> Co-authored-by: Mike Chrzanowski <mchrzanowski@nvidia.com> Co-authored-by: Mike Chrzanowski <mchrzanowski@gcp-nrt-cs-001-login-001.cm.cluster> Co-authored-by: Dingqing Yang <dingqingy@nvidia.com> Co-authored-by: liuzhenhai93 <liuzhenhai93@outlook.com> Co-authored-by: mathemakitten <helenn@nvidia.com> Co-authored-by: Jenny Chen <jennifchen@nvidia.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> Co-authored-by: kajalj22 <kajalj@nvidia.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: megnvidia <mmiranda@nvidia.com> Co-authored-by: Philip Petrakian <pgpetrak@gmail.com> Co-authored-by: Xuanteng Huang <44627253+xuantengh@users.noreply.github.com> Co-authored-by: Tailai Ma <58548582+xiaoyao0115@users.noreply.github.com> Co-authored-by: Santosh Bhavani <santosh.bhavani@live.com> Co-authored-by: Zhongbo Zhu <42691305+zhongbozhu@users.noreply.github.com> Co-authored-by: Nick Schank <nick@reflection.ai> Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com> Co-authored-by: Shivanjan Chakravorty <schakravorty846@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Tuomas Rintamaki <trintamaki@nvidia.com> Co-authored-by: Tyler Poon <tylerpoon@gmail.com> Co-authored-by: Collin McCarthy <cmccarthy@nvidia.com> Co-authored-by: Matthieu Le <matthieul@nvidia.com> Co-authored-by: Piotr Zelasko <pzelasko@nvidia.com> Co-authored-by: Ehsan Hosseini Asl <ehosseiniasl@nvidia.com> Co-authored-by: Siddharth Singh <sidsingh@nvidia.com> Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com> Co-authored-by: Xiaowei Ren <103958965+xrennvidia@users.noreply.github.com> Co-authored-by: wdykas <73254672+wdykas@users.noreply.github.com> Co-authored-by: William Dykas <wdykas@oci-hsg-cs-001-vscode-03.cm.cluster> Co-authored-by: kunlunl <kunlunl@nvidia.com> Co-authored-by: Xuesong Ye <xuesongyey@gmail.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: Daisy Gao <daisyg@nvidia.com> Co-authored-by: lichenlu <lichenlu_8618@163.com> Co-authored-by: peibli <lipeibao@126.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Co-authored-by: Gao Deng <gdeng@login-lyris02.lyris.clusters.nvidia.com> Co-authored-by: Pavel Gein <pavelgejn@yandex.ru> Co-authored-by: Gao Deng <gdeng@login-lyris01.lyris.clusters.nvidia.com> Co-authored-by: Eric Harper <eharper@nvidia.com> Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> Co-authored-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-01.cm.cluster> Co-authored-by: root <root@pool0-01849.cm.cluster> Co-authored-by: Kamran Jafari <kjafarisadeg@nvidia.com> Co-authored-by: Nan Zheng <80790206+nanz-nv@users.noreply.github.com> Co-authored-by: Li Tao <lit@nvidia.com> Co-authored-by: Zhongbo Zhu <zhongboz@nvidia.com> Co-authored-by: Robert Kirby <ArEsKay3@users.noreply.github.com> Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Co-authored-by: Cole Hawkins <chawkins@nvidia.com> Co-authored-by: Qi Zhang <qizhang@nvidia.com> Co-authored-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> Co-authored-by: tongliu <tongliu@nvidia.com> Co-authored-by: shurkat-nvidia <shurkat@nvidia.com> Co-authored-by: Rui WANG <51357798+rui23@users.noreply.github.com> Co-authored-by: rionawang <rionawang@tencent.com> Co-authored-by: Tom Long <tolong@nvidia.com> Co-authored-by: Lintch <44701395+returnL@users.noreply.github.com> Co-authored-by: Yan Bai <bayan@nvidia.com> Co-authored-by: Deyu Fu <deyuf@nvidia.com> Co-authored-by: Anish Mahishi <amahishi@nvidia.com> Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@users.noreply.github.com> Co-authored-by: jingqiny-99 <jingqiny@nvidia.com> Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com> Co-authored-by: Pranav Thombre <pthombre@nvidia.com> Co-authored-by: Dennis Liu <denliu@denliu.nvidia.com> Co-authored-by: Shijie <505749828@qq.com> Co-authored-by: Guihong Li <guihongl@nvidia.com> Co-authored-by: Guihong Li <guihongl@oci-hsg-cs-001-vscode-02.cm.cluster> Co-authored-by: Yan Xu <yxu1@nvidia.com> Co-authored-by: Li Jinliang <975761915@qq.com> Co-authored-by: Baibaifan <39549453+Baibaifan@users.noreply.github.com> Co-authored-by: yangfan.bai <yangfan.bai@shopee.com> Co-authored-by: HaochenYuan <106647990+HaochenYuan@users.noreply.github.com> Co-authored-by: Haochen Yuan <haocheny@login-eos01.eos.clusters.nvidia.com> Co-authored-by: ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟 <hollowman@opensuse.org> Co-authored-by: Lawrence McAfee <85179052+lmcafee-nvidia@users.noreply.github.com> Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com> Co-authored-by: Kaixiang Lei <5780122+shyoshyo@users.noreply.github.com> Signed-off-by: Jianbin Chang <jianbinc@nvidia.com>

wplf added the Run tests label May 12, 2026

wplf changed the title ~~feat(examples/multimodal_dev): add Qwen3.5-VL training example~~ [dev] [5/5] Qwen3.5 support: Qwen3.5-VL training example May 12, 2026

wplf force-pushed the feat/qwen35-vl-example branch from 0c608f0 to 9d9392a Compare May 13, 2026 03:10

wplf mentioned this pull request May 13, 2026

[dev] [follow-up] Qwen3.5 support: MoE aux loss padding_mask #4776

Merged

wplf force-pushed the feat/qwen35-vl-example branch from 9d9392a to 62a7890 Compare May 13, 2026 10:24

Victarry self-requested a review May 19, 2026 04:57

Victarry mentioned this pull request May 19, 2026

[ROADMAP][2026 Q2] Megatron Core MoE Roadmap #4815

Open

72 tasks

wplf marked this pull request as ready for review May 19, 2026 08:28

wplf requested review from a team as code owners May 19, 2026 08:28

svcnvidia-nemo-ci added the complexity: high label May 19, 2026

Victarry force-pushed the feat/qwen35-vl-example branch from 62a7890 to 0915f61 Compare May 20, 2026 02:00

wplf and others added 2 commits May 20, 2026 07:27

wplf force-pushed the feat/qwen35-vl-example branch from 36286c5 to 6d2e13c Compare May 20, 2026 15:39

copy-pr-bot Bot temporarily deployed to test May 27, 2026 06:24 Inactive

Victarry reviewed May 27, 2026

View reviewed changes

wplf mentioned this pull request May 28, 2026

examples/multimodal_dev: docs + Victarry review fixes + vision patch merger parity test wplf/Megatron-LM#3

Closed

wplf added 2 commits May 27, 2026 20:35

wplf force-pushed the feat/qwen35-vl-example branch from 3248b34 to 4dfdd06 Compare May 28, 2026 03:36

copy-pr-bot Bot temporarily deployed to test May 28, 2026 03:43 Inactive

Victarry approved these changes May 28, 2026

View reviewed changes

Victarry added this pull request to the merge queue May 29, 2026

Merged via the queue into NVIDIA:dev with commit 58f3e67 May 29, 2026
184 of 185 checks passed

wplf mentioned this pull request Jun 3, 2026

Merge GDN conv fusion (fused pre-gated-delta-rule kernels) wplf/Megatron-LM#13

Closed

This was referenced Jun 11, 2026

don't merge #5284

Draft

feat(fusions): fused mRoPE for Qwen3.5-VL #5294

Open

Uh oh!

Conversation

wplf commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Qwen3.5 support series

Summary

Dependency

Functionality support

CP / THD correctness verification

Checkpoint conversion (HF → Megatron-FSDP DTensor)

Risk

Test plan

Uh oh!

copy-pr-bot Bot commented May 12, 2026

Uh oh!

Victarry commented May 24, 2026

Uh oh!

BestJuly commented May 25, 2026

Uh oh!

wplf commented May 25, 2026

Uh oh!

wplf commented May 27, 2026

Uh oh!

cryoco commented May 27, 2026

Uh oh!

wplf commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cryoco commented May 27, 2026

Uh oh!

wplf commented May 27, 2026

Uh oh!

Victarry left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wplf commented May 28, 2026

Uh oh!

copy-pr-bot Bot commented May 28, 2026

Uh oh!

wplf commented May 28, 2026

Uh oh!

svcnvidia-nemo-ci commented May 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

wplf commented May 12, 2026 •

edited

Loading

wplf commented May 27, 2026 •

edited

Loading