chore: nightly sync main into dev (29_06_2026)

[svcnvidia-nemo-ci]

Summary

Nightly sync of main → dev, merging 51 commits from main (origin/dev..origin/main, merge-base fc4597c0c).

• Python lines: +6809 / -3 across 54 files
• Total: 66 files changed, 7484 insertions(+), 45 deletions(-)

⚠️ Conservative (dev-superset) resolution — please read

The branch's pre-push dev-feature-preservation guard aborts if the merged tree drops any dev-only line in a non-override file. This sync is unusual: main is genuinely ahead of dev in several subsystems (dev lacks log_probs_kernel/logprobs_mode, inter_document_masking, checkpoint process-group threading, etc.), and main refactored shared code in place. Incorporating those refactors verbatim would replace (drop) dev's versions of those lines and trip the guard (1101 flagged lines on a naive full merge).

To satisfy the guard, in-place divergences were resolved in dev's favor. This PR therefore contains:

• ✅ main's additive content — all new files + additions that do not replace dev lines (+6809 Python lines).
• ⏸️ deferred — main's in-place modifications to files dev also evolved (below). These need a follow-up (dev catching up, or a human reconciliation), as the guard cannot pass while they overwrite dev's lines.

Deferred main changes (kept dev's version to preserve dev-only lines)

• RL rollout submission/consumption granularity (Add RL rollout submission and consumption granularity controls #5306) — kept dev's rl_parallel_generation_tasks design.
• Per-sequence/per-document CP batch balancing + inter_document_masking data path (Add inter-document attention masking to GPTDataset #5298) — kept dev's get_batch + sequence-packing-scheduler.
• schedules.py aux-loss-scale generalization — kept dev's DSA-indexer scaling.
• Inference config refactor / to_inference_config slimming — kept dev's RL inference path.
• Checkpoint process-group threading (Thread process groups through training checkpoint paths #5486) — kept dev's mpu.get_* checkpointing.
• MTP full-model CUDA-graph caching (Add full model cuda graph support for MTP inference #4950) — kept dev's _decoder_hidden_states_cache.
• Absorbed-MLA projection refactor ([split 3/4] Refactor absorbed MLA projection handling #5245), sampling logprobs_mode (Add logprobs_mode (raw/processed) to inference config #5419), oncall script consolidation (Narrow oncall responsibilities #5490).
• Override-list files (training.py, initialize.py) were also kept at dev's version for consistency with the dev-design core.

main-new files brought in (dormant where their feature-core is deferred)

megatron/core/inference/disaggregation/*, megatron/training/config/inference_config.py, dsa_layout.py/dsa_masking.py, megatron/rl/rollout_granularity.py, FSDP experimental/{fully_shard,module,parameter_group}.py, examples/mimo/*, .github/scripts/{community_request_assignee,github_slack_utils}.py, new functional/perf test recipes.

Removed (half-integrated new files whose feature-core is deferred)

• examples/mimo/model_providers/nemotron_moe_vlm.py (+ test) — imports mamba_stack_spec absent on dev.
• tests/unit_tests/distributed/megatron_fsdp/test_{cuda_graph,experimental_fully_shard}.py — import fully_shard/Placements not exported by dev's experimental __init__.

Dependency triple & CODEOWNERS

uv.lock and docker/Dockerfile.ci.dev kept identical to dev. pyproject.toml differs from dev only by adding the launch_on_gb200 pytest marker (used by main's GB200 tests; does not affect uv.lock). .github/CODEOWNERS unchanged (identical to dev).

🤖 Generated with Claude Code

---

CI Status — ready for review

CI run on HEAD 7a7a72952. All non-exempt required checks are terminal and green (161 SUCCESS + 4 SKIPPED):

• ✅ Nemo_CICD_Test (aggregate gate): SUCCESS
• ✅ All unit tests, functional/integration tests (gpt / moe / hybrid — 100-step training + golden-value comparison), linting, cicd-container-build (aws + gcp), wheel builds, copyright-check, pre-flight.

Accepted pre-existing failures (verified against prior sync, not sync-caused)

• DCO — ACTION_REQUIRED: main carries 6 commits without a Signed-off-by trailer (e.g. [skip ci] / chore: rotate oncall) which the sync brings in but cannot rewrite. The merge commit + fix commit are signed off. Prior merged sync chore: nightly sync main into dev (22_06_2026) #5430 had identical DCO ACTION_REQUIRED and merged into dev.
• cicd-mbridge-testing — FAILURE: downstream Megatron-Bridge pipeline. This merged tree is a dev-superset (no Megatron-LM-side API break introduced), and the prior merged sync chore: nightly sync main into dev (22_06_2026) #5430 had identical cicd-mbridge-testing FAILURE with the same Run MBridge tests label and merged into dev.

[svcnvidia-nemo-ci]

/ok to test 7a7a729

[svcnvidia-nemo-ci]

✅ Ready for human review. CI is green on HEAD 7a7a72952 (Nemo_CICD_Test = SUCCESS).

Non-exempt checks (all terminal + green): 161 SUCCESS + 4 SKIPPED — all unit-test buckets, functional/integration tests (gpt / moe / hybrid, 100-step training + golden-value comparison), linting, cicd-container-build (aws + gcp), wheel builds, copyright-check, pre-flight. CI run: https://github.com/NVIDIA/Megatron-LM/actions/runs/28404169954

Documented pre-existing failures (verified, not sync-caused):

• DCO (ACTION_REQUIRED) — main carries 6 commits without Signed-off-by; the sync cannot rewrite main's history. Prior merged sync chore: nightly sync main into dev (22_06_2026) #5430 had identical DCO ACTION_REQUIRED and merged into dev.
• cicd-mbridge-testing (FAILURE) — downstream Megatron-Bridge pipeline. Prior merged sync chore: nightly sync main into dev (22_06_2026) #5430 had identical cicd-mbridge-testing FAILURE with the same Run MBridge tests label and merged into dev. This merged tree is a dev-superset, so no new Megatron-LM-side break was introduced.

Resolution approach. The branch's pre-push dev-feature-preservation guard aborts the push if any dev-only line is dropped (a naive full merge flagged 1101). Because main is genuinely ahead here with in-place refactors (not the usual dev-ahead sync), conflicts/divergences were resolved in dev's favour — this PR brings in main's additive content (new files + non-conflicting additions, +6.8k Python lines) and defers main's in-place modifications (RL granularity #5306, MTP CUDA-graph #4950, absorbed-MLA refactor #5245, inference logprobs_mode #5419, checkpoint PG-threading #5486, inter_document_masking #5298, oncall consolidation, etc.) — see PR description.

API mismatches detected & fixed during CI iteration (so the suite tests dev's code consistently):

• hybrid_layer_specs.py — main's DSA-CP dsa_layer used combined linear_kv_up_proj vs dev's split linear_k_up_proj/linear_v_up_proj.
• text_generation_controller.py / flashinfer_sampling.py — main's calculate_log_probs(sampling=…) vs dev's signature.
• install-test.yml — main's new megatron.training import-check hit dev's unguarded import yaml.
• Reverted main-version unit tests to dev / removed main-new tests for deferred features.

Files from main vs dev: No file was taken wholesale from main; the dependency triple (pyproject.toml, uv.lock, docker/Dockerfile.ci.dev) and .github/CODEOWNERS are byte-identical to dev. No [tool.uv.sources] reconciliation was needed (dev's git sources are a superset).

[svcnvidia-nemo-ci]

Superseded by today's nightly sync.

[svcnvidia-nemo-ci]

Superseded by today's nightly sync.

[svcnvidia-nemo-ci]

Superseded by today's nightly sync.