chore: nightly sync main into dev (08_07_2026)

[svcnvidia-nemo-ci]

Summary

Nightly sync of main → dev for 08_07_2026.

• 97 commits from main merged into dev (merge-base fc4597c0c).
• Merge commit parents: dev c6449f0b2 + main 509efe6c9.
• Python line-change stats: Python lines: +24206 / -5217 across 235 files
• 64 conflicted files resolved surgically (see conflict dispositions below); the remaining changes are clean auto-merges of main's incoming work.

Started from origin/dev and ran git merge origin/main --no-edit (no global -X theirs).

Files taken wholesale from main

Where main was the more-recent / self-consistent owner and dropped no referenced dev feature:

• Override-list (skill-sanctioned): megatron/training/training.py (see graft note below), megatron/training/initialize.py, megatron/training/datasets/data_samplers.py, megatron/core/optimizer/layer_wise_optimizer.py (auto-merged), and the utils package's common_utils.py kept dev's MultiStorageClientFeature import.
• Infra scripts (main refactored inline helpers into the new .github/scripts/github_slack_utils.py): oncall_manager.py, sync_team_usergroups.py, tools/trigger_internal_ci.py.
• Fine-grained activation offloading subsystem (fine_grained_activation_offload.py + its test, plus offload hunks in attention.py, multi_latent_attention.py, cuda_graphs.py): main's version is a superset that renamed the internal group_commit → group_offload; dev's max_inflight_offloads throttle is already present in main. The one dev fix (fused-group-MLP delay_offload, PR fix: restore fused group MLP offload in main2dev sync #5493) is preserved via dev's experts.py.
• RL granularity redesign (main Add RL rollout submission and consumption granularity controls #5306): rl_utils.py (union of imports), agent/api.py, agent/reward_only_agent.py, inference/megatron.py, train_rl.py, and the RL arg definitions in arguments.py. Main removed args.rl_parallel_generation_tasks in favour of rollout_granularity.get_rl_parallel_generation_tasks(...); the merged tree uses main's API consistently.
• Checkpointing (main Thread dp_cp/expt_dp process groups through checkpoint load path #5579 threads dp_cp/expt_dp process groups): megatron/training/checkpointing.py — identical function set on both branches; main is newer.
• ModelOpt finetune (main Add Auto Quantize in ModelOpt quantize example #4821): examples/post_training/modelopt/finetune.py — main's build_lm_batch refactor; its utils.py deps exist in the merged tree.
• Loss-scale helpers (main): schedules.py — main's _get_experimental_attention_variant_loss_scale_func generalizes DSA loss scaling and still covers dev's DSA.
• Misc supersets: dynamic_engine.py, config/container.py, config/__init__.py, argument_utils.py, distillation/utils_logits.py, yaml_arguments.py, models/hybrid.py, models/gpt.py (_te_or_local_layer_spec), post_training/checkpointing.py, test_absorbed_mla.py, test_mtp_cuda_graph_inference.py, bert_embedding/embed.py, run_inference_performance_test.py, examples/bert & gpt_dynamic_inference.

Files taken wholesale from dev (dev is the actively-developed owner)

• pretrain_gpt.py, pretrain_hybrid.py, pretrain_hybrid_flex.py — dev's data pipeline (25/3/1 commits vs main's 3/3/3). The merged parallel_state.py auto-merged to dev and provides get_dynamic_data_context_parallel_groups (main's get_hybrid_data_context_parallel_groups is absent), so dev's callers are the only consistent choice; the merged get_batch_on_this_tp_rank import resolves to dev's common_utils variant.
• transformer_layer.py — dev superset (37-commit history, 8 extra methods incl. _forward_mlp_output_with_bias, resume_moe_experts_after_partial_cudagraph); dev already has _te_cuda_graph_replay_impl that the auto-merged test_cuda_graphs.py requires.
• moe/experts.py — dev superset (37 commits). Preserves dev-only _make_fused_impl_post_forward_hook (used by test_grouped_mlp.py) and dev's fused-group-MLP delay_offload; grafted main's mxfp8/nvfp4 offload guard + Fp8Recipe/Fp4Recipe import.
• ssm/gated_delta_net.py — dev's context-parallel-layout imports are used in its body.

Manual (surgical) union merges

• experimental_attention_variant/ cluster (dsa.py, dsa_kernels.py, module_specs.py, test_attention_variant_dsa.py): bidirectional divergence — dev owns the DeepSeek-v4 chain (csa.py/deepseek_v4_hybrid_attention.py are dev-only), main added DSA CP/THD hooks ([split 4/4] Enable DSA CP and THD hooks #5246) + AbsorbedMLA. Built a union: dsa_kernels.py = union of dev's fused kernels + main's dispatch API; dsa.py exports the union of both public-symbol sets; module_specs.get_dsa_module_spec_for_backend returns AbsorbedMLASelfAttention (main) while keeping dev's dsv4_hybrid/csa variants; the test file is the union of both suites. Verified every symbol imported from dsa/dsa_kernels/module_specs across the repo resolves.
• transformer_config.py — union of dev's dsv4_hybrid validation + main's dsa/AbsorbedMLA validation + main's fused_group_mlp recompute set + dev's max_inflight_offloads assert + main's DSA-CP allgather assert.
• gpt_model.py / hybrid_model.py — union so both dev's _decoder_hidden_states_cache (read by the auto-merged text_generation_controller.py, spec-decode) and main's block-scope mtp_decoder_hidden_states (used by test_mtp_cuda_graph_inference.py) coexist.
• core/utils.py — main's per_sequence/per_document CP-balancing dispatch + dev's cp_group is None fallback (required by pretrain_hybrid_flex.py's no-arg call).
• transformer_block.py — combined imports: dev's CheckpointManager + main's checkpointed_forward/annotate_first_last_layer (all used).
• inference/utils.py — main's imports + enable_cuda_graphs, kept dev's reformatted kwargs block.
• copy-pr-bot.yaml — union of the trustee allowlists (preserves dev-only wplf, sanandaraj5597; adds main's 11 new trustees).

API-mismatch fixes (post-merge audit)

• training.py num_floating_point_operations: taking main's training.py would have failed the dev regression test test_num_floating_point_operations.py::test_mla_attention_variants_are_counted (main counts only Symbols.ATTENTION and reports zero FLOPs for C/H/W/D MLA-variant layers). Grafted dev's num_floating_point_operations and its module-level helper _dsv4_hybrid_self_attention_flops, plus the is_linear_attention_variant import. All calls verified resolvable.
• args.hybrid_context_parallel → args.dynamic_context_parallel: main's training.py referenced the deprecated arg (undefined in the merged arguments.py, which defines only dynamic_context_parallel). Renamed both references. (self.hybrid_context_parallel on ModelParallelConfig is a legitimate deprecated-alias field with a __post_init__ shim — left as-is.)
• experts.py _make_fused_impl_post_forward_hook: restored by taking dev's superset (see above).
• Offload interface: verified no group_commit callers remain on the renamed group_offload interface.

Files deleted in dev but NOT restored (intentional dev removals)

All existed at the merge-base, were removed on dev, are unreferenced in the merged tree, and are test/CI artifacts (not production code):

• .github/workflows/multi-approval-bot.yml — removed by dev PR ci: Remove multi-approval action from dev branch #3576.
• tests/test_utils/recipes/h100/bert.yaml, tests/test_utils/recipes/h100/t5.yaml — dev removed; main's ci: Update test configurations to unify legacy scope names  #5316 scope-rename touched them but nothing references them.
• tests/unit_tests/inference/engines/test_cg_admission_gating.py — dev removed; main never modified it after base.

Dependency triple & CODEOWNERS

• .github/CODEOWNERS, pyproject.toml, uv.lock, docker/Dockerfile.ci.dev are byte-identical to origin/dev (verified). No git-source reconciliation was needed: main's only divergent source is switching mamba-ssm to a git rev, but dev satisfies mamba-ssm~=2.2 via a newer PyPI wheel (2.3.1) and main's merged code uses only stock mamba APIs.

Pre-push advisory findings — disposition

• CODEOWNERS / dependency-triple: no differences (identical to dev).
• Dev-feature-removal audit: the raw exact-line audit reports a large count dominated by formatting/reordering false positives in files where main reformatted lines dev also had, plus one intentional main removal (TensorReusePool, removed by main PR Optimize memory usage of partial CUDA graphs #5451 "Optimize memory usage of partial CUDA graphs" — unreferenced in the merged tree). The two genuine dropped-dev-feature regressions it would have caught (_make_fused_impl_post_forward_hook and num_floating_point_operations/_dsv4_hybrid_self_attention_flops) were identified and fixed (see API-mismatch fixes above). A targeted "dev-only public API referenced elsewhere but now missing" re-scan returns no remaining real drops.

Conflict resolutions (remerge-diff)

The full git show --remerge-diff HEAD (only the conflict resolutions, ~11.8k lines) is large; the 63 files it touches are exactly the conflicted set listed above. Reviewers can inspect any single resolution with:

git show --remerge-diff HEAD -- <path>

🤖 Generated with Claude Code

[svcnvidia-nemo-ci]

/ok to test 7701041

[svcnvidia-nemo-ci]

/ok to test 592e21a

[svcnvidia-nemo-ci]

/ok to test ca090d5

[svcnvidia-nemo-ci]

/ok to test 19d5f0f

[svcnvidia-nemo-ci]

Superseded by today's nightly sync.

[svcnvidia-nemo-ci]

Superseded by today's nightly sync.