chore: nightly sync main into dev (16_05_2026)

[svcnvidia-nemo-ci]

Summary

Nightly sync of main into dev for 2026-05-16. Brings in 49 commits from main, including:

• A2A overlap for Megatron-FSDP ([Main][feat] Support A2A Overlap for Megatron-FSDP #3797)
• Reorder mtp_post_process after attention backward in 1F1B (Reorder mtp_post_process after attention backward in 1F1B schedule plan #4695)
• Use MSC for checking checkpoint existence ([fix] Use MSC for checking checkpoint existence #4251)
• Combine GEMM + SwiGLU fused MLP PRs (Combine GEMM + SwiGLU fused MLP PRs (3890, 4071, 4095, 4219, 4311, 4324) → main #4636)
• Strengthen test_checkpoint to verify distributed checkpoint behavior (Strengthen test_checkpoint to verify distributed checkpoint behavior #4711)
• Inference: Optimize Prefill Engine Steps for Nemotron (Inference: Optimize Prefill Engine Steps for Nemotron #4764)
• Bump nvidia-modelopt>=0.44.0 (Bump nvidia-modelopt>=0.44.0 #4803)
• Use Protocols to type-check linear_proj submodules of Attention (Use Protocols to type-check linear_proj submodules of Attention #3434)
• … and 41 more (see merge commit).

Python lines: +13121 / -3926 across 160 files

Files taken from main (per nightly-sync skill)

Files known to have semantic conflicts where dev's versions reference args
or APIs that main removed/renamed:

• megatron/training/training.py
• megatron/training/initialize.py
• megatron/training/utils.py
• megatron/training/datasets/data_samplers.py

These were explicitly checked out from origin/main after the -X theirs
merge so they capture main's caller signatures verbatim.

Files preserved from dev (must not change)

• .github/CODEOWNERS — dev's governance file; restored after -X theirs
  overrode it.
• pyproject.toml, uv.lock, docker/Dockerfile.ci.dev — tightly coupled
  triple; dev's versions retain fast-hadamard-transform,
  nvidia-resiliency-ext, and the --group no_pypi_wheels flag that main
  doesn't have.

Special handling: megatron/core/datasets/data_schedule.py

Main and dev have disjoint class sets in this file
(HybridCPDataLoaderWrapper on main vs BasePackingScheduler family on
dev). Verified the merged file retains BOTH sets after -X theirs, so no
manual surgery needed this cycle.

API mismatches detected and fixed (post-merge surgery)

1. megatron/core/optimizer/distrib_optimizer.py — copy_main_params_to_model_params:
   -X theirs produced a duplicate fp8 quantize-shard loop (main's gated
   if fp8 / elif fp4 / else pass block followed by dev's old unconditional
   fp8 loop). Removed the duplicate; kept main's else: pass structure and
   preserved dev-only OptimizerStateOffloader additions.

2. megatron/core/transformer/multi_latent_attention.py —
   forward(): leftover call to off_interface.group_commit(...) from
   main's version of the attn_proj section, but dev's
   FineGrainedActivationOffloadingInterface exposes group_offload(...)
   instead. Converted to dev's manager pattern.

3. megatron/core/transformer/attention.py — same fix as above for the
   attn_proj section.

Git source reconciliation

[tool.uv.sources] in dev's pyproject.toml covers everything main
declares plus dev-only sources (fast-hadamard-transform,
nvidia-resiliency-ext). Only revision difference is transformer-engine
(main 42b8400, dev f031cf8); kept dev's revision — will revisit if CI
reveals an API symbol that requires main's newer revision.

Files restored from main (deleted in dev)

None. Three files exist on main but not on the merged tree
(.github/workflows/multi-approval-bot.yml,
tests/test_utils/recipes/h100/bert.yaml,
tests/test_utils/recipes/h100/t5.yaml); all were verified as intentional
dev deletions (see commit 2b4b9c428 and 40d30942d).

Remerge-diff (file-level)

Click to expand merge-conflict file list

 .github/CODEOWNERS                                 |   53 -
 docker/Dockerfile.ci.dev                           |    2 +-
 megatron/core/distributed/fsdp/mcore_fsdp_adapter.py |   15 -
 megatron/core/extensions/transformer_engine.py     |  190 ---
 megatron/core/models/gpt/experimental_attention_variant_module_specs.py | 4 -
 megatron/core/models/gpt/fine_grained_callables.py |   13 -
 megatron/core/models/gpt/gpt_layer_specs.py        |   18 -
 megatron/core/models/gpt/gpt_model.py              |   17 -
 megatron/core/optimizer/distrib_optimizer.py       |   44 +-
 megatron/core/pipeline_parallel/combined_1f1b.py   |   13 -
 megatron/core/ssm/gated_delta_net.py               |   24 -
 megatron/core/transformer/attention.py             |   16 +-
 megatron/core/transformer/moe/experts.py           |  317 -----
 megatron/core/transformer/moe/moe_layer.py         |   14 -
 megatron/core/transformer/moe/router.py            |    5 -
 megatron/core/transformer/multi_latent_attention.py |  11 +-
 megatron/core/transformer/multi_token_prediction.py |  40 -
 megatron/core/transformer/transformer_block.py     |  174 ---
 megatron/core/transformer/transformer_config.py    |   12 -
 megatron/core/transformer/transformer_layer.py     |   50 -
 megatron/training/argument_utils.py                |   12 -
 megatron/training/arguments.py                     |  438 -------
 megatron/training/checkpointing.py                 |   28 -
 megatron/training/config/container.py              |    4 -
 megatron/training/datasets/data_samplers.py        |   70 +-
 megatron/training/initialize.py                    |    4 +-
 megatron/training/training.py                      | 1249 +-
 megatron/training/utils.py                         |  137 ++-
 pretrain_hybrid.py                                 |   17 -
 pretrain_vlm.py                                    |    8 -
 pyproject.toml                                     |   18 +-
 tests/.../model_config.yaml                        |   10 -
 tests/test_utils/recipes/h100/gpt.yaml             |    7 -
 tests/unit_tests/a2a_overlap/test_fsdp_1f1b_overlap.py |  12 -
 tests/unit_tests/inference/test_moe_dispatching_and_routing.py | 4 -
 tests/unit_tests/models/test_hybrid_moe_model.py   |    8 -
 tests/unit_tests/ssm/test_gated_delta_net.py       |   29 -
 tests/unit_tests/transformer/moe/test_routers.py   |   10 -
 tests/unit_tests/transformer/moe/test_token_dispatcher.py | 12 -
 tests/unit_tests/transformer/test_attention.py     |    4 -
 tests/unit_tests/transformer/test_cuda_graphs.py   |   10 -
 uv.lock                                            |  493 +-

Run git show --remerge-diff 906c09222 locally for full per-hunk detail.

🤖 Generated by the nightly main-to-dev sync bot.

[svcnvidia-nemo-ci]

/ok to test 247fbc0

[svcnvidia-nemo-ci]

/ok to test 247fbc0

[svcnvidia-nemo-ci]

/ok to test cd222aa

[svcnvidia-nemo-ci]

Superseded by today's nightly sync.