chore: nightly sync main into dev (18_05_2026)

[svcnvidia-nemo-ci]

Summary

Nightly main → dev sync for 18_05_2026.

• Synced 57 commits from origin/main into origin/dev.
• Python lines: +14033 / -3721 across 181 files
• 44 files had merge conflicts; resolved per the nightly-sync skill (favoring main on genuine conflicts, preserving dev-only additions, applying file-specific overrides).

Files where main's version was taken

Per the nightly-sync skill, these training-side files reference dev-only args/APIs main has since removed/renamed, so they were overridden with main's version:

• megatron/training/training.py
• megatron/training/initialize.py
• megatron/training/utils.py
• megatron/training/datasets/data_samplers.py

Files preserved from dev (override-from-dev)

Per the skill, the dependency-management triple and CODEOWNERS were kept at dev's version:

• .github/CODEOWNERS (governance: dev's reviewer assignments)
• pyproject.toml (dev superset: includes fast-hadamard-transform, nvidia-resiliency-ext)
• uv.lock
• docker/Dockerfile.ci.dev

No git source reconciliation was needed — dev's [tool.uv.sources] is already a superset of main's.

API mismatches detected and fixed

Per skill's API Mismatch Detection (Phase 1):

• megatron/core/transformer/multi_latent_attention.py — off_interface.group_commit(...) (main's API) was leaking into dev's group_offload-based interface. Rewrote the attn_proj block to use attn_proj_manager.group_offload(...) matching dev's FineGrainedActivationOffloadingInterface.

• megatron/core/models/gpt/gpt_model.py — main's version dropped the LinearCrossEntropyModule and paged_stash_init_chunk_handler imports but the merged file still uses both symbols (line 253 and 493). Restored the two imports from dev's version.

• megatron/core/models/gpt/gpt_layer_specs.py — main no longer uses TEFusedDenseMLP (replaced by not_none(TEFusedMLP)), but the import block still imported it (dev's). Removed the unused import and the corresponding None placeholder.

• megatron/core/transformer/moe/experts.py — dev's nullcontext import was orphaned after main removed the stash_context usage that relied on it. Removed the unused import.

Files restored after deletion

None this run — the three files missing from dev (multi-approval-bot.yml, tests/test_utils/recipes/h100/bert.yaml, .../t5.yaml) were intentionally absent on dev:

• multi-approval-bot.yml was explicitly removed by dev PR ci: Remove multi-approval action from dev branch #3576.
• The two recipe YAMLs are not referenced anywhere in the merged tree (CI only references unit-tests.yaml in that directory).

Formatting

Ran black --skip-magic-trailing-comma --skip-string-normalization (24.10.0) and isort (5.13.2) on all changed Python files. Verified both pass --check on the autoformat scope (megatron/core/ + tests/). Pylint and ruff also clean on changed files.

Merge conflict summary

44 conflicted files (click to expand)

.github/CODEOWNERS
docs/user-guide/features/cuda_graph.md
megatron/core/extensions/transformer_engine.py
megatron/core/models/gpt/experimental_attention_variant_module_specs.py
megatron/core/models/gpt/fine_grained_callables.py
megatron/core/models/gpt/gpt_layer_specs.py
megatron/core/models/gpt/gpt_model.py
megatron/core/optimizer/distrib_optimizer.py
megatron/core/pipeline_parallel/combined_1f1b.py
megatron/core/pipeline_parallel/schedules.py
megatron/core/ssm/gated_delta_net.py
megatron/core/transformer/attention.py
megatron/core/transformer/moe/experts.py
megatron/core/transformer/moe/moe_layer.py
megatron/core/transformer/moe/moe_utils.py
megatron/core/transformer/moe/router.py
megatron/core/transformer/multi_latent_attention.py
megatron/core/transformer/multi_token_prediction.py
megatron/core/transformer/transformer_block.py
megatron/core/transformer/transformer_config.py
megatron/core/transformer/transformer_layer.py
megatron/rl/rl_utils.py
megatron/training/argument_utils.py
megatron/training/arguments.py
megatron/training/checkpointing.py
megatron/training/config/container.py
megatron/training/training.py
pretrain_hybrid.py
pretrain_vlm.py
pyproject.toml
tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp1_pp1_dp8_583m_throughputtest_zmq/model_config.yaml
tests/functional_tests/test_cases/nemotron/nemotron3_super_release_gb200/model_config.yaml
tests/test_utils/recipes/h100/gpt.yaml
tests/unit_tests/inference/test_moe_dispatching_and_routing.py
tests/unit_tests/models/test_hybrid_moe_model.py
tests/unit_tests/ssm/test_gated_delta_net.py
tests/unit_tests/transformer/moe/test_routers.py
tests/unit_tests/transformer/moe/test_token_dispatcher.py
tests/unit_tests/transformer/test_attention.py
tests/unit_tests/transformer/test_cuda_graphs.py
tests/unit_tests/transformer/test_transformer_layer.py
tools/preprocess_mmdata.py
uv.lock
tests/unit_tests/a2a_overlap/test_fsdp_1f1b_overlap.py (add/add)

The full git show --remerge-diff HEAD is large (~8500 lines) and not inlined; reviewers can produce it locally with:

git fetch origin main2dev/18_05_2026
git show --remerge-diff origin/main2dev/18_05_2026

Test plan

• GitHub CI green (unit tests, linting, cicd-container-build, mbridge tests)
• Internal GitLab functional tests pass (Nemo_CICD_Test)
• No regressions in golden values

🤖 Generated by the nightly main-to-dev sync bot.

[svcnvidia-nemo-ci]

/ok to test 7acc30f

[svcnvidia-nemo-ci]

/ok to test c1edd9b

[svcnvidia-nemo-ci]

Superseded by today's nightly sync.