chore: nightly sync main into dev (15_05_2026) by svcnvidia-nemo-ci · Pull Request #4823 · NVIDIA/Megatron-LM

svcnvidia-nemo-ci · 2026-05-15T21:35:20Z

Summary

Nightly sync of main into dev. Merges 40 commits from main into the
current dev tip.

Python lines: +10866 / -2638 across 145 files
Merge strategy: git merge origin/main -X theirs --no-edit from origin/dev
40 conflict files resolved (full surgical diff inside <details> below)

Files restored from dev (must NOT take main's version)

.github/CODEOWNERS — dev's reviewer config differs intentionally
pyproject.toml, uv.lock, docker/Dockerfile.ci.dev — tightly coupled
dependency triple that includes dev-only entries (fast-hadamard-transform,
dev's TransformerEngine revision, etc.)

Files explicitly overridden from main

Per the nightly-sync skill, these are known sources of semantic conflicts
where dev's versions reference args/APIs that main removed or renamed:

megatron/training/training.py
megatron/training/initialize.py
megatron/training/utils.py
megatron/training/datasets/data_samplers.py
megatron/core/optimizer/layer_wise_optimizer.py

API mismatches detected and fixed

The merge produced two call sites where main's caller used the now-renamed
off_interface.group_commit(...) static method; dev's
FineGrainedActivationOffloadingInterface only exposes the instance-method
group_offload(...). Switched both to the dev-style pattern that saves the
context-manager handle and calls .group_offload(...) on it:

megatron/core/transformer/attention.py (linear_proj output offload)
megatron/core/transformer/multi_latent_attention.py (attn_proj output
offload)

Also removed a duplicate import of ArgumentGroupFactory in
megatron/training/arguments.py introduced by the merge.

Files in main intentionally NOT restored

.github/workflows/multi-approval-bot.yml — dev deleted in PR ci: Remove multi-approval action from dev branch #3576
("ci: Remove multi-approval action from dev branch")
tests/test_utils/recipes/h100/bert.yaml, t5.yaml — never referenced
by anything in dev's tree (documented in prior sync commit 40d3094)

pyproject.toml git source reconciliation

Compared [tool.uv.sources] in main vs dev. No NEW git sources appear in
main that are missing from dev. Differences (TransformerEngine and
nvidia-resiliency-ext revisions) are intentional dev pins; keeping dev's
versions.

Formatting

Ran black --skip-magic-trailing-comma --skip-string-normalization then
isort (pinned 5.13.2 to match CI) on every changed Python file. Both
tools verify idempotent.

Conflict resolutions (remerge-diff)

40 files — full surgical diff

(Diff omitted for brevity; reviewers can reproduce with
git show --remerge-diff <merge-sha>.)

Test plan

Unit tests pass
Functional tests pass
MBridge tests pass
Internal GitLab CI pass

🤖 Generated with Claude Code

Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>

…ename seq_len (#4094)" (#4718) Signed-off-by: oliver könig <okoenig@nvidia.com>

…al tests` (#4730) Signed-off-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Signed-off-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: Shivanjan Chakravorty <shivanjanc@nvidia.com> Co-authored-by: Cory Ye <44509866+cspades@users.noreply.github.com>

Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: Shivanjan Chakravorty <shivanjanc@nvidia.com>

Signed-off-by: Ajay Balasa <abalasa@nvidia.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Signed-off-by: Maanu Grover <maanug@nvidia.com>

Signed-off-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Tuomas Rintamaki <trintamaki@nvidia.com> Co-authored-by: Tyler Poon <tylerpoon@gmail.com> Co-authored-by: Collin McCarthy <cmccarthy@nvidia.com> Co-authored-by: Matthieu Le <matthieul@nvidia.com> Co-authored-by: Piotr Zelasko <pzelasko@nvidia.com> Co-authored-by: Ehsan Hosseini Asl <ehosseiniasl@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-authored-by: Siddharth Singh <sidsingh@nvidia.com>

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

…parallel_size * expert_tensor_parallel_size (#4678) Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com>

#4509) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Signed-off-by: Maanu Grover <maanug@nvidia.com>

#4492) Signed-off-by: Xiaowei Ren <xren@nvidia.com>

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-authored-by: William Dykas <wdykas@oci-hsg-cs-001-vscode-03.cm.cluster>

…parallelism correctly (#4687)

svcnvidia-nemo-ci · 2026-05-16T01:11:49Z

/ok to test 732f382

svcnvidia-nemo-ci · 2026-05-16T02:58:46Z

/ok to test 6d3abeb

svcnvidia-nemo-ci · 2026-05-16T21:14:13Z

Superseded by today's nightly sync.

nschank and others added 30 commits May 10, 2026 20:59

Create a Protocol for the MLP layer of TransformerLayer (#3435)

5e31514

Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>

Revert "Add Python-side guardrail for HybridEP InfiniBand limit and r…

a2ec5c1

…ename seq_len (#4094)" (#4718) Signed-off-by: oliver könig <okoenig@nvidia.com>

chore(beep boop 🤖): Bump (main) (2026-05-11)

e93755e

Add Python-side guardrail for DeepEP IB limits (#4719)

ad58411

ci: revert bad uv.lock bump and label future bumps with `Run function…

5123f6a

…al tests` (#4730) Signed-off-by: oliver könig <okoenig@nvidia.com>

[ci] fix: treat cancelled run-main-script step as failure (#4727)

33d47e0

Signed-off-by: oliver könig <okoenig@nvidia.com>

ci: Major refactor of release-workflows (#4602)

e42e2fa

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

build(deps): bump nvidia-modelopt to 0.43 (#4723)

434368c

Signed-off-by: oliver könig <okoenig@nvidia.com>

fix(fsdp): recognize legacy GDN TP metadata (#4664)

74687fe

Signed-off-by: Shivanjan Chakravorty <shivanjanc@nvidia.com> Co-authored-by: Cory Ye <44509866+cspades@users.noreply.github.com>

Fixes for Nemotron3 Super release test config (#4544)

9718f7d

Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com>

feat(gpt): add output postprocess hook (#4686)

97f3bce

Signed-off-by: Shivanjan Chakravorty <shivanjanc@nvidia.com>

Add bump-base-image skill and update golden value comparison (#4733)

f744215

Signed-off-by: Ajay Balasa <abalasa@nvidia.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Guard omegaconf imports (#4685)

6486d52

Signed-off-by: Maanu Grover <maanug@nvidia.com>

Fix a regression introduced by #4625 for nightly runs (#4734)

7d24b28

Support transfomers 5.x.x for text generation server (#4732)

86bf476

Update transformer-engine dependency to version 2.15.0 (#4682)

815c83d

Increase CG cover from max_requests to max_tokens (#4214)

72dd053

Co-authored-by: Siddharth Singh <sidsingh@nvidia.com>

chore: rotate oncall schedule

f8c942b

fully remove legacy code (#4759)

10b514b

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

fix legacy torch save when tensor_model_parallel_size > expert_model_…

fc41581

…parallel_size * expert_tensor_parallel_size (#4678) Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

Wire --rl-inference-parsers into MRL (#4768)

d802f09

Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com>

Integrate LayerWiseDistributedOptimizer with DDP buffer infrastructure (

c1e938b

#4509) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

[training migration] Migrate mamba builder (#4550)

0dc36df

Signed-off-by: Maanu Grover <maanug@nvidia.com>

NCCL UB fix: reduce memory cost and correctly deregister NCCL mem pool (

e35d4e5

#4492) Signed-off-by: Xiaowei Ren <xren@nvidia.com>

fix: use no_mask in local ViT layer spec (#4395)

1ba0aa9

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

refit clean up and refactoring (#4762)

20bf831

Co-authored-by: William Dykas <wdykas@oci-hsg-cs-001-vscode-03.cm.cluster>

Make weight and optimizer memory estimation take into account expert …

a7c9e8c

…parallelism correctly (#4687)

Support recomputing in HybridModel (#4496)

118933a

One single flag that determines if we are in inference (#4617)

925422c

copy-pr-bot Bot temporarily deployed to test May 15, 2026 23:55 Inactive

copy-pr-bot Bot temporarily deployed to public May 15, 2026 23:58 Inactive

copy-pr-bot Bot temporarily deployed to public May 16, 2026 00:06 Inactive

svcnvidia-nemo-ci force-pushed the main2dev/15_05_2026 branch from 5332458 to 732f382 Compare May 16, 2026 01:11

copy-pr-bot Bot temporarily deployed to public May 16, 2026 01:12 Inactive

copy-pr-bot Bot temporarily deployed to test May 16, 2026 01:14 Inactive

copy-pr-bot Bot temporarily deployed to public May 16, 2026 01:15 Inactive

copy-pr-bot Bot temporarily deployed to public May 16, 2026 01:24 Inactive

fix: post-CI corrections

6d3abeb

svcnvidia-nemo-ci force-pushed the main2dev/15_05_2026 branch from 732f382 to 6d3abeb Compare May 16, 2026 02:58

copy-pr-bot Bot temporarily deployed to public May 16, 2026 02:59 Inactive

copy-pr-bot Bot temporarily deployed to test May 16, 2026 03:00 Inactive

copy-pr-bot Bot temporarily deployed to public May 16, 2026 03:02 Inactive

copy-pr-bot Bot temporarily deployed to public May 16, 2026 03:11 Inactive

svcnvidia-nemo-ci closed this May 16, 2026

Navigation Menu

Uh oh!

chore: nightly sync main into dev (15_05_2026) - #4823

chore: nightly sync main into dev (15_05_2026)#4823
svcnvidia-nemo-ci wants to merge 42 commits into
devfrom
main2dev/15_05_2026

svcnvidia-nemo-ci commented May 15, 2026

Uh oh!

svcnvidia-nemo-ci commented May 16, 2026

Uh oh!

svcnvidia-nemo-ci commented May 16, 2026

Uh oh!

svcnvidia-nemo-ci commented May 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Uh oh!

Conversation

svcnvidia-nemo-ci commented May 15, 2026

Summary

Files restored from dev (must NOT take main's version)

Files explicitly overridden from main

API mismatches detected and fixed

Files in main intentionally NOT restored

pyproject.toml git source reconciliation

Formatting

Conflict resolutions (remerge-diff)

Test plan

Uh oh!

svcnvidia-nemo-ci commented May 16, 2026

Uh oh!

svcnvidia-nemo-ci commented May 16, 2026

Uh oh!

svcnvidia-nemo-ci commented May 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants