Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
451 commits
Select commit Hold shift + click to select a range
1879dc2
chore(beep boop 🤖): Bump (main) (2026-04-27)
github-actions[bot] Apr 27, 2026
970c254
checkpoint integrity verification (#4305)
dimapihtar Apr 27, 2026
ebd70d3
Fix cache gating (#4455)
wdykas Apr 27, 2026
0447347
[Main] Fix FusedAdam.use_decoupled_grad mis-set for Megatron-FSDP. (#…
cspades Apr 27, 2026
8c5cf05
add permute fusion into hybrid ep (#4089)
Autumn1998 Apr 28, 2026
42e396e
Add ColocatedBridgeCommunicator for heterogeneous TP/DP MIMO training…
yashaswikarnati Apr 28, 2026
6fd6652
Fix incorrect bias display in extra_repr of Column/RowParallelLinear …
HelloWorldBeginner Apr 28, 2026
c8a4bfd
Fix assertion logic in combined_1f1b_schedule_for_interleaved_pipelin…
joapolarbear Apr 28, 2026
374fa85
ci: Fix event name reference in CI workflow condition for merge group…
balasaajay Apr 28, 2026
9c15290
Add manual sync workflow from main to dev (#4165)
Phlip79 Apr 28, 2026
9816140
fix: handle list-format quant_cfg from ModelOpt PR #1094 (#4187)
ChenhanYu Apr 28, 2026
9e98259
ci: also add Run MBridge tests label in nightly sync workflow (#4499)
Phlip79 Apr 28, 2026
533dc75
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] Apr 29, 2026
1c4e537
[training migration] Add serialization features to config container (…
maanug-nv Apr 29, 2026
f4a49cf
Fix conflict with inference graphs (#4504)
tdene Apr 29, 2026
251c6e9
chore: rotate oncall schedule
github-actions[bot] Apr 29, 2026
c5201a0
Add tools/prepare_cache.py for offline GPT dataset cache preparation …
asolergi-nv Apr 29, 2026
cb3d5d9
[build] fix: move mamba-ssm and causal-conv1d to optional [ssm] extra…
ko3n1g Apr 29, 2026
4e208a8
mamba: avoid redundant HBM reloads in causal_conv1d_update shift loop…
wdykas Apr 29, 2026
3f59bbb
Standardize misc graph interface (#4485)
tdene Apr 29, 2026
29864b2
Fix inference graph override in RL flow (#4323)
tdene Apr 29, 2026
b23aa3f
Unify and refactor Megatron-FSDP documentation. (#4418)
cspades Apr 29, 2026
51ea07e
Revert "ci: add base_sha to codecov/codecov-action upload step (#4445…
chtruong814 Apr 29, 2026
cfee04e
Skills for running unit tests and working with slurm (#4502)
yashaswikarnati Apr 29, 2026
0d98cb8
Reorganize order of operations in inference context and text generati…
tdene Apr 29, 2026
0c52c39
ci: Update CI workflow conditions to include merge group handling (#4…
balasaajay Apr 30, 2026
6ba794b
ci: add base_sha to codecov/codecov-action upload step (#4540)
chtruong814 Apr 30, 2026
580d53a
Fix release tests: remove --global-batch-size conflicting with --step…
deepakn94 Apr 30, 2026
77afc60
docs: use @file-path notation for file references in skills (#4542)
ko3n1g Apr 30, 2026
1a83320
Support YAML quant recipe in PTQ and remove first/last layer modifier…
jenchen13 Apr 30, 2026
12f18da
Avoid nsys profile crash with CUDA graphs (#4541)
tdene Apr 30, 2026
dcb2bd2
fix(ci): add retry with backoff to approve-test-queue bot (#4559)
ko3n1g Apr 30, 2026
bfd4574
New allgathervdispatcher for inference and simplify old dispatcher. …
sidsingh-nvidia Apr 30, 2026
83e7466
Fixes for modelopt examples and SFTTokenizer for transformers v5 (#4450)
jenchen13 Apr 30, 2026
3460bba
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] May 1, 2026
2d862fe
Adding code for Flextron (#4429)
sheliang-nv May 1, 2026
3b1521e
Fix partial cudagraphs + HybridEP not properly triggering DDP hook (#…
jiemingz May 1, 2026
9776b58
Ignore pytorch link anchors (#4582)
maanug-nv May 1, 2026
4e0f636
MoE dispatcher fixes: size NVLS dispatcher buffers from actual tensor…
mathemakitten May 1, 2026
74c857b
Finalize all builders in preprocess_data, not just the last key (#4573)
sayalinvidia May 2, 2026
a6cf566
refactor(skills): add when_to_use frontmatter, split ci-test-system, …
ko3n1g May 2, 2026
396bee1
Make last_token_logits graphable (#4552)
tdene May 2, 2026
0031752
fix(ci): correct off-by-one in total_steps_evaluated formula (#4591)
ko3n1g May 2, 2026
0afcfbf
Add fault injection support via nvidia_resiliency_ext. (#4370)
hexinw-nvidia May 2, 2026
cf736dc
Guard vocab reduce_scatter on TP > 1 (#4565)
mathemakitten May 2, 2026
342dd59
Move inference context bookkeeping to CPU with ContextGPUView (#4306)
lmcafee-nvidia May 3, 2026
fc43cb8
Enable InJob restart on failures. (#4594)
hexinw-nvidia May 3, 2026
442a936
Enable shared expert overlap with allgatherv in inference (#4570)
sidsingh-nvidia May 4, 2026
bb979dd
Add vLLM grouped gemm backend for MoE inference (#4566)
santhnm2 May 4, 2026
99abdc8
Move KD teacher loading to after Float16Module (#4394)
AAnoosheh May 4, 2026
0efa47a
ci: update gpt3_7b_tp4_pp1_memory_speed gb200 golden values (#4601)
ko3n1g May 4, 2026
0c479ee
Fix inference unit test (#4589)
maanug-nv May 4, 2026
cf21d70
Checkpoint conversion between GPT_model and Hybrid_model (#4482)
guihong-nv May 4, 2026
c8fde51
ci: add cadence input for test filtering in CI workflows (#4561)
balasaajay May 4, 2026
fa9c714
Handle SSM sharded tensor merge OOM with CPU fallback (#4442)
returnL May 4, 2026
2194f51
Fix `mtp_use_repeated_layer` behavior for GPT models (#3965)
rkarimimahab May 4, 2026
878228f
FlashInfer sampling (#2456)
tdene May 5, 2026
7924242
Fix main2dev workflow (#4610)
Phlip79 May 5, 2026
f4a0710
Add logic to enable chunked MLP during training (#3656)
pengdurice May 5, 2026
c817dad
Inference bug-fixes: Re-enable EP syncs for the legacy A2A dispatcher…
sidsingh-nvidia May 5, 2026
0b2b572
Remove invalid `timeout` argument for dist.barrier (#4512)
zhaoyinglia May 5, 2026
4397e07
Fix buffers in refit (#4580)
wdykas May 5, 2026
4858caf
Named validation sets (#4578)
RPrenger May 5, 2026
ae65776
Fix Hang in tests (#4575)
wdykas May 5, 2026
b819ac7
Single commit for main2dev nightly (#4614)
Phlip79 May 5, 2026
40d024b
convert tokenizer args to config (#4406)
dimapihtar May 5, 2026
190c833
Siddharth/fix ep sync (#4607)
wdykas May 5, 2026
6e5fb47
mmiranda working on another set of broken links (#4534)
megnvidia May 5, 2026
b25a76e
Fix gradient corruption with layerwise param all-gather overlap (#4609)
deepakn94 May 6, 2026
c325855
test: mark TestFusedApplyMLARope::test_forward_backward_for_q flaky_i…
ko3n1g May 6, 2026
fd443f2
chore: rotate oncall schedule
github-actions[bot] May 6, 2026
39ec5eb
remove legacy GPT code (#4322)
dimapihtar May 6, 2026
008c3e9
ci: introduce L-tier scope vocabulary via parser (#4625)
balasaajay May 6, 2026
20f0936
Inference: Tune vLLM grouped gemm, moe_sum kernel, and enable shared …
sidsingh-nvidia May 7, 2026
5c7968a
Fix crash involving evicted requests and tpot (#4645)
tdene May 7, 2026
8de8238
remove legacy tranformer and modules (#4207)
dimapihtar May 7, 2026
5fe3f06
chore: Update Docker image version to 26.04-py3 (#4611)
balasaajay May 7, 2026
387ba42
Propagate errors for failed inference requests (#4679)
mathemakitten May 7, 2026
1df264c
Inference: Cache input + position ID views (#4634)
mathemakitten May 7, 2026
7cdf652
ci: Update Gitlab base image to 26.04 pytorch (#4688)
chtruong814 May 8, 2026
932d9ee
Add periodic GPU sniff tests to detect hardware stragglers (#4662)
deepakn94 May 8, 2026
8079f37
ci: Bump GHA versions (#4606)
chtruong814 May 8, 2026
a7bd379
build: widen flashinfer-python pin to <0.7.0 (#4700)
ko3n1g May 8, 2026
a08e259
Add Python-side guardrail for HybridEP InfiniBand limit and rename se…
Shreyas-S-809 May 8, 2026
dfd4a1a
Switch oncall (#4702)
janEbert May 8, 2026
4933b24
Update golden values for various functional tests (#4703)
balasaajay May 8, 2026
31fbdf0
chore: Update golden values for various functional tests (#4706)
balasaajay May 9, 2026
708c6f5
build: upgrade mamba-ssm to 2.3.2.post1, causal-conv1d to 1.6.2.post1…
ko3n1g May 9, 2026
1256299
ci: replace uuidgen with /proc/sys/kernel/random/uuid (#4714)
ko3n1g May 10, 2026
7106fe9
chore(codeowners): add megatron/inference/ ownership (#4704)
ko3n1g May 10, 2026
5e31514
Create a Protocol for the MLP layer of TransformerLayer (#3435)
nschank May 10, 2026
a2ec5c1
Revert "Add Python-side guardrail for HybridEP InfiniBand limit and r…
ko3n1g May 11, 2026
e93755e
chore(beep boop 🤖): Bump (main) (2026-05-11)
github-actions[bot] May 11, 2026
ad58411
Add Python-side guardrail for DeepEP IB limits (#4719)
janEbert May 11, 2026
5123f6a
ci: revert bad uv.lock bump and label future bumps with `Run function…
ko3n1g May 11, 2026
33d47e0
[ci] fix: treat cancelled run-main-script step as failure (#4727)
ko3n1g May 11, 2026
e42e2fa
ci: Major refactor of release-workflows (#4602)
ko3n1g May 11, 2026
434368c
build(deps): bump nvidia-modelopt to 0.43 (#4723)
ko3n1g May 11, 2026
74687fe
fix(fsdp): recognize legacy GDN TP metadata (#4664)
Glitchfix May 11, 2026
9718f7d
Fixes for Nemotron3 Super release test config (#4544)
maanug-nv May 11, 2026
97f3bce
feat(gpt): add output postprocess hook (#4686)
Glitchfix May 11, 2026
f744215
Add bump-base-image skill and update golden value comparison (#4733)
balasaajay May 11, 2026
6486d52
Guard omegaconf imports (#4685)
maanug-nv May 11, 2026
7d24b28
Fix a regression introduced by #4625 for nightly runs (#4734)
balasaajay May 12, 2026
6853b47
Add LLaVA audio (sound) model support (#4402)
cuichenx May 12, 2026
86bf476
Support transfomers 5.x.x for text generation server (#4732)
tdene May 12, 2026
815c83d
Update transformer-engine dependency to version 2.15.0 (#4682)
balasaajay May 12, 2026
72dd053
Increase CG cover from max_requests to max_tokens (#4214)
tdene May 13, 2026
f8c942b
chore: rotate oncall schedule
github-actions[bot] May 13, 2026
10b514b
fully remove legacy code (#4759)
dimapihtar May 13, 2026
fc41581
fix legacy torch save when tensor_model_parallel_size > expert_model_…
dimapihtar May 13, 2026
d802f09
Wire --rl-inference-parsers into MRL (#4768)
tdene May 13, 2026
c1e938b
Integrate LayerWiseDistributedOptimizer with DDP buffer infrastructur…
deepakn94 May 13, 2026
0dc36df
[training migration] Migrate mamba builder (#4550)
maanug-nv May 13, 2026
e35d4e5
NCCL UB fix: reduce memory cost and correctly deregister NCCL mem poo…
xrennvidia May 13, 2026
1ba0aa9
fix: use no_mask in local ViT layer spec (#4395)
Phlip79 May 14, 2026
20bf831
refit clean up and refactoring (#4762)
wdykas May 14, 2026
a7c9e8c
Make weight and optimizer memory estimation take into account expert …
YangFei1990 May 14, 2026
118933a
Support recomputing in HybridModel (#4496)
xuantengh May 14, 2026
925422c
One single flag that determines if we are in inference (#4617)
tdene May 14, 2026
2d1fa8d
[main] feat(moe): Support packed sequence for gated delta net (GDN) (…
yuzhongw-nvidia May 14, 2026
58d45bf
remove dead manual_release_grads code path in 1F1B overlap schedule (…
Wohox May 14, 2026
ada8dfe
Fix recompute checkpointing + training CGs (#3919)
tdene May 14, 2026
dbfc96b
Use Protocols to type-check linear_proj submodules of Attention (#3434)
nschank May 14, 2026
d167123
fix tokenizers in respect to newer transformers (#4608)
dimapihtar May 14, 2026
98031e1
Bump nvidia-modelopt>=0.44.0 (#4803)
kevalmorabia97 May 14, 2026
266562f
Update owners (#4794)
Phlip79 May 14, 2026
e9a0930
ci: Update workflow to use same commit for building docker image and …
balasaajay May 14, 2026
a53107c
chore: Update nightly tests golden values (#4805)
balasaajay May 14, 2026
9b4074b
Inference: Optimize Prefill Engine Steps for Nemotron (#4764)
sidsingh-nvidia May 14, 2026
493e2be
Strengthen test_checkpoint to verify distributed checkpoint behavior …
lichenlu May 15, 2026
6815c0f
Combine GEMM + SwiGLU fused MLP PRs (3890, 4071, 4095, 4219, 4311, 43…
Connor-XY May 15, 2026
95bba94
Disable MSC by default; opt in via --enable-msc (#4629)
asolergi-nv May 15, 2026
f2070ee
[fix] Use MSC for checking checkpoint existence (#4251)
pavelgein May 15, 2026
b3b6719
ci: tolerate git-gc race in /home/runner chown after checkout (#4808)
balasaajay May 15, 2026
212f885
Reorder mtp_post_process after attention backward in 1F1B schedule pl…
gdengk May 15, 2026
f92a207
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] May 16, 2026
f9952cb
additional tests for nvrx (#4522)
dimapihtar May 15, 2026
e9d9a4c
[Main][feat] Support A2A Overlap for Megatron-FSDP (#3797)
Wohox May 16, 2026
7371e01
add is_torch_min_version in fsdp src (#4812)
xrennvidia May 17, 2026
54f90af
Add high-priority A2A stream and HybridEP preprocessing SMs (#4694)
gdengk May 17, 2026
e41b370
Refactor CUDA graph API: decompose cuda_graph_scope into full_iterati…
buptzyb May 18, 2026
a1c5f5d
chore(beep boop 🤖): Bump (main) (2026-05-18)
github-actions[bot] May 18, 2026
ab28544
Tokenizers updates (#4780)
dimapihtar May 18, 2026
5096459
Fix no nvrx tests (#4847)
dimapihtar May 18, 2026
aa786b7
Thread custom process groups through MoE grad finalization (#4782)
yashaswikarnati May 18, 2026
648b916
Fix unit tests (#4689)
shanmugamr1992 May 18, 2026
6c3fa6d
Tests/dynamic inference functional coverage (#4761)
shanmugamr1992 May 18, 2026
74daec9
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] May 19, 2026
952efb4
Fix oncall references (#4722)
janEbert May 18, 2026
b2a8ec7
Update golden values for nightly functional tests (#4850)
balasaajay May 19, 2026
43fb2f5
fix(inference): size DynamicInferenceContext KV layer_map for non-uni…
athitten May 19, 2026
14aaa7e
Modernize post-training modelopt example scripts (#4807)
kevalmorabia97 May 19, 2026
38986a9
chore: rotate oncall schedule
github-actions[bot] May 20, 2026
23ba357
test: add inference performance test harness for GPT 583M, hybrid 2B,…
shanmugamr1992 May 20, 2026
1ffcd1d
ci: Prevent shell trace in parts of _run_training.sh (#4884)
chtruong814 May 20, 2026
14dabeb
Ignore Vim swap files (#4860)
wujingyue May 20, 2026
b5d143f
M-FSDP: Make fine_grained_param_gather configurable for MXFP8 to enab…
shjwudp May 20, 2026
68d64f7
MimoOptimizer: fix distributed checkpoint save and load for non-coloc…
kamran-nvidia May 20, 2026
5df3e4e
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] May 21, 2026
0044db1
Route non-Muon params through DistributedOptimizer (#4771)
deepakn94 May 21, 2026
990fa69
ci: Gate optional CI jobs with repository variables (#4907)
chtruong814 May 21, 2026
e69a1ea
Fix tokenizers bug in nightly (#4833)
Phlip79 May 20, 2026
daec17c
Allow optimizer CG to share the same pool as full-iter CG (#4698)
nanz-nv May 21, 2026
32a7e46
Use sharded_state_dict_default in MLP.sharded_state_dict (#4693)
gdengk May 21, 2026
2b77d32
Fix MTP recompute crash with packed sequences (#4593)
BestJuly May 21, 2026
c6096d9
Update PR template (#4904)
Phlip79 May 21, 2026
0588bf7
ci: Update perf test to output logs for tests to pass (#4906)
chtruong814 May 21, 2026
6576040
Also persist asymmetrical units for the MXFP8 transpose weight buffer…
cspades May 21, 2026
80a2d39
fix no_shard training convergency and add unittest for no_shard (#3754)
wplf May 21, 2026
e9a3184
Move policy epoch stats to the message object (#4533)
ArEsKay3 May 21, 2026
9a7cd17
Add a knob to throttle the max allowed inflight offload in fine grain…
nanz-nv May 21, 2026
5e4fc93
refactor(data): consolidate get_batch and enable PP for SFT THD (#4103)
asolergi-nv May 21, 2026
0d198cd
Allow YAML MoE configs to use model specs (#4822)
chawkins-nvidia May 21, 2026
547fb17
Move bert and t5 pretrain files (#4820)
Phlip79 May 21, 2026
e27607a
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] May 22, 2026
f007db7
Paged Stashing (#4247)
nanz-nv May 22, 2026
4c63602
make FP4 param gather work with the mixed precisions in NVFP4 recipe …
xrennvidia May 22, 2026
4db6fa4
fix: Fix multi-node functional test phase sync (#4924)
chtruong814 May 22, 2026
686aa8c
Perf tests (#4917)
shanmugamr1992 May 22, 2026
0beaa53
fix(cuda_graphs): handle TE 2.15 removal of FP8GlobalStateManager.set…
balasaajay May 22, 2026
6c1bd6e
Fix paged stashing test submodules lookup (#4925)
Phlip79 May 22, 2026
fa7a23b
Add TEFusedDenseMLP for Dense+Grouped GEMM fusion on SM100+ (#4318) (…
sraman-rgb May 22, 2026
5f79118
Fix mxfp8 param gather numerical issue when DP overlap is off (#4800)
WanZzzzzz May 22, 2026
08bad7a
[MXFP8/FP4-param-gather] Post processing after forced param AG in eva…
WanZzzzzz May 22, 2026
3fb34c6
ci: Update training script paths in BERT and T5 (#4939)
balasaajay May 22, 2026
34560c4
Various training utils (#4872)
maanug-nv May 22, 2026
f7f584d
ci: restore perf test torchrun logs (#4951)
chtruong814 May 23, 2026
4bd8bb3
Fix `get_batch` return order to ignore BlendedDataset provenance fiel…
deepakn94 May 23, 2026
be2b2cd
test(release): add release goldens for deepseekv3/nemotron3 and set t…
ko3n1g May 25, 2026
3b2b6e7
chore(beep boop 🤖): Bump (main) (2026-05-25)
github-actions[bot] May 25, 2026
2f754f4
test: enable NVTE_CUTEDSL_FUSED_GROUPED_MLP via pytest fixture (#4931)
ko3n1g May 25, 2026
2bd9fd5
Avoid offsetting functional test master port (#4973)
chtruong814 May 25, 2026
4415119
Fix elastification unwrap_model import (#4972)
Devil1716 May 25, 2026
432d76b
test: re-enable paged stashing MoE tests (#4978)
ko3n1g May 26, 2026
ff64743
test(ci): re-enable 8experts2parallel_multi_dist_optimizer_instances_…
ko3n1g May 26, 2026
0e5cd0e
ci: Add support for MBridge job gating based on PR labels (#4926)
balasaajay May 26, 2026
08c368a
test: re-enable test_pp2_create_cudagraphs_first_stage on TE 2.15+ (#…
ko3n1g May 26, 2026
6ce6fac
fix(tests): initialize num_microbatches calculator in vision cudagrap…
ko3n1g May 26, 2026
859b719
ci: Add allow_failure flag to gpt and moe recipes that are failing in…
balasaajay May 26, 2026
88e7ab0
Drain predecessor reduce-scatter at dispatch time (#4940)
deepakn94 May 27, 2026
e6b2bd8
nightly(ci): Update golden values for functional t5 tests (#4995)
balasaajay May 27, 2026
7521ecb
chore: rotate oncall schedule
github-actions[bot] May 27, 2026
873678a
[main] Refactor and Improve MoE Logginginit commit (#3431)
yanring May 27, 2026
4e52a9e
ci: validate release branch-rules (#4929)
ko3n1g May 27, 2026
67b2f38
[Megatron-FSDP] Add conditional param.grad dereferencing logic to sup…
cspades May 27, 2026
71223d5
test: restrict iter-time comparison to steady-state window (#5010)
ko3n1g May 27, 2026
a6d61fb
fix(test): pin eval-global-batch-size on 15b gb200 release configs (#…
ko3n1g May 27, 2026
286445c
[fix] Release MTP assertion when EP overlap with PP=1 (#4796)
Wohox May 27, 2026
0cb4034
fix(test): widen iter-time steady-state window for short tests (#5023)
ko3n1g May 27, 2026
805e24e
Perf fix (#4996)
shanmugamr1992 May 27, 2026
457e3f7
Add dev-feature preservation gate and change schedule (#4773)
Phlip79 May 27, 2026
146d171
chore(test): remove orphan nemotron3_super_release_g200 dir (#5024)
ko3n1g May 27, 2026
7be1748
Ignore Claude worktree directory (#5020)
Phlip79 May 27, 2026
9d23a73
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] May 28, 2026
3e00820
ci: update CI workflow conditions for integration tests (#4658)
balasaajay May 28, 2026
0010683
Add NVSkills CI request workflow (#5033)
Phlip79 May 28, 2026
2a4f820
DDP wrap pg size fixes (#5006)
maanug-nv May 28, 2026
e0afe89
fix(layer_wise): tag MTP-stage word_embeddings as is_embedding_or_out…
Wohox May 28, 2026
4f2e7ce
Move LTS dependencies from pyproject.toml to Dockerfile.ci.lts (#4877)
balasaajay May 28, 2026
88efdf7
Use shared ModelOpt calibration loop on 0.45+ with 0.44 fallback fix …
kevalmorabia97 May 28, 2026
a5c2d1b
test(release): skip golden comparison on intermediate resume windows …
ko3n1g May 28, 2026
2ebca1e
[mimo] Thread position_ids through MimoModel for multimodal RoPE (#4938)
liding-nv May 28, 2026
e6ff4c3
build: Switch DSv3 on H100 to HybridEP (#5039)
ko3n1g May 28, 2026
ba9e0ea
Fix: Import unwrap_model from megatron.core.utils in modelopt example…
kevalmorabia97 May 28, 2026
3c39d98
Simple and stable Inference APIs (#4697)
YangFei1990 May 29, 2026
f63a46e
ci: Add notification step for MBridge downstream test results (#5028)
balasaajay May 29, 2026
f1b5516
Delete output tensor early (#4742)
Phlip79 May 29, 2026
f8e3885
Support ScaledSReLU in TE grouped MLP fuser (#4859)
sraman-rgb May 29, 2026
1801316
Skip gradient updates when grad norm exceeds threshold (#3460)
yfw May 29, 2026
6e0d14a
Add 9 user skills (#5066)
Phlip79 May 29, 2026
54dc530
test(nemotron): align nemotron3 super GB200 goldens with exit-interva…
ko3n1g May 30, 2026
6e091e1
chore: Update transformer-engine dependency to version 2.16.0 (#4992)
balasaajay May 30, 2026
52d1d68
Update energon version requirement (#4572)
maanug-nv May 30, 2026
791a45f
Fix test failures for new inference APIs (#5068)
YangFei1990 May 30, 2026
f33a51f
fix(ci): set PYTHONUNBUFFERED=1 in JET workload env (#5072)
ko3n1g May 30, 2026
378d81f
Preserve non-FSDP-unit buckets across AllGatherPipeline reset (#4717)
wujingyue May 30, 2026
3f6a2ed
Add opt-in MXFP8 LM-head output projection (#4825)
gdengk May 31, 2026
24cc2a8
chore(beep boop 🤖): Bump (main) (2026-06-01)
github-actions[bot] Jun 1, 2026
0a1726a
fix(ci): bound JET pipeline polling with a watchdog to prevent indefi…
ko3n1g Jun 1, 2026
595f697
ci: prune old artifacts on cluster lustre during weekly/release runs …
ko3n1g Jun 1, 2026
46f1af7
ci(test): isolate ckpt-resume tensorboard per phase (#5074)
ko3n1g Jun 1, 2026
33da12c
test: unmark EP A2A activation offload test flaky (#5009)
lhb8125 Jun 1, 2026
e656940
Change ownership groups (#5021)
Phlip79 Jun 1, 2026
496e1ff
test: skip mfsdp_fully_shard cases when world_size < mesh size (#4487)
wujingyue Jun 1, 2026
80cf756
fix mimo optimizer checkpoint metadata restore (#4791)
liding-nv Jun 1, 2026
de030fc
[mimo] Support bridge fan-out for variable modality tokens (#5062)
liding-nv Jun 1, 2026
edbf26b
Merge commit 'de030fc5823e59d7e3c3a2c7c64a9a384566acfb' into sudhu/co…
sudhu2k Aug 24, 2026
1b7e986
Merge remote-tracking branch 'origin/rocm_dev' into sudhu/core_r0.18.…
sudhu2k Aug 24, 2026
0ee8e32
core_r0.18.0 IFU cleanup
sudhu2k Aug 25, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions .agents/skills
14 changes: 14 additions & 0 deletions .claude/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"hooks": {
"UserPromptSubmit": [
{
"hooks": [
{
"type": "command",
"command": "printf '{\"hookSpecificOutput\":{\"hookEventName\":\"UserPromptSubmit\",\"additionalContext\":\"MANDATORY WORKFLOW — never skip or reorder: (1) Read the artifact first (commit, file, error, PR). (2) Identify and invoke the relevant skill via the Skill tool BEFORE forming any answer or plan — even when the answer seems obvious. (3) Only then answer using the skill context. Skipping step 2 is not allowed.\"}}'"
}
]
}
]
}
}
1 change: 1 addition & 0 deletions .claude/skills
1 change: 1 addition & 0 deletions .cursorrules
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
See CLAUDE.md for all repository guidelines.
18 changes: 13 additions & 5 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1,36 +1,43 @@
megatron/core/ @NVIDIA/core-adlr @NVIDIA/core-nemo

megatron/core/models/common/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/gpt

megatron/core/models/gpt/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/gpt

megatron/core/models/multimodal/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/multi-modal

megatron/core/models/mamba/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/hybrid-mamba
megatron/core/ssm/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/hybrid-mamba
megatron/core/models/mamba/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/hybrid-model
megatron/core/ssm/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/hybrid-model

megatron/core/models/hybrid/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/hybrid-model

megatron/core/datasets/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/datasets

megatron/core/tokenizers/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/tokenizers

megatron/core/distributed/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/data-parallelism
megatron/core/distributed/fsdp/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/megatron-fsdp

megatron/core/transformer/fsdp_dtensor_checkpoint.py @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/megatron-fsdp

megatron/core/dist_checkpointing/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/dist-checkpointing

megatron/core/optimizer/distrib_optimizer/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/dist-optimizer
megatron/core/optimizer/distrib_optimizer.py @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/dist-optimizer

megatron/core/inference/modelopt_support @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/quantization-and-inference
megatron/core/inference/modelopt_support @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/post-training

megatron/core/datasets/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/datasets

megatron/core/pipeline_parallel/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/pipeline-parallelism

megatron/core/transformer/ @NVIDIA/core-adlr @NVIDIA/core-nemo
megatron/core/transformer/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/transformer

megatron/core/transformer/moe/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/mixture-of-experts-adlr @NVIDIA/mixture-of-experts-devtech

megatron/core/inference/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/inference

megatron/inference/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/inference-interface

megatron/core/parallel_state.py @NVIDIA/core-adlr @NVIDIA/core-nemo

megatron/core/post_training/ @NVIDIA/core-adlr @NVIDIA/core-nemo @NVIDIA/post-training
Expand All @@ -44,6 +51,7 @@ megatron/training/arguments.py

.gitlab/ @NVIDIA/ci
.github/ @NVIDIA/ci
.github/oncall_schedule.json @NVIDIA/mcore-oncall-rotation
.gitlab-ci.yml @NVIDIA/ci
docker/ @NVIDIA/ci
tests/functional_tests/python_test_utils/ @NVIDIA/ci
Expand Down
4 changes: 2 additions & 2 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ assignees: ''

**Describe the bug**

A clear and concise description of what the bug is. Tag the [@mcore-oncall](https://github.com/orgs/NVIDIA/teams/mcore-oncall)
A clear and concise description of what the bug is. Tag @NVIDIA/mcore-oncall
to get oncall's attention to this issue.

**Steps/Code to reproduce bug**
Expand All @@ -26,4 +26,4 @@ A clear and concise description of what you expected to happen.

**Additional context**

Add any other context about the problem here.
Add any other context about the problem here.
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ assignees: ''
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Tag the [@mcore-oncall](https://github.com/orgs/NVIDIA/teams/mcore-oncall)
Tag @NVIDIA/mcore-oncall
to get oncall's attention to this issue.

**Describe the solution you'd like**
Expand Down
4 changes: 2 additions & 2 deletions .github/ISSUE_TEMPLATE/question.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ assignees: ''
---

**Your question**
Ask a clear and concise question about Megatron-LM. Tag the [@mcore-oncall](https://github.com/orgs/NVIDIA/teams/mcore-oncall)
to get oncall's attention to this issue.
Ask a clear and concise question about Megatron-LM. Tag @NVIDIA/mcore-oncall
to get oncall's attention to this issue.
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/regression.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ assignees: ''
---

**Describe the regression**
A clear and concise description of what the regression is. Tag the [@mcore-oncall](https://github.com/orgs/NVIDIA/teams/mcore-oncall)
A clear and concise description of what the regression is. Tag @NVIDIA/mcore-oncall
to get oncall's attention to this issue.

**To Reproduce**
Expand Down
128 changes: 98 additions & 30 deletions .github/actions/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,14 @@ inputs:
description: "Platform to run tests on (e.g. dgx_h100, dgx_gb200)"
required: false
default: "dgx_h100"
cadence:
description: "Trigger cadence for cadence filter (pr|nightly|mergegroup). Empty disables filter."
required: false
default: ""
sha:
description: "Git ref to check out. Must match the SHA used by the upstream parse step so recipes don't diverge between scheduling and execution."
required: false
default: ""
runs:
using: "composite"
steps:
Expand All @@ -70,21 +78,43 @@ runs:

- name: Checkout repository
uses: actions/checkout@v6
with:
ref: ${{ inputs.sha }}

- name: Change ownership of /home/runner/
shell: bash
run: sudo chown -R $(whoami) /home/runner/
# Tolerate vanishing `.git/objects/pack/.tmp-*` files: the prior
# `actions/checkout` may leave a background `git gc --auto` running,
# whose `git pack-objects` renames/deletes temp files while `chown`
# is walking the tree. On failure, wait 5 s for gc to settle, retry,
# then succeed unconditionally.
run: |
sudo chown -R $(whoami) /home/runner/ 2>/dev/null && exit 0
sleep 5
sudo chown -R $(whoami) /home/runner/ 2>/dev/null || true

- name: Setup python
uses: actions/setup-python@v5
with:
python-version: 3.12
python-version: '3.12'

- name: Install uuid-runtime
shell: bash -x -e -u -o pipefail {0}
run: |
for i in 1 2 3; do
apt-get update && apt-get install -y uuid-runtime && break
echo "apt attempt $i failed, retrying..."
sleep 10
done

- name: Install uuidgen
- name: Install uv
shell: bash -x -e -u -o pipefail {0}
run: |
apt-get update
apt-get install -y uuid-runtime
for i in 1 2 3; do
curl -LsSf https://astral.sh/uv/install.sh | UV_INSTALL_DIR=/usr/local/bin sh && break
echo "uv install attempt $i failed, retrying..."
sleep 10
done

- name: Create run-script (unit test)
shell: bash -x -e -u -o pipefail {0}
Expand All @@ -97,7 +127,6 @@ runs:
export PYTHONPATH=$(pwd)
export NEMORUN_HOME=$(pwd)
export NCCL_DEBUG=INFO
pip install --no-cache-dir "uv<0.9.29"
uv venv .venv
uv cache clean
uv sync --no-cache --only-group test
Expand Down Expand Up @@ -134,10 +163,12 @@ runs:
if [ "${{ inputs.lightweight }}" == "true" ]; then
ARGS+=(--enable-lightweight-mode)
fi
if [ -n "${{ inputs.cadence }}" ]; then
ARGS+=(--cadence ${{ inputs.cadence }})
fi

export PYTHONPATH=$(pwd)
export NEMORUN_HOME=$(pwd)
pip install --no-cache-dir "uv<0.9.29"
uv venv .venv
uv cache clean
uv sync --no-cache --only-group test
Expand All @@ -159,10 +190,7 @@ runs:
- name: Set timeout
shell: bash -x -e -u -o pipefail {0}
id: timeout_in_seconds
run: |
echo "::group::Set timeout"
echo "main=$(( ${{ inputs.timeout }} * 60 ))" | tee -a "$GITHUB_OUTPUT"
echo "::endgroup::"
run: echo "main=$(( ${{ inputs.timeout }} * 60 ))" | tee -a "$GITHUB_OUTPUT"

- name: Pull container
shell: bash -x -e -u -o pipefail {0}
Expand All @@ -175,54 +203,94 @@ runs:
shell: bash -x -e -u -o pipefail {0}
id: run-main-script
run: |
echo "::group::Run main script"
{ set +x; } 2>/dev/null
echo -e "\033[1;34m┌─ launching test ─────────────────────────────────────────────────────────┐\033[0m"
echo -e "\033[1;34m│ test case : ${{ inputs.test_case }}\033[0m"
echo -e "\033[1;34m│ platform : ${{ inputs.platform }} scope: ${{ inputs.scope }}\033[0m"
echo -e "\033[1;34m│ container : ${{ inputs.container-image }}\033[0m"
echo -e "\033[1;34m└──────────────────────────────────────────────────────────────────────────┘\033[0m"
{ set -x; } 2>/dev/null
echo "::group::Logs"
EXIT_CODE=0
/bin/bash job.sh || EXIT_CODE=$?
echo "exit_code=$EXIT_CODE" | tee -a "$GITHUB_OUTPUT"
exit $EXIT_CODE
echo "::endgroup::"
exit $EXIT_CODE

- name: Check result
id: check
shell: bash -x -e -u -o pipefail {0}
shell: bash -e -u -o pipefail {0}
if: always()
env:
IS_UNIT_TEST: ${{ inputs.is_unit_test == 'true' }}
MAIN_CONCLUSION: ${{ steps.run-main-script.conclusion }}
MAIN_EXIT_CODE: ${{ steps.run-main-script.outputs.exit_code }}
run: |
echo "::group::Check result"

logs_report=logs-${{ inputs.test_case }}-${{ github.run_id }}-$(uuidgen)
logs_report=logs-${{ inputs.test_case }}-${{ github.run_id }}-$(cat /proc/sys/kernel/random/uuid)
echo "logs_report=$logs_report" | sed 's/\//-/g' | sed 's/\*/-/g' | tee -a "$GITHUB_OUTPUT"

if [[ "$IS_UNIT_TEST" == "true" ]]; then
coverage_report=coverage-${{ inputs.is_unit_test == 'true' && 'unit-test' || 'e2e' }}-${{ github.run_id }}-$(uuidgen)
coverage_report=coverage-unit-test-${{ github.run_id }}-$(cat /proc/sys/kernel/random/uuid)
else
coverage_report=none
fi
echo "coverage_report=$coverage_report" | tee -a "$GITHUB_OUTPUT"

EXIT_CODE=${{ steps.run-main-script.outputs.exit_code }}
IS_SUCCESS=$([[ "$EXIT_CODE" -eq 0 ]] && echo "true" || echo "false")
EXIT_CODE="${MAIN_EXIT_CODE:-${MAIN_CONCLUSION}}"
if [[ "$MAIN_CONCLUSION" == "success" ]]; then
IS_SUCCESS=true
else
IS_SUCCESS=false
fi

if [[ "$IS_SUCCESS" == "false" && "${{ inputs.is-optional }}" == "true" ]]; then
echo "::warning:: Test failed, but displayed as successful because it is marked as optional."
echo "::warning::Test failed but is marked optional — treating as success."
IS_SUCCESS=true
fi

if [[ "$IS_SUCCESS" == "false" ]]; then
echo Test did not finish successfully.
exit 1
LOG_BASE=$([[ "$IS_UNIT_TEST" == "true" ]] && echo "assets_dir/logs" || echo "assets_dir")
LATEST_LOG=""
if [[ -d "$LOG_BASE" ]]; then
LATEST_LOG=$(find "$LOG_BASE" -name "*.log" ! -name "nccl_debug.log" -type f 2>/dev/null \
| xargs -r ls -t 2>/dev/null | head -1 || true)
fi
if [[ -n "$LATEST_LOG" ]]; then
echo -e "\033[1;36m\n📋 ── log excerpt ───────────────────────────────────────────────────────\033[0m"
echo -e "\033[1;36m ${LATEST_LOG} — last 40 lines\033[0m"
echo -e "\033[1;36m────────────────────────────────────────────────────────────────────────\033[0m"
tail -40 "$LATEST_LOG"
echo -e "\033[1;36m────────────────────────────────────────────────────────────────────────\033[0m\n"
else
echo -e "\033[33m⚠ no log file found in ${LOG_BASE}\033[0m"
fi

if [[ "$coverage_report" != "none" ]]; then
uv run coverage report -i
echo "::group::Coverage report"
uv run coverage report -i || true
echo "::endgroup::"
fi

exit $EXIT_CODE
echo "::endgroup::"
if [[ "$IS_SUCCESS" == "true" ]]; then
echo -e "\033[1;32m╔══════════════════════════════════════════════════════════════════════════╗\033[0m"
echo -e "\033[1;32m║ ║\033[0m"
echo -e "\033[1;32m║ ✅ PASSED ║\033[0m"
echo -e "\033[1;32m║ ${{ inputs.test_case }}\033[0m"
echo -e "\033[1;32m║ ║\033[0m"
echo -e "\033[1;32m╚══════════════════════════════════════════════════════════════════════════╝\033[0m"
echo "::notice title=Result::✅ ${{ inputs.test_case }} — PASSED"
exit 0
else
echo -e "\033[1;31m╔══════════════════════════════════════════════════════════════════════════╗\033[0m"
echo -e "\033[1;31m║ ║\033[0m"
echo -e "\033[1;31m║ ❌ FAILED (exit code: ${EXIT_CODE}) ║\033[0m"
echo -e "\033[1;31m║ ${{ inputs.test_case }}\033[0m"
echo -e "\033[1;31m║ ║\033[0m"
echo -e "\033[1;31m╚══════════════════════════════════════════════════════════════════════════╝\033[0m"
echo "::error title=Result::❌ ${{ inputs.test_case }} — FAILED (exit $EXIT_CODE)"
exit 1
fi

- name: Upload coverage
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v6
if: ${{ always() && steps.check.outputs.coverage_report != 'none' }}
with:
name: ${{ steps.check.outputs.coverage_report }}
Expand All @@ -232,7 +300,7 @@ runs:
include-hidden-files: true

- name: Upload logs
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v6
if: always()
with:
name: ${{ steps.check.outputs.logs_report }}
Expand Down
2 changes: 1 addition & 1 deletion .github/copy-pr-bot.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
enabled: true
auto_sync_draft: false
auto_sync_ready: true
trustees_override: ["AAnoosheh", "ArEsKay3", "Autumn1998", "BestJuly", "BoxiangW", "CarlosGomes98", "ChenhanYu", "FDecaYed", "HaochenYuan", "ISEEKYAN", "JRD971000", "Phlip79", "QiZhangNV", "RPrenger", "ShriyaRishab", "Victarry", "WanZzzzzz", "Wohox", "ZhiyuLi-Nvidia", "ahmadki", "aklife97", "ananthsub", "asolergi-nv", "buptzyb", "chtruong814", "cjld", "cspades", "cuichenx", "deepakn94", "dimapihtar", "dingqingy-nv", "duncanriach", "erhoo82", "ericharper", "fanshiqing", "faradawn", "frsun-nvda", "gautham-kollu", "gdengk", "guyueh1", "hexinw-nvidia", "huvunvidia", "hxbai", "ilml", "jalbericiola", "janEbert", "jaredcasper", "jenchen13", "jiemingz", "jingqiny-99", "jkamalu", "jon-barker", "jstjohn", "kajalj22", "kanz-nv", "kevalmorabia97", "ko3n1g", "ksivaman", "kunlunl", "kvareddy", "kwyss-nvidia", "layalir", "lhb8125", "lmcafee-nvidia", "maanug-nv", "mathemakitten", "matthieule", "mchrzanowski", "mehraakash", "mkhona-nvidia", "nanz-nv", "parthmannan", "prajwal1210", "pthombre", "rhewett-nv", "rogerwaleffe", "sajadn", "sanandaraj5597", "sancha", "santhnm2", "sbak5", "shanmugamr1992", "sharathts", "shengf-nv", "shifangx", "shjwudp", "sidsingh-nvidia", "skyw", "sudhakarsingh27", "tdene", "theothermike", "thomasdhc", "tomlifu", "trintamaki", "tylerpoon", "wdykas", "wplf", "xiaoyao0115", "xuwchen", "yanring", "yaox12", "yaoyu-33", "yashaswikarnati", "yeyu-nvidia", "yobibyte", "youngeunkwon0405", "yueshen2016", "yuzhongw-nvidia", "zhongbozhu"]
trustees_override: ["AAnoosheh", "ArEsKay3", "Autumn1998", "BestJuly", "BoxiangW", "CarlosGomes98", "ChenhanYu", "Connor-XY", "FDecaYed", "HaochenYuan", "ISEEKYAN", "JF-D", "JRD971000", "Mellonta", "Phlip79", "QiZhangNV", "RPrenger", "ShriyaRishab", "Victarry", "WanZzzzzz", "Wohox", "YangFei1990", "ZhiyuLi-Nvidia", "adistomar", "ahmadki", "aklife97", "ananthsub", "aroshanghias-nvd", "asolergi-nv", "athitten", "balasaajay", "buptzyb", "chtruong814", "cjld", "cspades", "cuichenx", "deepakn94", "dimapihtar", "dingqingy-nv", "duncanriach", "erhoo82", "ericharper", "fanshiqing", "faradawn", "fitsumreda", "frsun-nvda", "gautham-kollu", "gdengk", "guihong-nv", "guyueh1", "hexinw-nvidia", "huvunvidia", "hxbai", "ilml", "jalbericiola", "janEbert", "jaredcasper", "jenchen13", "jiemingz", "jingqiny-99", "jkamalu", "jon-barker", "jstjohn", "kajalj22", "kevalmorabia97", "ko3n1g", "ksivaman", "kunlunl", "kvareddy", "kwyss-nvidia", "layalir", "lhb8125", "liding-nv", "lmcafee-nvidia", "maanug-nv", "macandro96", "mathemakitten", "matthieule", "mchrzanowski", "mehraakash", "minitu", "mkhona-nvidia", "nanz-nv", "ntajbakhsh", "parthmannan", "philipcmonk", "prajwal1210", "pthombre", "rapatel", "rhewett-nv", "rogerwaleffe", "sajadn", "sanandaraj5597", "sancha", "santhnm2", "sbak5", "shanmugamr1992", "sharathts", "sheliang-nv", "shengf-nv", "shifangx", "shjwudp", "sidsingh-nvidia", "skyw", "sraman-rgb", "sudhakarsingh27", "tdene", "theothermike", "thomasdhc", "tomlifu", "trintamaki", "tylerpoon", "wdykas", "wplf", "wujingyue", "xiaoyao0115", "xuantengh", "xuwchen", "yaox12", "yaoyu-33", "yashaswikarnati", "yeyu-nvidia", "yobibyte", "youngeunkwon0405", "yueshen2016", "yuzhongw-nvidia", "zhongbozhu"]
Loading
Loading