Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
276 commits
Select commit Hold shift + click to select a range
53a2dd5
IMA fix by making the copy of book keeping buffer to GPU blocking (#5…
shanmugamr1992 Jul 15, 2026
802da56
Add GPTModel to HybridModel migration guide (#5698)
Phlip79 Jul 16, 2026
38626c2
Pair frozen FSDP backward hooks (#5710)
wujingyue Jul 16, 2026
c2f0df5
Clarify NVIDIA email signing guidance (#5699)
wujingyue Jul 16, 2026
d981f66
Overlap FSDP communication with compute (#5719)
wujingyue Jul 16, 2026
7b2a97b
chore(skills): add Regent Open Plugin manifest (#5840)
ko3n1g Jul 16, 2026
2aa3645
chore(skills): remove Open Plugin manifest (superseded) (#5842)
ko3n1g Jul 16, 2026
edd4562
return prefix cache hits data from the chat completions api (#5609)
sidsingh-nvidia Jul 16, 2026
f8e1ac6
Refactor data parallel coordinator to enable modular handlers (#5550)
santhnm2 Jul 16, 2026
407ea4f
Pass device IDs to cleanup barrier (#5702)
wujingyue Jul 16, 2026
4abeecc
Fix averaging for MoE z-loss metric tracking (#3199)
Marks101 Jul 16, 2026
9a67ea0
test(mfsdp): AUT-881 mark test_overlaps_communication_and_compute fla…
svcnemo-autobot Jul 16, 2026
740c16e
Inference: Extend default cuda-graph coverage to 512 tokens (#5797)
sidsingh-nvidia Jul 16, 2026
e3fe550
Fix for sequence-level aux MoE loss being dependent on batch size (#5…
OlegSudakov Jul 17, 2026
61f3114
Allow parameterless FSDP root modules (#5711)
wujingyue Jul 17, 2026
27e312c
Fix issue where parameter groups with different min/max LRs get overr…
jstjohn Jul 17, 2026
acc7e64
fix bug where Gemma4 is not working with recompute_granularity = "ful…
warpuv Jul 17, 2026
4a1f743
Avoid extra MFSDP v2 model-weight sync memcpy (#5834)
wujingyue Jul 17, 2026
167d51d
[experimental] Add experimental/agent_compose placeholder with previe…
ISEEKYAN Jul 17, 2026
53492eb
Stabilize mfsdp_v2 overlap test by enlarging the model (#5846)
wujingyue Jul 17, 2026
dc57c2b
Print important dependencies (#5814)
gautham-kollu Jul 18, 2026
f258d4f
Test zero-CTA copy-engine all-gather (#5858)
wujingyue Jul 18, 2026
b4ad280
Route Lion through DistributedOptimizer and support single-moment che…
deepakn94 Jul 18, 2026
3219f38
Fix broken remove_sharded_tensors public API and re-enable its unit t…
huthvincent Jul 19, 2026
adfb8ec
Make the model larger and higher mb size to make reduce flakiness (#5…
gautham-kollu Jul 20, 2026
207be1f
chore(beep boop 🤖): Bump (main) (2026-07-20)
github-actions[bot] Jul 20, 2026
1c74286
Update active oncall to Phlip79 this week (#5896)
Phlip79 Jul 20, 2026
f41ec54
Overlap async scheduling phases (#5549)
lmcafee-nvidia Jul 20, 2026
e443bd3
ci: integrate nemo-ci-triage with linear issues management for gitlab…
balasaajay Jul 20, 2026
bd0872d
Reduce boilerplate around MultiStorageClient feature checks (#5269)
Randl Jul 20, 2026
cfb116f
Add NeMo waveform audio processor (data-side feature extractor) (#5570)
yqwangustc Jul 20, 2026
5a88c57
Support HSDP deferred DP-outer gradient reduction (#5743)
Achyuthan-S Jul 20, 2026
337c061
Add fully_shard_optimizer for mixed-precision FSDP (#5411)
wujingyue Jul 20, 2026
7012adb
Stabilize perf warmup (#5913)
Phlip79 Jul 21, 2026
12c05a2
Add compatibility between training CGs and CP>1 (#5894)
tdene Jul 21, 2026
368fa88
Log app_finish_time and app_train_loop_finish_time on early-exit path…
aliardaeker Jul 21, 2026
58bf14e
Test mFSDP v2 overlap with default and symmetric memory (#5859)
wujingyue Jul 21, 2026
ddaa315
fix: Harden Claude GitHub workflows (#5408)
chtruong814 Jul 22, 2026
a046281
chore: rotate oncall schedule
github-actions[bot] Jul 22, 2026
d207685
Reuse profiler helpers in mFSDP v2 symmetric memory tests (#5873)
wujingyue Jul 22, 2026
cc5b092
Reduce MimoOptimizer update-success across the world for cross-grid c…
yashaswikarnati Jul 22, 2026
96d4159
Refresh BERT H100 golden values (#5953)
Phlip79 Jul 22, 2026
890247c
fix(resharding): stabilize NVSHMEM refit copy service (#5915)
wdykas Jul 22, 2026
e4ea859
[Main] Numerical fix for FC2 expert bias scales when using `use_trans…
zhongbozhu Jul 22, 2026
b842f59
ci: Integrates the latest config-driven nemo-ci-triage Slack and Line…
balasaajay Jul 22, 2026
8d16c67
[Main] Numerical fix for moe single grouped weight with fp8 fp4 prima…
zhongbozhu Jul 22, 2026
602fad0
Inference: Do not let prompt tokens return from the engine, unless re…
sidsingh-nvidia Jul 22, 2026
52cf20d
Make LRU prefix caching eviction policy only evict child blocks (#5822)
santhnm2 Jul 23, 2026
411a5d8
Inference: Reduce mamba scratch space size by an order of magnitude. …
sidsingh-nvidia Jul 23, 2026
c5398de
Fix TE grouped MLP fused main-grad setup (#5209)
Wong4j Jul 23, 2026
bb5647a
fix(ci): AUT-957 support golden checks in merge queue (#5989)
svcnemo-autobot Jul 23, 2026
6cd6ea5
Fix formatting error in qwen3_30b_a3b config (#5978)
jon-barker Jul 23, 2026
51e915a
Batch-invariant train/inference logprob parity (#5897)
wdykas Jul 23, 2026
15d4b34
Move FSDP model weight sync to optimizer post-step (#5949)
wujingyue Jul 23, 2026
acd0e35
fix: allow mtp_num_layers=0 with overlap_moe_expert_parallel_comm (#5…
cuichenx Jul 23, 2026
4464d1c
Port Multi-Latent Attention to `HybridModel` (#4452)
janEbert Jul 23, 2026
4f72cc0
docs(skills): clarify container::lts is the older LTS PyTorch base (#…
ko3n1g Jul 23, 2026
ff60360
test(hybrid): AUT-971 quarantine Nemotron QAD functional test (#6013)
svcnemo-autobot Jul 24, 2026
81770cb
[main] add thd sequence packing dispatcher support for main (#5008)
HaochenYuan Jul 23, 2026
a69bbb0
Add encoder prefetch for heterogeneous MIMO training (#5833)
yashaswikarnati Jul 24, 2026
0a7f372
Use explicit process groups for dataloader checkpoints (#5988)
yashaswikarnati Jul 24, 2026
4f65717
[2/2] Add TileLang fused DSA kernels support with THD and CP & Clean …
HollowMan6 Jul 24, 2026
6f0db6f
Refit: local plan building, node-add cache key, and NIXL backend (#5872)
wdykas Jul 24, 2026
72411e5
Fix FSDP2 SwiGLU checkpointing. (#5714)
cspades Jul 24, 2026
2d754e6
Enable DDP communication overlap for MIMO training (#5979)
yashaswikarnati Jul 24, 2026
af3d240
rl: release G-submission gate slots on consumption instead of assembl…
lauradang Jul 24, 2026
648bc01
Inference: Optimized triton kernels to extract mamba states in prefix…
sidsingh-nvidia Jul 24, 2026
066145e
[Inference] Set different random seeds for each DP rank for generatio…
cspades Jul 24, 2026
dfe9d04
Ensure Mamba prefix cache snapshots are recorded for multi-chunk prom…
santhnm2 Jul 24, 2026
50cc27a
ci(actions): AUT-977 retry transient log artifact uploads (#6027)
svcnemo-autobot Jul 25, 2026
21fe0fe
fix: add additional error checks for flaky failures (#6029)
balasaajay Jul 25, 2026
7ee2580
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] Jul 25, 2026
650b783
Prevent FlashInfer sampling from running with CUDA graphs (#5791)
santhnm2 Jul 24, 2026
10c9383
fix(inference): AUT-980 disable fp8 primary weights in graph tests (#…
svcnemo-autobot Jul 25, 2026
7e6a045
chore(deps): AUT-967 stabilize Transformer Engine 2.18 upgrade (#5997)
svcnemo-autobot Jul 25, 2026
c5ff22b
[feat] Generalized Tensor Parallelism (GTP) (#4967)
fanshiqing Jul 25, 2026
cd4afff
Fix gradient-norm undercounting when using EP and TP (#5916)
philipcmonk Jul 25, 2026
7ec1bce
fix(inference): MCORE-536 report dropped prompt token lengths (#6051)
svcnemo-autobot Jul 26, 2026
543b803
[GTP][Feat] Add one-block-ahead prefetch for GTP grouped-expert weigh…
fanshiqing Jul 27, 2026
076f61f
chore(codeowners): AUT-1094 add GTP owners (#6062)
svcnemo-autobot Jul 27, 2026
10cd30c
Deprecate GPTModel in favor of HybridModel (#5911)
Phlip79 Jul 27, 2026
4b18b26
Fix CUDA graph correctness issues due to memory bugs (#5975)
jiemingz Jul 27, 2026
876016a
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] Jul 28, 2026
fc57daa
fix(dist-ckpt): AUT-1100 restore default strategy factories (#6065)
svcnemo-autobot Jul 27, 2026
07e958f
refactor: extract and split common logic between GDN & GDN2 (#5843)
xuantengh Jul 28, 2026
c4b4d59
Add load-time GPT-to-Hybrid checkpoint translation (#5675) (#5792)
guihong-nv Jul 28, 2026
866efa4
dist_ckpt: add --stream-ckpt-dequant to fix OOM on large FP8/MXFP8 lo…
asolergi-nv Jul 28, 2026
6f7bcd4
Populate dp process group in auto-built ProcessGroupCollection in pip…
ilml Jul 28, 2026
bacd340
fix(cuda-graphs): MB-928 align DDP initialization with capture stream…
svcnemo-autobot Jul 28, 2026
8a424b8
Enforce that the number of optimizer shards used in layout computatio…
deepakn94 Jul 28, 2026
4da9212
NCCL EP zero copy (#5735)
YangFei1990 Jul 28, 2026
7557c02
Optimize unit metadata for fused shared experts (#6053)
sraman-rgb Jul 28, 2026
053fa07
build: AUT-1117 serialize uv dependency installation (#6090)
svcnemo-autobot Jul 28, 2026
541d5ee
Extend dynamic inference asynchronous scheduling support (#5939)
lmcafee-nvidia Jul 28, 2026
c922805
Add support for non-Gym multi-turn environments (#5312)
tdene Jul 28, 2026
2576c15
Correct prefix-caching ref-count accounting (#6047)
tdene Jul 28, 2026
818b258
send pg group for distributed checkpoint validation (#6092)
wdykas Jul 28, 2026
2658704
fix(ci): AUT-1080 retry live NCCL watchdog timeouts (#6054)
svcnemo-autobot Jul 28, 2026
157c023
Prevent coordinator crash if an engine disconnects (#6025)
tdene Jul 28, 2026
bc67abd
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] Jul 29, 2026
8a63dd5
chore: rotate oncall schedule
github-actions[bot] Jul 29, 2026
0a447f5
Make nightly sync checks advisory (#5713)
Phlip79 Jul 29, 2026
f43ff6b
test(optimizer): MCORE-560 cover MoE gradient zero counts (#6050)
svcnemo-autobot Jul 29, 2026
b78cfd5
RL: fix logging issues from failed rollouts (#6061)
tdene Jul 29, 2026
e1b2a2f
ci: AUT-1141 skip MBridge dispatch for docs-only changes (#6116)
svcnemo-autobot Jul 29, 2026
3ff70c0
chore(ci): AUT-1135 pin GitHub Actions to commit SHAs (#6100)
svcnemo-autobot Jul 29, 2026
54045a5
chore(deps): AUT-1080 pin merged NeMo Run cancellation fix (#6110)
svcnemo-autobot Jul 29, 2026
efb3963
chore: AUT-1142 bump package versions (#6120)
svcnemo-autobot Jul 29, 2026
c7d1865
test(inference): mark prefix caching CUDA graph test flaky (#6131)
ko3n1g Jul 29, 2026
e598899
Make BertModel lm_head optional. (#5690)
bbuschkaemper Jul 29, 2026
eab6551
test(inference): MCORE-561 trust FP8 metadata in DeepSeek checkpoints…
svcnemo-autobot Jul 29, 2026
3f88509
ci: fix test notifications for weekly/release tests (#6055)
balasaajay Jul 29, 2026
8f1c99c
Allow the MInf sync API to run serve() (#6123)
tdene Jul 29, 2026
77c8772
Add disaggregated KV transfer backends (#5861)
nvcsathe Jul 29, 2026
542ff53
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] Jul 30, 2026
6183f9d
Fix gradient counting for muon+expert biases (#6099)
philipcmonk Jul 29, 2026
9436656
Reorganize a few docs (#6109)
Phlip79 Jul 30, 2026
8e57bb6
Perf: skip per-param copy_ dispatch in the MXFP8 param copy-back (#6094)
fanshiqing Jul 30, 2026
354b5d2
test: add async scheduling inference coverage (#6104)
lmcafee-nvidia Jul 30, 2026
ab16414
test(ci): AUT-1173 set functional tests to ft_launcher (#6128)
svcnemo-autobot Jul 30, 2026
ff6b92a
Fix --freeze-all-layers for new model builder path + unit tests (#5926)
AAnoosheh Jul 30, 2026
3fd8f33
Fix inference overflow into dummy blocks (#5950)
tdene Jul 30, 2026
175b4cc
fix(tests): prefer staged release assets (#6149)
ko3n1g Jul 30, 2026
ce8a8a7
Fix Megatron FSDP optimizer under-update (#5976)
wujingyue Jul 30, 2026
9d416ee
Rebind MFSDP v2 sharded grads once per gradient reduction (#6041)
wujingyue Jul 30, 2026
0081c84
Fix MLA dynamic inference decode flag (#4902)
cuichenx Jul 30, 2026
dc3c9ab
Normalize MFSDP data-parallel axes at fully_shard (#6067)
wujingyue Jul 30, 2026
ab20a60
Add Agent Compose package and runtime interface skeleton (#5937)
ISEEKYAN Jul 30, 2026
ec53920
Fix GTP full-iteration CUDA-graph capture regression (#6077)
prajwal1210 Jul 30, 2026
d04e7a0
Harden dynamic prefix cache allocation - mamba mostly (#6091)
wdykas Jul 30, 2026
6513e3e
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] Jul 31, 2026
8543ae4
ci(cache): AUT-1188 harden container cache selection (#6160)
svcnemo-autobot Jul 30, 2026
b19b1f4
ci(nvskills): AUT-1188 fix reusable workflow startup failure (#6162)
svcnemo-autobot Jul 31, 2026
d3519d5
Add MFSDP runtime schedule design (#6031)
wujingyue Jul 31, 2026
8339200
Feature/enable inference optimized qwen moe (#5700)
utkarsh530 Jul 31, 2026
57deb43
Bs invar moe (#4871)
wdykas Jul 31, 2026
4ac5ad0
chore: update pytest version to 9.1.1 in pyproject.toml and uv.lock (…
balasaajay Jul 31, 2026
ee38649
Fix LRU block accounting for resumed requests (#5995)
lmcafee-nvidia Jul 31, 2026
d378bfb
[Main] update moe single weight unit test (#6035)
zhongbozhu Jul 31, 2026
801d7e6
fix ckpt conversion issue (#6151)
dimapihtar Jul 31, 2026
07fdc08
Add determinism developer documentation (#5718)
Connor-XY Jul 31, 2026
4d129d8
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] Aug 1, 2026
313f6e9
Add MFSDP v2 MCore integration with FullyShardedOptimizer (#5865)
wujingyue Jul 31, 2026
1addfbd
Add streaming replies for dynamic engine (#5727)
nvcsathe Aug 1, 2026
2cbe71d
deps: Update urllib3 to version 2.7.0 (#6182)
balasaajay Aug 1, 2026
f2d4dfa
deps: Update black dependency to version 26.3.0 (#6180)
balasaajay Aug 1, 2026
1678db5
Add meta-parameter support (#5369)
wujingyue Aug 1, 2026
714c24a
Fix tied parameters (#6164)
wujingyue Aug 1, 2026
9829b3f
chore(ci): AUT-1231 bump preflight workflow template (#6198)
svcnemo-autobot Aug 2, 2026
42460a7
Clarify single communication stream benefits (#6177)
wujingyue Aug 3, 2026
df2da78
Bump black version in pre-commit config to match pyproject.toml (#6201)
xuantengh Aug 3, 2026
dac95f3
set weights_only=True (#6150)
dimapihtar Aug 3, 2026
c2fd827
[fix] Give same-key GTP chain neighbours distinct gather buffers (#6207)
fanshiqing Aug 3, 2026
abf04f4
Fix gradient reduction issue when EP=1, EP=TP, and EGTP != GTP (#6080)
philipcmonk Aug 3, 2026
b4587a3
Apply black 26.3.0 formatting to four files it now reformats (#6210)
deepakn94 Aug 3, 2026
708f0a5
Add context-parallel causal conv1d support (#5917)
Mellonta Aug 3, 2026
43b1fc8
Refactor TE fused ops integration into mixin (#4630)
CarlosGomes98 Aug 3, 2026
23b2ff2
[feat] Add reduce-scatter-with-fp32-accumulation support for GTP (#6200)
fanshiqing Aug 4, 2026
5adbee2
chore(ci): AUT-1298 bump community workflow to v1.8.7 (#6219)
svcnemo-autobot Aug 4, 2026
0896be0
fix: pretrain_hybrid.py --help (#6211)
paulwambergue Aug 4, 2026
22dfc28
refactor(ci): migrate triage consumer to Cerno (#6250)
thomasdhc Aug 4, 2026
46fe6d2
Validate NIXL CUDA transports and share transfer agents (#6221)
wdykas Aug 4, 2026
3fe1519
Add prefix-cache lifecycle hooks (#5860)
nvcsathe Aug 4, 2026
9a3c40b
Make Bridge Communicator aware of GTP (#6263)
yashaswikarnati Aug 5, 2026
9fa816b
Fix GTP reduce-scatter overlap across local CUDA graphs (#6060)
JF-D Aug 5, 2026
6699509
revert(dist_checkpointing): AUT-1325 remove streaming checkpoint dequ…
svcnemo-autobot Aug 5, 2026
7a6f683
chore: rotate oncall schedule
github-actions[bot] Aug 5, 2026
bf71c14
avoid circular dependency usage between mcore and modelopt- #5644 (#6…
dimapihtar Aug 5, 2026
aa4624e
Update DEEPEP_COMMIT in Dockerfile.ci.dev to commit de0dd1… (#6261)
balasaajay Aug 5, 2026
b8ff565
revert: revert community-bot bump to v1.8.7 (5adbee2) (#6288)
kajalj22 Aug 5, 2026
9f3fe8c
[GTP] Cleanup usage of pg_collection in gtp (#6234)
fanshiqing Aug 5, 2026
3aee84c
[Copy to main] Update Nemotron 3 Super GB200 release config (#6282)
svcnvidia-nemo-ci Aug 5, 2026
8aec64d
Revert "Add Agent Compose package and runtime interface skeleton" (#6…
Phlip79 Aug 5, 2026
19c50e8
Release MFSDP storage after model deletion (#6230)
wujingyue Aug 5, 2026
d180b7c
Document MFSDP optimizer design (#6186)
wujingyue Aug 5, 2026
ef3ce11
Handle checkpoint and text-only batches in MIMO encoder prefetch (#6039)
yashaswikarnati Aug 5, 2026
a54bd8c
Revert "avoid circular dependency usage between mcore and modelopt- #…
Phlip79 Aug 6, 2026
3d3323f
Fix multimodal MTP loss masking for modality tokens (#6010)
kevjshih Aug 6, 2026
59b72fa
Replace lazy MFSDP context initialization with an explicit scope (#6190)
wujingyue Aug 6, 2026
d62ff04
chore(ci): bump community workflow to v1.8.8 (#6322)
ko3n1g Aug 7, 2026
07a0de6
fix(moe): free activations immediately in full recompute mode (#5197)
liuyun7345 Aug 7, 2026
fa0758c
Allocate main_grad on the reduce-scatter stream (#6187)
wujingyue Aug 7, 2026
cde3b5c
Encapsulate symmetric-memory AVG in DBuffer (#6169)
wujingyue Aug 7, 2026
c4968aa
fix(ci): remove community workflow PAT (#6350)
ko3n1g Aug 7, 2026
628c2a5
Add multimodal argument and config plumbing (#6332)
matthieule Aug 7, 2026
07a2319
Fix FSDP activation recompute prefetch (#6153)
shjwudp Aug 7, 2026
d2715d7
Add Nemotron 3.5 Lightning GB200 nightly test (#6312)
Phlip79 Aug 7, 2026
a6fb162
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] Aug 8, 2026
e04ed19
Exit non-zero when --exit-on-missing-checkpoint triggers (#6321)
shanmugamr1992 Aug 8, 2026
f8ec15e
Divide expert gradients by the expert-parallel size (#5956)
wujingyue Aug 8, 2026
79e6b6a
[feat] GTP+MTP (#6242)
fanshiqing Aug 8, 2026
67700f7
fix(mtp): scatter the MTP embedding when the model's embedding does n…
DanialTaheri Aug 8, 2026
0ced206
Apply the layerwise param layout on the ModelBuilder DDP path (#6286)
deepakn94 Aug 8, 2026
14499ef
test: AUT-1363 prevent recipe fetch-token exposure (#6364)
svcnemo-autobot Aug 8, 2026
ccf686e
fix(ci): AUT-1396 use one global external queue worker (#6366)
svcnemo-autobot Aug 8, 2026
d12f6c8
Add generic interface for SSM inference (#5382)
santhnm2 Aug 8, 2026
6518b75
Add PyTorch DCP checkpoint save/load for MFSDP v2 (#6024)
ahmadki Aug 9, 2026
3656752
chore(beep boop 🤖): Bump (main) (2026-08-10)
github-actions[bot] Aug 10, 2026
24bad8e
Skip save-side sharding validation via --no-ckpt-load-validate-shardi…
asolergi-nv Aug 10, 2026
fb24e87
feat(gtp): opt-in GTP_remat sharding support for 'moe-latent-proj' (#…
fanshiqing Aug 10, 2026
db1ea81
Support RL eval-only runs (#6043)
tdene Aug 10, 2026
894928a
fix(ci): redact sensitive environment values from test output (#6406)
ko3n1g Aug 10, 2026
fa83195
Add --muon-use-syrk option for Triton SYRK kernel in Newton-Schulz (#…
deepakn94 Aug 10, 2026
7d08fca
Add 1-node GB200 functional test for checkpoint load with load_optim=…
asolergi-nv Aug 10, 2026
86e928a
Selective-FP32 runtime and native-FP32 optimizer checkpoint lifecycle…
FDecaYed Aug 10, 2026
0c1fc4c
Bump FLA to 0.5.1 and integrate GDN2 kernels (#5765)
xuantengh Aug 11, 2026
67a355e
Update docstring for moe_router_topk_scaling_factor (#2178)
JavaZeroo Aug 11, 2026
105445d
Add --ckpt-drop-redundant-extra-state to skip persisting redundant TE…
asolergi-nv Aug 10, 2026
78901d8
Add construction-scoped MFSDP v2 fully shard options (#6127)
wujingyue Aug 11, 2026
5e86dfa
Support explicit Hugging Face finetune data files in finetune.sh (#6413)
jenchen13 Aug 11, 2026
d7d7a6a
Add megatron.core.telemetry base layer and observability guide (#6415)
deepakn94 Aug 11, 2026
8971d19
[NCCL EP] Support eager mode and overflow handling (#6229)
YangFei1990 Aug 11, 2026
db57c15
Gate expensive work behind log levels (#5805)
shurkat-nvidia Aug 11, 2026
8018459
Fixes for SP + MTP all-gathers (#5748)
santhnm2 Aug 11, 2026
7a76dff
TE Layernorm dtype guard with fp32 residuals (#6272)
mkhona-nvidia Aug 11, 2026
e0ccd26
Treat LayerWise bucket_size as a soft minimum and fill would-be paddi…
deepakn94 Aug 11, 2026
8e70f69
Gated Delta Product (GDP) implementation (#6074)
devnkong Aug 11, 2026
1d402ad
fix(mfsdp): refresh V2 compute weights after optimizer step (#6336)
wujingyue Aug 11, 2026
c3d44dc
[core] Clarify GPT output_processor callback contract (#5739)
chengcuiping Aug 11, 2026
03318a8
Validate MFSDP module phase transitions (#6359)
wujingyue Aug 11, 2026
80cda5d
Refine MFSDP v2 configuration validation (#6333)
wujingyue Aug 11, 2026
f3eb0df
Add hybrid layer config definitions (#6410)
Phlip79 Aug 11, 2026
da18ed6
Propagate RL runtime config updates to module configs (#6423)
Phlip79 Aug 11, 2026
e22bcb0
[fix] GTP+recompute: keep adjacent GTP recompute weights off the same…
fanshiqing Aug 11, 2026
a0394fe
Update copy-pr-bot.yaml [skip ci]
github-actions[bot] Aug 12, 2026
49a49a9
Add MFSDP v2 design document (#6431)
wujingyue Aug 11, 2026
8af9b04
Make core-nemo fallback owner for top-level files (#6358)
Phlip79 Aug 12, 2026
725949a
ci(auth): treat svcnemo-autobot as internal (#6436)
ko3n1g Aug 12, 2026
897649d
Add mixed-precision (FP32) LM output logits (#6252)
mkhona-nvidia Aug 12, 2026
14346b6
Eliminate the ShardedObject all_gather_object on the FullyParallel lo…
asolergi-nv Aug 12, 2026
15c83d2
chore: rotate oncall schedule
github-actions[bot] Aug 12, 2026
81fe7c7
GTP + PartialCG: Classify latent projections for partial CUDA graphs …
JF-D Aug 12, 2026
8534490
Balance LayerWise optimizer shards by Newton-Schulz cost, not paramet…
deepakn94 Aug 12, 2026
fad6111
Suppress noisy external-library log lines on non-rank-0 processes (#5…
deepakn94 Aug 12, 2026
b2f2888
feat(ci): enhance triage script to assign on-call assignee dynamicall…
balasaajay Aug 12, 2026
5436289
Skip attention mask materialization in RL training (#5885)
tdene Aug 13, 2026
a34880f
Avoid unnecessary MoE router host synchronization (#6432)
JF-D Aug 13, 2026
3cd8cf3
Refactor GDP dynamic inference onto the shared SSM interface (#6443)
santhnm2 Aug 13, 2026
3ce4b4a
test(fsdp): mark default overlap check flaky (#6493)
wujingyue Aug 13, 2026
301e0d6
Paged stashing no longer assumes single layer config (#6419)
Phlip79 Aug 13, 2026
5714a20
Respect `profile_ranks` on log interval memory snapshots (#6354)
maanug-nv Aug 13, 2026
fecb29f
Add unit tests for GDP inference (#6505)
santhnm2 Aug 13, 2026
8ba5c60
[fix] fix GTP+DCP ckpt saving/loading for GDP module (#6503)
fanshiqing Aug 13, 2026
281075d
feat(fsdp): support expert parallelism in MFSDP v2 adapter (#6450)
wujingyue Aug 13, 2026
2a75ac1
Complete clamped SwiGLU across expert paths (#5940)
FDecaYed Aug 13, 2026
87bf520
RL: unify rollout generation into one pipeline (#6495)
tdene Aug 13, 2026
487bb39
Avoid passing -1 sentinels into inference kernels (#6481)
tdene Aug 13, 2026
09921c0
RL: bound per-env wandb keys to the char limit (#6477)
tdene Aug 13, 2026
579e0ea
Merge remote-tracking branch 'origin/main' into main2dev/13_08_2026
github-actions[bot] Aug 13, 2026
6d20ae6
fix: post-CI corrections
github-actions[bot] Aug 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
10 changes: 5 additions & 5 deletions .github/actions/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ runs:
run: echo "node_name=$NODE_NAME" | tee -a "$GITHUB_OUTPUT"

- name: Checkout repository
uses: actions/checkout@v6
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
ref: ${{ inputs.sha }}

Expand All @@ -94,7 +94,7 @@ runs:
sudo chown -R $(whoami) /home/runner/ 2>/dev/null || true

- name: Setup python
uses: actions/setup-python@v5
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: '3.12'

Expand Down Expand Up @@ -296,7 +296,7 @@ runs:
fi

- name: Upload coverage
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
if: ${{ always() && steps.check.outputs.coverage_report != 'none' }}
with:
name: ${{ steps.check.outputs.coverage_report }}
Expand All @@ -307,7 +307,7 @@ runs:

- name: Upload logs
id: upload-logs
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
if: always()
continue-on-error: true
with:
Expand All @@ -321,7 +321,7 @@ runs:
run: sleep 10

- name: Retry log upload
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
if: ${{ always() && steps.upload-logs.outcome == 'failure' }}
with:
name: ${{ steps.check.outputs.logs_report }}-retry
Expand Down
2 changes: 1 addition & 1 deletion .github/copy-pr-bot.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
enabled: true
auto_sync_draft: false
auto_sync_ready: true
trustees_override: ["AAnoosheh", "ArEsKay3", "Autumn1998", "BestJuly", "BoxiangW", "CarlosGomes98", "ChenhanYu", "Connor-XY", "FDecaYed", "HaochenYuan", "HollowMan6", "ISEEKYAN", "JRD971000", "Leili", "Mellonta", "Phlip79", "QiZhangNV", "RPrenger", "ShriyaRishab", "Victarry", "WanZzzzzz", "Wohox", "YangFei1990", "ZhiyuLi-Nvidia", "adistomar", "ahmadki", "aklife97", "alokpathy", "ananthsub", "anlthms", "aroshanghias-nvd", "ashehper", "asolergi-nv", "athitten", "balasaajay", "buptzyb", "chtruong814", "cjld", "cspades", "cuichenx", "deepakn94", "desh2608", "dimapihtar", "dingqingy-nv", "duncanriach", "erhoo82", "ericharper", "fanshiqing", "faradawn", "fitsumreda", "freewym", "frsun-nvda", "gautham-kollu", "gdengk", "goelarushi", "guihong-nv", "guyueh1", "hexinw-nvidia", "huvunvidia", "hxbai", "ilml", "jalbericiola", "janEbert", "jaredcasper", "jenchen13", "jiaji-huang", "jiemingz", "jingqiny-99", "jkamalu", "jon-barker", "jstjohn", "kajalj22", "kamran-nvidia", "kevalmorabia97", "kingformatty", "ko3n1g", "ksivaman", "kunlunl", "kvareddy", "kwyss-nvidia", "lauradang", "layalir", "lhb8125", "liding-nv", "lmcafee-nvidia", "maanug-nv", "macandro96", "mathemakitten", "matthieule", "mchrzanowski", "mehraakash", "minitu", "mkhona-nvidia", "nanz-nv", "ntajbakhsh", "parthmannan", "philipcmonk", "prajwal1210", "pthombre", "rapatel", "rhewett-nv", "rogerwaleffe", "sajadn", "sanandaraj5597", "sancha", "santhnm2", "sbak5", "shanmugamr1992", "sharathts", "sheliang-nv", "shengf-nv", "shifangx", "shjwudp", "sidsingh-nvidia", "skyw", "sraman-rgb", "sudhakarsingh27", "svcnemo-autobot", "tdene", "theothermike", "thomasdhc", "tomlifu", "trintamaki", "tylerpoon", "wdykas", "wplf", "wujingyue", "xiaoyao0115", "xuantengh", "xuwchen", "yaox12", "yaoyu-33", "yashaswikarnati", "yeyu-nvidia", "yobibyte", "youngeunkwon0405", "yqwangustc", "yueshen2016", "yuzhongw-nvidia", "zhehuaichen", "zhongbozhu"]
trustees_override: ["AAnoosheh", "ArEsKay3", "Autumn1998", "BestJuly", "BoxiangW", "CarlosGomes98", "ChenhanYu", "Connor-XY", "DanialTaheri", "FDecaYed", "HaochenYuan", "HollowMan6", "ISEEKYAN", "JF-D", "JRD971000", "Leili", "Mellonta", "Phlip79", "QiZhangNV", "RPrenger", "ShriyaRishab", "Victarry", "WanZzzzzz", "Wohox", "YangFei1990", "ZhiyuLi-Nvidia", "adistomar", "ahmadki", "aklife97", "alokpathy", "ananthsub", "anlthms", "aroshanghias-nvd", "ashehper", "asolergi-nv", "athitten", "balasaajay", "buptzyb", "chtruong814", "cjld", "cspades", "cuichenx", "deepakn94", "desh2608", "dimapihtar", "dingqingy-nv", "duncanriach", "ehosseiniasl", "erhoo82", "ericharper", "fanshiqing", "faradawn", "fitsumreda", "freewym", "frsun-nvda", "gautham-kollu", "gdengk", "goelarushi", "guihong-nv", "guyueh1", "hexinw-nvidia", "huvunvidia", "hxbai", "ilml", "jalbericiola", "janEbert", "jaredcasper", "jenchen13", "jiaji-huang", "jiemingz", "jingqiny-99", "jkamalu", "jon-barker", "jstjohn", "kajalj22", "kamran-nvidia", "kevalmorabia97", "kevjshih", "kingformatty", "ko3n1g", "ksivaman", "kunlunl", "kvareddy", "kwyss-nvidia", "lauradang", "layalir", "lhb8125", "liding-nv", "lmcafee-nvidia", "maanug-nv", "macandro96", "mathemakitten", "matthieule", "mchrzanowski", "mehraakash", "minitu", "mkhona-nvidia", "nanz-nv", "niyunsheng", "ntajbakhsh", "nvcsathe", "parthmannan", "philipcmonk", "prajwal1210", "pthombre", "rapatel", "rhewett-nv", "rogerwaleffe", "sajadn", "sanandaraj5597", "sancha", "santhnm2", "sbak5", "shanmugamr1992", "sharathts", "sheliang-nv", "shengf-nv", "shifangx", "shjwudp", "sidsingh-nvidia", "skyw", "sraman-rgb", "sudhakarsingh27", "svcnemo-autobot", "svcnvidia-nemo-ci", "tdene", "theothermike", "thomasdhc", "tomlifu", "trintamaki", "tylerpoon", "vasunvidia", "wdykas", "wplf", "wujingyue", "xiaoyao0115", "xuantengh", "xuwchen", "yaox12", "yaoyu-33", "yashaswikarnati", "yeyu-nvidia", "yobibyte", "youngeunkwon0405", "yqwangustc", "yueshen2016", "yuzhongw-nvidia", "zhehuaichen", "zhongbozhu"]
40 changes: 20 additions & 20 deletions .github/oncall_schedule.json
Original file line number Diff line number Diff line change
@@ -1,24 +1,4 @@
[
{
"user": "cspades",
"date": "2026-07-08"
},
{
"user": "dimapihtar",
"date": "2026-07-15"
},
{
"user": "guihong-nv",
"date": "2026-07-22"
},
{
"user": "ilml",
"date": "2026-07-29"
},
{
"user": "janEbert",
"date": "2026-08-05"
},
{
"user": "maanug-nv",
"date": "2026-08-12"
Expand Down Expand Up @@ -46,5 +26,25 @@
{
"user": "cspades",
"date": "2026-09-23"
},
{
"user": "dimapihtar",
"date": "2026-09-30"
},
{
"user": "guihong-nv",
"date": "2026-10-07"
},
{
"user": "ilml",
"date": "2026-10-14"
},
{
"user": "janEbert",
"date": "2026-10-21"
},
{
"user": "maanug-nv",
"date": "2026-10-28"
}
]
56 changes: 56 additions & 0 deletions .github/scripts/test_cache_keys.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/usr/bin/env bash
set -euo pipefail

compute_keys() {
local base_ref=$1
local cache_variant=$2
local pr_number=$3
local merge_group_head_ref=$4
local github_ref=$5
local event_name=$6
local ref_name=$7

BASE_REF="${base_ref#refs/heads/}"
CACHE_VARIANT=$cache_variant
CACHE_NAMESPACE=$(printf '%s' "$BASE_REF" | tr '/:@' '-' | tr -cd '[:alnum:]_.-')
if [ -z "$CACHE_NAMESPACE" ]; then
return 1
fi

PR_NUMBER=$pr_number
if [ "$PR_NUMBER" = "0" ] && [ -n "$merge_group_head_ref" ]; then
PR_NUMBER=$(printf '%s' "$merge_group_head_ref" | sed -nE 's#.*pr-([0-9]+)-.*#\1#p')
fi

BASELINE_KEY="${CACHE_NAMESPACE}-${CACHE_VARIANT}-baseline"
if [ -n "$PR_NUMBER" ] && [ "$PR_NUMBER" != "0" ]; then
KEY="${CACHE_NAMESPACE}-${CACHE_VARIANT}-${PR_NUMBER}"
elif [ "$github_ref" = "refs/heads/$BASE_REF" ] || [ "$event_name" = "schedule" ]; then
KEY="$BASELINE_KEY"
else
BRANCH_NAMESPACE=$(printf '%s' "$ref_name" | tr '/:@' '-' | tr -cd '[:alnum:]_.-')
KEY="${CACHE_NAMESPACE}-${CACHE_VARIANT}-${BRANCH_NAMESPACE}"
fi

if [ "${#KEY}" -gt 100 ]; then
KEY="${KEY:0:83}-$(printf '%s' "$KEY" | sha256sum | cut -c1-16)"
fi
if [ "${#BASELINE_KEY}" -gt 100 ]; then
BASELINE_KEY="${BASELINE_KEY:0:83}-$(printf '%s' "$BASELINE_KEY" | sha256sum | cut -c1-16)"
fi
}

assert_keys() {
local expected_key=$1
local expected_baseline=$2
shift 2
compute_keys "$@"
test "$KEY" = "$expected_key"
test "$BASELINE_KEY" = "$expected_baseline"
}

assert_keys main-dev-6077 main-dev-baseline refs/heads/main dev 0 pr-6077-ff6b refs/heads/gh-readonly-queue/main/pr-6077 merge_group gh-readonly-queue/main/pr-6077
assert_keys dev-dev-6072 dev-dev-baseline refs/heads/dev dev 0 pr-6072-95e4 refs/heads/gh-readonly-queue/dev/pr-6072 merge_group gh-readonly-queue/dev/pr-6072
assert_keys main-lts-6159 main-lts-baseline main lts 6159 '' refs/heads/pull-request/6159 push pull-request/6159
assert_keys main-dev-baseline main-dev-baseline main dev 0 '' refs/heads/main schedule main
assert_keys main-dev-deploy-release-1.2 main-dev-baseline main dev 0 '' refs/heads/deploy-release/1.2 push deploy-release/1.2
6 changes: 3 additions & 3 deletions .github/workflows/_build_test_publish_wheel.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ jobs:
PUBLISH_DRYRUN: ${{ inputs.dry-run }}
steps:
- name: Checkout repository
uses: actions/checkout@v6
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
ref: ${{ inputs.ref }}

Expand Down Expand Up @@ -141,7 +141,7 @@ jobs:
"

- name: Upload wheels
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: wheels-${{ matrix.PACKAGE }}-${{ matrix.PLATFORM }}-${{ inputs.dry-run && 'dry-run' || 'release' }}
path: dist/
Expand All @@ -165,7 +165,7 @@ jobs:
PACKAGE: ${{ matrix.PACKAGE }}
steps:
- name: Download wheels
uses: actions/download-artifact@v7
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: wheels-${{ matrix.PACKAGE }}-${{ matrix.PLATFORM }}-${{ inputs.dry-run && 'dry-run' || 'release' }}
path: dist/
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/_update_dependencies.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
TARGET_BRANCH: ${{ inputs.target-branch }}
steps:
- name: Checkout repo
uses: actions/checkout@v6
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
ref: ${{ env.TARGET_BRANCH }}

Expand All @@ -60,7 +60,7 @@ jobs:
fi

- name: Checkout repo
uses: actions/checkout@v6
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
ref: ${{ env.SOURCE_BRANCH }}

Expand All @@ -77,7 +77,7 @@ jobs:
bash -c 'uv lock --upgrade'

- name: Upload lock file
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: lock-file-${{ env.SOURCE_BRANCH }}
path: uv.lock
Expand All @@ -90,7 +90,7 @@ jobs:
TARGET_BRANCH: ${{ inputs.target-branch }}
steps:
- name: Checkout code
uses: actions/checkout@v6
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
token: ${{ secrets.PAT }}
ref: ${{ env.TARGET_BRANCH }}
Expand All @@ -103,12 +103,12 @@ jobs:
fi

- name: Download lock file
uses: actions/download-artifact@v7
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: lock-file-${{ env.SOURCE_BRANCH }}

- name: Create Bump PR
uses: peter-evans/create-pull-request@v8
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8
id: create-pull-request
env:
title: "chore(beep boop 🤖): Bump `uv.lock` (${{ inputs.target-branch}}) (${{ needs.pre-flight.outputs.date }})"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/auto-assign-milestone.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
- name: Get PR info
id: get-pr-info
if: startsWith(github.ref, 'refs/heads/pull-request/')
uses: nv-gha-runners/get-pr-info@main
uses: nv-gha-runners/get-pr-info@090577647b8ddc4e06e809e264f7881650ecdccf # main

- name: Check if PR has milestone
id: check_milestone
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/auto-reminder-bot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ jobs:
if: github.repository == 'NVIDIA/Megatron-LM'
steps:
- name: Check out repository code
uses: actions/checkout@v6
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6

- name: Set up Python
uses: actions/setup-python@v6
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.10"

Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/auto-swap-labels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
id: get-pr
if: github.event_name == 'workflow_run'
continue-on-error: true
uses: actions/download-artifact@v7
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: pr-number
path: pr-number
Expand All @@ -54,11 +54,11 @@ jobs:

- name: Check out repository code
if: steps.pr.outputs.number
uses: actions/checkout@v6
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6

- name: Set up Python
if: steps.pr.outputs.number
uses: actions/setup-python@v6
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.10"

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/auto-update-copy-pr-bot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
if: github.repository == 'NVIDIA/Megatron-LM'
steps:
- name: Checkout code
uses: actions/checkout@v6
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
token: ${{ secrets.PAT }}
ref: main
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/cherry-pick-release-commit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ on:

jobs:
cherry-pick:
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_cherry_pick.yml@v0.65.9
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_cherry_pick.yml@0cb71cd98aa47ba338d8e38514387d5ceecfedff # v0.65.9
if: github.repository == 'NVIDIA/Megatron-LM'
with:
target-branches-pattern: 'core_(*dev_)?r[0-9]+\.[0-9]+\.[0-9]+'
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/cicd-approve-test-queue.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,10 @@ jobs:
contributor_type: external
steps:
- name: Checkout repository
uses: actions/checkout@v6
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6

- name: Set up Python
uses: actions/setup-python@v6
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.12"

Expand Down
Loading
Loading