Skip to content

[main] moe(feat): support chunkwise context parallelism for GDN. - #5363

Closed
yuzhongw-nvidia wants to merge 516 commits into
NVIDIA:mainfrom
yuzhongw-nvidia:yuzhongw/fla_cp
Closed

[main] moe(feat): support chunkwise context parallelism for GDN.#5363
yuzhongw-nvidia wants to merge 516 commits into
NVIDIA:mainfrom
yuzhongw-nvidia:yuzhongw/fla_cp

Conversation

@yuzhongw-nvidia

@yuzhongw-nvidia yuzhongw-nvidia commented Jun 16, 2026

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do ?

PR for dev: #3282

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact @NVIDIA/mcore-oncall.

Issue tracking

For PRs from open-source community contributors:

  • New features: a linked issue is required. Please open a feature request and reference it here before submitting the PR.
  • Small updates (bug fixes, minor improvements): a linked issue is recommended and will accelerate the PR review process.

Linked issue:

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment @NVIDIA/mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

  1. When your PR is ready, click Ready for Review.
  2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
    • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

FDecaYed and others added 30 commits January 27, 2026 14:08
@FDecaYed
Merge branch 'main' into deyuf/dev_pull_main_260122_fix_git
da56650
@ko3n1g @Phlip79
@github-actions @sidsingh-nvidia @tdene @buptzyb @jaredcasper @lmcafee-nvidia @sbhavani @chtruong814 @maanug-nv @wdykas William Dykas root root @HaochenYuan @jenchen13 @asolergi-nv @deepakn94 @tomlifu Lifu Zhang @shanmugamr1992 Shanmugam Ramasamy @santhnm2 @kanz-nv @mathemakitten @youngeunkwon0405 @nschank @ericharper @WineChord @yaox12 @ChenhanYu @lhb8125 @Wohox @jstjohn @kwyss-nvidia @ankurv-nvidia @jon-barker root root root root root root root root root root root root root root root root root root @yuzhongw-nvidia @kunlunl @shjwudp @cspades @yobibyte @cuichenx @yaoyu-33 Mcore Bot @dimapihtar @zhongbozhu @yanring @skyw @BoxiangW @mkhona-nvidia root root root @AAnoosheh @hexinw-nvidia @rhewett-nv @BestJuly @shifangx @Deepak-J0shi Hongbin Liu root @jkamalu @nvda-bnorick @duncanriach @xuwchen @parthmannan @xiaoyao0115 @jiemingz @yashaswikarnati @thomasdhc
[dev] fix git history for dev pull main 260122 (NVIDIA#3094)
08357d8 
Signed-off-by: Robin Zhang <robinz@nvidia.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Maanu Grover <maanug@nvidia.com>
Signed-off-by: Jennifer Chen <jennifchen@nvidia.com>
Signed-off-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>
Signed-off-by: Lifu Zhang <lifuz@login-lyris01.lyris.clusters.nvidia.com>
Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Signed-off-by: Pingtian Li <pingtianl@nvidia.com>
Signed-off-by: John St. John <jstjohn@nvidia.com>
Signed-off-by: John St John <jstjohn@nvidia.com>
Signed-off-by: kunlunl <kunlunl@nvidia.com>
Signed-off-by: jianbinc <shjwudp@gmail.com>
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
Signed-off-by: dimapihtar <dpihtar@gmail.com>
Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com>
Signed-off-by: Boxiang Wang <boxiangw@nvidia.com>
Signed-off-by: Deyu Fu <deyuf@nvidia.com>
Signed-off-by: Hao Wu <skyw@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Li Tao <lit@nvidia.com>
Signed-off-by: lit <lit@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@cw-dfw-cs-001-login-02.cm.cluster>
Signed-off-by: root <root@eos0321.eos.clusters.nvidia.com>
Signed-off-by: tailaim <tailaim@nvidia.com>
Signed-off-by: Parth Mannan <pmannan@nvidia.com>
Signed-off-by: Cory Ye <cye@nvidia.com>
Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com>
Signed-off-by: Jieming Zhang <jiemingz@nvidia.com>
Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com>
Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Co-authored-by: Robin Zhang <robinz@nvidia.com>
Co-authored-by: Jared Casper <155158+jaredcasper@users.noreply.github.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Lawrence McAfee <85179052+lmcafee-nvidia@users.noreply.github.com>
Co-authored-by: Santosh Bhavani <santosh.bhavani@live.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Maanu Grover <109391026+maanug-nv@users.noreply.github.com>
Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com>
Co-authored-by: wdykas <73254672+wdykas@users.noreply.github.com>
Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster>
Co-authored-by: root <root@gpu-h100-0371.cm.cluster>
Co-authored-by: root <root@gpu-h100-0159.cm.cluster>
Co-authored-by: HaochenYuan <106647990+HaochenYuan@users.noreply.github.com>
Co-authored-by: Philip Petrakian <pgpetrak@gmail.com>
Co-authored-by: Jenny Chen <jennifchen@nvidia.com>
Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>
Co-authored-by: Deepak Narayanan <2724038+deepakn94@users.noreply.github.com>
Co-authored-by: Lifu Zhang <lifuz@nvidia.com>
Co-authored-by: Lifu Zhang <lifuz@login-lyris01.lyris.clusters.nvidia.com>
Co-authored-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com>
Co-authored-by: Shanmugam Ramasamy <shanmugamr@cw-dfw-cs-001-login-01.cm.cluster>
Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com>
Co-authored-by: Kan Zhu <kanz@nvidia.com>
Co-authored-by: helen ngo <helenn@nvidia.com>
Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com>
Co-authored-by: Nick Schank <nicolas.schank@gmail.com>
Co-authored-by: Eric Harper <eharper@nvidia.com>
Co-authored-by: Nick Schank <nick@reflection.ai>
Co-authored-by: wineandchord <guoqizhou123123@qq.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Chenhan D. Yu <5185878+ChenhanYu@users.noreply.github.com>
Co-authored-by: Hongbin Liu <lhb8125@users.noreply.github.com>
Co-authored-by: Pingtian Li <pingtianl@nvidia.com>
Co-authored-by: John St. John <jstjohn@users.noreply.github.com>
Co-authored-by: kwyss-nvidia <kwyss@nvidia.com>
Co-authored-by: ankurv-nvidia <ankurv@nvidia.com>
Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com>
Co-authored-by: Jon Barker <jbarker@nvidia.com>
Co-authored-by: root <root@gpu-h100-0348.cm.cluster>
Co-authored-by: root <root@gpu-h100-0193.cm.cluster>
Co-authored-by: root <root@gpu-h100-0082.cm.cluster>
Co-authored-by: root <root@gpu-h100-0495.cm.cluster>
Co-authored-by: root <root@gpu-h100-0213.cm.cluster>
Co-authored-by: root <root@gpu-h100-0435.cm.cluster>
Co-authored-by: root <root@gpu-h100-0188.cm.cluster>
Co-authored-by: root <root@gpu-h100-0032.cm.cluster>
Co-authored-by: root <root@gpu-h100-0023.cm.cluster>
Co-authored-by: root <root@gpu-h100-0368.cm.cluster>
Co-authored-by: root <root@gpu-h100-0203.cm.cluster>
Co-authored-by: root <root@gpu-h100-0229.cm.cluster>
Co-authored-by: root <root@gpu-h100-0123.cm.cluster>
Co-authored-by: root <root@gpu-h100-0217.cm.cluster>
Co-authored-by: root <root@gpu-h100-0496.cm.cluster>
Co-authored-by: root <root@gpu-h100-0022.cm.cluster>
Co-authored-by: root <root@gpu-h100-0176.cm.cluster>
Co-authored-by: root <root@gpu-h100-0190.cm.cluster>
Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com>
Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com>
Co-authored-by: jianbinc <shjwudp@gmail.com>
Co-authored-by: Cory Ye <44509866+cspades@users.noreply.github.com>
Co-authored-by: shanmugamr1992 <shanmugamr1992@gmail.com>
Co-authored-by: yobi byte <yobibyte@users.noreply.github.com>
Co-authored-by: Chen Cui <chcui@nvidia.com>
Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com>
Co-authored-by: Mcore Bot <mcore-bot@nvidia.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: dimapihtar <dpihtar@gmail.com>
Co-authored-by: Zhongbo Zhu <42691305+zhongbozhu@users.noreply.github.com>
Co-authored-by: Zijie Yan <zijiey@nvidia.com>
Co-authored-by: Hao Wu <skyw@nvidia.com>
Co-authored-by: Boxiang Wang <boxiangw@nvidia.com>
Co-authored-by: mikail <mkhona@nvidia.com>
Co-authored-by: root <root@gpu-h100-0261.cm.cluster>
Co-authored-by: root <root@gpu-h100-0029.cm.cluster>
Co-authored-by: root <root@gpu-h100-0215.cm.cluster>
Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com>
Co-authored-by: Hexin Wang <160587990+hexinw-nvidia@users.noreply.github.com>
Co-authored-by: Russell Hewett <rhewett@nvidia.com>
Co-authored-by: Li Tao <lit@nvidia.com>
Co-authored-by: shifangx <shifangx@nvidia.com>
Co-authored-by: Deepak Joshi <151525321+Deepak-J0shi@users.noreply.github.com>
Co-authored-by: Hongbin Liu <hongbinl@cw-dfw-cs-001-login-02.cm.cluster>
Co-authored-by: root <root@eos0321.eos.clusters.nvidia.com>
Co-authored-by: John Kamalu <jkamalu@users.noreply.github.com>
Co-authored-by: Brandon Norick <bnorick@nvidia.com>
Co-authored-by: Pingtian Li <158665726+Wohox@users.noreply.github.com>
Co-authored-by: Duncan Riach <33532941+duncanriach@users.noreply.github.com>
Co-authored-by: xuwchen <79835960+xuwchen@users.noreply.github.com>
Co-authored-by: John St. John <jstjohn@nvidia.com>
Co-authored-by: Parth Mannan <38387286+parthmannan@users.noreply.github.com>
Co-authored-by: tailaim <tailaim@nvidia.com>
Co-authored-by: kunlunl <kunlunl@nvidia.com>
Co-authored-by: Jimmy Zhang <133159885+jiemingz@users.noreply.github.com>
Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com>
Co-authored-by: Dong Hyuk Chang <donghyukc@nvidia.com>
@FDecaYed @yuzhongw-nvidia @ko3n1g
[dev] fixes for pull main 260122 (NVIDIA#3103)
0f82f05 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Deyu Fu <deyuf@nvidia.com>
Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Disable broken test (NVIDIA#3121)
0ceb698 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@BestJuly @skydoorkai @kunlunl
[Dev] Param offset in _ParamAndGradBucket should be aligned (NVIDIA#3010
f6f2abe 
)

Signed-off-by: skydoorkai <htsantaclara@163.com>
Co-authored-by: skydoorkai <htsantaclara@163.com>
Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com>
@Wohox
[Dev] fix cg missing wgrad hook (NVIDIA#2999)
d587dd1
@shjwudp @yanring
[Megatron-FSDP] Add fsdp_all_gather_in_start_param_sync option in DDP…
8f8f735 
… Config (NVIDIA#2627)

Co-authored-by: Zijie Yan <zijiey@nvidia.com>
@wplf Jinliang Li
Jinliang Li @shjwudp
[Dev] Support EP with HSDP (NVIDIA#2800)
bde9e32 
Signed-off-by: jinliangl <jinliangl@nvidia.com>
Co-authored-by: Jinliang Li <jinliangl@pool0-01676.cm.cluster>
Co-authored-by: Jinliang Li <jinliangl@cw-dfw-cs-001-vscode-01.cm.cluster>
Co-authored-by: Jianbin Chang <shjwudp@gmail.com>
@ko3n1g
Cherrypick CI improvements to dev branch (NVIDIA#3118)
27fcfb2 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@FDecaYed
Merge branch 'main' into deyuf/dev_pull_main_260130
a9fb6c8
@ko3n1g
[dev] ci: Add DSv3 proxy (NVIDIA#3144)
55e3a0a 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
[dev] ci: Fix DSv3 (NVIDIA#3187)
a78ae49 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@youngeunkwon0405 @yaox12
Fix: nccl-ub in ddp path (NVIDIA#3181)
9375be4 
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
@yuzhongw-nvidia
[dev] perf(moe): Refine gated delta net implementation (NVIDIA#3040)
0f73a8a
@BestJuly
[Dev] Add the missing part to support 1F1B overlap for Qwen3-Next (NV…
5035cbe 
…IDIA#2996)
@Autumn1998
Use the latest hybrid-ep (NVIDIA#3092)
4aac3fe
@Autumn1998
[BUG FIX] Try to enable cuda graph ut (NVIDIA#3192)
bfa1d31
@shjwudp @yaox12
[Dev] Fix Linear-Cross-Entropy Convergence Issue (NVIDIA#2739)
13ad653 
Co-authored-by: Xin Yao <xiny@nvidia.com>
@chtruong814
Revert "[Dev] Fix Linear-Cross-Entropy Convergence Issue (NVIDIA#2739)…
b8b8662 
…" (NVIDIA#3218)

Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@parthmannan
Fix missing PackedSeqParams import (NVIDIA#3215)
2ab74ab
@FDecaYed
fix merge main issues
20e8ac8 
Signed-off-by: Deyu Fu <deyuf@nvidia.com>
@ko3n1g @shanmugamr1992
@JavaZeroo @yaox12 @Skylion007 @HaochenYuan @Phlip79 @jon-barker Jon Barker @thomasdhc @santhnm2 @github-actions @yaoyu-33 @shifangx Mcore Bot @deepakn94 @chtruong814 @sidsingh-nvidia @maanug-nv @hexinw-nvidia @jiemingz @liuyun7345 @shjwudp @jeffnvidia @ChenhanYu @jenchen13 @AAnoosheh @litianjian litianjian @ISEEKYAN @mathemakitten Dennis(Zhenhuan) Liu @kunlunl @parthmannan @megnvidia @faradawn @cspades @tdene @Autumn1998 @dimapihtar @kwyss-nvidia @jaredcasper @yobibyte @asolergi-nv
[dev] pull main 260130 (NVIDIA#3166)
77b5a3d 
Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Maanu Grover <maanug@nvidia.com>
Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: jenchen13 <jennifchen@nvidia.com>
Signed-off-by: Jennifer Chen <jennifchen@nvidia.com>
Signed-off-by: kunlunl <kunlunl@nvidia.com>
Signed-off-by: meg miranda <mmiranda@nvidia.com>
Signed-off-by: dimapihtar <dpihtar@gmail.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Deyu Fu <deyuf@nvidia.com>
Co-authored-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com>
Co-authored-by: JavaZero <71128095+JavaZeroo@users.noreply.github.com>
Co-authored-by: Xin Yao <yaox12@outlook.com>
Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: HaochenYuan <106647990+HaochenYuan@users.noreply.github.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
Co-authored-by: Jon Barker <jbarker@nvidia.com>
Co-authored-by: Jon Barker <jbarker@oci-hsg-cs-001-vscode-01.cm.cluster>
Co-authored-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com>
Co-authored-by: shifangx <shifangx@nvidia.com>
Co-authored-by: Mcore Bot <mcore-bot@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com>
Co-authored-by: Maanu Grover <109391026+maanug-nv@users.noreply.github.com>
Co-authored-by: Hexin Wang <160587990+hexinw-nvidia@users.noreply.github.com>
Co-authored-by: Jimmy Zhang <133159885+jiemingz@users.noreply.github.com>
Co-authored-by: liuyun7345 <51505092+liuyun7345@users.noreply.github.com>
Co-authored-by: Jianbin Chang <shjwudp@gmail.com>
Co-authored-by: jeffnvidia <152798700+jeffnvidia@users.noreply.github.com>
Co-authored-by: Chenhan D. Yu <5185878+ChenhanYu@users.noreply.github.com>
Co-authored-by: jenchen13 <jennifchen@nvidia.com>
Co-authored-by: Jenny Chen <jc4686@columbia.edu>
Co-authored-by: Asha Anoosheh <ashaanoosheh1@gmail.com>
Co-authored-by: litianjian <45817262+litianjian@users.noreply.github.com>
Co-authored-by: litianjian <litianjian@bytedance.com>
Co-authored-by: Yan Bai <baiyan1996@icloud.com>
Co-authored-by: helen ngo <helenn@nvidia.com>
Co-authored-by: Dennis(Zhenhuan) Liu <denliu@nvidia.com>
Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com>
Co-authored-by: Parth Mannan <38387286+parthmannan@users.noreply.github.com>
Co-authored-by: megnvidia <mmiranda@nvidia.com>
Co-authored-by: Philip Petrakian <pgpetrak@gmail.com>
Co-authored-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
Co-authored-by: Cory Ye <44509866+cspades@users.noreply.github.com>
Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com>
Co-authored-by: Tong Liu <tongliu@nvidia.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: kwyss-nvidia <kwyss@nvidia.com>
Co-authored-by: Jared Casper <155158+jaredcasper@users.noreply.github.com>
Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com>
Co-authored-by: yobi byte <yobibyte@users.noreply.github.com>
Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>
@ko3n1g
ci(hotfix): Pin uv (NVIDIA#3233) (NVIDIA#3234)
c5b282b 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@shjwudp
[DEV] Reapply fix Linear CE Fusion (NVIDIA#3226)
8a29fd5
@parthmannan
Missing import fix (NVIDIA#3242)
dd17acc
@Wohox
[Dev] Fix EP Overlap Bugs for Full-Iter CG (NVIDIA#3163)
fa5bcf6
@laixinn @kunlunl
[Refactor] Decouple topk and loss from DSA Indexer (NVIDIA#3013)
a592819 
Co-authored-by: kunlunl <kunlunl@nvidia.com>
Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com>
@chtruong814 @Phlip79
cp: Fix uv install for GH actions (NVIDIA#3259) (NVIDIA#3261)
54f4feb 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
@Wohox
[Dev] Fix EP Overlap missing record stream for shared expert (NVIDIA#…
ef336ca 
…3244)
@shjwudp @yaox12
Restore missing linear-cross-entropy option accidentally removed from…
ec94d63 
… arguments.py (NVIDIA#3266)

Co-authored-by: Xin Yao <xiny@nvidia.com>
ISEEKYAN and others added 15 commits June 11, 2026 07:23
@ISEEKYAN @FDecaYed
[dev] Add experimental Megatron Lite as agentic exploration (NVIDIA#4885
08518ad 
)

Co-authored-by: Deyu Fu <deyuf@nvidia.com>
@svcnvidia-nemo-ci
Merge remote-tracking branch 'origin/main' into main2dev/12_06_2026
ae955ca 
# Conflicts:
#	README.md
#	megatron/core/optimizer/optimizer.py
#	megatron/core/ssm/gated_delta_net.py
#	megatron/core/transformer/multi_token_prediction.py
#	megatron/core/transformer/transformer_config.py
#	megatron/elastification/pretrain_hybrid_flex.py
#	megatron/training/training.py
#	megatron/training/utils/common_utils.py
#	tests/test_utils/recipes/h100/t5.yaml
#	tests/test_utils/recipes/moe2.0.yaml
#	tests/unit_tests/ssm/test_gated_delta_net.py
#	tests/unit_tests/transformer/test_multi_token_prediction.py
#	uv.lock
@balasaajay @chtruong814
ci: Allow DCO check in merge queue and add DCO requirement (NVIDIA#5305)
c700379 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
@svcnvidia-nemo-ci
fix: restore dev-only training.py features dropped by the main override
b0d49c9 
The merge takes main's training.py wholesale (per the sync skill's override
list), which silently reverted four dev-only features whose supporting core
code the merge keeps at dev's version. Each surfaced as a CI failure:

1. Dynamic context-parallel API rename. Dev renamed
   get_hybrid_data_context_parallel_groups -> get_dynamic_data_context_parallel_groups
   (identical signature) and args.hybrid_context_parallel -> dynamic_context_parallel.
   Point training.py at dev's names. Fixes the conftest.py ImportError that
   cascaded to every unit-test bucket.

2. Dynamic-CP / sequence-packing data loading. Replace main's setup-time
   HybridCPDataLoaderWrapper wrap with dev's per-step wrap_data_iterator
   (gated on config.sequence_packing_scheduler), which returns a
   RerunDataIterator-compatible iterator. Fixes the *_cp4_dcp RerunDataIterator
   assertion.

3. MTP loss logging scale. Dev's MTPLossLoggingHelper stores raw loss sums and
   token counts and computes the per-token loss after reduction, so the log
   scale must be 1.0; main's 1/get_num_microbatches() divided the reported
   mtp_N loss by num_microbatches. Fixes moe gpt3_..._scoped_cudagraph mtp_1
   loss (was 16x too small: 0.682 vs golden 10.915).

4. DSA indexer loss cross-PP reduction. dsa.py requires num_layers (and
   csa_compress_ratios) so first-pipeline-stage ranks lazily initialize the
   tracker and join the cross-PP all_reduce in reduce_loss_in_tracker; main's
   call omitted them, so stage 0 returned early and the last stage hung on an
   unmatched all_reduce. Fixes the gpt3_..._dsv4_hybrid_mhc_mtp NCCL timeout.

Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@users.noreply.github.com>
@jingqiny-99 @claude
[dev]: faster implementation of mHC fused kernels (NVIDIA#4624)
9d46c92 
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Dennis(Zhenhuan) Liu Dennis Liu
[Dev] Add MoE recipe performance summary (NVIDIA#5289)
e5ad098 
Signed-off-by: Dennis Liu <denliu@denliu.nvidia.com>
Co-authored-by: Dennis Liu <denliu@denliu.nvidia.com>
@Autumn1998 Dennis(Zhenhuan) Liu
[Dev] Add DeepEP v2 flex dispatcher backend (NVIDIA#4793)
c049020 
Signed-off-by: tongliu <tongliu@nvidia.com>
Co-authored-by: Dennis(Zhenhuan) Liu <denliu@nvidia.com>
@xiaoyao0115
fix tflops calculation when sequence_packing_scheduler is not none (N…
15a4ffc 
…VIDIA#5342)

Signed-off-by: xiaoyao0115 <1804647152@qq.com>
@FDecaYed
[dev] bump emerging optimizers to v0.3.0 (NVIDIA#5320)
3d8b95c 
Signed-off-by: Deyu Fu <deyuf@nvidia.com>
@Phlip79
Merge branch 'dev' into main2dev/12_06_2026
0302fac
@Phlip79
fix: preserve seqlen stats in train_step
95654c9 
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
@guihong-nv Guihong Li
@Connor-XY @hxbai @claude
Enable Deepseek-v4 hybrid_model in dev branch Part (1/N) (NVIDIA#5042)
377b2e0 
Signed-off-by: guihong-nv <guihongl@nvidia.com>
Signed-off-by: Guihong Li <guihongl@oci-hsg-cs-001-vscode-02.cm.cluster>
Signed-off-by: Yan Xu <yxu1@nvidia.com>
Signed-off-by: Guihong Li <guihongl@nvidia.com>
Co-authored-by: Guihong Li <guihongl@oci-hsg-cs-001-vscode-02.cm.cluster>
Co-authored-by: Yan Xu <yxu1@nvidia.com>
Co-authored-by: hx <hongxiaob@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@balasaajay @megnvidia
@Phlip79 @xuantengh @claude @Wohox @github-actions @dimapihtar @sbhavani @cspades @yfw @Connor-XY @yashaswikarnati @wujingyue @lichenlu @peibli @chtruong814 @shurkat-nvidia @ko3n1g @returnL @janEbert @sraman-rgb @shanmugamr1992 @macandro96 @svcnvidia-nemo-ci
chore: nightly sync main into dev (12_06_2026) (NVIDIA#5314)
b6bea7e 
Signed-off-by: meg miranda <mmiranda@nvidia.com>
Signed-off-by: dimapihtar <dpykhtar@nvidia.com>
Signed-off-by: Cory Ye <cye@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Skand Hurkat <shurkat@nvidia.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Yan Xu <yxu1@nvidia.com>
Signed-off-by: Siddhartha Raman Sundara Raman <270218152+sraman-rgb@users.noreply.github.com>
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@users.noreply.github.com>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Co-authored-by: megnvidia <mmiranda@nvidia.com>
Co-authored-by: Philip Petrakian <pgpetrak@gmail.com>
Co-authored-by: Xuanteng Huang <44627253+xuantengh@users.noreply.github.com>
Co-authored-by: Ajay <abalasa@nvidia.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Pingtian Li <158665726+Wohox@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Santosh Bhavani <santosh.bhavani@live.com>
Co-authored-by: Cory Ye <44509866+cspades@users.noreply.github.com>
Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com>
Co-authored-by: Yan Xu <45385219+Connor-XY@users.noreply.github.com>
Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com>
Co-authored-by: Jingyue Wu <wujingyue@gmail.com>
Co-authored-by: lichenlu <lichenlu_8618@163.com>
Co-authored-by: peibli <lipeibao@126.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: shurkat-nvidia <shurkat@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Lintch <44701395+returnL@users.noreply.github.com>
Co-authored-by: janEbert <janpabloe@nvidia.com>
Co-authored-by: Siddhartha Raman Sundara Raman <sraman@nvidia.com>
Co-authored-by: Siddhartha Raman Sundara Raman <270218152+sraman-rgb@users.noreply.github.com>
Co-authored-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com>
Co-authored-by: Anish Mahishi <amahishi@nvidia.com>
Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@users.noreply.github.com>
@wplf @claude @yaox12
feat(ssm): whole-module 'gdn' selective recompute for GatedDeltaNet (N…
8dc6e66 
…VIDIA#5296)

Signed-off-by: jinliangl <975761915@qq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
@Baibaifan yangfan.bai @yaox12
[Fix] Fix optimizer parameter override bugs. (NVIDIA#5213)
232c478 
Signed-off-by: yangfan.bai <yangfan.bai@shopee.com>
Co-authored-by: yangfan.bai <yangfan.bai@shopee.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jun 16, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yuzhongw-nvidia yuzhongw-nvidia mentioned this pull request Jun 16, 2026
Merged
6 tasks
@yuzhongw-nvidia
Refactor and rename GDN CP layout handling
0565caa 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.