[Community][Main] feat(moe): Adding context parallel support to eager attention implementation - #2295

Draft

yuzhongw-nvidia wants to merge 154 commits into

NVIDIA:mainfrom

yuzhongw-nvidia:eagercp

Commits on Sep 24, 2025

chore: Add codeowners file
ko3n1g
committed
5c2ae01
ADLR/megatron-lm!4065 - ci: Add main/dev branching to queuemanager
ko3n1g
committed
454e7b5

Commits on Sep 29, 2025

ADLR/megatron-lm!4090 - cp: `!4084 - ci: Send dev alerts to separate channel`
ko3n1g
committed
c0188dc

Commits on Sep 30, 2025

ci(hotfix): Nightly runs
ko3n1g
committed
4808e33

Commits on Oct 3, 2025

Commits on Oct 9, 2025

Merge remote-tracking branch 'origin/main' into dev_pull_main_Oct_09
yanring
committed
372fc9f

Commits on Oct 10, 2025

ci: Re-add safe-imports
ko3n1g
committed
f9bb58c
ci: No legacy for unit test extended
ko3n1g
committed
2a6ca17
ci: Reduce number of repeats
ko3n1g
committed
54825ab
ci: Fix notification channel
ko3n1g
committed
15819b6
ADLR/megatron-lm!4106 - [DEV] Add muon and layer-wise distributed optimizer

committed
879a7a1

Commits on Oct 11, 2025

ADLR/megatron-lm!4060 - Update dev branch README

sbhavani
authored and
yanring
committed
4bdc4e2

Commits on Oct 12, 2025

ADLR/megatron-lm!4223 - Ko3n1g/cp/4213 to dev

ko3n1g
and
Mcore Bot
committed
eff3f6a

Commits on Oct 13, 2025

ADLR/megatron-lm!4070 - [DEV] Support Qwen3next

yuzhongw-nvidia
authored and
yanring
committed
c7590d8

Commits on Oct 14, 2025

!4236 - [Dev] Formatting dev branch code to avoid linting pipeline failures
ko3n1g
committed
c7dee4b
!4211 - ci(fix): Cherrypicking from forks
ko3n1g
committed
4c3a1be
!4239 - ci: Check out dev for formatting
ko3n1g
committed
7c350f5
ci: Fix formatting
ko3n1g
committed
46687cd
ci: Fix linting
ko3n1g
committed
50ed5eb
ADLR/megatron-lm!4225 - [Dev][NVFP4][MOE] Proper NVFP4 Zero Padding for MOE

authored and
yanring
committed
b01ad5b

Commits on Oct 15, 2025

ADLR/megatron-lm!4248 - ADLR/megatron-lm!4159 - Fix ProcessGroupCollection missing initialization

skyw
authored and
yanring
committed
061bc37
ADLR/megatron-lm!4207 - Refactor dev functional tests.

Dennis Liu
authored and
yanring
committed
b007b91
ci: No batch short anymore
ko3n1g
committed
32e9518
fallback to eager attn config
nrailg
committed
9568a15
adding cp support to eager attn
nrailg
committed
b4c7979
ADLR/megatron-lm!4231 - [Dev] fix(dataset): fix the divergence when using dsv3 tokenizer after !3646; Have datasets account for tokenizers which incorrectly define PAD;

authored and
yanring
committed
bbc762d

Commits on Oct 16, 2025

refine the code

yuzhongw-nvidia
authored and
nrailg
committed
ac0fc23
Refine dot_product_attention_context_parallel.py

yuzhongw-nvidia
authored and
nrailg
committed
370aa4a
ADLR/megatron-lm!4254 - [Dev] Fix dev nightly functional tests.

Dennis Liu
authored and
ko3n1g
committed
df41a69

Commits on Oct 17, 2025

ADLR/megatron-lm!4232 - [DEV] improve muon and layer-wise dist opt unit tests

FDecaYed
authored and
yanring
committed
bf1a503

Commits on Oct 20, 2025

ADLR/megatron-lm!4296 - [DEV] fix(MoE): Fix parameter initialization

cuichenx
authored and
ko3n1g
committed
6802bec

Commits on Oct 21, 2025

[Dev] Fix attention output gate for TE2.8
yanring
committed
a6ca591
Cleanup UT and toml
yanring
committed
7843324
Clean up functional test
yanring
committed
a48a416
Merge remote-tracking branch 'origin/main' into zijiey/pull-main-Oct-20
yanring
committed
1a6a98e
build: Fix jet
ko3n1g
committed
12839ed
ADLR/megatron-lm!4312 - [dev] Set tensor-parallel attributes irrespective of perform_initialization

authored and
yanring
committed
a8bad4b

Commits on Oct 22, 2025

ADLR/megatron-lm!4237 - [Dev] perf(MoE): Add the Hybrid-EP backend to the Flex Dispatcher

Autumn1998
authored and
yanring
committed
d9153a5
ADLR/megatron-lm!4235 - [dev] Support multimodule communication

authored and
yanring
committed
2e56506

Commits on Oct 23, 2025

Merge branch 'dev' into 'zijiey/pull-main-Oct-20'
yanring
committed
1c8bf21
chore: Update golden values.
yanring
committed
97ef777
cp: !4298 - ci: Refactor testsytem - Removal of JET Artifacts
ko3n1g
committed
3bf9b87
ci: Add copyright-checker for GitHub
ko3n1g
committed
265f4ee
ADLR/megatron-lm!4298 - ci: Refactor testsytem - Removal of JET Artifacts
ko3n1g
committed
bec6582
ci: Fix copyright checker (#1889) (#1890)
ko3n1g
committed
855aa49
ci: Run on dev
ko3n1g
committed
6068fa9
ci: Bump copyright header (#1894)
ko3n1g
committed
8856794
ci: Allow runs on dev branch
ko3n1g
committed
beceec6
ci: Linting on push
ko3n1g
committed
13b6a36
ci: Run linting only on PR
ko3n1g
committed
14a0a23
ci(fix): HAS_RUN_TESTS_LABEL
ko3n1g
committed
8e03549
ci: Fix linting
ko3n1g
committed
da84298
ci: Add codeowners to dev branch (#1898)
ko3n1g
authored
38166a6
ci(fix): dynamic inference tests
ko3n1g
committed
620826b
ci(fix): No copyright on push
ko3n1g
committed
829ae2f

Commits on Oct 24, 2025

ci: Move test optimizer into its own bucket (#1909)
ko3n1g
committed
f737697
ci: Update container image tags to use github.sha
ko3n1g
authored
176a2ed
Ko3n1g/chore/merge main into dev (#1903)

authored
d3d2048
ci: Fix approval bot
ko3n1g
committed
1ef95d9
ci: Fix dev branch CI (#1922)

chtruong814
and
ko3n1g
authored
9b8d703
Ko3n1g/ci/cherrypick automation dev (#1926)
ko3n1g
authored
10d280a
ci: Fix dev
ko3n1g
committed
017c7b3

Commits on Oct 25, 2025

Ko3n1g/chore/merge main into dev20251025 (#1943)
ko3n1g
authored
598d41f
ci: Fix branch of approval bot (#1945)
ko3n1g
authored
4fc8520

Commits on Oct 26, 2025

ci(fix): Approval gate
ko3n1g
committed
574a009
ci: Approval gate
ko3n1g
committed
8243834
ci: Approval gate rule
ko3n1g
committed
106516c
ci: Update golden values dev
ko3n1g
committed
ef48a13
ci: Approval gate
ko3n1g
committed
3281c01
ci: Approval bot
ko3n1g
committed
8fe0c3a
ci: Increase time limit for main tests
ko3n1g
committed
a33936d
ci: Auto-assign milestone (#1952)
ko3n1g
authored
5176823
ci: Run on push to release branch (#1960) (#1962)
ko3n1g
authored
4b6ba60

Commits on Oct 27, 2025

[DEV] support split qkv in muon (#1915)
FDecaYed
authored
221747d
Merge branch 'dev' into nrwu/eagercp
yuzhongw-nvidia
authored
c2410fc
[Dev] feat(moe): Fine-grained activation offloading (#1912)
lhb8125
authored
9069e12
ci: Add golden values for gpt_dynamic_inference_tp1_pp1_583m_cuda_graphs_logitsmatch_decode_graphs_only
ko3n1g
committed
a0a1866
ci: Add more golden values
ko3n1g
committed
c9fb78b
ci: Aggregate throughput
ko3n1g
committed
6f51284
Update dev branch codeowners (#1963)
chtruong814
authored
eb07b69
[Dev] JIT for MoE router and preprocess (#1918)
yaox12
authored
fa384d2
tests: Fix paths for test_cases
ko3n1g
committed
65c8f40
Revert "[Dev] feat(moe): Fine-grained activation offloading (#1912)"
ko3n1g
committed
2155c47
fix: Missing logger (#1966)
ko3n1g
authored
d95e86a
ci: Update copyright checker (#1974)
ko3n1g
authored
113cefb

Commits on Oct 28, 2025

Commits on Oct 29, 2025

ci(fix): `Run tests` label (#1970) (#2006)
ko3n1g
authored
bada8f9
Renaming golden values (#2020)
lhb8125
authored
ccf794e
Ko3n1g/chore/sync main to dev (#2018)

authored
7342f67
Ko3n1g/fix/golden values (#2037)
ko3n1g
authored
0d0f29c

Commits on Oct 30, 2025

cp: `Megatron-FSDP Expert Parallel (DeepSeek-v3) Support` into `dev` (#2007)

authored
1d1ac73

Commits on Oct 31, 2025

Update golden values due to PR #2007 (#2057)
chtruong814
authored
2c85448
Add DeepSeek-V3 GB200 NVL72 optimization guide (#2059)

sbhavani
and
yaox12
authored
402bc50
ci(hotfix): Disable flaky tests
ko3n1g
committed
4959f1f
[DEV] feat(MoE): Refactor cuda_graph_scope (#1917)
buptzyb
authored
4008d3d
Add mirror to main workflow (#2042)
chtruong814
authored
44539d1
ci: Remove cluster specific golden values (#2069)

ko3n1g
and
mathemakitten
authored
41c6e66

Commits on Nov 2, 2025

ci: Disable inference test
ko3n1g
committed
9d05926
[Dev] feat(moe): Support placing MTP layers into standalone stages (#1916)

authored
effebd8

Commits on Nov 3, 2025

Commits on Nov 4, 2025

Commits on Nov 5, 2025

add device and dtype to empty inv_dt init (#2137)
maanug-nv
authored
79e8592

Commits on Nov 6, 2025

Remove DS-V3 doc - draft being updated (#2155)
sbhavani
authored
3c1b98e
[DEV] torch_dist fixes, speed improvements and memory reduction for layerwise distopt (#1965)
FDecaYed
authored
b2fdd94
[Dev] Fix Qwen3-Next hang on Blackwell, add a flag to control torch.compile (#2058)

yuzhongw-nvidia
and
Dennis(Zhenhuan) Liu
authored
6cc224d
[dev] Fix cuda graph scope check in `language_model.py` (#2158)
ananthsub
authored
f4bd87e

Commits on Nov 10, 2025

[Dev] Remove experimental tags for fused kernels (#2172)
Dennis(Zhenhuan) Liu
authored
3207c23

Commits on Nov 11, 2025

Merge branch 'main' into dev
FDecaYed
committed
ae4cda5
[DEV] pull changes from main(f150f42e3929f7f2171e3687e67990332a76285b) (#2188)

authored
3cbe5c6
fallback to eager attn config

nrailg
authored and
yuzhongw-nvidia
committed
f545d76
adding cp support to eager attn

nrailg
authored and
yuzhongw-nvidia
committed
53cac6d
refine the code
yuzhongw-nvidia
committed
f943607
Refine dot_product_attention_context_parallel.py
yuzhongw-nvidia
committed
b65cb5c
fix CI: copyright, linting, and import
yuzhongw-nvidia
committed
6f5f12f
fix(transformer_config): Initialize cuda_graph_scope if not set regardless of cuda_graph_impl (#2166)
cuichenx
authored
6b59d71
[Dev] fix(offloading): Accuracy mismatch when offloading and recomputing same module (#2123)
lhb8125
authored
442a7f2
refine the code
yuzhongw-nvidia
committed
4abf166
Merge branch 'nrwu/eagercp' into eagercp
yuzhongw-nvidia
authored
b5e8f58
Ko3n1g/chore/update dev release settings (#2099)
ko3n1g
authored
6b01330
Merge remote-tracking branch 'github/main' into dev
ko3n1g
committed
d7d71e0
[DEV] Cherry-pick: M4 + Dist Checkpoint: Replace global parallel state with explicit group parameters (#2152)

yaoyu-33
and
dimapihtar
authored
56019e6

Commits on Nov 12, 2025

Commits on Nov 13, 2025

Commits on Nov 14, 2025

Commits on Nov 17, 2025

Revert "[20251111] Ko3n1g/chore/main to dev (#2211)" (#2266)
chtruong814
authored
71fa2e6
chroe: [Dev]Disable muon test for now (#2275)
BoxiangW
authored
0bf9ff9
Fixes of Merge main into dev
ko3n1g
committed
565202f
Replay: [20251111] Ko3n1g/chore/main to dev (#2267)

authored
4b78163

Commits on Nov 18, 2025