Skip to content

fix: skip FSDP DTensor boundary validation under fake process group - #3669

Closed
Victarry wants to merge 319 commits into
NVIDIA:mainfrom
Victarry:denliu/fix-fsdp-fake-pg-compat
Closed

fix: skip FSDP DTensor boundary validation under fake process group#3669
Victarry wants to merge 319 commits into
NVIDIA:mainfrom
Victarry:denliu/fix-fsdp-fake-pg-compat

Conversation

@Victarry

@Victarry Victarry commented Mar 3, 2026

Copy link
Copy Markdown

Summary

  • Skip validate_uneven_dtensor boundary check when the distributed backend is 'fake' (fake process group)
  • The validation uses all_reduce(MAX) to verify all ranks' shards cover the full tensor, but under fake PG all collectives are no-ops — only rank 0's boundaries are visible, causing the end-boundary assertion to always fail
  • Fake process group is only used for single-GPU memory profiling where numerical correctness is irrelevant, so the validation can be safely skipped

Target branch: main

Test plan

  • Run Megatron-LM with --fake-process-group --use-megatron-fsdp --data-parallel-sharding-strategy optim_grads_params --init-model-with-meta-device on a single GPU
  • Verify training completes 3 iterations without DTensor assertion error
  • Verify memory snapshot is correctly captured

ko3n1g and others added 30 commits October 26, 2025 19:01
@ko3n1g
ci: Run on push to release branch (NVIDIA#1960) (NVIDIA#1962)
4b6ba60 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@FDecaYed
[DEV] support split qkv in muon (NVIDIA#1915)
221747d
@lhb8125
[Dev] feat(moe): Fine-grained activation offloading (NVIDIA#1912)
9069e12 
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
@ko3n1g
ci: Add golden values for gpt_dynamic_inference_tp1_pp1_583m_cuda_gra…
a0a1866 
…phs_logitsmatch_decode_graphs_only

Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Add more golden values
c9fb78b 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Aggregate throughput
6f51284 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@chtruong814
Update dev branch codeowners (NVIDIA#1963)
eb07b69 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@yaox12
[Dev] JIT for MoE router and preprocess (NVIDIA#1918)
fa384d2 
Signed-off-by: Xin Yao <xiny@nvidia.com>
@ko3n1g
tests: Fix paths for test_cases
65c8f40 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
Revert "[Dev] feat(moe): Fine-grained activation offloading (NVIDIA#1912
2155c47 
)"

This reverts commit 9069e12.

Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
fix: Missing logger (NVIDIA#1966)
d95e86a 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Update copyright checker (NVIDIA#1974)
113cefb 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@youngeunkwon0405
[Dev] Update symmetric registration interface to sync-up with upstrea…
d9e0806 
…m pytorch change (NVIDIA#1930)

Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
Signed-off-by: Youngeun <kyeg9404@gmail.com>
@chtruong814 @shjwudp @xuwchen
cp: Megatron-FSDP Expert Parallel (DeepSeek-v3) Support into dev (N…
cc33e00 
…VIDIA#1987)

Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Jack Chang <jianbinc@nvidia.com>
Co-authored-by: jianbinc <shjwudp@gmail.com>
Co-authored-by: xuwenc <xuwenc@nvidia.com>
@ko3n1g
Revert "cp: Megatron-FSDP Expert Parallel (DeepSeek-v3) Support int…
13edb58 
…o `dev` (NVIDIA#1987)"

This reverts commit cc33e00.
@lhb8125 Hongbin Liu
[Was PR1912][Dev] feat(moe): Fine-grained activation offloading (NVID…
c22c2aa 
…IA#1969)

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@cw-dfw-cs-001-login-02.cm.cluster>
Co-authored-by: Hongbin Liu <hongbinl@cw-dfw-cs-001-login-02.cm.cluster>
@ko3n1g
ci(fix): Run tests label (NVIDIA#1970) (NVIDIA#2006)
bada8f9 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@lhb8125
Renaming golden values (NVIDIA#2020)
ccf794e 
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
@ko3n1g @yueshen2016
Chen-Han Yu @shanmugamr1992 Shanmugam Ramasamy Mcore Bot Shanmugam Ramasamy @sidsingh-nvidia Shanmugam Ramasamy @youngeunkwon0405 @shunjiad @maanug-nv @shjwudp @xuwchen @tdene
Ko3n1g/chore/sync main to dev (NVIDIA#2018)
7342f67 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: James Shen <yueshen@nvidia.com>
Co-authored-by: Chen-Han Yu <chenhany@cw-dfw-cs-001-login-01.cm.cluster>
Co-authored-by: Shanmugam Ramasamy <shanmugamr@nvidia.com>
Co-authored-by: Shanmugam Ramasamy <shanmugamr@cw-dfw-cs-001-vscode-01.cm.cluster>
Co-authored-by: Mcore Bot <mcore-bot@nvidia.com>
Co-authored-by: Shanmugam Ramasamy <shanmugamr@shanmugamr-mlt.client.nvidia.com>
Co-authored-by: Siddharth Singh <sidsingh@nvidia.com>
Co-authored-by: Shanmugam Ramasamy <shanmugamr@cw-dfw-cs-001-login-01.cm.cluster>
Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com>
Co-authored-by: Shunjia Ding <shunjiad@nvidia.com>
Co-authored-by: Maanu Grover <maanug@nvidia.com>
Co-authored-by: Jack Chang <jianbinc@nvidia.com>
Co-authored-by: jianbinc <shjwudp@gmail.com>
Co-authored-by: xuwenc <xuwenc@nvidia.com>
Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com>
@ko3n1g
Ko3n1g/fix/golden values (NVIDIA#2037)
0d0f29c 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@chtruong814 @shjwudp @xuwchen
cp: Megatron-FSDP Expert Parallel (DeepSeek-v3) Support into dev (N…
1d1ac73 
…VIDIA#2007)

Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Jack Chang <jianbinc@nvidia.com>
Co-authored-by: jianbinc <shjwudp@gmail.com>
Co-authored-by: xuwenc <xuwenc@nvidia.com>
@chtruong814
Update golden values due to PR NVIDIA#2007 (NVIDIA#2057)
2c85448 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@sbhavani @yaox12
Add DeepSeek-V3 GB200 NVL72 optimization guide (NVIDIA#2059)
402bc50 
Co-authored-by: Xin Yao <xiny@nvidia.com>
@ko3n1g
ci(hotfix): Disable flaky tests
4959f1f 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@buptzyb
[DEV] feat(MoE): Refactor cuda_graph_scope (NVIDIA#1917)
4008d3d 
Signed-off-by: Robin Zhang <robinz@nvidia.com>
@chtruong814
Add mirror to main workflow (NVIDIA#2042)
44539d1 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@ko3n1g @mathemakitten
ci: Remove cluster specific golden values (NVIDIA#2069)
41c6e66 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: helen ngo <helenn@nvidia.com>
@ko3n1g
ci: Disable inference test
9d05926 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@BestJuly @shifangx @Wohox
[Dev] feat(moe): Support placing MTP layers into standalone stages (N…
effebd8 
…VIDIA#1916)

Signed-off-by: Li Tao <lit@nvidia.com>
Co-authored-by: shifangx <shifangx@nvidia.com>
Co-authored-by: Pingtian Li <pingtianl@nvidia.com>
@yaox12
[Dev] A Guide to Reproduce DeepSeek-V3 Pre-training Performance on GB…
1d9edcf 
…200 (NVIDIA#1996)

Signed-off-by: Xin Yao <xiny@nvidia.com>
kunlunl and others added 13 commits February 13, 2026 14:36
@kunlunl
Add absorbed-mla (NVIDIA#3193)
6059f36
@chtruong814
cp: Remove gpu sanity check (NVIDIA#3420) into dev (NVIDIA#3421)
9f2ca96 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@ko3n1g
[dev] ci: Fix merge queue (NVIDIA#3385)
1dcf0da 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
[dev] cp: Cherrypick CI changes (NVIDIA#3543)
cd1c215 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Dennis(Zhenhuan) Liu
[Dev] Fix MoE aux loss tracker hang with MTP enabled (NVIDIA#3400)
aa86018
@chtruong814
ci: Remove multi-approval action from dev branch (NVIDIA#3576)
2b4b9c4 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@FDecaYed
Merge branch 'main' into dev
0ab47fa
@ko3n1g @HaochenYuan
@Phlip79 @duncanriach @yobibyte @chtruong814 @wdykas root root root root William Dykas root root root root root root root root root root root root @github-actions @1195343015 @yuzhongw-nvidia @lhb8125 @youngeunkwon0405 @santhnm2 @jiemingz @tgkyrie @dimapihtar @yaox12 @rkarimimahab Rabeeh Mahabadi @sancha @deepakn94 @sbhavani @ahmadki @BestJuly @maanug-nv @mvirts @asolergi-nv @HollowMan6 @buptzyb @shengf-nv @venmugil @mathemakitten @jaredcasper @parthmannan @tdene @Autumn1998 @wplf Jinliang Li Jinliang Li @yashaswikarnati @nschank @Glabred @janEbert @rj42 @juntaowww @Wohox @chrisgrimm @ChenhanYu @ericharper @xiaoxi-wangfj @shjwudp @c1lovez1 @skydoorkai @yeyu-nvidia @kwyss-nvidia @jon-barker @AAnoosheh @sidsingh-nvidia @megnvidia @thecaptain789 litianjian @ISEEKYAN @xuwchen @jstjohn @lmcafee-nvidia @claude @ArEsKay3 Robert Kirby Dennis(Zhenhuan) Liu @cursoragent @kevalmorabia97 @shanmugamr1992 @vasunvidia @sajadn @kunlunl @laixinn @dndnda liming127 Jon Barker @jenchen13 @yueshen2016 @faradawn @cspades @arendu Soumye Singhal @seonjinn @sbak5 @mkhona-nvidia
[dev] pull main 260220 (NVIDIA#3574)
a1a73f8 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>
Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com>
Signed-off-by: Santosh Bhavani <santosh.bhavani@live.com>
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
Signed-off-by: Hollow Man <hollowman@opensuse.org>
Signed-off-by: Robin Zhang <robinz@nvidia.com>
Signed-off-by: jinliangl <jinliangl@nvidia.com>
Signed-off-by: Maanu Grover <maanug@nvidia.com>
Signed-off-by: dimapihtar <dpihtar@gmail.com>
Signed-off-by: xiaoxi-wangfj <690912414@qq.com>
Signed-off-by: skydoorkai <htsantaclara@163.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: meg miranda <mmiranda@nvidia.com>
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Signed-off-by: sajadn <snorouzi@nvidia.com>
Signed-off-by: lit <lit@nvidia.com>
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
Signed-off-by: Cory Ye <cye@nvidia.com>
Signed-off-by: adithyare <adithyare@nvidia.com>
Signed-off-by: Soumye Singhal <soumyes@cw-dfw-cs-001-dc-01.cm.cluster>
Signed-off-by: Ahmad Kiswani <kiswani.ahmad@gmail.com>
Signed-off-by: mikail <mkhona@nvidia.com>
Co-authored-by: HaochenYuan <106647990+HaochenYuan@users.noreply.github.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Duncan Riach <33532941+duncanriach@users.noreply.github.com>
Co-authored-by: yobi byte <yobibyte@users.noreply.github.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: wdykas <73254672+wdykas@users.noreply.github.com>
Co-authored-by: root <root@gpu-h100-0348.cm.cluster>
Co-authored-by: root <root@gpu-h100-0193.cm.cluster>
Co-authored-by: root <root@gpu-h100-0082.cm.cluster>
Co-authored-by: root <root@gpu-h100-0495.cm.cluster>
Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster>
Co-authored-by: root <root@gpu-h100-0213.cm.cluster>
Co-authored-by: root <root@gpu-h100-0435.cm.cluster>
Co-authored-by: root <root@gpu-h100-0188.cm.cluster>
Co-authored-by: root <root@gpu-h100-0032.cm.cluster>
Co-authored-by: root <root@gpu-h100-0023.cm.cluster>
Co-authored-by: root <root@gpu-h100-0368.cm.cluster>
Co-authored-by: root <root@gpu-h100-0203.cm.cluster>
Co-authored-by: root <root@gpu-h100-0229.cm.cluster>
Co-authored-by: root <root@gpu-h100-0123.cm.cluster>
Co-authored-by: root <root@gpu-h100-0217.cm.cluster>
Co-authored-by: root <root@gpu-h100-0496.cm.cluster>
Co-authored-by: root <root@gpu-h100-0261.cm.cluster>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jiayi Yan <66017932+1195343015@users.noreply.github.com>
Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com>
Co-authored-by: Hongbin Liu <lhb8125@users.noreply.github.com>
Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com>
Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com>
Co-authored-by: Jimmy Zhang <133159885+jiemingz@users.noreply.github.com>
Co-authored-by: tgkyrie <74066353+tgkyrie@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: rkarimimahab <rkarimimahab@nvidia.com>
Co-authored-by: Rabeeh Mahabadi <rkarimimahab@nb-hel-cs-001-vscode-02.cm.cluster>
Co-authored-by: Sanjeev Satheesh <sasatheesh@nvidia.com>
Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com>
Co-authored-by: Santosh Bhavani <santosh.bhavani@live.com>
Co-authored-by: Ahmad Kiswani <kiswani.ahmad@gmail.com>
Co-authored-by: Li Tao <lit@nvidia.com>
Co-authored-by: Maanu Grover <109391026+maanug-nv@users.noreply.github.com>
Co-authored-by: mvirts <mvirts@gmail.com>
Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>
Co-authored-by: ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟 <hollowman@opensuse.org>
Co-authored-by: Robin Zhang <robinz@nvidia.com>
Co-authored-by: Sheng Fu <shengf@nvidia.com>
Co-authored-by: Venmugil Elango <498703+venmugil@users.noreply.github.com>
Co-authored-by: mathemakitten <helenn@nvidia.com>
Co-authored-by: Jared Casper <155158+jaredcasper@users.noreply.github.com>
Co-authored-by: Parth Mannan <38387286+parthmannan@users.noreply.github.com>
Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com>
Co-authored-by: Tong Liu <tongliu@nvidia.com>
Co-authored-by: Li Jinliang <jinliangl@nvidia.com>
Co-authored-by: Jinliang Li <jinliangl@pool0-01676.cm.cluster>
Co-authored-by: Jinliang Li <jinliangl@cw-dfw-cs-001-vscode-01.cm.cluster>
Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com>
Co-authored-by: Nick Schank <nick@reflection.ai>
Co-authored-by: Jeffrey Chen <jeffrey@reflection.ai>
Co-authored-by: janEbert <janpabloe@nvidia.com>
Co-authored-by: rj42 <lbkzman@gmail.com>
Co-authored-by: Juntao Wang <juntaow@nvidia.com>
Co-authored-by: Pingtian Li <158665726+Wohox@users.noreply.github.com>
Co-authored-by: Chris Grimm <chris@reflection.ai>
Co-authored-by: Chenhan D. Yu <5185878+ChenhanYu@users.noreply.github.com>
Co-authored-by: Eric Harper <eharper@nvidia.com>
Co-authored-by: xiaoxi-wangfj <690912414@qq.com>
Co-authored-by: Jianbin Chang <shjwudp@gmail.com>
Co-authored-by: c1lovez1 <141424951+c1lovez1@users.noreply.github.com>
Co-authored-by: Zhang Haitao <htsantaclara@163.com>
Co-authored-by: yeyu-nvidia <yeyu@nvidia.com>
Co-authored-by: kwyss-nvidia <kwyss@nvidia.com>
Co-authored-by: Jon Barker <jbarker@nvidia.com>
Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com>
Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com>
Co-authored-by: megnvidia <mmiranda@nvidia.com>
Co-authored-by: thecaptain789 <257642323+thecaptain789@users.noreply.github.com>
Co-authored-by: thecaptain789 <thecaptain789@users.noreply.github.com>
Co-authored-by: litianjian <litianjian@bytedance.com>
Co-authored-by: Yan Bai <baiyan1996@icloud.com>
Co-authored-by: xuwchen <xuwenc@nvidia.com>
Co-authored-by: John St. John <jstjohn@users.noreply.github.com>
Co-authored-by: Lawrence McAfee <85179052+lmcafee-nvidia@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Robert Kirby <ArEsKay3@users.noreply.github.com>
Co-authored-by: Siddharth Singh <sidsingh@nvidia.com>
Co-authored-by: Robert Kirby <rkirby@cw-dfw-cs-001-vscode-01.cm.cluster>
Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Co-authored-by: Dennis(Zhenhuan) Liu <denliu@nvidia.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Co-authored-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com>
Co-authored-by: vasunvidia <108759426+vasunvidia@users.noreply.github.com>
Co-authored-by: Philip Petrakian <pgpetrak@gmail.com>
Co-authored-by: Sajad Norouzi <sajad.n@gmail.com>
Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com>
Co-authored-by: xielaixin <xielx@shanghaitech.edu.cn>
Co-authored-by: Robert Kirby <rkirby@nvidia.com>
Co-authored-by: Ming <93323717+dndnda@users.noreply.github.com>
Co-authored-by: liming127 <liming127@meituan.com>
Co-authored-by: Jon Barker <jbarker@oci-hsg-cs-001-vscode-01.cm.cluster>
Co-authored-by: helen ngo <helen.ngo14@gmail.com>
Co-authored-by: Jenny Chen <jennifchen@nvidia.com>
Co-authored-by: yueshen2016 <39203804+yueshen2016@users.noreply.github.com>
Co-authored-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
Co-authored-by: Cory Ye <44509866+cspades@users.noreply.github.com>
Co-authored-by: Adi Renduchintala <adithya.r@gmail.com>
Co-authored-by: Soumye Singhal <soumyes@cw-dfw-cs-001-dc-01.cm.cluster>
Co-authored-by: Seonjin Na <sna@nvidia.com>
Co-authored-by: Seonmyeong Bak <sbak@nvidia.com>
Co-authored-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com>
@LiuXTao xiaotaoliu
@yuzhongw-nvidia @yanring
[dev] fix(moe): fix the bug where gate was not sliced when kv_head < …
2e4a5d4 
…tp_size. (NVIDIA#3529)

Co-authored-by: xiaotaoliu <xiaotaoliu@tencent.com>
Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com>
Co-authored-by: Zijie Yan <zijiey@nvidia.com>
@kunlunl
Add unit test for THD (NVIDIA#3608)
d0e0cf0
Dennis(Zhenhuan) Liu @claude
[Dev] feat(checkpoint): zero-copy storage sharing in CheckpointWithou…
bc9298c 
…tOutput (NVIDIA#3641)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@xiaoyao0115 @kunlunl
[Dev] Add E2E support for THD format (NVIDIA#2924)
5c613ab 
Signed-off-by: xiaoyao0115 <1804647152@qq.com>
Signed-off-by: tailaim <tailaim@nvidia.com>
Co-authored-by: kunlunl <kunlunl@nvidia.com>
Dennis Liu @claude
fix: skip FSDP DTensor boundary validation under fake process group
ffdf587 
The `validate_uneven_dtensor` function uses `all_reduce(MAX)` across
all ranks to verify that local shards collectively cover the full
global tensor. Under fake process group (backend='fake'), all
collective operations are no-ops, so only rank 0's boundaries are
visible — the end-boundary check always fails.

Skip the boundary validation when the distributed backend is 'fake',
since fake process group is only used for memory profiling where
numerical correctness is irrelevant.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Victarry
Victarry requested review from a team as code owners March 3, 2026 04:03
@copy-pr-bot

copy-pr-bot Bot commented Mar 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci
svcnvidia-nemo-ci requested a review from a team March 3, 2026 04:03
@Victarry Victarry mentioned this pull request Mar 4, 2026
Merged
@Victarry Victarry closed this Mar 4, 2026
@Victarry
Victarry deleted the denliu/fix-fsdp-fake-pg-compat branch March 4, 2026 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.