Perf Config for 1 node GB200 DSV3 by gautham-kollu · Pull Request #3545 · NVIDIA-NeMo/Megatron-Bridge

gautham-kollu · 2026-04-27T22:20:38Z

What does this PR do ?

Perf Config for 1 node GB200 DSV3 with all the perk knobs

Changelog

Add specific line by line info of high level changes in this PR.

GitHub Actions CI

See the CI sectionin the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

Related to # (issue)

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

copy-pr-bot · 2026-04-27T22:20:41Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

gautham-kollu · 2026-04-27T22:26:19Z

/ok to test b55883a

cspades

Already LGTM but I think you turned on an argument that isn't relevant to MFSDP.

cspades · 2026-04-27T22:40:43Z

Also, let's wait until this PR merges and you can use the precision-aware optimization path: NVIDIA/Megatron-LM#4427

Loss should go down, if you can triple-check for me!

rapatel · 2026-04-27T22:56:36Z

LGTM. Thanks @gautham-kollu

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

gautham-kollu · 2026-04-27T23:12:59Z

/ok to test 8ea7c14

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

gautham-kollu · 2026-04-27T23:44:51Z

/ok to test 77cfa8c

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

gautham-kollu · 2026-04-28T00:43:12Z

/ok to test 0146b09

cspades

LGTM

cspades · 2026-04-28T01:26:46Z


+        # Validate reuse_grad_buf_for_mxfp8_param_ag when FSDP is not enabled
+        is_fsdp = self.dist.use_megatron_fsdp or self.ddp.use_megatron_fsdp
+        if not is_fsdp and self.mixed_precision.fp8_param_gather and self.mixed_precision.fp8_recipe == "mxfp8":


self.mixed_precision.fp8_param_gather -> self.ddp.fp8_param_gather IIRC?

Config valid failure

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

gautham-kollu · 2026-04-28T04:33:42Z

/ok to test eb0f20b

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

Signed-off-by: gautham-kollu <gkollu@nvidia.com>

gautham-kollu · 2026-04-28T17:14:25Z

/ok to test cf09f4b

) Signed-off-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: lonexreb <reach2shubhankar@gmail.com>

…or FLOPs calculator (#3695) Signed-off-by: lonexreb <reach2shubhankar@gmail.com> Co-authored-by: Chen Cui <chcui@nvidia.com>

Signed-off-by: lonexreb <reach2shubhankar@gmail.com> Co-authored-by: Chen Cui <chcui@nvidia.com>

…#3743) Signed-off-by: Chen Cui <chcui@nvidia.com>

Signed-off-by: qiyuw <qiyuw@nvidia.com> Co-authored-by: gautham-kollu <gkollu@nvidia.com>

…#3764) Signed-off-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: oliver könig <okoenig@nvidia.com>

…256x gpu scale expendable segments addition due to CUDA OOM issue (#3759) Signed-off-by: Rahul Salagame <rsalagame@nvidia.com>

Signed-off-by: Oliver Koenig <okoenig@nvidia.com> Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

Signed-off-by: Malay Nagda <malayn@nvidia.com>

Signed-off-by: oliver könig <okoenig@nvidia.com>

…d generation (#3758) Signed-off-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Signed-off-by: Lianglipeng <lianglipeng@didiglobal.com> Signed-off-by: hbhflw2000 <417911774@qq.com> Co-authored-by: Huy Vu <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: Yuekai Zhang <zhangyuekai@foxmail.com> Co-authored-by: Chen Cui <chcui@nvidia.com>

…workflow (#3779) Signed-off-by: Yu Yao <yaoyu.094@gmail.com>

Signed-off-by: Malay Nagda <malayn@nvidia.com>

Signed-off-by: gautham-kollu <gkollu@nvidia.com>

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

Squashed version of PR #3545 — perf config for 1-node GB200 DeepSeek V3 with all perk knobs. - Add GB200 DSV3 perf functional test (L0_Launch_recipes_deepseek_perf.sh, test_deepseek_recipes_pretrain_perf.py, gb200 golden values) - Wire reuse_grad_buf_for_mxfp8_param_ag for FSDP and handle None mixed_precision case in training/config.py - Drop now-unused branch in training/mixed_precision.py - Update unit tests for the above - Extend scripts/performance argument_parser and evaluate utilities Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

gautham-kollu · 2026-05-13T19:00:27Z

Closing this in favor of #3796

first draft

b55883a

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

gautham-kollu marked this pull request as ready for review April 27, 2026 22:20

copy-pr-bot Bot temporarily deployed to test April 27, 2026 22:27 Inactive

cspades reviewed Apr 27, 2026

View reviewed changes

Comment thread tests/functional_tests/test_groups/recipes/test_deepseek_recipes_pretrain_perf.py Outdated

rapatel reviewed Apr 27, 2026

View reviewed changes

Comment thread tests/functional_tests/test_groups/recipes/test_deepseek_recipes_pretrain_perf.py

add debug code

8ea7c14

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

copy-pr-bot Bot temporarily deployed to test April 27, 2026 23:13 Inactive

gautham-kollu added 2 commits April 27, 2026 16:22

cleanup

ea40b22

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

gb200 x4

ac0319d

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

gautham-kollu requested a review from a team as a code owner April 27, 2026 23:41

review comments

77cfa8c

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

copy-pr-bot Bot temporarily deployed to test April 27, 2026 23:46 Inactive

reuse_grad_buf_for_mxfp8_param_ag false for fsdp

0146b09

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

copy-pr-bot Bot temporarily deployed to test April 28, 2026 00:43 Inactive

cspades previously approved these changes Apr 28, 2026

View reviewed changes

cspades reviewed Apr 28, 2026

View reviewed changes

chck if mixed_precision is none

eb0f20b

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

copy-pr-bot Bot temporarily deployed to test April 28, 2026 04:34 Inactive

gautham-kollu added 2 commits April 28, 2026 09:29

update uts

b67f61a

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

Merge branch 'main' into gk/1node_gb200_dsv3

cf09f4b

Signed-off-by: gautham-kollu <gkollu@nvidia.com>

ko3n1g and others added 25 commits May 12, 2026 15:50

build: pin flashinfer-python + flashinfer-cubin in pyproject.toml (#3750

a2b0a85

) Signed-off-by: oliver könig <okoenig@nvidia.com>

[training] fix: aggregate TB memory metrics across PP group (#3645)

75799e7

Signed-off-by: lonexreb <reach2shubhankar@gmail.com>

[training, test] test: add MLA, MTP, and provider-override coverage f…

5f9447c

…or FLOPs calculator (#3695) Signed-off-by: lonexreb <reach2shubhankar@gmail.com> Co-authored-by: Chen Cui <chcui@nvidia.com>

[training, test] feat: add unit tests for pg_utils helpers (#3650)

5c12732

Signed-off-by: lonexreb <reach2shubhankar@gmail.com> Co-authored-by: Chen Cui <chcui@nvidia.com>

[recipe, test] feat: add unit tests for CometPlugin (#3652)

0573ee0

Signed-off-by: lonexreb <reach2shubhankar@gmail.com> Co-authored-by: Chen Cui <chcui@nvidia.com>

[model, ckpt] fix: align GPT-OSS BF16 down_proj orientation on import (…

71ebb51

…#3743) Signed-off-by: Chen Cui <chcui@nvidia.com>

[MXFP8]Update param buffer before AG in eval (#3727)

6aebe35

Signed-off-by: qiyuw <qiyuw@nvidia.com> Co-authored-by: gautham-kollu <gkollu@nvidia.com>

chore(beep boop 🤖): Bump (main, mcore-main) (2026-05-09)

bd8f9f4

build: upgrade mamba-ssm to 2.3.2.post1, causal-conv1d to 1.6.2.post1 (…

35ab5a1

…#3764) Signed-off-by: oliver könig <okoenig@nvidia.com>

[docs] feat: add cherrypick policy for release branches (#3739)

5975d6f

Signed-off-by: oliver könig <okoenig@nvidia.com>

chore(beep boop 🤖): Bump (main, mcore-main) (2026-05-10)

51453a1

[performance] feat: add 405B B200/B300 V2 aliases + 405B GB200 NVFP4 …

9230619

…256x gpu scale expendable segments addition due to CUDA OOM issue (#3759) Signed-off-by: Rahul Salagame <rsalagame@nvidia.com>

ci: Major refactor of release-workflows (#3725)

986c646

Signed-off-by: Oliver Koenig <okoenig@nvidia.com> Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

nccl-algo reset

2f44bd3

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

1

b706332

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

run as module

9659e4a

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

llama31 405b gb200 nvfp4 no pg overlap (#3713)

7831e35

Signed-off-by: Malay Nagda <malayn@nvidia.com>

[ci] fix: check-imports action lookup on ARC container runner (#3774)

23cfc35

Signed-off-by: oliver könig <okoenig@nvidia.com>

[build] chore: bump transformer-engine to release_v2.15 (#3672)

b358ee1

Signed-off-by: oliver könig <okoenig@nvidia.com>

[ci, test] chore: Split L0 converter launch script into conversion an…

1dc260b

…d generation (#3758) Signed-off-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

[docs, ci] chore: rename state labels and retire remove-needs-author …

8109cf3

…workflow (#3779) Signed-off-by: Yu Yao <yaoyu.094@gmail.com>

Malay/move expandable flag (#3781)

b565504

Signed-off-by: Malay Nagda <malayn@nvidia.com>

Update L0_Launch_recipes_deepseek_perf.sh

b27acc5

Signed-off-by: gautham-kollu <gkollu@nvidia.com>

1

d9f7c49

Signed-off-by: Gautham Kollu <gkollu@nvidia.com>

gautham-kollu force-pushed the gk/1node_gb200_dsv3 branch from eda5670 to d9f7c49 Compare May 12, 2026 22:50

gautham-kollu mentioned this pull request May 12, 2026

Perf Config for 1 node GB200 DSV3 #3796

Merged

5 tasks

gautham-kollu closed this May 13, 2026

Uh oh!

Conversation

gautham-kollu commented Apr 27, 2026

What does this PR do ?

Changelog

GitHub Actions CI

Before your PR is "Ready for review"

Additional Information

Uh oh!

copy-pr-bot Bot commented Apr 27, 2026

Uh oh!

gautham-kollu commented Apr 27, 2026

Uh oh!

cspades left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cspades commented Apr 27, 2026

Uh oh!

Uh oh!

rapatel commented Apr 27, 2026

Uh oh!

gautham-kollu commented Apr 27, 2026

Uh oh!

gautham-kollu commented Apr 27, 2026

Uh oh!

gautham-kollu commented Apr 28, 2026

Uh oh!

cspades left a comment

Choose a reason for hiding this comment

Uh oh!

cspades Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gautham-kollu commented Apr 28, 2026

Uh oh!

gautham-kollu commented Apr 28, 2026

Uh oh!

gautham-kollu commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

18 participants

cspades Apr 28, 2026 •

edited

Loading