Skip to content

nemotron-3-super model support#2912

Merged
yaoyu-33 merged 16 commits intomainfrom
super-v3-pr
Mar 26, 2026
Merged

nemotron-3-super model support#2912
yaoyu-33 merged 16 commits intomainfrom
super-v3-pr

Conversation

@liding-nv
Copy link
Copy Markdown
Contributor

@liding-nv liding-nv commented Mar 20, 2026 •
edited by coderabbitai bot
Loading

Add Nemotron-3 Super model support to Megatron Bridge

  • Add bridge, provider, and recipe for the Nemotron-3 Super model
  • Include packed parquet data format PR
  • Reorganize examples/models/nemotron_3/ into nano/ and super/ subdirectories. Super examples will be added later
  • Add initial GB300 performance benchmark scripts

Summary by CodeRabbit

Release Notes

  • New Features

    • Added Nemotron 3 Super model support with pretraining, finetuning, and quantization-aware training capabilities
    • Introduced Parquet-backed packed sequence datasets for improved data handling
    • Added multi-token prediction (MTP) support for Mamba models
    • New mixed-precision recipe with NVFP4 quantization for the Super model

  • Improvements

    • Enhanced HuggingFace model loading with improved device placement strategy
    • Expanded Mixture of Experts (MoE) configuration options
    • New performance baseline configurations for GB300 hardware

  • Dependencies

    • Added optional pyarrow dependency for Parquet dataset support

@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot bot commented Mar 20, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 20, 2026

/ok to test cbd87cc

@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 20, 2026

/ok to test 34da894

@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 20, 2026

/claude review

claude[bot]
claude bot reviewed Mar 20, 2026
View reviewed changes
claude[bot]
claude bot reviewed Mar 20, 2026
View reviewed changes
claude[bot]
claude bot reviewed Mar 20, 2026
View reviewed changes
claude[bot]
claude bot reviewed Mar 20, 2026
View reviewed changes
examples/conversion/compare_hf_and_megatron/compare.py Outdated
Comment on lines +718 to +719
print(f"{input_ids.shape=}")
print(f"{input_ids=}")
Copy link
Copy Markdown
Contributor

@claude claude bot Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover debug prints. Consider removing before merge.

@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 20, 2026

/ok to test 2130901

@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 20, 2026

/ok to test c908ebe

@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 20, 2026

/claude review

@liding-nv
nemotron 3 super model support
72c5b89 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
update
02c4646 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
lint
7618457 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
clean up
8fc0f7d 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
clean up
b30a725 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
src/megatron/bridge/models/nemotronh/nemotron_h_bridge.py
2fe99b1 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
comments
13d58bb 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 23, 2026

/ok to test 13d58bb

@liding-nv
update
ce3d13e 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 24, 2026

/ok to test ce3d13e

@liding-nv
fix tests
f313561 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 24, 2026

/ok to test f313561

@cuichenx cuichenx mentioned this pull request Mar 24, 2026
Closed
5 tasks
cuichenx
cuichenx previously approved these changes Mar 24, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@cuichenx cuichenx left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
please follow up

  • can we merge nemotron training scripts with run_recipe.py
  • example launch scripts and W&B plots
  • add new MTP flags to MTP documentation

@liding-nv
tests
772b6a6 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
Merge branch 'main' into super-v3-pr
af7187a
@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 24, 2026

/ok to test af7187a

@liding-nv
update
4d22705 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
Merge branch 'main' into super-v3-pr
62776b9
@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 25, 2026

/ok to test 62776b9

yaoyu-33
yaoyu-33 previously approved these changes Mar 25, 2026
View reviewed changes
cuichenx
cuichenx reviewed Mar 25, 2026
View reviewed changes
@liding-nv
add cp>1 examples
564e071 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
Merge branch 'main' into super-v3-pr
20955dd
@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 25, 2026

/ok to test 20955dd

@liding-nv
readme
a7a52a5 
Signed-off-by: Li Ding <liding@nvidia.com>
@liding-nv
Copy link
Copy Markdown
Contributor Author

liding-nv commented Mar 25, 2026

/ok to test a7a52a5

cuichenx
cuichenx approved these changes Mar 25, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@cuichenx cuichenx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

@claude claude[bot] claude[bot] left review comments

@cuichenx cuichenx cuichenx approved these changes

@chtruong814 chtruong814 chtruong814 approved these changes

@yaoyu-33 yaoyu-33 yaoyu-33 left review comments

Assignees

No one assigned

Labels

area:model Model implementations and HF bridge logic feature New capabilities, enhancements, or enablement work ready-to-merge PR is approved, current, and only waiting for CI to pass before merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liding-nv @cuichenx @yaoyu-33 @chtruong814