Skip to content

Qwen3-Next Model Provider + Pretraining Support - #1027

Merged
yaoyu-33 merged 25 commits into
mainfrom
avavre/qwen3next_support
Nov 3, 2025
Merged

Qwen3-Next Model Provider + Pretraining Support#1027
yaoyu-33 merged 25 commits into
mainfrom
avavre/qwen3next_support

Conversation

@adityavavreNVDA

@adityavavreNVDA adityavavreNVDA commented Oct 21, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

This PR adds Qwen3-Next model support and has a dependecy on Megatron-LM's dev branch.

  1. Pre-training loss curves match Megatron-LM's training curves as shown below:
image
  1. Unfused Attention Pretraining logs comparison with MLM
image

  1. Able to successfully store a checkpoint and resume training from where it left off.

  2. DeepEP works in 2509 container with DeepEP package installed.

  3. Add unit tests for model provider. [Done]

  4. IMPORTANT EDIT: Qwen3-Next Pre-training uses Global Auxiliary Loss for load balancing. Updated the recipe and compared with MLM again:

image

@copy-pr-bot

copy-pr-bot Bot commented Oct 21, 2025

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@adityavavreNVDA adityavavreNVDA mentioned this pull request Oct 21, 2025
Closed
9 tasks
ananthsub
ananthsub reviewed Oct 21, 2025
View reviewed changes
Comment thread src/megatron/bridge/models/qwen/qwen_provider.py Outdated
Comment thread src/megatron/bridge/models/qwen/qwen_provider.py Outdated
Comment thread src/megatron/bridge/models/qwen/qwen_provider.py Outdated
@adityavavreNVDA adityavavreNVDA changed the title Qwen3-Next Model Support Qwen3-Next Model Provider + Pretraining Support Oct 21, 2025
@adityavavreNVDA
adityavavreNVDA marked this pull request as ready for review October 25, 2025 18:14
@adityavavreNVDA
adityavavreNVDA requested a review from a team as a code owner October 25, 2025 18:15
@adityavavreNVDA
adityavavreNVDA changed the base branch from main to bridge/qwen3next October 27, 2025 17:29
Base automatically changed from bridge/qwen3next to main October 28, 2025 22:30
yaoyu-33
yaoyu-33 reviewed Oct 28, 2025
View reviewed changes
Comment thread src/megatron/bridge/training/utils/weight_decay_utils.py Outdated
yaoyu-33
yaoyu-33 reviewed Oct 28, 2025
View reviewed changes
Comment thread src/megatron/bridge/training/setup.py
yaoyu-33
yaoyu-33 reviewed Oct 29, 2025
View reviewed changes
Comment thread src/megatron/bridge/training/setup.py
yaoyu-33
yaoyu-33 reviewed Oct 29, 2025
View reviewed changes
Comment thread src/megatron/bridge/recipes/qwen/qwen3_next.py Outdated
@adityavavreNVDA

adityavavreNVDA commented Nov 2, 2025

Copy link
Copy Markdown
Contributor Author

/ok to test 8bf18c2

@yaoyu-33
yaoyu-33 merged commit 4cad5c6 into main Nov 3, 2025
29 checks passed
@yaoyu-33
yaoyu-33 deleted the avavre/qwen3next_support branch November 3, 2025 17:43
adityavavreNVDA added a commit that referenced this pull request Nov 3, 2025
@adityavavreNVDA
Qwen3-Next Model Provider + Pretraining Support (#1027)
aefa289 
Signed-off-by: adityavavreNVDA <avavre@nvidia.com>
@adityavavreNVDA adityavavreNVDA mentioned this pull request Nov 3, 2025
Merged
ananthsub pushed a commit that referenced this pull request Nov 4, 2025
@adityavavreNVDA
Cherry Pick Qwen3-Next Model Provider + Pretraining Support (#1027) (#…
f872e23 
…1173)

Signed-off-by: adityavavreNVDA <avavre@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yaoyu-33 yaoyu-33 yaoyu-33 approved these changes
@ananthsub ananthsub Awaiting requested review from ananthsub

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@adityavavreNVDA @yaoyu-33 @ananthsub