Qwen3 finetune recipes - #975
Merged
Merged
Conversation
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Contributor
Author
|
/ok to test d0ecaf0 |
yaoyu-33
reviewed
Oct 16, 2025
yaoyu-33
reviewed
Oct 16, 2025
| return _qwen3_finetune_common(**combined_kwargs) | ||
|
|
||
|
|
||
| def _qwen3_finetune_common( |
Contributor
There was a problem hiding this comment.
how is finetune common different from common? do we need 2 of them? @cuichenx as well
Contributor
Author
There was a problem hiding this comment.
i find it easier to read when they're split up. otherwise the user has to keep track of branching between pretraining vs sft vs peft across parallelisms, lr, dataset, and tokenizers. in addition, the finetuning recipe configs will have a lot of code in common, at least based on https://github.com/NVIDIA-NeMo/NeMo/blob/main/nemo/collections/llm/recipes/finetune_default.py
if anything, having _qwen3_sft_common and _qwen3_peft_common helpers will make the split even more obvious to users/developers. wdyt?
Contributor
There was a problem hiding this comment.
i think it makes sense to separate finetune and pretrain. the data config, checkpoint config and tokenizer are all different. i think it even makes sense to have a common finetune helper for all models
cuichenx
reviewed
Oct 17, 2025
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Contributor
Author
|
/ok to test 37e0a2f |
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Contributor
Author
|
/ok to test 8bb57ab |
cuichenx
approved these changes
Oct 22, 2025
chtruong814
pushed a commit
that referenced
this pull request
Oct 22, 2025
* qwen3 finetuning configs Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * qwen3 finetuning configs Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * qwen3 finetune Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * qwen3 finetune recipes Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * updates Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * revert changes to gpt step Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * address feedback Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * add load dir Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
ananthsub
added a commit
that referenced
this pull request
Oct 23, 2025
* qwen3 finetuning configs * qwen3 finetuning configs * qwen3 finetune * qwen3 finetune recipes * updates * revert changes to gpt step * address feedback * add load dir --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Ananth Subramaniam <ansubramania@nvidia.com>
paul-gibbons
pushed a commit
to paul-gibbons/Megatron-Bridge
that referenced
this pull request
Oct 29, 2025
* qwen3 finetuning configs Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * qwen3 finetuning configs Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * qwen3 finetune Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * qwen3 finetune recipes Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * updates Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * revert changes to gpt step Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * address feedback Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * add load dir Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Paul Gibbons <pgibbons@nvidia.com>
nv-mollys
pushed a commit
that referenced
this pull request
Oct 31, 2025
* qwen3 finetuning configs Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * qwen3 finetuning configs Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * qwen3 finetune Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * qwen3 finetune recipes Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * updates Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * revert changes to gpt step Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * address feedback Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * add load dir Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: mollys <mollys@mollys.nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Import the pretrained checkpoints using AutoBridge: https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/conversion/convert_checkpoints.py
In the user train script, import qwen3 finetuning configs + specify paths to pretrained checkpoints from step 1 to begin finetuning
train script: https://gist.github.com/ananthsub/69139c3a8e30f57b4aa09eef422559cb