ModelOpt Knowledge-Distillation support - #880
Conversation
19640dd to
c352594
Compare
ananthsub
left a comment
ananthsub
left a comment
There was a problem hiding this comment.
this LGTM. we also need:
- a functional test for
distill - updated docs to describe distillation, similar to https://docs.nvidia.com/nemo-framework/user-guide/latest/model-optimization/distillation/distillation.html
| @@ -0,0 +1,201 @@ | |||
| #!/usr/bin/env python3 | |||
There was a problem hiding this comment.
I'd strongly recommend keeping the examples to be as python-based as possible with minimal config dependencies. I am planning to rework the llama recipe example as it was added initially to demonstrate config overrides, but i think it ends up detracting from seeing the main APIs at work.
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
This reverts commit 9f0fc9e. Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
c352594 to
1abcf1d
Compare
| "nvidia-modelopt[torch]>=0.37.0,<0.39.0", | ||
| "torch; sys_platform == 'never'", |
There was a problem hiding this comment.
Why do we need this override?
There was a problem hiding this comment.
Need newer version without dealing with MLM
There was a problem hiding this comment.
At least if @ananthsub wants to try and cherry-pick this PR by end of week
There was a problem hiding this comment.
I'm lifting the pin here: NVIDIA/Megatron-LM#2046
There was a problem hiding this comment.
This will be reflected in CICD build by tomorrow?
|
/ok to test 1abcf1d |
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
|
/ok to test 7bd8165 |
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
|
|
||
| ## Limitations | ||
|
|
||
| * Only GPT-based checkpoints are currently supported. |
There was a problem hiding this comment.
below it says only GPT and Mamba based models are supported, but here it says only GPT
There was a problem hiding this comment.
I guess there's a naming ambiguity here. In MCore the Mamba models are subclasses of GPTModel, so it's kind of referring to that. Though I should make it more clear anyway, thanks
| model.teacher.tensor_model_parallel_size=2 | ||
| ``` | ||
|
|
||
| ### Combining YAML and CLI Overrides |
There was a problem hiding this comment.
when would a user need to combine yaml config with CLI overrides? would it be cleaner to enforce that the user must update the yaml to change any parameter?
There was a problem hiding this comment.
See the other examples, I just followed their format.
4 participants
#659
Integrate Disitllation support from Nvidia Model Optimizer into Megatron-Bridge, similarly to how it was done in Megatron-LM and NeMo