Fix fast-glu activation in change partitions #6909
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: hsiehjackson <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>
hsiehjackson
changed the title
titu1994
requested changes
Jun 23, 2023
| @@ -199,7 +199,7 @@ def compute_tp_splits( | |||
| # alias the global index to idx | |||
| idx = global_idx | |||
|
|
|||
| swiglu_activation = 'swiglu' in str(model_cfg.get('activation', '')).lower() | |||
| fast_glu_activation = str(model_cfg.get('activation', '')).lower() in ['fast-geglu', 'fast-swiglu', 'fast-reglu'] | |||
There was a problem hiding this comment.
swiglu didn't use the torch chunk tricks, so we don't need to handle the partition. Only fast_glu_activation need.
https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/modules/common/megatron/mlp.py#L230-L231
There was a problem hiding this comment.
From Megatron-LM, they do special handing for swiglu for partition conversion (code) is because they have chunk operation for swiglu in their implementation (code). In NeMo, we also have chunk operation but it only used when our activation is fast_glu_activation (code). Therefore, I change the partition conversion script from swiglu to fast_glu_activation
I explain the reason why the chunk operation needs special handling for partition conversion:
TP=2:
GPU0: tensor A [a1, a2] -> chunk to tensor a1 and tensor a2 -> activation(a1) * a2
GPU1: tensor B [b1, b2] -> chunk to tensor b1 and tensor b2 -> activation(b1) * b2
(Wrong) TP = 1
GPU0: tensor C = [a1, a2, b1, b2] (normal TP concatenation) -> chunk to tensor [a1,a2] and tensor [b1,b2] -> activation([a1,a2]) * [b1,b2]
(Correct) TP = 1
GPU0: tensor C = [a1, b1, a2, b2] (special handling) -> chunk to tensor [a1,b1] and tensor [a1,b2] -> activation([a1,b1]) * [a2,b2]
titu1994
approved these changes
Jun 23, 2023
| @@ -199,7 +199,7 @@ def compute_tp_splits( | |||
| # alias the global index to idx | |||
| idx = global_idx | |||
|
|
|||
| swiglu_activation = 'swiglu' in str(model_cfg.get('activation', '')).lower() | |||
| fast_glu_activation = str(model_cfg.get('activation', '')).lower() in ['fast-geglu', 'fast-swiglu', 'fast-reglu'] | |||
sudhakarsingh27
pushed a commit
that referenced
this pull request
Jun 27, 2023
* Fix fast-swiglu Signed-off-by: hsiehjackson <[email protected]> * change to all fast glu activation Signed-off-by: hsiehjackson <[email protected]> --------- Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Sudhakar Singh <[email protected]>
sudhakarsingh27
pushed a commit
that referenced
this pull request
Jun 27, 2023
* Fix fast-swiglu Signed-off-by: hsiehjackson <[email protected]> * change to all fast glu activation Signed-off-by: hsiehjackson <[email protected]> --------- Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Sudhakar Singh <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Change
swigluto all fast glu activation for partition conversion.Collection: [NLP]
Changelog
Usage
# Add a code snippet demonstrating how to use thisBefore your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information