Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Debug Transformer Engine FP8 support with Megatron-core infrastructure #6740

Merged
merged 4 commits into from
Jun 1, 2023
Merged

Debug Transformer Engine FP8 support with Megatron-core infrastructure #6740

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
748b721
Construct FP8 amax reduction group
timmoon10 May 26, 2023
26369ee
update core for CI
aklife97 May 26, 2023
23c1994
Merge branch 'r1.19.0' into tmoon/fp8-amax-group-debug
timmoon10 May 31, 2023
6ef3df1
merge r1.19 to branch
aklife97 Jun 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion nemo/collections/nlp/parts/nlp_overrides.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ def init_model_parallel(self, global_rank: int, world_size: int) -> None:
pipeline_model_parallel_size=app_state.pipeline_model_parallel_size,
virtual_pipeline_model_parallel_size=app_state.virtual_pipeline_model_parallel_size,
pipeline_model_parallel_split_rank=app_state.pipeline_model_parallel_split_rank,
use_fp8=app_state.use_fp8,
)

# assert that fake tp and pp rank match after model parallel init
Expand Down Expand Up @@ -405,7 +406,7 @@ class PEFTSaveRestoreConnector(NLPSaveRestoreConnector):
Args:
peft_model_nemo_path: Used to provide the .nemo file corresponding to a PEFT model (which will only contain a small set of params)
peft_model_ckpt_path: Used to provide the path to .ckpt files of a PEFt model. This is required when no .nemo is available (yet) such as during resumed training.
If both are provided the peft_model_ckpt_path takes precedence.
If both are provided the peft_model_ckpt_path takes precedence.
If neither are provided, PEFT params are initialized at random (not loaded from any external source).
"""

Expand Down
Loading