Upgrade NeMo to latest mcore and TE #7862

dimapihtar · 2023-11-06T14:47:27Z

What does this PR do ?

Upgrades NeMo to latest mcore and TE versions.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: dimapihtar <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: dimapihtar <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: dimapihtar <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: dimapihtar <[email protected]>

examples/nlp/language_modeling/conf/megatron_gpt_config.yaml

examples/nlp/language_modeling/conf/megatron_gpt_inference.yaml

Signed-off-by: dimapihtar <[email protected]>

Signed-off-by: Dmytro Pykhtar <[email protected]>

ericharper · 2023-11-15T16:58:51Z

jenkins

Signed-off-by: dimapihtar <[email protected]>

dimapihtar · 2023-11-16T20:24:37Z

jenkins

Signed-off-by: dimapihtar <[email protected]>

dimapihtar · 2023-11-16T21:39:27Z

jenkins

dimapihtar · 2023-11-16T21:45:43Z

jenkins

dimapihtar · 2023-11-16T22:32:19Z

jenkins

Signed-off-by: dimapihtar <[email protected]>

dimapihtar · 2023-11-16T22:53:05Z

jenkins

Jenkinsfile

ericharper · 2023-11-16T22:57:25Z

Jenkinsfile

@@ -3150,51 +3160,51 @@ assert_frame_equal(training_curve, gt_curve, rtol=1e-3, atol=1e-3)"'''
        sh "rm -rf examples/nlp/language_modeling/gpt_index_mappings"
      }
    }
-    stage('L2: Megatron GPT with Rope Pretraining and Resume Training TP=2') {


Do we need to remove this test?

@ericharper this one uses position_embedding_type=rope which causes an error (TE bug).

This test is not even using mcore_gpt or transformer_engine flags though....

Signed-off-by: dimapihtar <[email protected]>

dimapihtar · 2023-11-16T23:02:02Z

jenkins

Signed-off-by: dimapihtar <[email protected]>

dimapihtar · 2023-11-16T23:20:08Z

jenkins

Signed-off-by: dimapihtar <[email protected]>

dimapihtar · 2023-11-16T23:53:21Z

jenkins

Signed-off-by: dimapihtar <[email protected]>

dimapihtar · 2023-11-17T02:05:54Z

jenkins

Signed-off-by: eharper <[email protected]>

ericharper · 2023-11-17T03:36:26Z

jenkins

Signed-off-by: eharper <[email protected]>

ericharper · 2023-11-17T06:45:52Z

jenkins

ericharper

LGTM. Thanks!

* mcore upgrade Signed-off-by: dimapihtar <[email protected]> * remove GPTEmbedding import (deprecated) Signed-off-by: dimapihtar <[email protected]> * switch to LanguageModelEmbedding Signed-off-by: dimapihtar <[email protected]> * reset config Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * pass attn_mask-type through the forward method Signed-off-by: dimapihtar <[email protected]> * reset conf Signed-off-by: dimapihtar <[email protected]> * add more attn_mask_type params Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * attn_,ask_type fixes Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove attn_mask_type param Signed-off-by: dimapihtar <[email protected]> * t5/mt5 fix Signed-off-by: dimapihtar <[email protected]> * revert configs Signed-off-by: dimapihtar <[email protected]> * revert configs Signed-off-by: dimapihtar <[email protected]> * remove attn_mask_type param Signed-off-by: dimapihtar <[email protected]> * update mcore commit Signed-off-by: dimapihtar <[email protected]> * change mcore commit Signed-off-by: dimapihtar <[email protected]> * add TE installation Signed-off-by: dimapihtar <[email protected]> * add env var Signed-off-by: dimapihtar <[email protected]> * comment out rope test Signed-off-by: dimapihtar <[email protected]> * change mcore installation Signed-off-by: dimapihtar <[email protected]> * update mcore commit Signed-off-by: dimapihtar <[email protected]> * change mcore installation Signed-off-by: dimapihtar <[email protected]> * change mcore commit Signed-off-by: dimapihtar <[email protected]> * add -e Signed-off-by: eharper <[email protected]> * revert Signed-off-by: eharper <[email protected]> * revert jenkins test comment Signed-off-by: eharper <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: eharper <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: eharper <[email protected]> Signed-off-by: Piotr Żelasko <[email protected]>

* mcore upgrade Signed-off-by: dimapihtar <[email protected]> * remove GPTEmbedding import (deprecated) Signed-off-by: dimapihtar <[email protected]> * switch to LanguageModelEmbedding Signed-off-by: dimapihtar <[email protected]> * reset config Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * pass attn_mask-type through the forward method Signed-off-by: dimapihtar <[email protected]> * reset conf Signed-off-by: dimapihtar <[email protected]> * add more attn_mask_type params Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * attn_,ask_type fixes Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove attn_mask_type param Signed-off-by: dimapihtar <[email protected]> * t5/mt5 fix Signed-off-by: dimapihtar <[email protected]> * revert configs Signed-off-by: dimapihtar <[email protected]> * revert configs Signed-off-by: dimapihtar <[email protected]> * remove attn_mask_type param Signed-off-by: dimapihtar <[email protected]> * update mcore commit Signed-off-by: dimapihtar <[email protected]> * change mcore commit Signed-off-by: dimapihtar <[email protected]> * add TE installation Signed-off-by: dimapihtar <[email protected]> * add env var Signed-off-by: dimapihtar <[email protected]> * comment out rope test Signed-off-by: dimapihtar <[email protected]> * change mcore installation Signed-off-by: dimapihtar <[email protected]> * update mcore commit Signed-off-by: dimapihtar <[email protected]> * change mcore installation Signed-off-by: dimapihtar <[email protected]> * change mcore commit Signed-off-by: dimapihtar <[email protected]> * add -e Signed-off-by: eharper <[email protected]> * revert Signed-off-by: eharper <[email protected]> * revert jenkins test comment Signed-off-by: eharper <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: eharper <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: eharper <[email protected]>

dimapihtar and others added 6 commits October 13, 2023 07:14

mcore upgrade

3e235df

Signed-off-by: dimapihtar <[email protected]>

remove GPTEmbedding import (deprecated)

3f10321

Signed-off-by: dimapihtar <[email protected]>

Merge branch 'main' into dpykhtar/te_mcore_upgrade

f1329f8

switch to LanguageModelEmbedding

6a0b36d

Signed-off-by: dimapihtar <[email protected]>

reset config

a1c8e41

Signed-off-by: dimapihtar <[email protected]>

Merge branch 'main' into dpykhtar/te_mcore_upgrade

148e331

github-actions bot added the NLP label Nov 6, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

5dccef6

for more information, see https://pre-commit.ci

dimapihtar requested a review from ericharper November 6, 2023 15:59

ericharper and others added 9 commits November 6, 2023 19:00

Merge branch 'main' into dpykhtar/te_mcore_upgrade

177cec7

pass attn_mask-type through the forward method

22ee169

Signed-off-by: dimapihtar <[email protected]>

reset conf

5ab2655

Signed-off-by: dimapihtar <[email protected]>

add more attn_mask_type params

79f95b2

Signed-off-by: dimapihtar <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

d02ec45

for more information, see https://pre-commit.ci

attn_,ask_type fixes

a7368fc

Signed-off-by: dimapihtar <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

8ed2cf1

for more information, see https://pre-commit.ci

remove attn_mask_type param

989b9a5

Signed-off-by: dimapihtar <[email protected]>

t5/mt5 fix

d1b7189

Signed-off-by: dimapihtar <[email protected]>

ericharper reviewed Nov 10, 2023

View reviewed changes

examples/nlp/language_modeling/conf/megatron_gpt_config.yaml Outdated Show resolved Hide resolved

ericharper reviewed Nov 10, 2023

View reviewed changes

examples/nlp/language_modeling/conf/megatron_gpt_inference.yaml Outdated Show resolved Hide resolved

ericharper reviewed Nov 10, 2023

View reviewed changes

examples/nlp/language_modeling/conf/megatron_gpt_inference.yaml Outdated Show resolved Hide resolved

dimapihtar and others added 4 commits November 10, 2023 06:37

revert configs

683b277

Signed-off-by: dimapihtar <[email protected]>

revert configs

6518f88

Signed-off-by: dimapihtar <[email protected]>

Merge branch 'main' into dpykhtar/te_mcore_upgrade

d11cadd

Signed-off-by: Dmytro Pykhtar <[email protected]>

Merge branch 'main' into dpykhtar/te_mcore_upgrade

3acaf9c

ericharper requested review from aklife97 and gshennvm November 15, 2023 17:16

dimapihtar and others added 2 commits November 16, 2023 06:52

remove attn_mask_type param

6963155

Signed-off-by: dimapihtar <[email protected]>

Merge branch 'main' into dpykhtar/te_mcore_upgrade

3693888

add TE installation

57c2487

Signed-off-by: dimapihtar <[email protected]>

dimapihtar and others added 2 commits November 16, 2023 13:37

add env var

68895b9

Signed-off-by: dimapihtar <[email protected]>

Merge branch 'main' into dpykhtar/te_mcore_upgrade

83c708b

comment out rope test

4aff223

Signed-off-by: dimapihtar <[email protected]>

ericharper reviewed Nov 16, 2023

View reviewed changes

Jenkinsfile Show resolved Hide resolved

ericharper reviewed Nov 16, 2023

View reviewed changes

change mcore installation

197f26a

Signed-off-by: dimapihtar <[email protected]>

update mcore commit

a5390c9

Signed-off-by: dimapihtar <[email protected]>

change mcore installation

fbfbe3a

Signed-off-by: dimapihtar <[email protected]>

change mcore commit

d5d0a6f

Signed-off-by: dimapihtar <[email protected]>

ericharper and others added 3 commits November 16, 2023 20:04

Merge branch 'main' into dpykhtar/te_mcore_upgrade

447507a

add -e

0ad5d7f

Signed-off-by: eharper <[email protected]>

revert

fda6c93

Signed-off-by: eharper <[email protected]>

revert jenkins test comment

d102831

Signed-off-by: eharper <[email protected]>

ericharper approved these changes Nov 17, 2023

View reviewed changes

dimapihtar merged commit ce6afd7 into main Nov 17, 2023
15 checks passed

dimapihtar deleted the dpykhtar/te_mcore_upgrade branch November 17, 2023 13:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade NeMo to latest mcore and TE #7862

Upgrade NeMo to latest mcore and TE #7862

dimapihtar commented Nov 6, 2023

ericharper commented Nov 15, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

ericharper Nov 16, 2023

dimapihtar Nov 16, 2023 •

edited

Loading

ericharper Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 17, 2023

ericharper commented Nov 17, 2023

ericharper commented Nov 17, 2023

ericharper left a comment

Upgrade NeMo to latest mcore and TE #7862

Upgrade NeMo to latest mcore and TE #7862

Conversation

dimapihtar commented Nov 6, 2023

What does this PR do ?

Changelog

Usage

Before your PR is "Ready for review"

Who can review?

Additional Information

ericharper commented Nov 15, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

ericharper Nov 16, 2023

Choose a reason for hiding this comment

dimapihtar Nov 16, 2023 • edited Loading

Choose a reason for hiding this comment

ericharper Nov 16, 2023

Choose a reason for hiding this comment

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 17, 2023

ericharper commented Nov 17, 2023

ericharper commented Nov 17, 2023

ericharper left a comment

Choose a reason for hiding this comment

dimapihtar Nov 16, 2023 •

edited

Loading