Neural Machine Translation with Megatron Transformer Models (Tensor Parallel and Tarred Datasets Only) #3861

MaximumEntropy · 2022-03-19T02:56:33Z

What does this PR do ?

Adds support to train Neural Machine Translation models with Megatron Transformer models that support tensor parallelism.

Collection: NLP

Changelog

Add a MegatronNMT class that inherits from MegatronLMEncoderDecoderModel and overrides appropriate functions and calls classmethods from MTEncDecModel
Change appropriate methods in MTEncDecModel to classmethods so they can be called in MegatronNMTModel.
Add a config file and training script.

Usage

examples/nlp/machine_translation/megatron_nmt_training.py -cn aayn_base_megatron model.tensor_model_parallel_size=2

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-03-19T06:17:46Z

This pull request introduces 1 alert when merging 9d9bd77 into c4ec081 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-03-19T06:30:55Z

This pull request introduces 1 alert when merging 8968a5b into c4ec081 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

lgtm-com · 2022-03-19T06:47:07Z

This pull request introduces 1 alert when merging 5816680 into c4ec081 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-03-19T08:47:59Z

This pull request introduces 1 alert when merging fdac001 into c4ec081 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-03-19T22:52:41Z

This pull request introduces 1 alert when merging 2dcb808 into c4ec081 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-03-19T23:47:30Z

This pull request introduces 4 alerts when merging af80f37 into c4ec081 - view on LGTM.com

new alerts:

2 for Unused local variable
1 for Signature mismatch in overriding method
1 for Unreachable code

lgtm-com · 2022-03-20T00:02:01Z

This pull request introduces 4 alerts when merging 6becb3c into c4ec081 - view on LGTM.com

new alerts:

2 for Unused local variable
1 for Signature mismatch in overriding method
1 for Unreachable code

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-03-20T00:25:26Z

This pull request introduces 4 alerts when merging 9e08de6 into c4ec081 - view on LGTM.com

new alerts:

2 for Unused local variable
1 for Signature mismatch in overriding method
1 for Unreachable code

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-03-20T04:52:44Z

This pull request introduces 4 alerts when merging 21f32d1 into c4ec081 - view on LGTM.com

new alerts:

2 for Unused local variable
1 for Signature mismatch in overriding method
1 for Unreachable code

lgtm-com · 2022-03-28T20:36:41Z

This pull request introduces 4 alerts when merging 3960a0f into 41c4ab7 - view on LGTM.com

new alerts:

2 for Unused local variable
1 for Signature mismatch in overriding method
1 for Unreachable code

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-03-28T22:17:27Z

This pull request introduces 4 alerts when merging 7f805fc into 41c4ab7 - view on LGTM.com

new alerts:

2 for Unused local variable
1 for Signature mismatch in overriding method
1 for Unreachable code

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-03-28T22:41:23Z

This pull request introduces 2 alerts when merging 0dd0012 into 41c4ab7 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method
1 for Unused import

lgtm-com · 2022-03-28T22:55:59Z

This pull request introduces 1 alert when merging b632b83 into 41c4ab7 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

lgtm-com · 2022-03-29T00:10:52Z

This pull request introduces 1 alert when merging 770a7a8 into 45f8f32 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-03-29T17:06:06Z

This pull request introduces 1 alert when merging 7b0d8a5 into 45f8f32 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

lgtm-com · 2022-03-29T18:14:18Z

This pull request introduces 1 alert when merging 0ca7883 into 92dd783 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

lgtm-com · 2022-03-29T19:27:05Z

This pull request introduces 1 alert when merging 2268232 into 77acfee - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

ericharper

LGTM. Thanks!

lgtm-com · 2022-03-29T21:18:05Z

This pull request introduces 1 alert when merging 86d7099 into 70963a2 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

Signed-off-by: MaximumEntropy <[email protected]>

michalivne

LGTM! see minor comments

nemo/collections/nlp/models/machine_translation/megatron_nmt_model.py

michalivne · 2022-03-29T21:23:03Z

examples/nlp/machine_translation/noisy_channel_reranking.py

@@ -296,7 +296,12 @@ def main():
            src_lines = []
            for line in src_f:
                src_lines.append(line.strip().split('\t'))
-            assert len(all_reverse_scores) == len(all_lm_scores) == len(all_forward_scores) == len(src_lines)
+            assert len(all_reverse_scores) == len(all_lm_scores) == len(all_forward_scores) == len(src_lines), (


Remove printing out or add meaningful message

lgtm-com · 2022-03-29T21:52:29Z

This pull request introduces 1 alert when merging ea2e3a5 into 70963a2 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

lgtm-com · 2022-03-29T23:22:27Z

This pull request introduces 1 alert when merging 0e2f636 into 679f14c - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

michalivne

LGTM!

MaximumEntropy added 13 commits March 10, 2022 23:18

Initial megatron nmt

bccb554

Signed-off-by: MaximumEntropy <[email protected]>

Val step changes

f28e7bd

Signed-off-by: MaximumEntropy <[email protected]>

Minor

a2d7f53

Signed-off-by: MaximumEntropy <[email protected]>

More changes

34592af

Signed-off-by: MaximumEntropy <[email protected]>

More fixes and setup() method

3622a45

Signed-off-by: MaximumEntropy <[email protected]>

Add training script and yaml config

d5255f0

Signed-off-by: MaximumEntropy <[email protected]>

Initial working MegatronNMT

4ff0f69

Signed-off-by: MaximumEntropy <[email protected]>

Working TP=2 and train, validation sampler fix

9e3fbe1

Signed-off-by: MaximumEntropy <[email protected]>

Style fixes

82c0baf

Signed-off-by: MaximumEntropy <[email protected]>

Merge branch 'main' into megatron_nmt

2fc735b

Merge branch 'megatron_nmt' of github.com:NVIDIA/NeMo into megatron_nmt

70a8270

BLEU score changes

9d9bd77

Signed-off-by: MaximumEntropy <[email protected]>

Change config

8968a5b

Signed-off-by: MaximumEntropy <[email protected]>

Style

5816680

Signed-off-by: MaximumEntropy <[email protected]>

Remove checkpoint filename

fdac001

Signed-off-by: MaximumEntropy <[email protected]>

Fixes for multilingual and better logging

2dcb808

Signed-off-by: MaximumEntropy <[email protected]>

MaximumEntropy added 2 commits March 19, 2022 16:32

Minor changes

af80f37

Signed-off-by: MaximumEntropy <[email protected]>

Average metric logging

6becb3c

Signed-off-by: MaximumEntropy <[email protected]>

Style

9e08de6

Signed-off-by: MaximumEntropy <[email protected]>

MaximumEntropy added 2 commits March 19, 2022 21:34

Fix loss logging for multilingual

d4a1f12

Signed-off-by: MaximumEntropy <[email protected]>

lowercase false

21f32d1

Signed-off-by: MaximumEntropy <[email protected]>

MaximumEntropy added 3 commits March 28, 2022 15:00

Address PR comments

c96ac68

Signed-off-by: MaximumEntropy <[email protected]>

Merge branch 'megatron_nmt' of github.com:NVIDIA/NeMo into megatron_nmt

95154ff

Style

7f805fc

Signed-off-by: MaximumEntropy <[email protected]>

MaximumEntropy added 2 commits March 28, 2022 15:25

Undo dataloader change

0dd0012

Signed-off-by: MaximumEntropy <[email protected]>

Remove unused imports

b632b83

Signed-off-by: MaximumEntropy <[email protected]>

Merge branch 'main' into megatron_nmt

770a7a8

MaximumEntropy added 2 commits March 29, 2022 09:47

Empty commit to restart CI

356a862

Signed-off-by: MaximumEntropy <[email protected]>

Merge branch 'megatron_nmt' of github.com:NVIDIA/NeMo into megatron_nmt

7b0d8a5

Merge branch 'main' into megatron_nmt

0ca7883

Merge branch 'main' into megatron_nmt

2268232

Merge branch 'main' into megatron_nmt

86d7099

ericharper previously approved these changes Mar 29, 2022

View reviewed changes

Improve assert

ea2e3a5

Signed-off-by: MaximumEntropy <[email protected]>

michalivne reviewed Mar 29, 2022

View reviewed changes

MaximumEntropy dismissed ericharper’s stale review via ea2e3a5 March 29, 2022 21:31

Merge branch 'main' into megatron_nmt

0e2f636

michalivne approved these changes Mar 29, 2022

View reviewed changes

michalivne merged commit ff91628 into main Mar 29, 2022

michalivne deleted the megatron_nmt branch March 29, 2022 23:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Neural Machine Translation with Megatron Transformer Models (Tensor Parallel and Tarred Datasets Only) #3861

Neural Machine Translation with Megatron Transformer Models (Tensor Parallel and Tarred Datasets Only) #3861

MaximumEntropy commented Mar 19, 2022

lgtm-com bot commented Mar 19, 2022

lgtm-com bot commented Mar 19, 2022

lgtm-com bot commented Mar 19, 2022

lgtm-com bot commented Mar 19, 2022

lgtm-com bot commented Mar 19, 2022

lgtm-com bot commented Mar 19, 2022

lgtm-com bot commented Mar 20, 2022

lgtm-com bot commented Mar 20, 2022

lgtm-com bot commented Mar 20, 2022

lgtm-com bot commented Mar 28, 2022

lgtm-com bot commented Mar 28, 2022

lgtm-com bot commented Mar 28, 2022

lgtm-com bot commented Mar 28, 2022

lgtm-com bot commented Mar 29, 2022

lgtm-com bot commented Mar 29, 2022

lgtm-com bot commented Mar 29, 2022

lgtm-com bot commented Mar 29, 2022

ericharper left a comment

lgtm-com bot commented Mar 29, 2022

michalivne left a comment

michalivne Mar 29, 2022

lgtm-com bot commented Mar 29, 2022

lgtm-com bot commented Mar 29, 2022

michalivne left a comment

Neural Machine Translation with Megatron Transformer Models (Tensor Parallel and Tarred Datasets Only) #3861

Neural Machine Translation with Megatron Transformer Models (Tensor Parallel and Tarred Datasets Only) #3861

Conversation

MaximumEntropy commented Mar 19, 2022

What does this PR do ?

Changelog

Usage

Before your PR is "Ready for review"

Who can review?

Additional Information

lgtm-com bot commented Mar 19, 2022

lgtm-com bot commented Mar 19, 2022

lgtm-com bot commented Mar 19, 2022

lgtm-com bot commented Mar 19, 2022

lgtm-com bot commented Mar 19, 2022

lgtm-com bot commented Mar 19, 2022

lgtm-com bot commented Mar 20, 2022

lgtm-com bot commented Mar 20, 2022

lgtm-com bot commented Mar 20, 2022

lgtm-com bot commented Mar 28, 2022

lgtm-com bot commented Mar 28, 2022

lgtm-com bot commented Mar 28, 2022

lgtm-com bot commented Mar 28, 2022

lgtm-com bot commented Mar 29, 2022

lgtm-com bot commented Mar 29, 2022

lgtm-com bot commented Mar 29, 2022

lgtm-com bot commented Mar 29, 2022

ericharper left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Mar 29, 2022

michalivne left a comment

Choose a reason for hiding this comment

michalivne Mar 29, 2022

Choose a reason for hiding this comment

lgtm-com bot commented Mar 29, 2022

lgtm-com bot commented Mar 29, 2022

michalivne left a comment

Choose a reason for hiding this comment