Megatron Encoder Decoder models with RPE and PP > 2 #4663

MaximumEntropy · 2022-08-02T23:26:21Z

Signed-off-by: MaximumEntropy [email protected]

What does this PR do ?

Adds support for Enc-Dec models with RPE and PP > 2

Collection: NLP

Changelog

Add specific line by line info of high level changes in this PR.

Usage

model.encoder.position_embedding_type="relative"
model.decoder.position_embedding_type="relative"
model.pipeline_model_parallel_size=4
model.pipeline_model_parallel_split_rank=2

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: MaximumEntropy <[email protected]>

…n_rpe_pp

Signed-off-by: ericharper <[email protected]>

Signed-off-by: Jason <[email protected]>

@blisc

* [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]>

Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]>

* [TTS] incremented the version number to 22.08 in tutorials. Signed-off-by: Xuesong Yang <[email protected]>

* Fix for RPE Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]>

Signed-off-by: nithinraok <[email protected]>

Signed-off-by: MaximumEntropy <[email protected]>

khcs

Outside of those where I left comments, we made a few changes to the files to make it work.

nemo/collections/nlp/models/language_modeling/megatron_lm_encoder_decoder_model.py

nemo/collections/nlp/modules/common/megatron/token_level_encoder_decoder.py

… megatron_rpe_pp

khcs · 2022-08-15T23:49:40Z

Currently working with PP>2.
Continuing testing with different configs like bf16, O2, PP>4, different encoder/decoder RPE settings, etc.

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-08-16T23:43:44Z

This pull request introduces 2 alerts when merging f5a6d03 into 06ad865 - view on LGTM.com

new alerts:

2 for Unused import

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-08-17T18:48:44Z

This pull request introduces 2 alerts when merging 7711620 into 4bf54b7 - view on LGTM.com

new alerts:

2 for Unused import

Signed-off-by: MaximumEntropy <[email protected]>

…n_rpe_pp

lgtm-com · 2022-08-18T00:17:00Z

This pull request introduces 2 alerts when merging 187d1fd into 8845add - view on LGTM.com

new alerts:

2 for Unused import

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-08-18T18:05:34Z

This pull request introduces 2 alerts when merging 4526417 into 8845add - view on LGTM.com

new alerts:

2 for Unused import

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-08-18T21:12:50Z

This pull request introduces 2 alerts when merging 9b7a9ee into 6abfbbf - view on LGTM.com

new alerts:

2 for Unused import

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-08-22T22:41:36Z

This pull request introduces 2 alerts when merging bba0335 into 4a1fcc6 - view on LGTM.com

new alerts:

2 for Unused import

Signed-off-by: MaximumEntropy <[email protected]>

nemo/collections/nlp/models/language_modeling/megatron_lm_encoder_decoder_model.py

Signed-off-by: MaximumEntropy <[email protected]>

khcs

ready to be merged.

@blisc

* Enc-Dec RPE PP > 2 Signed-off-by: MaximumEntropy <[email protected]> * Syle Signed-off-by: MaximumEntropy <[email protected]> * update branch Signed-off-by: ericharper <[email protected]> * update package info and dockerfile Signed-off-by: ericharper <[email protected]> * fix fastpitch export (#4676) Signed-off-by: Jason <[email protected]> * [TTS] fixed wrong pronunciations for r1.11. (#4677) * [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]> * Fix for incorrect batch size issue while decoding (#4675) Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]> * [TTS] incremented the version number to 22.08 in tutorials. (#4684) * [TTS] incremented the version number to 22.08 in tutorials. Signed-off-by: Xuesong Yang <[email protected]> * Megatron encode function with RPE fix (#4692) * Fix for RPE Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * fix to fetch config file (#4699) Signed-off-by: nithinraok <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix rpe check Signed-off-by: MaximumEntropy <[email protected]> * Revert to main Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Revert more files Signed-off-by: MaximumEntropy <[email protected]> * Additional changes to make PP>2 with RPE work. * Fixes for O2 Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * Update Signed-off-by: MaximumEntropy <[email protected]> * Fix cross attention sync check Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix zero init Signed-off-by: MaximumEntropy <[email protected]> * Modify .data instead of the weight Signed-off-by: MaximumEntropy <[email protected]> * Add worker changes Signed-off-by: MaximumEntropy <[email protected]> * Set workers to 0 for validation and test Signed-off-by: MaximumEntropy <[email protected]> * Remove unused imports and allreduce func Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Revert workers change Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: ericharper <[email protected]> Signed-off-by: Jason <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: nithinraok <[email protected]> Co-authored-by: ericharper <[email protected]> Co-authored-by: Jason <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Rajesh Ilango <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Hoo Chang Shin <[email protected]> Co-authored-by: khcs <[email protected]>

@blisc

* Enc-Dec RPE PP > 2 Signed-off-by: MaximumEntropy <[email protected]> * Syle Signed-off-by: MaximumEntropy <[email protected]> * update branch Signed-off-by: ericharper <[email protected]> * update package info and dockerfile Signed-off-by: ericharper <[email protected]> * fix fastpitch export (NVIDIA#4676) Signed-off-by: Jason <[email protected]> * [TTS] fixed wrong pronunciations for r1.11. (NVIDIA#4677) * [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]> * Fix for incorrect batch size issue while decoding (NVIDIA#4675) Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]> * [TTS] incremented the version number to 22.08 in tutorials. (NVIDIA#4684) * [TTS] incremented the version number to 22.08 in tutorials. Signed-off-by: Xuesong Yang <[email protected]> * Megatron encode function with RPE fix (NVIDIA#4692) * Fix for RPE Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * fix to fetch config file (NVIDIA#4699) Signed-off-by: nithinraok <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix rpe check Signed-off-by: MaximumEntropy <[email protected]> * Revert to main Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Revert more files Signed-off-by: MaximumEntropy <[email protected]> * Additional changes to make PP>2 with RPE work. * Fixes for O2 Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * Update Signed-off-by: MaximumEntropy <[email protected]> * Fix cross attention sync check Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix zero init Signed-off-by: MaximumEntropy <[email protected]> * Modify .data instead of the weight Signed-off-by: MaximumEntropy <[email protected]> * Add worker changes Signed-off-by: MaximumEntropy <[email protected]> * Set workers to 0 for validation and test Signed-off-by: MaximumEntropy <[email protected]> * Remove unused imports and allreduce func Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Revert workers change Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: ericharper <[email protected]> Signed-off-by: Jason <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: nithinraok <[email protected]> Co-authored-by: ericharper <[email protected]> Co-authored-by: Jason <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Rajesh Ilango <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Hoo Chang Shin <[email protected]> Co-authored-by: khcs <[email protected]> Signed-off-by: Matvei Novikov <[email protected]>

@blisc

* Enc-Dec RPE PP > 2 Signed-off-by: MaximumEntropy <[email protected]> * Syle Signed-off-by: MaximumEntropy <[email protected]> * update branch Signed-off-by: ericharper <[email protected]> * update package info and dockerfile Signed-off-by: ericharper <[email protected]> * fix fastpitch export (NVIDIA#4676) Signed-off-by: Jason <[email protected]> * [TTS] fixed wrong pronunciations for r1.11. (NVIDIA#4677) * [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]> * Fix for incorrect batch size issue while decoding (NVIDIA#4675) Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]> * [TTS] incremented the version number to 22.08 in tutorials. (NVIDIA#4684) * [TTS] incremented the version number to 22.08 in tutorials. Signed-off-by: Xuesong Yang <[email protected]> * Megatron encode function with RPE fix (NVIDIA#4692) * Fix for RPE Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * fix to fetch config file (NVIDIA#4699) Signed-off-by: nithinraok <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix rpe check Signed-off-by: MaximumEntropy <[email protected]> * Revert to main Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Revert more files Signed-off-by: MaximumEntropy <[email protected]> * Additional changes to make PP>2 with RPE work. * Fixes for O2 Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * Update Signed-off-by: MaximumEntropy <[email protected]> * Fix cross attention sync check Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix zero init Signed-off-by: MaximumEntropy <[email protected]> * Modify .data instead of the weight Signed-off-by: MaximumEntropy <[email protected]> * Add worker changes Signed-off-by: MaximumEntropy <[email protected]> * Set workers to 0 for validation and test Signed-off-by: MaximumEntropy <[email protected]> * Remove unused imports and allreduce func Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Revert workers change Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: ericharper <[email protected]> Signed-off-by: Jason <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: nithinraok <[email protected]> Co-authored-by: ericharper <[email protected]> Co-authored-by: Jason <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Rajesh Ilango <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Hoo Chang Shin <[email protected]> Co-authored-by: khcs <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

@blisc

* Enc-Dec RPE PP > 2 Signed-off-by: MaximumEntropy <[email protected]> * Syle Signed-off-by: MaximumEntropy <[email protected]> * update branch Signed-off-by: ericharper <[email protected]> * update package info and dockerfile Signed-off-by: ericharper <[email protected]> * fix fastpitch export (NVIDIA#4676) Signed-off-by: Jason <[email protected]> * [TTS] fixed wrong pronunciations for r1.11. (NVIDIA#4677) * [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]> * Fix for incorrect batch size issue while decoding (NVIDIA#4675) Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]> * [TTS] incremented the version number to 22.08 in tutorials. (NVIDIA#4684) * [TTS] incremented the version number to 22.08 in tutorials. Signed-off-by: Xuesong Yang <[email protected]> * Megatron encode function with RPE fix (NVIDIA#4692) * Fix for RPE Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * fix to fetch config file (NVIDIA#4699) Signed-off-by: nithinraok <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix rpe check Signed-off-by: MaximumEntropy <[email protected]> * Revert to main Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Revert more files Signed-off-by: MaximumEntropy <[email protected]> * Additional changes to make PP>2 with RPE work. * Fixes for O2 Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * Update Signed-off-by: MaximumEntropy <[email protected]> * Fix cross attention sync check Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix zero init Signed-off-by: MaximumEntropy <[email protected]> * Modify .data instead of the weight Signed-off-by: MaximumEntropy <[email protected]> * Add worker changes Signed-off-by: MaximumEntropy <[email protected]> * Set workers to 0 for validation and test Signed-off-by: MaximumEntropy <[email protected]> * Remove unused imports and allreduce func Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Revert workers change Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: ericharper <[email protected]> Signed-off-by: Jason <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: nithinraok <[email protected]> Co-authored-by: ericharper <[email protected]> Co-authored-by: Jason <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Rajesh Ilango <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Hoo Chang Shin <[email protected]> Co-authored-by: khcs <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

MaximumEntropy and others added 21 commits August 2, 2022 16:18

Enc-Dec RPE PP > 2

d38f9f1

Signed-off-by: MaximumEntropy <[email protected]>

Syle

7a5c07e

Signed-off-by: MaximumEntropy <[email protected]>

Merge branch 'main' into megatron_rpe_pp

6db7dd3

Merge branch 'megatron_rpe_pp' of github.com:NVIDIA/NeMo into megatro…

177c55e

…n_rpe_pp

update branch

fef39e0

Signed-off-by: ericharper <[email protected]>

update package info and dockerfile

5bc1f11

Signed-off-by: ericharper <[email protected]>

fix fastpitch export (#4676)

7450b4a

Signed-off-by: Jason <[email protected]>

Fix for incorrect batch size issue while decoding (#4675)

816ffda

Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]>

[TTS] incremented the version number to 22.08 in tutorials. (#4684)

4b33f6e

* [TTS] incremented the version number to 22.08 in tutorials. Signed-off-by: Xuesong Yang <[email protected]>

Megatron encode function with RPE fix (#4692)

7dbdddd

* Fix for RPE Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]>

fix to fetch config file (#4699)

d24e330

Signed-off-by: nithinraok <[email protected]>

Merge branch 'r1.11.0' of github.com:NVIDIA/NeMo into megatron_rpe_pp

3c6aeee

Fix

09b48d8

Signed-off-by: MaximumEntropy <[email protected]>

Style

62c3ea0

Signed-off-by: MaximumEntropy <[email protected]>

Fix

f351658

Signed-off-by: MaximumEntropy <[email protected]>

Fix rpe check

f08216a

Signed-off-by: MaximumEntropy <[email protected]>

Merge branch 'main' of github.com:NVIDIA/NeMo into megatron_rpe_pp

aad1b44

Revert to main

cb81b64

Signed-off-by: MaximumEntropy <[email protected]>

Fix

75cbe4c

Signed-off-by: MaximumEntropy <[email protected]>

Revert more files

b5cf83e

Signed-off-by: MaximumEntropy <[email protected]>

khcs self-requested a review August 15, 2022 22:49

khcs reviewed Aug 15, 2022

View reviewed changes

nemo/collections/nlp/models/language_modeling/megatron_lm_encoder_decoder_model.py Outdated Show resolved Hide resolved

nemo/collections/nlp/modules/common/megatron/token_level_encoder_decoder.py Outdated Show resolved Hide resolved

khcs added 2 commits August 15, 2022 23:01

Additional changes to make PP>2 with RPE work.

93949b6

Merge branch 'megatron_rpe_pp' of https://github.com/NVIDIA/NeMo into…

2ff39cb

… megatron_rpe_pp

khcs and others added 2 commits August 16, 2022 10:09

Merge branch 'main' into megatron_rpe_pp

9110084

Fixes for O2

f5a6d03

Signed-off-by: MaximumEntropy <[email protected]>

Fixes

9511092

Signed-off-by: MaximumEntropy <[email protected]>

Merge branch 'main' into megatron_rpe_pp

7711620

MaximumEntropy added 2 commits August 17, 2022 16:40

Fix

0dbef69

Signed-off-by: MaximumEntropy <[email protected]>

Merge branch 'megatron_rpe_pp' of github.com:NVIDIA/NeMo into megatro…

187d1fd

…n_rpe_pp

Fix

4526417

Signed-off-by: MaximumEntropy <[email protected]>

MaximumEntropy added 2 commits August 18, 2022 13:45

Fix zero init

2565f2a

Signed-off-by: MaximumEntropy <[email protected]>

Modify .data instead of the weight

9b7a9ee

Signed-off-by: MaximumEntropy <[email protected]>

MaximumEntropy added 3 commits August 22, 2022 15:22

Add worker changes

dc1abe4

Signed-off-by: MaximumEntropy <[email protected]>

Set workers to 0 for validation and test

01e099a

Signed-off-by: MaximumEntropy <[email protected]>

Merge and fix

bba0335

Signed-off-by: MaximumEntropy <[email protected]>

MaximumEntropy added 2 commits August 24, 2022 12:08

Merge branch 'main' into megatron_rpe_pp

28be040

Remove unused imports and allreduce func

d70089c

Signed-off-by: MaximumEntropy <[email protected]>

MaximumEntropy requested a review from ericharper August 24, 2022 19:11

ericharper reviewed Aug 24, 2022

View reviewed changes

nemo/collections/nlp/models/language_modeling/megatron_lm_encoder_decoder_model.py Outdated Show resolved Hide resolved

ericharper reviewed Aug 24, 2022

View reviewed changes

nemo/collections/nlp/models/language_modeling/megatron_lm_encoder_decoder_model.py Outdated Show resolved Hide resolved

MaximumEntropy and others added 3 commits August 24, 2022 14:12

Style

787d3d8

Signed-off-by: MaximumEntropy <[email protected]>

Revert workers change

ce0ca35

Signed-off-by: MaximumEntropy <[email protected]>

Merge branch 'main' into megatron_rpe_pp

c91f921

khcs self-requested a review August 26, 2022 17:58

khcs approved these changes Aug 26, 2022

View reviewed changes

khcs merged commit 4d3e335 into main Aug 26, 2022

ericharper deleted the megatron_rpe_pp branch September 20, 2022 23:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Megatron Encoder Decoder models with RPE and PP > 2 #4663

Megatron Encoder Decoder models with RPE and PP > 2 #4663

MaximumEntropy commented Aug 2, 2022

khcs left a comment

khcs commented Aug 15, 2022

lgtm-com bot commented Aug 16, 2022

lgtm-com bot commented Aug 17, 2022

lgtm-com bot commented Aug 18, 2022

lgtm-com bot commented Aug 18, 2022

lgtm-com bot commented Aug 18, 2022

lgtm-com bot commented Aug 22, 2022

khcs left a comment

Megatron Encoder Decoder models with RPE and PP > 2 #4663

Megatron Encoder Decoder models with RPE and PP > 2 #4663

Conversation

MaximumEntropy commented Aug 2, 2022

What does this PR do ?

Changelog

Usage

Before your PR is "Ready for review"

Who can review?

Additional Information

khcs left a comment

Choose a reason for hiding this comment

khcs commented Aug 15, 2022

lgtm-com bot commented Aug 16, 2022

lgtm-com bot commented Aug 17, 2022

lgtm-com bot commented Aug 18, 2022

lgtm-com bot commented Aug 18, 2022

lgtm-com bot commented Aug 18, 2022

lgtm-com bot commented Aug 22, 2022

khcs left a comment

Choose a reason for hiding this comment