Prompt learning of Huggingface T5v1.1 converted checkpoints #4746

MaximumEntropy · 2022-08-17T03:24:23Z

What does this PR do ?

Makes the changes necessary to do prompt learning of T5v1.1-converted checkpoints from HF.

Collection: NLP

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: ericharper <[email protected]>

Signed-off-by: Jason <[email protected]>

@blisc

* [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]>

Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]>

Signed-off-by: MaximumEntropy <[email protected]>

Signed-off-by: ericharper <[email protected]>

Signed-off-by: Jason <[email protected]>

@blisc

* [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]>

Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]>

…Mo into megatron_t51_1_compat

Signed-off-by: MaximumEntropy <[email protected]>

…egatron_t51_1_compat

Signed-off-by: MaximumEntropy <[email protected]>

Signed-off-by: Abhinav Khattar <[email protected]>

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-10-04T22:44:05Z

This pull request introduces 3 alerts when merging a724c34 into a71712b - view on LGTM.com

new alerts:

2 for Unused import
1 for Variable defined multiple times

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-10-06T23:00:02Z

This pull request introduces 3 alerts when merging cd31acc into 0d5ed9b - view on LGTM.com

new alerts:

2 for Unused import
1 for Variable defined multiple times

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-10-10T21:14:08Z

This pull request introduces 3 alerts when merging 79543ea into cfd3682 - view on LGTM.com

new alerts:

2 for Unused import
1 for Variable defined multiple times

Signed-off-by: MaximumEntropy <[email protected]>

for more information, see https://pre-commit.ci

lgtm-com · 2022-10-12T23:03:53Z

This pull request introduces 2 alerts and fixes 2 when merging ae46376 into bd377b7 - view on LGTM.com

new alerts:

1 for Unused import
1 for Variable defined multiple times

fixed alerts:

2 for Unused import

Signed-off-by: MaximumEntropy <[email protected]>

lgtm-com · 2022-10-13T21:31:24Z

This pull request introduces 2 alerts and fixes 2 when merging ac9dcf6 into 4366699 - view on LGTM.com

new alerts:

1 for Unused import
1 for Variable defined multiple times

fixed alerts:

2 for Unused import

vadam5

Looks great! Just left a few comments

nemo/collections/nlp/models/language_modeling/megatron_finetune_model.py

vadam5 · 2022-10-13T23:15:43Z

nemo/collections/nlp/data/language_modeling/megatron/t5_prompt_learning_dataset.py


            # Add BOS/EOS to the input of encoder if desired, adds EOS by default
+            if self.ul2_prompt_token is not None:


Should probably add the ul2_prompt_token after the virtual prompt tokens if there are some at the beginning of the text input.

nemo/collections/nlp/models/language_modeling/megatron_t5_adapter_model.py

vadam5 · 2022-10-13T23:32:03Z

nemo/collections/nlp/models/language_modeling/megatron_t5_adapter_model.py

-            'predicted_token_ids': processed_preds,
-            'log_probs': log_probs,
-            'labels': processed_labels,
+            'input_text': input_text,


Some down stream code might expect these key values, we should check with @arendu that this doesn't break anything he is aware of.

scripts/nlp_language_modeling/hf_t5-v1_1_to_nemo.py

Signed-off-by: MaximumEntropy <[email protected]>

for more information, see https://pre-commit.ci

lgtm-com · 2022-10-14T02:29:26Z

This pull request introduces 3 alerts and fixes 2 when merging 203c67b into 4fc5385 - view on LGTM.com

new alerts:

2 for Unused import
1 for Variable defined multiple times

fixed alerts:

2 for Unused import

vadam5

LGTM!

@blisc

) * update branch Signed-off-by: ericharper <[email protected]> * update package info and dockerfile Signed-off-by: ericharper <[email protected]> * fix fastpitch export (NVIDIA#4676) Signed-off-by: Jason <[email protected]> * [TTS] fixed wrong pronunciations for r1.11. (NVIDIA#4677) * [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]> * Fix for incorrect batch size issue while decoding (NVIDIA#4675) Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Initial Signed-off-by: MaximumEntropy <[email protected]> * Fix for RPE Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * update branch Signed-off-by: ericharper <[email protected]> * update package info and dockerfile Signed-off-by: ericharper <[email protected]> * fix fastpitch export (NVIDIA#4676) Signed-off-by: Jason <[email protected]> * [TTS] fixed wrong pronunciations for r1.11. (NVIDIA#4677) * [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]> * Fix for incorrect batch size issue while decoding (NVIDIA#4675) Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Make megatron legacy configurable Signed-off-by: MaximumEntropy <[email protected]> * Enc-Dec checksum matching Signed-off-by: MaximumEntropy <[email protected]> * Add conversion script Signed-off-by: MaximumEntropy <[email protected]> * Reset files Signed-off-by: MaximumEntropy <[email protected]> * Reset docker and jenkinsfile Signed-off-by: MaximumEntropy <[email protected]> * Reset README Signed-off-by: MaximumEntropy <[email protected]> * Remove tts scripts files Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Update finetuning script Signed-off-by: MaximumEntropy <[email protected]> * add cloning Signed-off-by: Abhinav Khattar <[email protected]> * map to cpu Signed-off-by: Abhinav Khattar <[email protected]> * Fix TP change for HF exported models Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Update conversion script and style Signed-off-by: MaximumEntropy <[email protected]> * Add base config Signed-off-by: MaximumEntropy <[email protected]> * Add arg Signed-off-by: MaximumEntropy <[email protected]> * Change partition comment update Signed-off-by: MaximumEntropy <[email protected]> * Update base config Signed-off-by: MaximumEntropy <[email protected]> * Minor fix for prompt learning Signed-off-by: MaximumEntropy <[email protected]> * style Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix default Signed-off-by: MaximumEntropy <[email protected]> * Fix to latest ptl Signed-off-by: MaximumEntropy <[email protected]> * Add arg to perceiver Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Temporarily add Signed-off-by: MaximumEntropy <[email protected]> * Restore Signed-off-by: MaximumEntropy <[email protected]> * Move tokens head bias to cfg population Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * Empty Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fixes to get decode to work. Signed-off-by: MaximumEntropy <[email protected]> * More changes Signed-off-by: MaximumEntropy <[email protected]> * Update base config Signed-off-by: MaximumEntropy <[email protected]> * Test Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Update config to 0 dropout Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Reset file Signed-off-by: MaximumEntropy <[email protected]> * Remove scheduler Signed-off-by: MaximumEntropy <[email protected]> * Changes Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Support generic bos id Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Minor Signed-off-by: MaximumEntropy <[email protected]> * Minor Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * Minor changes Signed-off-by: MaximumEntropy <[email protected]> * Add embedding dropout Signed-off-by: MaximumEntropy <[email protected]> * Changes for ul2 Signed-off-by: MaximumEntropy <[email protected]> * Fix for pad id Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Update models that can be converted Signed-off-by: MaximumEntropy <[email protected]> * Fix inference Signed-off-by: MaximumEntropy <[email protected]> * Remove ipdb Signed-off-by: MaximumEntropy <[email protected]> * Fix typo Signed-off-by: MaximumEntropy <[email protected]> * Load ul2 in bf16 Signed-off-by: MaximumEntropy <[email protected]> * Add amp o2 arg Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Tmp Signed-off-by: MaximumEntropy <[email protected]> * Fix rmsnorm Signed-off-by: MaximumEntropy <[email protected]> * Reset config Signed-off-by: MaximumEntropy <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix eval for converted models Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Update predict step for adapters Signed-off-by: MaximumEntropy <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Minor Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: ericharper <[email protected]> Signed-off-by: Jason <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: ericharper <[email protected]> Co-authored-by: Jason <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Rajesh Ilango <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Co-authored-by: Oleksii Kuchaiev <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: 1-800-bad-code <[email protected]>

@blisc

) * update branch Signed-off-by: ericharper <[email protected]> * update package info and dockerfile Signed-off-by: ericharper <[email protected]> * fix fastpitch export (NVIDIA#4676) Signed-off-by: Jason <[email protected]> * [TTS] fixed wrong pronunciations for r1.11. (NVIDIA#4677) * [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]> * Fix for incorrect batch size issue while decoding (NVIDIA#4675) Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Initial Signed-off-by: MaximumEntropy <[email protected]> * Fix for RPE Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * update branch Signed-off-by: ericharper <[email protected]> * update package info and dockerfile Signed-off-by: ericharper <[email protected]> * fix fastpitch export (NVIDIA#4676) Signed-off-by: Jason <[email protected]> * [TTS] fixed wrong pronunciations for r1.11. (NVIDIA#4677) * [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]> * Fix for incorrect batch size issue while decoding (NVIDIA#4675) Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Make megatron legacy configurable Signed-off-by: MaximumEntropy <[email protected]> * Enc-Dec checksum matching Signed-off-by: MaximumEntropy <[email protected]> * Add conversion script Signed-off-by: MaximumEntropy <[email protected]> * Reset files Signed-off-by: MaximumEntropy <[email protected]> * Reset docker and jenkinsfile Signed-off-by: MaximumEntropy <[email protected]> * Reset README Signed-off-by: MaximumEntropy <[email protected]> * Remove tts scripts files Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Update finetuning script Signed-off-by: MaximumEntropy <[email protected]> * add cloning Signed-off-by: Abhinav Khattar <[email protected]> * map to cpu Signed-off-by: Abhinav Khattar <[email protected]> * Fix TP change for HF exported models Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Update conversion script and style Signed-off-by: MaximumEntropy <[email protected]> * Add base config Signed-off-by: MaximumEntropy <[email protected]> * Add arg Signed-off-by: MaximumEntropy <[email protected]> * Change partition comment update Signed-off-by: MaximumEntropy <[email protected]> * Update base config Signed-off-by: MaximumEntropy <[email protected]> * Minor fix for prompt learning Signed-off-by: MaximumEntropy <[email protected]> * style Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix default Signed-off-by: MaximumEntropy <[email protected]> * Fix to latest ptl Signed-off-by: MaximumEntropy <[email protected]> * Add arg to perceiver Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Temporarily add Signed-off-by: MaximumEntropy <[email protected]> * Restore Signed-off-by: MaximumEntropy <[email protected]> * Move tokens head bias to cfg population Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * Empty Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fixes to get decode to work. Signed-off-by: MaximumEntropy <[email protected]> * More changes Signed-off-by: MaximumEntropy <[email protected]> * Update base config Signed-off-by: MaximumEntropy <[email protected]> * Test Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Update config to 0 dropout Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Reset file Signed-off-by: MaximumEntropy <[email protected]> * Remove scheduler Signed-off-by: MaximumEntropy <[email protected]> * Changes Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Support generic bos id Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Minor Signed-off-by: MaximumEntropy <[email protected]> * Minor Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * Minor changes Signed-off-by: MaximumEntropy <[email protected]> * Add embedding dropout Signed-off-by: MaximumEntropy <[email protected]> * Changes for ul2 Signed-off-by: MaximumEntropy <[email protected]> * Fix for pad id Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Update models that can be converted Signed-off-by: MaximumEntropy <[email protected]> * Fix inference Signed-off-by: MaximumEntropy <[email protected]> * Remove ipdb Signed-off-by: MaximumEntropy <[email protected]> * Fix typo Signed-off-by: MaximumEntropy <[email protected]> * Load ul2 in bf16 Signed-off-by: MaximumEntropy <[email protected]> * Add amp o2 arg Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Tmp Signed-off-by: MaximumEntropy <[email protected]> * Fix rmsnorm Signed-off-by: MaximumEntropy <[email protected]> * Reset config Signed-off-by: MaximumEntropy <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix eval for converted models Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Update predict step for adapters Signed-off-by: MaximumEntropy <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Minor Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: ericharper <[email protected]> Signed-off-by: Jason <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: ericharper <[email protected]> Co-authored-by: Jason <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Rajesh Ilango <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Co-authored-by: Oleksii Kuchaiev <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Hainan Xu <[email protected]>

@blisc

) * update branch Signed-off-by: ericharper <[email protected]> * update package info and dockerfile Signed-off-by: ericharper <[email protected]> * fix fastpitch export (NVIDIA#4676) Signed-off-by: Jason <[email protected]> * [TTS] fixed wrong pronunciations for r1.11. (NVIDIA#4677) * [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]> * Fix for incorrect batch size issue while decoding (NVIDIA#4675) Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Initial Signed-off-by: MaximumEntropy <[email protected]> * Fix for RPE Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * update branch Signed-off-by: ericharper <[email protected]> * update package info and dockerfile Signed-off-by: ericharper <[email protected]> * fix fastpitch export (NVIDIA#4676) Signed-off-by: Jason <[email protected]> * [TTS] fixed wrong pronunciations for r1.11. (NVIDIA#4677) * [TTS] fixed wrong pronunciations. Signed-off-by: Xuesong Yang <[email protected]> * incremented the version number to 22.08 as @blisc suggested. Signed-off-by: Xuesong Yang <[email protected]> * correct cmudict versions in world-wide places. Signed-off-by: Xuesong Yang <[email protected]> * Fix for incorrect batch size issue while decoding (NVIDIA#4675) Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Make megatron legacy configurable Signed-off-by: MaximumEntropy <[email protected]> * Enc-Dec checksum matching Signed-off-by: MaximumEntropy <[email protected]> * Add conversion script Signed-off-by: MaximumEntropy <[email protected]> * Reset files Signed-off-by: MaximumEntropy <[email protected]> * Reset docker and jenkinsfile Signed-off-by: MaximumEntropy <[email protected]> * Reset README Signed-off-by: MaximumEntropy <[email protected]> * Remove tts scripts files Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Update finetuning script Signed-off-by: MaximumEntropy <[email protected]> * add cloning Signed-off-by: Abhinav Khattar <[email protected]> * map to cpu Signed-off-by: Abhinav Khattar <[email protected]> * Fix TP change for HF exported models Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Update conversion script and style Signed-off-by: MaximumEntropy <[email protected]> * Add base config Signed-off-by: MaximumEntropy <[email protected]> * Add arg Signed-off-by: MaximumEntropy <[email protected]> * Change partition comment update Signed-off-by: MaximumEntropy <[email protected]> * Update base config Signed-off-by: MaximumEntropy <[email protected]> * Minor fix for prompt learning Signed-off-by: MaximumEntropy <[email protected]> * style Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix default Signed-off-by: MaximumEntropy <[email protected]> * Fix to latest ptl Signed-off-by: MaximumEntropy <[email protected]> * Add arg to perceiver Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Temporarily add Signed-off-by: MaximumEntropy <[email protected]> * Restore Signed-off-by: MaximumEntropy <[email protected]> * Move tokens head bias to cfg population Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * Empty Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fixes to get decode to work. Signed-off-by: MaximumEntropy <[email protected]> * More changes Signed-off-by: MaximumEntropy <[email protected]> * Update base config Signed-off-by: MaximumEntropy <[email protected]> * Test Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Update config to 0 dropout Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Reset file Signed-off-by: MaximumEntropy <[email protected]> * Remove scheduler Signed-off-by: MaximumEntropy <[email protected]> * Changes Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Support generic bos id Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Minor Signed-off-by: MaximumEntropy <[email protected]> * Minor Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * Minor changes Signed-off-by: MaximumEntropy <[email protected]> * Add embedding dropout Signed-off-by: MaximumEntropy <[email protected]> * Changes for ul2 Signed-off-by: MaximumEntropy <[email protected]> * Fix for pad id Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Update models that can be converted Signed-off-by: MaximumEntropy <[email protected]> * Fix inference Signed-off-by: MaximumEntropy <[email protected]> * Remove ipdb Signed-off-by: MaximumEntropy <[email protected]> * Fix typo Signed-off-by: MaximumEntropy <[email protected]> * Load ul2 in bf16 Signed-off-by: MaximumEntropy <[email protected]> * Add amp o2 arg Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Tmp Signed-off-by: MaximumEntropy <[email protected]> * Fix rmsnorm Signed-off-by: MaximumEntropy <[email protected]> * Reset config Signed-off-by: MaximumEntropy <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix eval for converted models Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Update predict step for adapters Signed-off-by: MaximumEntropy <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Minor Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: ericharper <[email protected]> Signed-off-by: Jason <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: ericharper <[email protected]> Co-authored-by: Jason <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Rajesh Ilango <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Co-authored-by: Oleksii Kuchaiev <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Hainan Xu <[email protected]>

ericharper and others added 30 commits August 4, 2022 11:08

update branch

fef39e0

Signed-off-by: ericharper <[email protected]>

update package info and dockerfile

5bc1f11

Signed-off-by: ericharper <[email protected]>

fix fastpitch export (#4676)

7450b4a

Signed-off-by: Jason <[email protected]>

Fix for incorrect batch size issue while decoding (#4675)

816ffda

Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]>

Initial

e0815ff

Signed-off-by: MaximumEntropy <[email protected]>

Fix for RPE

c00319a

Signed-off-by: MaximumEntropy <[email protected]>

Style

a0935dc

Signed-off-by: MaximumEntropy <[email protected]>

update branch

cae78e2

Signed-off-by: ericharper <[email protected]>

update package info and dockerfile

aa32455

Signed-off-by: ericharper <[email protected]>

fix fastpitch export (#4676)

98dd24f

Signed-off-by: Jason <[email protected]>

Fix for incorrect batch size issue while decoding (#4675)

1fd7883

Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Eric Harper <[email protected]>

Merge branch 'megatron_encode_decode_rpe_fix' of github.com:NVIDIA/Ne…

419f1fd

…Mo into megatron_t51_1_compat

Make megatron legacy configurable

0e7850c

Signed-off-by: MaximumEntropy <[email protected]>

Merge branch 'megatron_t51_1_compat' of github.com:NVIDIA/NeMo into m…

f5a91c3

…egatron_t51_1_compat

Enc-Dec checksum matching

147778e

Signed-off-by: MaximumEntropy <[email protected]>

Add conversion script

e3c734e

Signed-off-by: MaximumEntropy <[email protected]>

Merge branch 'main' of github.com:NVIDIA/NeMo into megatron_t51_1_compat

0c0bf31

Reset files

0fb6823

Signed-off-by: MaximumEntropy <[email protected]>

Reset docker and jenkinsfile

c122bb1

Signed-off-by: MaximumEntropy <[email protected]>

Reset README

5fd0d21

Signed-off-by: MaximumEntropy <[email protected]>

Remove tts scripts files

5ec1abf

Signed-off-by: MaximumEntropy <[email protected]>

Style

88fb0a5

Signed-off-by: MaximumEntropy <[email protected]>

Update finetuning script

984621d

Signed-off-by: MaximumEntropy <[email protected]>

add cloning

1b83847

Signed-off-by: Abhinav Khattar <[email protected]>

map to cpu

0874f69

Signed-off-by: Abhinav Khattar <[email protected]>

Fix TP change for HF exported models

69e5865

Signed-off-by: MaximumEntropy <[email protected]>

Fix

479aa66

Signed-off-by: MaximumEntropy <[email protected]>

Fix

73d3cc1

Signed-off-by: MaximumEntropy <[email protected]>

MaximumEntropy added 3 commits October 4, 2022 15:29

merge main

4a545b5

Signed-off-by: MaximumEntropy <[email protected]>

Fix

2166d3e

Signed-off-by: MaximumEntropy <[email protected]>

Style

a724c34

Signed-off-by: MaximumEntropy <[email protected]>

MaximumEntropy marked this pull request as ready for review October 6, 2022 22:41

Fix

cd31acc

Signed-off-by: MaximumEntropy <[email protected]>

MaximumEntropy requested review from vadam5 and ericharper October 6, 2022 22:45

merge

79543ea

Signed-off-by: MaximumEntropy <[email protected]>

MaximumEntropy and others added 3 commits October 12, 2022 15:07

Merge branch 'main' of github.com:NVIDIA/NeMo into ul2_prompt_learning

096c210

Update predict step for adapters

af8fb06

Signed-off-by: MaximumEntropy <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

ae46376

for more information, see https://pre-commit.ci

Minor

ac9dcf6

Signed-off-by: MaximumEntropy <[email protected]>

vadam5 suggested changes Oct 13, 2022

View reviewed changes

MaximumEntropy and others added 3 commits October 13, 2022 19:15

Fixes

3e7652b

Signed-off-by: MaximumEntropy <[email protected]>

Merge branch 'main' into ul2_prompt_learning

cb66ffe

[pre-commit.ci] auto fixes from pre-commit.com hooks

203c67b

for more information, see https://pre-commit.ci

vadam5 self-requested a review October 14, 2022 18:21

vadam5 approved these changes Oct 14, 2022

View reviewed changes

MaximumEntropy merged commit bc48db3 into main Oct 15, 2022

MaximumEntropy deleted the ul2_prompt_learning branch October 15, 2022 01:10

arendu mentioned this pull request Oct 18, 2022

fix to t5 adapter and ia3 eval scripts #5193

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prompt learning of Huggingface T5v1.1 converted checkpoints #4746

Prompt learning of Huggingface T5v1.1 converted checkpoints #4746

MaximumEntropy commented Aug 17, 2022

lgtm-com bot commented Oct 4, 2022

lgtm-com bot commented Oct 6, 2022

lgtm-com bot commented Oct 10, 2022

lgtm-com bot commented Oct 12, 2022

lgtm-com bot commented Oct 13, 2022

vadam5 left a comment

vadam5 Oct 13, 2022

vadam5 Oct 13, 2022 •

edited

Loading

lgtm-com bot commented Oct 14, 2022

vadam5 left a comment


		# Add BOS/EOS to the input of encoder if desired, adds EOS by default
		if self.ul2_prompt_token is not None:

Prompt learning of Huggingface T5v1.1 converted checkpoints #4746

Prompt learning of Huggingface T5v1.1 converted checkpoints #4746

Conversation

MaximumEntropy commented Aug 17, 2022

What does this PR do ?

Changelog

Usage

Before your PR is "Ready for review"

Who can review?

Additional Information

lgtm-com bot commented Oct 4, 2022

lgtm-com bot commented Oct 6, 2022

lgtm-com bot commented Oct 10, 2022

lgtm-com bot commented Oct 12, 2022

lgtm-com bot commented Oct 13, 2022

vadam5 left a comment

Choose a reason for hiding this comment

vadam5 Oct 13, 2022

Choose a reason for hiding this comment

vadam5 Oct 13, 2022 • edited Loading

Choose a reason for hiding this comment

lgtm-com bot commented Oct 14, 2022

vadam5 left a comment

Choose a reason for hiding this comment

vadam5 Oct 13, 2022 •

edited

Loading