Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generalized the P-tuning method to support various NLP tasks #3623

Merged
merged 31 commits into from
Feb 14, 2022
Merged

Generalized the P-tuning method to support various NLP tasks #3623

merged 31 commits into from
Feb 14, 2022

Conversation

yidong72
Copy link
Collaborator

@yidong72 yidong72 commented Feb 7, 2022 •
edited
Loading

What does this PR do ?

It generalizes the P-tuning method to support various NLP tasks including text-cls, NER, text summation, Q/A etc. It supports multiple task with one prompt encoder model. It also includes a tutorial notebook to show how to use it.

Changelog

  • Added the decoder for the p-tuning model so it can generate any length of text.
  • The prompt encoder can be conditioned on the task-tags, so it supports multiple tasks simultaneously.
  • Added the p-tune dataset. The user just need to write their own data processor to generate customized prompts for different tasks.

Usage

Check the notebook https://github.com/NVIDIA/NeMo/blob/generalized_gpt_ptuning/tutorials/nlp/PTune_multiple_NLP_tasks.ipynb

PR Type:

  • New Feature
  • Bugfix
  • Documentation

@yidong72
memory fix
12b2aa0 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
Merge branch 'main' into generalized_gpt_ptuning
bab8c6c
@yidong72
fix exp cause inf
a370b9d 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
clean comments
0a7b206 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
added the ptune gpt model
78839cb 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
fix the decode
36ce831 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
text gen loss is done
933962f 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
working now
62dfe6b 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
fix style
e87ba8d 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
auto infer decoder len
6106733 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
evaluate on the best checkpoint
5f8410b 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
update the dataset to handle multiple tasks
6c9b908 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
added the inference logics
15ac60f 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
simplify the data processor
18f5fc6 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
task dependent
7e25803 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
multiple task working
b8cb63d 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
performance improvement
a9a78b3 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
fix the old ptune classification
9219983 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
added tutorial notebook
ce31f2a 
Signed-off-by: Yi Dong <[email protected]>
@yidong72
Merge branch 'main' into generalized_gpt_ptuning
844fc83
@lgtm-com
Copy link

lgtm-com bot commented Feb 7, 2022

This pull request introduces 10 alerts when merging 844fc83 into 4697662 - view on LGTM.com

new alerts:

  • 4 for Unused import
  • 3 for Signature mismatch in overriding method
  • 1 for First parameter of a method is not named 'self'
  • 1 for Unused local variable
  • 1 for Module is imported with 'import' and 'import from'

@okuchaiev okuchaiev requested a review from khcs February 7, 2022 22:33
@lgtm-com
Copy link

lgtm-com bot commented Feb 7, 2022

This pull request introduces 10 alerts when merging 846cf4d into 4697662 - view on LGTM.com

new alerts:

  • 4 for Unused import
  • 3 for Signature mismatch in overriding method
  • 1 for First parameter of a method is not named 'self'
  • 1 for Unused local variable
  • 1 for Module is imported with 'import' and 'import from'

@yidong72
remove the outputs
a94c8af 
Signed-off-by: Yi Dong <[email protected]>
@lgtm-com
Copy link

lgtm-com bot commented Feb 8, 2022

This pull request introduces 10 alerts when merging a94c8af into 4697662 - view on LGTM.com

new alerts:

  • 4 for Unused import
  • 3 for Signature mismatch in overriding method
  • 1 for First parameter of a method is not named 'self'
  • 1 for Unused local variable
  • 1 for Module is imported with 'import' and 'import from'

@lgtm-com
Copy link

lgtm-com bot commented Feb 9, 2022

This pull request introduces 11 alerts when merging 3a7b4fb into c3101cc - view on LGTM.com

new alerts:

  • 5 for Unused import
  • 3 for Signature mismatch in overriding method
  • 1 for First parameter of a method is not named 'self'
  • 1 for Unused local variable
  • 1 for Module is imported with 'import' and 'import from'

@lgtm-com
Copy link

lgtm-com bot commented Feb 10, 2022

This pull request introduces 11 alerts when merging 306d9bc into 8e15ba4 - view on LGTM.com

new alerts:

  • 5 for Unused import
  • 3 for Signature mismatch in overriding method
  • 1 for First parameter of a method is not named 'self'
  • 1 for Unused local variable
  • 1 for Module is imported with 'import' and 'import from'

@ericharper
Copy link
Collaborator

This pull request introduces 11 alerts when merging 306d9bc into 8e15ba4 - view on LGTM.com

new alerts:

* 5 for Unused import

* 3 for Signature mismatch in overriding method

* 1 for First parameter of a method is not named 'self'

* 1 for Unused local variable

* 1 for Module is imported with 'import' and 'import from'

Can you take a look at these?

@lgtm-com
Copy link

lgtm-com bot commented Feb 11, 2022

This pull request introduces 11 alerts when merging a6bfd92 into 64eb620 - view on LGTM.com

new alerts:

  • 5 for Unused import
  • 3 for Signature mismatch in overriding method
  • 1 for First parameter of a method is not named 'self'
  • 1 for Unused local variable
  • 1 for Module is imported with 'import' and 'import from'

ericharper
ericharper reviewed Feb 11, 2022
View reviewed changes
nemo/collections/nlp/models/language_modeling/megatron_ptune_gpt_model.py Outdated
Comment on lines 64 to 66
# shared params for dataset and data loaders
# tokenizer needs to get initialized before the super.__init__()
# as dataloaders and datasets need it to process the data
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean up comment

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleaned.

ericharper
ericharper reviewed Feb 11, 2022
View reviewed changes
nemo/collections/nlp/models/language_modeling/megatron_ptune_gpt_model.py Outdated
# register the file containing the labels into the artifacts to get stored in the '.nemo' file later
self.embeddings = self.model.model.language_model.embedding.word_embeddings

# self.vocab = self.tokenizer.tokenizer.get_vocab()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clean up comment

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleaned

ericharper
ericharper reviewed Feb 11, 2022
View reviewed changes
nemo/collections/nlp/models/language_modeling/megatron_ptune_gpt_model.py Outdated
Comment on lines 185 to 187
# _, returned_pred = self.get_prediction(batch_size, label_position, logits)
# returned_label = self.get_ground_truth_labels(batch_size, label_ids)
# return floss, returned_pred, returned_label
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clean up comment

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleaned

ericharper
ericharper reviewed Feb 11, 2022
View reviewed changes
nemo/collections/nlp/models/language_modeling/megatron_ptune_gpt_model.py Outdated
Comment on lines 275 to 276
# predicted_tokens_dec = torch.cat([predicted_tokens_dec, token_ids[:, -1].unsqueeze(1)], 1)
# new_pred = torch.zeros_like(token_ids[:, 0:1])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clean up comments

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleaned

ericharper
ericharper reviewed Feb 11, 2022
View reviewed changes
nemo/collections/nlp/models/language_modeling/megatron_ptune_gpt_model.py
Comment on lines +148 to +152
# slow version of above scatter logics
# for bidx in range(bz):
# position = blocked_indices[bidx].nonzero()[:, 0]
# for i in range(len(position)):
# raw_embeds[bidx, position[i], :] = replace_embeds[bidx, i, :]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clean up comments

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will leave it since it helps people to understand the code.

ericharper
ericharper reviewed Feb 11, 2022
View reviewed changes
tutorials/nlp/PTune_multiple_NLP_tasks.ipynb Outdated
"id": "0b0f69a9",
"metadata": {},
"source": [
"## Download the SQuDA dataset\n",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

ericharper
ericharper previously approved these changes Feb 11, 2022
View reviewed changes
Copy link
Collaborator

@ericharper ericharper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@lgtm-com
Copy link

lgtm-com bot commented Feb 11, 2022

This pull request introduces 11 alerts when merging 4d3ebc6 into 9aef14f - view on LGTM.com

new alerts:

  • 5 for Unused import
  • 3 for Signature mismatch in overriding method
  • 1 for First parameter of a method is not named 'self'
  • 1 for Unused local variable
  • 1 for Module is imported with 'import' and 'import from'

@yidong72
comments clean up
f60f7ab 
Signed-off-by: Yi Dong <[email protected]>
@lgtm-com
Copy link

lgtm-com bot commented Feb 11, 2022

This pull request introduces 11 alerts when merging f60f7ab into 298b686 - view on LGTM.com

new alerts:

  • 5 for Unused import
  • 3 for Signature mismatch in overriding method
  • 1 for First parameter of a method is not named 'self'
  • 1 for Unused local variable
  • 1 for Module is imported with 'import' and 'import from'

@lgtm-com
Copy link

lgtm-com bot commented Feb 11, 2022

This pull request introduces 11 alerts when merging 6e6a306 into c645c4c - view on LGTM.com

new alerts:

  • 5 for Unused import
  • 3 for Signature mismatch in overriding method
  • 1 for First parameter of a method is not named 'self'
  • 1 for Unused local variable
  • 1 for Module is imported with 'import' and 'import from'

@lgtm-com
Copy link

lgtm-com bot commented Feb 12, 2022

This pull request introduces 11 alerts when merging 85100b4 into 461a866 - view on LGTM.com

new alerts:

  • 5 for Unused import
  • 3 for Signature mismatch in overriding method
  • 1 for First parameter of a method is not named 'self'
  • 1 for Unused local variable
  • 1 for Module is imported with 'import' and 'import from'

@lgtm-com
Copy link

lgtm-com bot commented Feb 14, 2022

This pull request introduces 11 alerts when merging 14739bc into d364b3f - view on LGTM.com

new alerts:

  • 5 for Unused import
  • 3 for Signature mismatch in overriding method
  • 1 for First parameter of a method is not named 'self'
  • 1 for Unused local variable
  • 1 for Module is imported with 'import' and 'import from'

ericharper
ericharper approved these changes Feb 14, 2022
View reviewed changes
Copy link
Collaborator

@ericharper ericharper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@ericharper ericharper merged commit 9e9aae2 into main Feb 14, 2022
@ericharper ericharper deleted the generalized_gpt_ptuning branch February 14, 2022 18:23
fayejf pushed a commit that referenced this pull request Mar 2, 2022
@yidong72 @ericharper @fayejf
Generalized the P-tuning method to support various NLP tasks (#3623)
ba92da4 
* memory fix

Signed-off-by: Yi Dong <[email protected]>

* fix exp cause inf

Signed-off-by: Yi Dong <[email protected]>

* clean comments

Signed-off-by: Yi Dong <[email protected]>

* added the ptune gpt model

Signed-off-by: Yi Dong <[email protected]>

* fix the decode

Signed-off-by: Yi Dong <[email protected]>

* text gen loss is done

Signed-off-by: Yi Dong <[email protected]>

* working now

Signed-off-by: Yi Dong <[email protected]>

* fix style

Signed-off-by: Yi Dong <[email protected]>

* auto infer decoder len

Signed-off-by: Yi Dong <[email protected]>

* evaluate on the best checkpoint

Signed-off-by: Yi Dong <[email protected]>

* update the dataset to handle multiple tasks

Signed-off-by: Yi Dong <[email protected]>

* added the inference logics

Signed-off-by: Yi Dong <[email protected]>

* simplify the data processor

Signed-off-by: Yi Dong <[email protected]>

* task dependent

Signed-off-by: Yi Dong <[email protected]>

* multiple task working

Signed-off-by: Yi Dong <[email protected]>

* performance improvement

Signed-off-by: Yi Dong <[email protected]>

* fix the old ptune classification

Signed-off-by: Yi Dong <[email protected]>

* added tutorial notebook

Signed-off-by: Yi Dong <[email protected]>

* remove the outputs

Signed-off-by: Yi Dong <[email protected]>

* added template data processor

Signed-off-by: Yi Dong <[email protected]>

* fix bug of non task depedent and detects negative cut

Signed-off-by: Yi Dong <[email protected]>

* remove the sentiment analysis notebook

Signed-off-by: Yi Dong <[email protected]>

* comments clean up

Signed-off-by: Yi Dong <[email protected]>

Co-authored-by: Eric Harper <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericharper ericharper ericharper approved these changes

@MaximumEntropy MaximumEntropy Awaiting requested review from MaximumEntropy

@okuchaiev okuchaiev Awaiting requested review from okuchaiev

@khcs khcs Awaiting requested review from khcs

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@yidong72 @ericharper