-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prompt Learning Inference Improvements #4566
Conversation
Signed-off-by: Virginia Adams <[email protected]>
Signed-off-by: Virginia Adams <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some minor comments
|
||
for dataset in sentences: | ||
for line in open(dataset, 'r', encoding='utf-8').readlines(): | ||
self.sentences.append(line) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here you still load all the sentences from the files into the memory. Why it avoids OOM?
Also consider add the doc string for RequestDataSet. the sentences can be multiple things now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ended up making a separate prompt learning eval script and removing the use of request dataset for prompt learning inference all together.
Previously, the inference code was trying to load and run inference on the entire test set at once without breaking it into batches. This was causing OOM for large test sets. This change was to make sure the data was loaded and passed to the model in batches. Though, like I said above, I ended up removing this part all together.
nemo/collections/nlp/data/language_modeling/megatron/gpt_prompt_learning_dataset.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Virginia Adams <[email protected]>
This pull request introduces 2 alerts when merging 3482a32 into fea3775 - view on LGTM.com new alerts:
|
This pull request introduces 2 alerts when merging d74d42c into 6b9617d - view on LGTM.com new alerts:
|
Signed-off-by: Virginia Adams <[email protected]>
This pull request introduces 2 alerts when merging a1f342c into 6b9617d - view on LGTM.com new alerts:
|
Signed-off-by: Virginia Adams <[email protected]>
This pull request introduces 3 alerts when merging ccbe930 into 793cf48 - view on LGTM.com new alerts:
|
This pull request introduces 3 alerts when merging 150ceab into 1a9daa5 - view on LGTM.com new alerts:
|
Signed-off-by: Virginia Adams <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some suggestion about api improvements
examples/nlp/language_modeling/megatron_gpt_prompt_learning_eval.py
Outdated
Show resolved
Hide resolved
nemo/collections/nlp/data/language_modeling/megatron/gpt_prompt_learning_dataset.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Virginia Adams <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
see my comment.
examples/nlp/language_modeling/megatron_gpt_prompt_learning_eval.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Virginia Adams <[email protected]>
This pull request introduces 1 alert when merging 3f84478 into 08a623b - view on LGTM.com new alerts:
|
Signed-off-by: Virginia Adams <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Signed-off-by: Virginia Adams <[email protected]>
* Improved prompt learning inference to avoid OOM Signed-off-by: Virginia Adams <[email protected]> * Python style fix Signed-off-by: Virginia Adams <[email protected]> * Made two prompt learning eval script seprate Signed-off-by: Virginia Adams <[email protected]> * updated CI tests and documentation Signed-off-by: Virginia Adams <[email protected]> * Removed request dataset from eval script Signed-off-by: Virginia Adams <[email protected]> * Inference runs as expected Signed-off-by: Virginia Adams <[email protected]> * address comments Signed-off-by: Virginia Adams <[email protected]> * Changed input format for generate method Signed-off-by: Virginia Adams <[email protected]> * Updated documentation, change dataset vairable name Signed-off-by: Virginia Adams <[email protected]> * Updated unit tests Signed-off-by: Virginia Adams <[email protected]> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: David Mosallanezhad <[email protected]>
* Improved prompt learning inference to avoid OOM Signed-off-by: Virginia Adams <[email protected]> * Python style fix Signed-off-by: Virginia Adams <[email protected]> * Made two prompt learning eval script seprate Signed-off-by: Virginia Adams <[email protected]> * updated CI tests and documentation Signed-off-by: Virginia Adams <[email protected]> * Removed request dataset from eval script Signed-off-by: Virginia Adams <[email protected]> * Inference runs as expected Signed-off-by: Virginia Adams <[email protected]> * address comments Signed-off-by: Virginia Adams <[email protected]> * Changed input format for generate method Signed-off-by: Virginia Adams <[email protected]> * Updated documentation, change dataset vairable name Signed-off-by: Virginia Adams <[email protected]> * Updated unit tests Signed-off-by: Virginia Adams <[email protected]> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: George <[email protected]>
* Improved prompt learning inference to avoid OOM Signed-off-by: Virginia Adams <[email protected]> * Python style fix Signed-off-by: Virginia Adams <[email protected]> * Made two prompt learning eval script seprate Signed-off-by: Virginia Adams <[email protected]> * updated CI tests and documentation Signed-off-by: Virginia Adams <[email protected]> * Removed request dataset from eval script Signed-off-by: Virginia Adams <[email protected]> * Inference runs as expected Signed-off-by: Virginia Adams <[email protected]> * address comments Signed-off-by: Virginia Adams <[email protected]> * Changed input format for generate method Signed-off-by: Virginia Adams <[email protected]> * Updated documentation, change dataset vairable name Signed-off-by: Virginia Adams <[email protected]> * Updated unit tests Signed-off-by: Virginia Adams <[email protected]> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: Anas Abou Allaban <[email protected]>
* Improved prompt learning inference to avoid OOM Signed-off-by: Virginia Adams <[email protected]> * Python style fix Signed-off-by: Virginia Adams <[email protected]> * Made two prompt learning eval script seprate Signed-off-by: Virginia Adams <[email protected]> * updated CI tests and documentation Signed-off-by: Virginia Adams <[email protected]> * Removed request dataset from eval script Signed-off-by: Virginia Adams <[email protected]> * Inference runs as expected Signed-off-by: Virginia Adams <[email protected]> * address comments Signed-off-by: Virginia Adams <[email protected]> * Changed input format for generate method Signed-off-by: Virginia Adams <[email protected]> * Updated documentation, change dataset vairable name Signed-off-by: Virginia Adams <[email protected]> * Updated unit tests Signed-off-by: Virginia Adams <[email protected]> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: Hainan Xu <[email protected]>
Signed-off-by: Virginia Adams [email protected]
Separates the gpt prompt learning eval script from the gpt model eval script to simplify inference script files and match t5 prompt learning inference workflow.
Collection: NLP
Changelog
Usage
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information