Add end_strings to SamplingParams #6986

markelsanz14 · 2023-07-06T19:44:49Z

What does this PR do ?

Makes end_strings part of SamplingParams and makes it configurable.

Collection: Text Generation Server

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

MaximumEntropy

Minor comment about including <extra_id_1> as a default end string everywhere.

examples/nlp/language_modeling/conf/megatron_gpt_inference.yaml

nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py

tests/collections/nlp/test_gpt_eval.py

Signed-off-by: Gerald Shen <[email protected]>

for more information, see https://pre-commit.ci Signed-off-by: Gerald Shen <[email protected]>

Signed-off-by: Gerald Shen <[email protected]>

Signed-off-by: Sergii Dymchenko <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

Signed-off-by: Gerald Shen <[email protected]>

Signed-off-by: Yi Dong <[email protected]> Co-authored-by: Yi Dong <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

* add support for max_total_length=4096 for 43b Signed-off-by: Zhilin Wang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Zhilin Wang <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Gerald Shen <[email protected]>

#6989) Signed-off-by: Vadim Kantorov <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

* Cache handling without input tensors mutation * Cleanup * Cleanup#2 * Cleanup#3 --------- Signed-off-by: Boris Fomitchev <[email protected]> Co-authored-by: Boris Fomitchev <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

* Implemented generic kv-pair setting of export_config from args * Hybrid conformer export * Hybrid decoder export * Cleanup * Changed from **kwargs * Docstring * Docs added * Stringify args * Added docs for ASR export configs * lowercase ctc --------- Signed-off-by: Boris Fomitchev <[email protected]> Co-authored-by: Boris Fomitchev <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

* Bug fix for the confidence ensembles * Relax constraints for the test --------- Signed-off-by: Igor Gitman <[email protected]> Co-authored-by: Igor Gitman <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

* [TTS] Add cosine distance option to TTS aligner Signed-off-by: Ryan <[email protected]> * [TTS] Update aligner comments Signed-off-by: Ryan <[email protected]> --------- Signed-off-by: Ryan <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

* Initial commit of minor MPT-7B fixes Signed-off-by: Daniel Egert <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Daniel Egert <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Gerald Shen <[email protected]>

* change timeout Signed-off-by: ericharper <[email protected]> * change to 8 hours Signed-off-by: ericharper <[email protected]> --------- Signed-off-by: ericharper <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

* remove hard coded input and output fields Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: arendu <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Gerald Shen <[email protected]>

* Push changes Signed-off-by: MaximumEntropy <[email protected]> * Fixes Signed-off-by: MaximumEntropy <[email protected]> * add continue training script Signed-off-by: MaximumEntropy <[email protected]> * [WIP] nonlinear interp Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * override encoder_seq_len Signed-off-by: MaximumEntropy <[email protected]> * Remove nonlinear Signed-off-by: MaximumEntropy <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * sft with pi (#7006) * sft with pi Signed-off-by: Evelina <[email protected]> * update values only if not None" Signed-off-by: Evelina <[email protected]> --------- Signed-off-by: Evelina <[email protected]> * Address comments Signed-off-by: MaximumEntropy <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add info Signed-off-by: MaximumEntropy <[email protected]> * Empty Signed-off-by: MaximumEntropy <[email protected]> --------- Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: Evelina <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Evelina <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

Signed-off-by: Gerald Shen <[email protected]>

for more information, see https://pre-commit.ci Signed-off-by: Gerald Shen <[email protected]>

Signed-off-by: Gerald Shen <[email protected]>

…o markelsanz14/end_strings

ericharper

Re-approving.

github-actions bot added the NLP label Jul 6, 2023

markelsanz14 requested review from yidong72, MaximumEntropy and ericharper and removed request for yidong72 July 6, 2023 22:29

MaximumEntropy suggested changes Jul 6, 2023

View reviewed changes

examples/nlp/language_modeling/conf/megatron_gpt_inference.yaml Outdated Show resolved Hide resolved

nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py Outdated Show resolved Hide resolved

tests/collections/nlp/test_gpt_eval.py Outdated Show resolved Hide resolved

MaximumEntropy previously approved these changes Jul 7, 2023

View reviewed changes

yidong72 previously approved these changes Jul 7, 2023

View reviewed changes

gshennvm dismissed stale reviews from yidong72 and MaximumEntropy via 0d4c995 July 11, 2023 23:45

gshennvm force-pushed the markelsanz14/end_strings branch from f7a1bdf to 57f345d Compare July 12, 2023 00:24

github-actions bot added core Changes to NeMo Core TTS ASR CI and removed core Changes to NeMo Core TTS ASR CI labels Jul 12, 2023

markelsanz14 and others added 10 commits July 12, 2023 11:02

Add end_strings to SamplingParams

979b4c0

Signed-off-by: Gerald Shen <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

8a7368c

for more information, see https://pre-commit.ci Signed-off-by: Gerald Shen <[email protected]>

Add end_strings to megatron_gpt_inference.yaml

744205a

Signed-off-by: Gerald Shen <[email protected]>

Add end_strings to sampling params

e223ba4

Signed-off-by: Gerald Shen <[email protected]>

Remove extra_id_1 from default end_strings

1ab4a4b

Signed-off-by: Gerald Shen <[email protected]>

Fix require_grad typos (#6930)

e408b20

Signed-off-by: Sergii Dymchenko <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

fix syntax error

67004af

Signed-off-by: Gerald Shen <[email protected]>

fix the mpt chatbot (#6957) (#6968)

dd1d48f

Signed-off-by: Yi Dong <[email protected]> Co-authored-by: Yi Dong <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

rnnt_greedy_decoding.py: typos? auto-repressively -> auto-regressively (

41e789c

#6989) Signed-off-by: Vadim Kantorov <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

github-actions bot and others added 9 commits July 12, 2023 11:02

Fixing an issue with confidence ensembles (#6987) (#7004)

fd53ae9

* Bug fix for the confidence ensembles * Relax constraints for the test --------- Signed-off-by: Igor Gitman <[email protected]> Co-authored-by: Igor Gitman <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

Change Jenkins timeout (#6997)

5551858

* change timeout Signed-off-by: ericharper <[email protected]> * change to 8 hours Signed-off-by: ericharper <[email protected]> --------- Signed-off-by: ericharper <[email protected]> Signed-off-by: Gerald Shen <[email protected]>

use proper config

a05ac06

Signed-off-by: Gerald Shen <[email protected]>

gshennvm force-pushed the markelsanz14/end_strings branch from 0bd3d63 to a05ac06 Compare July 12, 2023 18:02

github-actions bot added core Changes to NeMo Core TTS ASR CI labels Jul 12, 2023

markelsanz14 and others added 8 commits July 12, 2023 11:23

Add end_strings to SamplingParams

cdb820f

Signed-off-by: Gerald Shen <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

2c2b46d

for more information, see https://pre-commit.ci Signed-off-by: Gerald Shen <[email protected]>

Add end_strings to megatron_gpt_inference.yaml

5cb4b22

Signed-off-by: Gerald Shen <[email protected]>

Add end_strings to sampling params

67de2fb

Signed-off-by: Gerald Shen <[email protected]>

Remove extra_id_1 from default end_strings

da2d77b

Signed-off-by: Gerald Shen <[email protected]>

fix syntax error

a26970d

Signed-off-by: Gerald Shen <[email protected]>

use proper config

4c6d8a1

Signed-off-by: Gerald Shen <[email protected]>

Merge branch 'markelsanz14/end_strings' of github.com:NVIDIA/NeMo int…

26af6c3

…o markelsanz14/end_strings

github-actions bot removed core Changes to NeMo Core TTS ASR CI labels Jul 12, 2023

ericharper approved these changes Jul 13, 2023

View reviewed changes

ericharper merged commit f7e33fc into main Jul 13, 2023
15 checks passed

ericharper deleted the markelsanz14/end_strings branch July 13, 2023 19:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add end_strings to SamplingParams #6986

Add end_strings to SamplingParams #6986

markelsanz14 commented Jul 6, 2023

MaximumEntropy left a comment

ericharper left a comment

Add end_strings to SamplingParams #6986

Add end_strings to SamplingParams #6986

Conversation

markelsanz14 commented Jul 6, 2023

What does this PR do ?

Changelog

Usage

Before your PR is "Ready for review"

Who can review?

Additional Information

MaximumEntropy left a comment

Choose a reason for hiding this comment

ericharper left a comment

Choose a reason for hiding this comment