-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dehardcode test string #8865
dehardcode test string #8865
Conversation
3619adb
to
635a28b
Compare
Signed-off-by: Jimmy Zhang <[email protected]>
52b7d8d
to
845dc17
Compare
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a minor request to update the comment.
Besides this lgtm, but I just want to point out that using <extra_id_1>
as stop word with the Llama2 tokenizer is a bad idea since as you mentioned it may be tokenized in various ways when merged with what comes before. When using the text generation from NeMo at least we fall back to the (slower) string matching (here), but with TRT-LLM I don't know if such a string matching exists (?) => one solution could be to postprocess the output to be sure we stop on the first stop word (this has nothing to do with this PR though -- just mentioning it in case this would be relevant to your work)
Signed-off-by: JimmyZhang12 <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
Can this be merged @oyilmaz-nvidia ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks!
jenkins |
* dehardcode test string Signed-off-by: Jimmy Zhang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update nemo_utils.py Signed-off-by: JimmyZhang12 <[email protected]> --------- Signed-off-by: Jimmy Zhang <[email protected]> Signed-off-by: JimmyZhang12 <[email protected]> Co-authored-by: Jimmy Zhang <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Onur Yilmaz <[email protected]>
What does this PR do ?
By default
to_word_list_format
uses '<extra_id_1>' as its test string to map a string to its tokens, but using '<extra_id_1>' may not map correctly, so add option to specifiy the test string.Collection: [Note which collection this PR will affect]
Changelog
Usage
# Add a code snippet demonstrating how to use this
Jenkins CI
To run Jenkins, a NeMo User with write access must comment
jenkins
on the PR.Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information