Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Word boosting for Flashlight decoder #6367

Merged
merged 4 commits into from
Apr 5, 2023
Merged

Word boosting for Flashlight decoder #6367

merged 4 commits into from
Apr 5, 2023

Conversation

trias702
Copy link
Collaborator

@trias702 trias702 commented Apr 4, 2023 •
edited
Loading

What does this PR do ?

Adds word boosting to the Flashlight decoder in Nemo to mirror this capability in Riva

For more info see here: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-python-advanced-wordboosting.html

And here: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-customizing.html#word-boosting

Collection: ASR

Changelog

  • Added word boosting to collections/asr/modules/flashlight_decoder.py
  • Added new config line item to flashlight_cfg in ctc_beam_decoding.py

(I noticed that flashlight_decoder.py had Windows EOLs, so I changed them to Unix, but unfortunately this makes git show as if I had changed the entire file which isn't the case, sorry about this)

Usage

  • In order to boost certain words, you need to create a flat file which is tab separated. The format of this file is:
<word>\t<boost score>

For example, here is a valid format boost file on disk, not that it is a single tab separated file:

nvidia	40
geforce	50
riva	80
turing	30
badword	-100

Positive scores boost words higher in the LM decoding step so they show up more frequently, whereas negative scores squelch words so they show up less frequently. The recommended range for the boost score is 20 to 100.

The boost file handles both in-vocabulary words and OOV words just fine, so feel free to mix and match.

You can then add it to your flashlight config object in Nemo like this:

ctc_decoding.beam.flashlight_cfg.boost_path='/path/to/my_boost_file.boost'

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

@github-actions github-actions bot added the ASR label Apr 4, 2023
titu1994
titu1994 reviewed Apr 4, 2023
View reviewed changes
Copy link
Collaborator

@titu1994 titu1994 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dunno why there's such a huge diff, but I'll assume the code works for now.

The docs needs to be part of the Nemo docs page - however to avoid duplication you can add a small section in the ASR LM docs section explaning that the docs for boosting are located in Riva - and then link there

@trias702
Copy link
Collaborator Author

trias702 commented Apr 4, 2023

Dunno why there's such a huge diff, but I'll assume the code works for now.

It's because the flashlight_decoder.py file had Windows style EOL markings, and I changed them to UNIX to bring it into line with all other .py files in Nemo. Sorry about that. If Nemo is okay with Windows EOL markings, I can resubmit the Windows one which will then correctly show only the changes I made, the entire file won't be showing up red.

github-advanced-security[bot]
github-advanced-security bot found potential problems Apr 4, 2023
View reviewed changes
nemo/collections/asr/modules/flashlight_decoder.py
import torch

from nemo.collections.common.tokenizers.tokenizer_spec import TokenizerSpec
from nemo.core.classes import NeuralModule, typecheck

Check notice

Code scanning / CodeQL

Unused import

Import of 'typecheck' is not used.
nemo/collections/asr/modules/flashlight_decoder.py

import itertools
import math
from typing import Iterable, List, Optional, Tuple, Union

Check notice

Code scanning / CodeQL

Unused import

Import of 'Iterable' is not used. Import of 'Tuple' is not used.
nemo/collections/asr/modules/flashlight_decoder.py

from nemo.collections.common.tokenizers.tokenizer_spec import TokenizerSpec
from nemo.core.classes import NeuralModule, typecheck
from nemo.core.neural_types import LengthsType, LogprobsType, NeuralType, PredictionsType

Check notice

Code scanning / CodeQL

Unused import

Import of 'LengthsType' is not used. Import of 'LogprobsType' is not used. Import of 'NeuralType' is not used. Import of 'PredictionsType' is not used.
@trias702
Copy link
Collaborator Author

trias702 commented Apr 4, 2023

The docs needs to be part of the Nemo docs page - however to avoid duplication you can add a small section in the ASR LM docs section explaning that the docs for boosting are located in Riva - and then link there

I have done this and updated the commit, please let me know if it is sufficient

titu1994
titu1994 approved these changes Apr 4, 2023
View reviewed changes
Copy link
Collaborator

@titu1994 titu1994 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks ! Looks great

docs/source/asr/asr_language_modeling.rst
@@ -393,3 +392,45 @@ parameter, the plot of WER% for different values of the parameter is also shown.

It is recommended to first use the linear search for both parameters on a validation set by not providing any values for `--alpha` and `--beta`.
Then check the WER curves and decide on the best values for each parameter. Finally, evaluate the best values on the test set.


Word Boosting
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks !

titu1994
titu1994 approved these changes Apr 5, 2023
View reviewed changes
Copy link
Collaborator

@titu1994 titu1994 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks !

@titu1994 titu1994 merged commit 98ba0b7 into NVIDIA:main Apr 5, 2023
hsiehjackson pushed a commit to hsiehjackson/NeMo that referenced this pull request Jun 2, 2023
@trias702 @pre-commit-ci @hsiehjackson
Word boosting for Flashlight decoder (NVIDIA#6367)
6315cb3 
* Initial commit of word boosting for flashlight

Signed-off-by: Daniel Egert <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated docs for word boosting

Signed-off-by: Daniel Egert <[email protected]>

---------

Signed-off-by: Daniel Egert <[email protected]>
Co-authored-by: Daniel Egert <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@titu1994 titu1994 titu1994 approved these changes

Assignees
No one assigned
Labels
ASR
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@trias702 @titu1994