Adding tutorial for confidence ensembles #6932
Conversation
|
Update - fixed all links, except for the confidence tutorial, added docs and also a few simple standalone tests. Bug fixes are removed, since they are merged in another PR |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
This is a preliminary reviewas I was unable to finish the Colab run.
Overall, great PR!
Please confider the following comments.
It would also be great to somehow highlight the difference between the last and penultimate cells, as they long but mostly similar.
|
Thanks for the review, @GNroy! I addressed all of the feedback - also changed the history to remove the image I originally pushed and moved it over to the release files, so that it's accessible in colab and not in the git history. Sorry about the colab issue - only tested locally because was developing that way. I now verified that it works well in both colab and local mode. Just make sure to restart the kernel after running the first cell to pick up the installed nemo package (there is a comment at the bottom of that cell explaining this) |
There was a problem hiding this comment.
Seems a few images in the tutorial are missing. You can add them to the latest release
There was a problem hiding this comment.
Could you put the SDP download and install at the very top of the notebook
There was a problem hiding this comment.
Which images are missing?
There was a problem hiding this comment.
Oh in the preview the image in the first section of the tutorial (explanation of confidence) was missing. Maybe it will be fixed upob merge c
Signed-off-by: Igor Gitman <[email protected]>
Signed-off-by: Igor Gitman <[email protected]>
Signed-off-by: Igor Gitman <[email protected]>
Signed-off-by: Igor Gitman <[email protected]>
There was a problem hiding this comment.
Oh in the preview the image in the first section of the tutorial (explanation of confidence) was missing. Maybe it will be fixed upob merge c
| "\n", | ||
| "A short answer — you can use any ASR models. E.g., you can combine a number of CTC models, or Transducer models, or even mix-and-match. \n", | ||
| "\n", | ||
| "A more detailed answer is that hte performance of the confidence ensemble is upper-bounded by the performance of the best model on each of the input examples. Thus you will benefit if some of your models work really well on part of the input compared to other models. This way you will get more gains compared to each separate model, and it will also make correct model identification easier.\n", |
There was a problem hiding this comment.
Imo this section is quite heavy to read in one sitting. Breaking apart the subsection into digestible chunks may be better c
This is fine as well, just a bit heavy.
There was a problem hiding this comment.
I was not sure how much details to put, so decided to put enough for people to understand a bit more than just examples, but also encourage to skip around the text and go straight to code and return if they want to learn more. Maybe I should make it more explicit - right now I just say
Each cell is mostly self-contained, so feel free to skip around or jump directly to the code part if you want to see usage examples right away.
at the end of the first text cell.
| "Let's now talk about the \"model selection block\". First of all — you don't need to know the details to use confidence ensembles, calibration is always automatically performed when you build the model. But if you want to learn more, read on!\n", | ||
| "\n", | ||
| "First, let's discuss why we need a separate \"model selection block\" to pick the most confident model. If we had an access to the perfect confidence, which would exactly equal to the probability of the model's output being correct, we wouldn't need this block. In this idealized case we can simply take the model with the maximum confidence score. But in practice, models tend to be over- or under-confident, which means that their confidence scores need to be calibrated together to be comparable. E.g., one model might mostly produce scores from 0 to 0.8, while another model tend to produce scores from 0 to 0.5, even though they have the same average accuracy. So we want to multiply the first model's score by 1.25 and the second model's score by 2.0 to put the on the same \"scale\".\n", | ||
| "\n", |
There was a problem hiding this comment.
Wall of text, hard to maintain concentration for so much text. Maybe split apart or use some form of visual aid to give a high level understanding then dig into details.
It's fine for now but very heavy section.
There was a problem hiding this comment.
Yeah, it's a bit too much, but that's why I have
First of all — you don't need to know the details to use confidence ensembles, calibration is always automatically performed when you build the model. But if you want to learn more, read on!
in the first paragraph :)
What does this PR do?
There are a couple of bugfixes to the confidence computation and also a tutorial.
It currently misses a few links:
Collection: ASR
Changelog
Usage
# Add a code snippet demonstrating how to use thisBefore your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information