Add AudioBench and Librispeech-PC benchmarks for speech and audio language models#1043
Closed
Jorjeous wants to merge 23 commits intoNVIDIA-NeMo:mainfrom
Closed
Add AudioBench and Librispeech-PC benchmarks for speech and audio language models#1043Jorjeous wants to merge 23 commits intoNVIDIA-NeMo:mainfrom
Jorjeous wants to merge 23 commits intoNVIDIA-NeMo:mainfrom
Conversation
Member
Author
|
This PR adding evaluation on set's from audiobench and apply's minor fix to manifest format in mmau-pro To achieve this WER, BlUE score calculation was implemented. |
melllinia
force-pushed
the
audiobench-benchmark
branch
from
75c51eb to
97ca5b8
Compare
Resolve conflict Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com> Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Signed-off-by: Sadegh Mahdavi <smahdavi@nvidia.com> Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Signed-off-by: msamadi <msamadi@nvidia.com> Co-authored-by: msamadi <msamadi@nvidia.com> Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Signed-off-by: mmkrtchyan <mmkrtchyan@nvidia.com> Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Author did not signed commit This reverts commit ecfafd1. Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Author sign off is incorrect This reverts commit 353c202. Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
Jorjeous
force-pushed
the
audiobench-benchmark
branch
from
2e81d8e to
2990929
Compare
Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
melllinia
reviewed
Nov 21, 2025
| # Judge configuration matching AudioBench official implementation | ||
| # Using Llama-3.1-70B with vllm (can be overridden in run scripts) | ||
| JUDGE_PIPELINE_ARGS = { | ||
| "model": "meta-llama/Meta-Llama-3.1-70B-Instruct", |
Member
There was a problem hiding this comment.
Please try to add NVIDIA deployed model instead from this link and check if it works: https://build.nvidia.com/meta/llama-3_1-70b-instruct
Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>
- Add comprehensive documentation for LibriSpeech-PC benchmark in speech-audio.md - Fix jiwer import to be lazy (only import when needed for ASR evaluation) Signed-off-by: mmkrtchyan <mmkrtchyan@nvidia.com>
Signed-off-by: mmkrtchyan <mmkrtchyan@nvidia.com>
melllinia
changed the title
Collaborator
|
Closed in favor of #1060 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolve conflict
Signed-off-by: George Zelenfroind gzelenfroind@nvidia.com
Add audiobench
and fix prepare.py for MMAU-pro