Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ASR] Added a script for evaluating metrics for audio-to-audio #5971

Merged
merged 1 commit into from
Feb 24, 2023

Conversation

anteju
Copy link
Collaborator

@anteju anteju commented Feb 8, 2023

Added a script for evaluating audio-to-audio metrics for a manifest file (audio_to_audio_eval.py)

Signed-off-by: Ante Jukić [email protected]

What does this PR do ?

This PR adds a script for evaluation of audio-to-audio metrics.
It is similar to speech_to_text_eval.py

This scripts depends on the process_audio.py script and inherits all its arguments.

Collection: ASR

Changelog

  • Added audio_to_audio_eval.py
  • Minor change in proces_audio
  • Added an option to get_full_path to specify data_dir directly instead of using dirname(manifest_file)

Usage

  • You can potentially add a usage example below

To score a dataset with a manifest file that contains the input audio which needs to be processed and target audio

python audio_to_audio_eval.py \
    model_path=null \
    pretrained_model=null \
    dataset_manifest=<Mandatory: path to a dataset manifest file> \
    output_dir=<Optional: Directory where processed audio will be saved> \
    processed_channel_selector=<Optional: list of channels to select from the processed audio file> \
    target_key=<Optional: key for the target audio in the dataset manifest. Default: target_audio_filepath> \
    target_channel_selector=<Optional: list of channels to select from the target audio file> \
    metrics=<Optional: list of metrics to evaluate. Defaults to [sdr,estoi]>
    batch_size=32 \
    amp=True

To score a manifest file which has been previously processed and contains both processed audio and target audio

python audio_to_audio_eval.py \
    dataset_manifest=<Mandatory: path to a dataset manifest file> \
    processed_key=<Optional: key for the target audio in the dataset manifest. Default: processed_audio_filepath>
    processed_channel_selector=<Optional: list of channels to select from the processed audio file> \
    target_key=<Optional: key for the target audio in the dataset manifest. Default: target_audio_filepath> \
    target_channel_selector=<Optional: list of channels to select from the target audio file> \
    metrics=<Optional: list of metrics to evaluate. Defaults to [sdr,estoi]>
    batch_size=32 \
    amp=True

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

@anteju anteju force-pushed the dev/audio-to-audio-eval-script branch from d1e324b to 96b2a89 Compare February 9, 2023 01:12
@anteju anteju requested a review from jbalam-nv February 9, 2023 18:20
@anteju anteju force-pushed the dev/audio-to-audio-eval-script branch 7 times, most recently from f973be3 to f34f1b4 Compare February 13, 2023 21:50
@anteju anteju force-pushed the dev/audio-to-audio-eval-script branch 3 times, most recently from 38bda00 to 68c4f26 Compare February 21, 2023 19:05
@anteju anteju requested a review from titu1994 February 21, 2023 19:18
…fest file (audio_to_audio_eval.py)

Signed-off-by: Ante Jukić <[email protected]>
@anteju anteju force-pushed the dev/audio-to-audio-eval-script branch from 68c4f26 to d8cf738 Compare February 24, 2023 18:23
Copy link
Collaborator

@jbalam-nv jbalam-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jbalam-nv jbalam-nv merged commit 2f11f05 into NVIDIA:main Feb 24, 2023
titu1994 pushed a commit to titu1994/NeMo that referenced this pull request Mar 24, 2023
…fest file (audio_to_audio_eval.py) (NVIDIA#5971)

Signed-off-by: Ante Jukić <[email protected]>
hsiehjackson pushed a commit to hsiehjackson/NeMo that referenced this pull request Jun 2, 2023
…fest file (audio_to_audio_eval.py) (NVIDIA#5971)

Signed-off-by: Ante Jukić <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants