[TTS] Implement new vocoder dataset

[rlangman]

What does this PR do ?

Implement new VocoderDataset to support multi-manifest training, which follows the same design pattern as the new TextToSpeechDataset (#6575).

Collection: [TTS]

Changelog

• Add VocoderDataset implementation
• Add example config for using data loader
• Add logic in HiFi-GAN recipe for switching to new data loader (matching changes to support UnivNet recipe will be in a future PR).
• Add vocoder logging callback
• Use modified configure_optimizers() implementation from UnivNet recipe (https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/models/univnet.py#L106) to support schedulers other than CosineAnnealing. New config uses scheduler from original HiFi-GAN paper.

Usage

Example of training on 2 datasets, LJSpeech with 10% training examples and VCTK with 90%:

/home/rlangman/miniconda3/envs/nemo_dev/bin/python /home/rlangman/Code/NeMo/examples/tts/hifigan.py
--config-path=/home/rlangman/Code/NeMo/examples/tts/conf/hifigan
--config-dir=/home/rlangman/Code/NeMo/examples/tts/conf
--config-name=hifigan_22050_data.yaml
sample=22050
max_epochs=1000
weighted_sample_steps=1000
batch_size=16
trainer.devices=1
log_dir=/home/rlangman/exps/ljspeech_vctk/HifiGan/test/samples
exp_manager.exp_dir=/home/rlangman/exps/ljspeech_vctk
+exp_manager.version=test
+train_ds_meta.ljspeech.manifest_path=/home/rlangman/fs_mount/data/ljspeech/train_manifest.json
+train_ds_meta.ljspeech.audio_dir=/home/rlangman/fs_mount/data/ljspeech/audio
+train_ds_meta.ljspeech.sample_weight=0.1
+val_ds_meta.ljspeech.manifest_path=/home/rlangman/fs_mount/data/ljspeech/val_manifest.json
+val_ds_meta.ljspeech.audio_dir=/home/rlangman/fs_mount/data/ljspeech/audio
+log_ds_meta.ljspeech._target_=nemo.collections.tts.data.vocoder_dataset.DatasetMeta
+log_ds_meta.ljspeech.manifest_path=/home/rlangman/fs_mount/data/ljspeech/log_manifest.json
+log_ds_meta.ljspeech.audio_dir=/home/rlangman/fs_mount/data/ljspeech/audio
+train_ds_meta.vctk.manifest_path=/home/rlangman/fs_mount/data/vctk/train_manifest.json
+train_ds_meta.vctk.audio_dir=/home/rlangman/fs_mount/data/vctk/audio
+train_ds_meta.vctk.sample_weight=0.9
+val_ds_meta.vctk.manifest_path=/home/rlangman/fs_mount/data/vctk/val_manifest.json
+val_ds_meta.vctk.audio_dir=/home/rlangman/fs_mount/data/vctk/audio
+log_ds_meta.vctk._target_=nemo.collections.tts.data.vocoder_dataset.DatasetMeta
+log_ds_meta.vctk.manifest_path=/home/rlangman/fs_mount/data/vctk/log_manifest.json
+log_ds_meta.vctk.audio_dir=/home/rlangman/fs_mount/data/vctk/audio
exp_manager.create_wandb_logger=true
+exp_manager.wandb_logger_kwargs.project=hifigan_ljspeech_vctk
+exp_manager.wandb_logger_kwargs.name=test

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

[racoiaws]

What does weighted_sample_steps mean?

[rlangman]

Dataloader docstring: Optional int, If provided, then data will be sampled (with replacement) based on the sample weights provided in the dataset metadata. If None, then sample weights will be ignored.

In other words, with weighted sampling (and with large datasets in general) the standard definition of an epoch no longer works so you refer to an epoch as a fixed number of training steps.

The name is not as clear as I would like, but alternatives I thought of were either too long (like weighted_sampling_steps_per_epoch) or required breaking it into 2 arguments like:

WeightedSamplingConfig:
  use_weighted_sampling: true
  steps_per_epoch: 1000

Where steps_per_epoch does nothing if use_weighted_sampling: false.

[racoiaws]

Thanks

In other words, with weighted sampling (and with large datasets in general) the standard definition of an epoch no longer works so you refer to an epoch as a fixed number of training steps.

Can't really agree, I always thought of epoch as of something dependent on size of a specific dataset. Number of training examples in the dataset still works, with the caveat that some samples might not have been utilized in a given epoch run.

(Not a blocker)

but alternatives I thought of were either too long (like weighted_sampling_steps_per_epoch)

True, this is not something that can be expressed consisely. But I see no harm in a long-named argument. I would even suggest steps_per_epoch_if_using_weighted_sampling

Do you thing your could you rename this into a longer and more descriptive version?

[rlangman]

Can't really agree, I always thought of epoch as of something dependent on size of a specific dataset. Number of training examples in the dataset still works, with the caveat that some samples might not have been utilized in a given epoch run.

The main issue is that as datasets become large having epoch equal to dataset size means it may take days to complete 1 epoch, which is impractical when you only run validation and save checkpoints at the end of an epoch. This is a problem right now with multi-language ASR training which NeMo provides no good solution for.

So while tracking epochs is nice, you usually want things like checkpointing to happen at fixed intervals (eg. every few hours).

Do you thing your could you rename this into a longer and more descriptive version?

Sure.

[rlangman]

I also tried adding in alternate config which could optionally set epoch_size = dataset_size (even if not all training examples are seen each epoch), but the way that an epoch is defined for distributed training is relatively complicated (https://github.com/NVIDIA/NeMo/blob/main/nemo/core/optim/lr_scheduler.py#L934-L936). Getting it wrong will create a mismatch with the LR scheduler. So I would rather not support it for now unless there is an urgent need to.

[racoiaws]

Apologies if this is taking too long, after resolving two threads above this PR should be good to go.