add rampup batch size support for Megatron GPT #6424

dimapihtar · 2023-04-13T15:36:17Z

What does this PR do ?

This PR adds ramp up batch size support for GPT models. It allows to increase the batch size up to global batch size linearly during training in a given number of samples.

Collection: NLP

Changelog

Add rampup_batch_size value to the training config.
Input rampup_batch_size to the apex microbatch_calculator.
Add current global_batch_size value to the progress bar when using rampup_batch_size.
Update the apex microbatch_calculator after each training step.

Usage

Can be set via config or CLI.

Config:

rampup_batch_size: [8, 4, 10000] # Should be a list of 3 values: [<start_batch_size>, <batch_size_increment>, <rampup_samples>]

CLI:

model.rampup_batch_size=[8,4,10000]

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Dmytro Pykhtar <[email protected]>

ericharper · 2023-04-13T17:11:17Z

Are there any tests that you can add for this?

Signed-off-by: Dmytro Pykhtar <[email protected]>

ericharper · 2023-04-14T15:17:52Z

tests/collections/nlp/test_rampup_bath_size.py

typo in the file name, bath -> batch

Signed-off-by: Dmytro Pykhtar <[email protected]>

tests/collections/nlp/test_rampup_batch_size.py

Signed-off-by: Dmytro Pykhtar <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Dmytro Pykhtar <[email protected]>

…to batch_size_rampup

ericharper

LGTM. Thanks!

tests/collections/nlp/test_rampup_batch_size.py

+    import apex.transformer.pipeline_parallel.utils
+    from apex.transformer.pipeline_parallel.utils import get_num_microbatches
+
+    HAVE_APEX = True


tests/collections/nlp/test_rampup_batch_size.py

+
+except (ImportError, ModuleNotFoundError):
+
+    HAVE_APEX = False


Signed-off-by: Dmytro Pykhtar <[email protected]>

…to batch_size_rampup

ericharper

LGTM. Thanks!

* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- Signed-off-by: hsiehjackson <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. Signed-off-by: Xuesong Yang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Xuesong Yang <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] FastPitch speaker encoder (#6417) * Add initial codes Signed-off-by: hsiehjackson <[email protected]> * Remove wemb Signed-off-by: hsiehjackson <[email protected]> * Fix import Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss Signed-off-by: hsiehjackson <[email protected]> * Add ConditionalInput Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test Signed-off-by: hsiehjackson <[email protected]> * Add initial codes Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix import error Signed-off-by: hsiehjackson <[email protected]> * Add initial codes Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix dataset error Signed-off-by: hsiehjackson <[email protected]> * Remove reference speaker embedding Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Remove SV encoder Signed-off-by: hsiehjackson <[email protected]> * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix length type Signed-off-by: hsiehjackson <[email protected]> * Fix append Signed-off-by: hsiehjackson <[email protected]> * Move error msg Signed-off-by: hsiehjackson <[email protected]> * Add look-up into speaker encoder Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Add valueerror msg Signed-off-by: hsiehjackson <[email protected]> * Move lookup Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Remove unused Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix error Signed-off-by: hsiehjackson <[email protected]> * Rebase and Fix error Signed-off-by: hsiehjackson <[email protected]> * Fix spk encoder Signed-off-by: hsiehjackson <[email protected]> * Rename n_speakers Signed-off-by: hsiehjackson <[email protected]> * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error Signed-off-by: hsiehjackson <[email protected]> --------- Signed-off-by: hsiehjackson <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Sharded manifests for tarred datasets (#6395) * testing sharded manifests Signed-off-by: Dima Rekesh <[email protected]> * compatibility Signed-off-by: Dima Rekesh <[email protected]> * proper fixes Signed-off-by: Dima Rekesh <[email protected]> * adding flag tot convert_to_tarred_audio_dataset Signed-off-by: Dima Rekesh <[email protected]> * shard_manifests conf param Signed-off-by: Dima Rekesh <[email protected]> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <[email protected]> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <[email protected]> * distributed checks Signed-off-by: Dima Rekesh <[email protected]> * typo Signed-off-by: Dima Rekesh <[email protected]> * typo Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py Signed-off-by: Dima Rekesh <[email protected]> * reversing manifest shards flag Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests Signed-off-by: Dima Rekesh <[email protected]> * excluding manifests from webdataset url expansion Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore Signed-off-by: Dima Rekesh <[email protected]> * explicit use of UTF-8 for manifest i/o Signed-off-by: Dima Rekesh <[email protected]> --------- Signed-off-by: Dima Rekesh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) Signed-off-by: Jim O’Regan <[email protected]> * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. Signed-off-by: Mostafa Ghorbandoost <[email protected]> * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. Signed-off-by: Mostafa Ghorbandoost <[email protected]> --------- Signed-off-by: Mostafa Ghorbandoost <[email protected]> * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support Signed-off-by: Dmytro Pykhtar <[email protected]> * added tests for rampup batch size Signed-off-by: Dmytro Pykhtar <[email protected]> * fixed the typos Signed-off-by: Dmytro Pykhtar <[email protected]> * added assertions Signed-off-by: Dmytro Pykhtar <[email protected]> * changed assertion rules Signed-off-by: Dmytro Pykhtar <[email protected]> * deleted unused imports Signed-off-by: Dmytro Pykhtar <[email protected]> * changed tests for rampup batch size Signed-off-by: Dmytro Pykhtar <[email protected]> * updated rampup batch size tests Signed-off-by: Dmytro Pykhtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling Signed-off-by: Dmytro Pykhtar <[email protected]> * rampup batch size tests changes Signed-off-by: Dmytro Pykhtar <[email protected]> --------- Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- Signed-off-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default Signed-off-by: Kunal Dhawan <[email protected]> * updated README with the new agg_tokenizer_manifest flag Signed-off-by: Kunal Dhawan <[email protected]> * fixed typo in scripts/speech_recognition/code_switching/README.md Signed-off-by: Kunal Dhawan <[email protected]> * changed agg_tokenizer_manifest to is_lid_manifest Signed-off-by: Kunal Dhawan <[email protected]> --------- Signed-off-by: Kunal Dhawan <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script Signed-off-by: andrusenkoau <[email protected]> * add return encoding embedding option Signed-off-by: andrusenkoau <[email protected]> * update script Signed-off-by: andrusenkoau <[email protected]> * add rnnt and hat ngram decoding script Signed-off-by: andrusenkoau <[email protected]> * add some parameters Signed-off-by: andrusenkoau <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig Signed-off-by: andrusenkoau <[email protected]> * replace return_encoder_embeddings parameter Signed-off-by: andrusenkoau <[email protected]> * generalization of scipt behavior Signed-off-by: andrusenkoau <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <[email protected]> * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <[email protected]> * add manual encoder_embeddings calculation Signed-off-by: andrusenkoau <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 Signed-off-by: Andrei Andrusenko <[email protected]> * fix rescoring description Signed-off-by: Andrei Andrusenko <[email protected]> --------- Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * fix replace_bos_with_pad not found (#6443) (#6450) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * reduce workers on NMT CI (#6472) (#6474) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * 1. Added KERPLE positional embeddings to encoder-decoder. Signed-off-by: Micha Livne <[email protected]> * 1. Added a missing file. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: Jim O’Regan <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Kunal Dhawan <[email protected]> Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Cheng-Ping Hsieh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> Co-authored-by: Jim O’Regan <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: Mostafa Ghorbandoost <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]>

* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- * [TTS] FastPitch speaker encoder (#6417) * Add initial codes * Remove wemb * Fix import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss * Add ConditionalInput * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test * Add initial codes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix import error * Add initial codes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix dataset error * Remove reference speaker embedding * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove SV encoder * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix length type * Fix append * Move error msg * Add look-up into speaker encoder * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add valueerror msg * Move lookup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unused * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error * Rebase and Fix error * Fix spk encoder * Rename n_speakers * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error --------- * Sharded manifests for tarred datasets (#6395) * testing sharded manifests * compatibility * proper fixes * adding flag tot convert_to_tarred_audio_dataset * shard_manifests conf param * propagating the shard_manifests param * propagating the shard_manifests param * distributed checks * typo * typo * fixes * fixes * fixes * fixes * fixes * fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py * reversing manifest shards flag * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests * excluding manifests from webdataset url expansion * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore * explicit use of UTF-8 for manifest i/o --------- * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. --------- * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support * added tests for rampup batch size * fixed the typos * added assertions * changed assertion rules * deleted unused imports * changed tests for rampup batch size * updated rampup batch size tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling * rampup batch size tests changes --------- * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default * updated README with the new agg_tokenizer_manifest flag * fixed typo in scripts/speech_recognition/code_switching/README.md * changed agg_tokenizer_manifest to is_lid_manifest --------- * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script * add return encoding embedding option * update script * add rnnt and hat ngram decoding script * add some parameters * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig * replace return_encoder_embeddings parameter * generalization of scipt behavior * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter * remove return_encoder_embeddings parameter * add manual encoder_embeddings calculation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 * fix rescoring description --------- * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- * fix replace_bos_with_pad not found (#6443) (#6450) * reduce workers on NMT CI (#6472) (#6474) * 1. Added KERPLE positional embeddings to encoder-decoder. * 1. Added a missing file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: Jim O’Regan <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Kunal Dhawan <[email protected]> Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Cheng-Ping Hsieh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> Co-authored-by: Jim O’Regan <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: Mostafa Ghorbandoost <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]>

* added rampup batch size support Signed-off-by: Dmytro Pykhtar <[email protected]> * added tests for rampup batch size Signed-off-by: Dmytro Pykhtar <[email protected]> * fixed the typos Signed-off-by: Dmytro Pykhtar <[email protected]> * added assertions Signed-off-by: Dmytro Pykhtar <[email protected]> * changed assertion rules Signed-off-by: Dmytro Pykhtar <[email protected]> * deleted unused imports Signed-off-by: Dmytro Pykhtar <[email protected]> * changed tests for rampup batch size Signed-off-by: Dmytro Pykhtar <[email protected]> * updated rampup batch size tests Signed-off-by: Dmytro Pykhtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling Signed-off-by: Dmytro Pykhtar <[email protected]> * rampup batch size tests changes Signed-off-by: Dmytro Pykhtar <[email protected]> --------- Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: hsiehjackson <[email protected]>

* [TTS] FastPitch adapter fine-tune and conditional layer normalization (NVIDIA#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (NVIDIA#6416) --------- * [TTS] whitelist broken path fix. (NVIDIA#6412) * [TTS] whitelist broken path fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- * [TTS] FastPitch speaker encoder (NVIDIA#6417) * Add initial codes * Remove wemb * Fix import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss * Add ConditionalInput * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test * Add initial codes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix import error * Add initial codes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix dataset error * Remove reference speaker embedding * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove SV encoder * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix length type * Fix append * Move error msg * Add look-up into speaker encoder * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add valueerror msg * Move lookup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unused * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error * Rebase and Fix error * Fix spk encoder * Rename n_speakers * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error --------- * Sharded manifests for tarred datasets (NVIDIA#6395) * testing sharded manifests * compatibility * proper fixes * adding flag tot convert_to_tarred_audio_dataset * shard_manifests conf param * propagating the shard_manifests param * propagating the shard_manifests param * distributed checks * typo * typo * fixes * fixes * fixes * fixes * fixes * fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py * reversing manifest shards flag * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests * excluding manifests from webdataset url expansion * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore * explicit use of UTF-8 for manifest i/o --------- * Update wfst_text_normalization.rst (NVIDIA#6374) Add Hungarian (incoming in NeMo-text-processing) * Support Swiglu in TP PP Conversion (NVIDIA#6437) (NVIDIA#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- * Update NeMo_TTS_Primer.ipynb (NVIDIA#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. --------- * add rampup batch size support for Megatron GPT (NVIDIA#6424) * added rampup batch size support * added tests for rampup batch size * fixed the typos * added assertions * changed assertion rules * deleted unused imports * changed tests for rampup batch size * updated rampup batch size tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling * rampup batch size tests changes --------- * Meagtron encoder decoder fix for empty validation outputs (NVIDIA#6459) (NVIDIA#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (NVIDIA#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default * updated README with the new agg_tokenizer_manifest flag * fixed typo in scripts/speech_recognition/code_switching/README.md * changed agg_tokenizer_manifest to is_lid_manifest --------- * Added/updated new Conformer configs (NVIDIA#6426) (NVIDIA#6467) * Update script for ngram rnnt and hat beam search decoding (NVIDIA#6370) * add rnnt ngram beamsearch script * add return encoding embedding option * update script * add rnnt and hat ngram decoding script * add some parameters * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig * replace return_encoder_embeddings parameter * generalization of scipt behavior * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter * remove return_encoder_embeddings parameter * add manual encoder_embeddings calculation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 * fix rescoring description --------- * BERT pre-training mp fork to spawn (NVIDIA#6442) (NVIDIA#6454) * change bert fork to spawn * num_workers=0 fix --------- * fix replace_bos_with_pad not found (NVIDIA#6443) (NVIDIA#6450) * reduce workers on NMT CI (NVIDIA#6472) (NVIDIA#6474) * 1. Added KERPLE positional embeddings to encoder-decoder. * 1. Added a missing file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: Jim O’Regan <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Kunal Dhawan <[email protected]> Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Cheng-Ping Hsieh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> Co-authored-by: Jim O’Regan <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: Mostafa Ghorbandoost <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]>

…d Flash Attention (#6666) * move to nvidia megatron repo (#6465) (#6475) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Megatron KERPLE positional embeddings (#6478) (#6480) * [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- * [TTS] FastPitch speaker encoder (#6417) * Add initial codes * Remove wemb * Fix import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss * Add ConditionalInput * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test * Add initial codes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix import error * Add initial codes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix dataset error * Remove reference speaker embedding * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove SV encoder * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix length type * Fix append * Move error msg * Add look-up into speaker encoder * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add valueerror msg * Move lookup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unused * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error * Rebase and Fix error * Fix spk encoder * Rename n_speakers * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error --------- * Sharded manifests for tarred datasets (#6395) * testing sharded manifests * compatibility * proper fixes * adding flag tot convert_to_tarred_audio_dataset * shard_manifests conf param * propagating the shard_manifests param * propagating the shard_manifests param * distributed checks * typo * typo * fixes * fixes * fixes * fixes * fixes * fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py * reversing manifest shards flag * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests * excluding manifests from webdataset url expansion * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore * explicit use of UTF-8 for manifest i/o --------- * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. --------- * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support * added tests for rampup batch size * fixed the typos * added assertions * changed assertion rules * deleted unused imports * changed tests for rampup batch size * updated rampup batch size tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling * rampup batch size tests changes --------- * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default * updated README with the new agg_tokenizer_manifest flag * fixed typo in scripts/speech_recognition/code_switching/README.md * changed agg_tokenizer_manifest to is_lid_manifest --------- * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script * add return encoding embedding option * update script * add rnnt and hat ngram decoding script * add some parameters * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig * replace return_encoder_embeddings parameter * generalization of scipt behavior * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter * remove return_encoder_embeddings parameter * add manual encoder_embeddings calculation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 * fix rescoring description --------- * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- * fix replace_bos_with_pad not found (#6443) (#6450) * reduce workers on NMT CI (#6472) (#6474) * 1. Added KERPLE positional embeddings to encoder-decoder. * 1. Added a missing file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: Jim O’Regan <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Kunal Dhawan <[email protected]> Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Cheng-Ping Hsieh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> Co-authored-by: Jim O’Regan <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: Mostafa Ghorbandoost <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Fix an invalid link in get_data.py of ljspeech (#6456) Usage of the link in line 63 leads to downloading a html file not a tsv file, so we need to change it to a raw link. Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * 1. Added external index sample. (#6462) (#6483) Signed-off-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Update README to add core installation (#6488) (#6489) * update README for megatron-core * fix --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Fix cache aware hybrid bugs (#6466) (#6484) Signed-off-by: hsiehjackson <[email protected]> * Fix typos (#6494) (#6495) Signed-off-by: smajumdar <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Add disclaimer about dataset for ASR (#6496) Signed-off-by: smajumdar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * fix (#6502) datastore_path_to_webdataset_url(p) if is_datastore_path(p) and is_tarred_path(p) else p NameError: name 'is_tarred_path' is not defined Co-authored-by: George <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * fix broken links r1.18.0 (#6501) (#6504) * fix broken links * fix broken links --------- Signed-off-by: Evelina <[email protected]> Co-authored-by: Evelina <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * [TTS] Create functions for TTS preprocessing without dataloader (#6317) * [TTS] Create functions for TTS preprocessing without dataloader Signed-off-by: Ryan <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Cache aware streaming nfa (#6209) * add cache aware streaming to nemo aligner Signed-off-by: Slyne Deng <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator (#6499) * [BugFix] _get_batch_preds() is forced to keep logits in decoder timestamps generators Signed-off-by: Taejin Park <[email protected]> * Ingnore keep_logits boolean in FrameASRBatchLogits Signed-off-by: Taejin Park <[email protected]> --------- Signed-off-by: Taejin Park <[email protected]> Co-authored-by: Jagadeesh Balam <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * [TTS] Fix FastPitch energy code (#6511) Signed-off-by: Ryan <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * fix custom forward_torch_softmax (#6512) (#6517) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * [TTS] fixed broken path. (#6514) (#6518) Signed-off-by: Xuesong Yang <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Fix normalization of impulse response in ImpulsePerturbation (#6505) Signed-off-by: Ante Jukić <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Add interleaved pp support (#6498) * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <[email protected]> * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <[email protected]> * Switch to megatron core Signed-off-by: smajumdar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * Fix typos (#6523) * Fix typos Signed-off-by: smajumdar <[email protected]> * Fix typos Signed-off-by: smajumdar <[email protected]> --------- Signed-off-by: smajumdar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * New noise_norm perturbation based on Riva work (#6445) * Initial commit for new noise_norm perturbation Signed-off-by: Daniel Egert <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Minor fix to random seed in perturb Signed-off-by: Daniel Egert <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated code to reflect feedback Signed-off-by: Daniel Egert <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates for feedback given by code reviewers Signed-off-by: Daniel Egert <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates in response to PR feedback Signed-off-by: Daniel Egert <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added comment about ref_mic being None Signed-off-by: Daniel Egert <[email protected]> * Updated perturb to use inspect module Signed-off-by: Daniel Egert <[email protected]> --------- Signed-off-by: Daniel Egert <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * [TTS] Add script for computing feature stats (#6508) * [TTS] Add script for computing feature stats Signed-off-by: Ryan <[email protected]> * [TTS] Add overwrite config Signed-off-by: Ryan <[email protected]> --------- Signed-off-by: Ryan <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Add Frame-VAD model and datasets (#6441) * add model, dataset, necessary utils and tests Signed-off-by: stevehuang52 <[email protected]> * fix tarred data Signed-off-by: stevehuang52 <[email protected]> * fix typo Signed-off-by: stevehuang52 <[email protected]> * update docstring Signed-off-by: stevehuang52 <[email protected]> * update doc Signed-off-by: stevehuang52 <[email protected]> * update doc Signed-off-by: stevehuang52 <[email protected]> * update pretrained model info Signed-off-by: stevehuang52 <[email protected]> --------- Signed-off-by: stevehuang52 <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Support dynamic length batches with GPT SFT (#6510) * Support synamic length with GPT SFT Signed-off-by: Abhinav Khattar <[email protected]> * make branch functional Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * added back the fast emit section to the configs. (#6540) (#6542) * added back the fast emit section to the configs. * added back the fast emit section to the configs. --------- Signed-off-by: Vahid <[email protected]> Co-authored-by: Vahid Noroozi <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * removing unnessary avoid_bfloat16_autocast_context (#6481) Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * FC models in menu (#6473) * FC models in menu Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Dima Rekesh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * [TTS] Add tutorials for FastPitch TTS speaker adaptation with adapters (#6431) * Add tts adapter tutorial Signed-off-by: hsiehjackson <[email protected]> * Update main tutorial Signed-off-by: hsiehjackson <[email protected]> * Add tts adapter tutorial Signed-off-by: hsiehjackson <[email protected]> * Update main tutorial Signed-off-by: hsiehjackson <[email protected]> * Update tutorial Signed-off-by: hsiehjackson <[email protected]> * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments Signed-off-by: hsiehjackson <[email protected]> * Fix load .nemo error Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Support multi-speaker fine-tune Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments Signed-off-by: hsiehjackson <[email protected]> * Use .nemo Signed-off-by: hsiehjackson <[email protected]> * Follow Comments Signed-off-by: hsiehjackson <[email protected]> * Fix bug Signed-off-by: hsiehjackson <[email protected]> * Fix bug Signed-off-by: hsiehjackson <[email protected]> * Fix bug Signed-off-by: hsiehjackson <[email protected]> * Add precomputed speaker emb Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix space Signed-off-by: hsiehjackson <[email protected]> * Remove repeated argument Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * optional batch size Signed-off-by: hsiehjackson <[email protected]> * Fix comments in notebook Signed-off-by: hsiehjackson <[email protected]> --------- Signed-off-by: hsiehjackson <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * [TTS] Create initial TTS dataset feature processors (#6507) Signed-off-by: Ryan <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * fix (#6529) (#6546) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) (#6553) * Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by * updated ASR docs with the fastconformer hybrid checkpoints * added the fastconformer RNNT and CTC models --------- Signed-off-by: KunalDhawan <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Add scores for FastConformer models (#6557) (#6558) Signed-off-by: smajumdar <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Fix fp16 (#6543) (#6544) Signed-off-by: MaximumEntropy <[email protected]> Co-authored-by: Sandeep Subramanian <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Patch transcribe and support offline transcribe for hybrid model (#6550) (#6559) Signed-off-by: fayejf <[email protected]> Co-authored-by: fayejf <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Fix notebook bad json (#6561) Signed-off-by: smajumdar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Change Megatron Enc Dec model to use persistent_workers (#6548) (#6552) * persistent workers * fix --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Make KenLM with PC for AggregateTokenizer and merge it (#6081) * do_lowercase, rm_punctuation Signed-off-by: Nikolay Karpov <[email protected]> * support beam_strategy = beam Signed-off-by: Nikolay Karpov <[email protected]> * black Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix config and^Cunctuation capitalization Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * rm math Signed-off-by: Nikolay Karpov <[email protected]> * update kenlm Signed-off-by: Nikolay Karpov <[email protected]> * black Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add opengrm Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * mv install_beamsearch_decoders Signed-off-by: Nikolay Karpov <[email protected]> * punctuation_to_preserve Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only tikenizer opion Signed-off-by: Nikolay Karpov <[email protected]> * Black Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * DEFAULT_TOKEN_OFFSET Signed-off-by: Nikolay Karpov <[email protected]> * aggregate_tokenizer Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install kenlm with more than 5gram Signed-off-by: Nikolay Karpov <[email protected]> * install_beamsearch_decoders Signed-off-by: Nikolay Karpov <[email protected]> * ngram_bin_path kenlm_bin_path Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * black Signed-off-by: Nikolay Karpov <[email protected]> * fix greedy PC bug Signed-off-by: Nikolay Karpov <[email protected]> * move global params Signed-off-by: Nikolay Karpov <[email protected]> * fix description and perplexity Signed-off-by: Nikolay Karpov <[email protected]> * fix description Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * NEMO_PATH Signed-off-by: Nikolay Karpov <[email protected]> * nemo:23.01 Signed-off-by: Nikolay Karpov <[email protected]> * License Signed-off-by: Nikolay Karpov <[email protected]> * description Signed-off-by: Nikolay Karpov <[email protected]> * isinstance Signed-off-by: Nikolay Karpov <[email protected]> * refactor kenlm stdin Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * black Signed-off-by: Nikolay Karpov <[email protected]> * add cmd arg Signed-off-by: Nikolay Karpov <[email protected]> * use new iter_files Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * EncDecHybridRNNTCTCModel Signed-off-by: Nikolay Karpov <[email protected]> * punctuation Signed-off-by: Nikolay Karpov <[email protected]> * train_kenlm args Signed-off-by: Nikolay Karpov <[email protected]> * add docstrings Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add ngram_merge docs Signed-off-by: Nikolay Karpov <[email protected]> * ngram_prune Signed-off-by: Nikolay Karpov <[email protected]> * rename to ngram_merge Signed-off-by: Nikolay Karpov <[email protected]> * rename to ngram Signed-off-by: Nikolay Karpov <[email protected]> * add comments Signed-off-by: Nikolay Karpov <[email protected]> * Ngram Signed-off-by: Nikolay Karpov <[email protected]> * nemo_model_file Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install_opengrm_ngram Signed-off-by: Nikolay Karpov <[email protected]> * install opengrm Signed-off-by: Nikolay Karpov <[email protected]> * rename to install_opengrm.sh Signed-off-by: Nikolay Karpov <[email protected]> * rm extra import Signed-off-by: Nikolay Karpov <[email protected]> * train_paths Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * text_processing Signed-off-by: Nikolay Karpov <[email protected]> * fix ngram_bin_path Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * DECODERS_PATH Signed-off-by: Nikolay Karpov <[email protected]> * farcompile Signed-off-by: Nikolay Karpov <[email protected]> * rm text processing Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * text_processing Signed-off-by: Nikolay Karpov <[email protected]> * AggregateTokenizer.DummyTokenizer Signed-off-by: Nikolay Karpov <[email protected]> * comments Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * TextProcessingConfig Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * typo Signed-off-by: Nikolay Karpov <[email protected]> * doc Signed-off-by: Nikolay Karpov <[email protected]> * types Signed-off-by: Nikolay Karpov <[email protected]> * nemo_model_file Signed-off-by: Nikolay Karpov <[email protected]> * rm assert Signed-off-by: Nikolay Karpov <[email protected]> * import kenlm_utils Signed-off-by: Nikolay Karpov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * return None Signed-off-by: Nikolay Karpov <[email protected]> * Copyright Signed-off-by: Nikolay Karpov <[email protected]> * 2022 Signed-off-by: Nikolay Karpov <[email protected]> * 2023 Signed-off-by: Nikolay Karpov <[email protected]> --------- Signed-off-by: Nikolay Karpov <[email protected]> Signed-off-by: Nikolay Karpov <[email protected]> Co-authored-by: Nikolay Karpov <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * fix for running on 1 GPU. Signed-off-by: hsiehjackson <[email protected]> * temp rtd fix (#6568) (#6569) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * [TTS] Add script for mapping speaker names to indices (#6509) Signed-off-by: Ryan <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * whitespace (#6574) Signed-off-by: Nikolay Karpov <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Update manifest.py for speedup (#6565) (#6573) * Update manifest.py Re-order the checks for faster processing audio filepaths that are already absolute paths * Update manifest.py --------- Signed-off-by: He Huang (Steve) <[email protected]> Co-authored-by: He Huang (Steve) <[email protected]> Co-authored-by: Vahid Noroozi <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * More streaming conformer export fixes (#6567) (#6578) Signed-off-by: Greg Clark <[email protected]> Co-authored-by: Greg Clark <[email protected]> Co-authored-by: Vahid Noroozi <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * user selected max_seq_len should be less than model's max_seq_len (#6333) (#6386) * user selection should not break model max limit * eval max seq length --------- Signed-off-by: arendu <[email protected]> Signed-off-by: Adi Renduchintala <[email protected]> Co-authored-by: Adi Renduchintala <[email protected]> Co-authored-by: Sandeep Subramanian <[email protected]> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Framework for PEFT via mixins (#6391) * init commit ptuning via mixin Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates Signed-off-by: arendu <[email protected]> * gpt ptuning places virtual tokens on the left only Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * encoder input modified when pre_process is true Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * optimizer group and state dict updates Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adapter ptuning working for pp>1 Signed-off-by: arendu <[email protected]> * adapter defaults Signed-off-by: arendu <[email protected]> * adapter ptuining config defaults Signed-off-by: arendu <[email protected]> * training works Signed-off-by: arendu <[email protected]> * loading and saving adapter only params during training Signed-off-by: arendu <[email protected]> * added checks and comments Signed-off-by: arendu <[email protected]> * clean up Signed-off-by: arendu <[email protected]> * checks for grad is None before calling all_reduce Signed-off-by: arendu <[email protected]> * load adapter .nemo file working Signed-off-by: arendu <[email protected]> * resume training for adapters Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * peft tuning Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor Signed-off-by: arendu <[email protected]> * file not needed Signed-off-by: arendu <[email protected]> * undo prompt learning dataset changes Signed-off-by: arendu <[email protected]> * undo updates to gpt prompt learning model Signed-off-by: arendu <[email protected]> * naming updates Signed-off-by: arendu <[email protected]> * decoding Signed-off-by: arendu <[email protected]> * predict_step in gpt_sft_model Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed inference from tuning config Signed-off-by: arendu <[email protected]> * no test in peft training Signed-off-by: arendu <[email protected]> * answer only loss and correct defaults for val_loss Signed-off-by: arendu <[email protected]> * hybrid adapters and ptuning Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * eval working.. Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * prepending tokens for ptuning Signed-off-by: arendu <[email protected]> * cleaned up eval config Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * clean up Signed-off-by: arendu <[email protected]> * update Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * default prompt template Signed-off-by: arendu <[email protected]> * Lora added Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Support synamic length with GPT SFT Signed-off-by: Abhinav Khattar <[email protected]> * make branch functional Signed-off-by: Abhinav Khattar <[email protected]> * defaults to max_pad_length=False in GPT SFT dataset Signed-off-by: arendu <[email protected]> * adapter parallel_adapters to support Lora Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * added early stopping by default Signed-off-by: arendu <[email protected]> * eval script for peft and eval config. bug fixes in predict step and added out_features to t5 adapter config Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docs Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * better defaults Signed-off-by: arendu <[email protected]> * updates Signed-off-by: arendu <[email protected]> * update Signed-off-by: arendu <[email protected]> * docs Signed-off-by: arendu <[email protected]> --------- Signed-off-by: arendu <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Signed-off-by: Adi Renduchintala <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * cache and reuse inputs (#6422) (#6452) Co-authored-by: Sangkug Lym <[email protected]> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Add patches for Virtual Parallel conversion (#6589) * Add patches for Virtual Parllel conversion Signed-off-by: smajumdar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * Pass `.scale` instead of scaler object to core (#6551) * pass .scale instead of scaler object to core (#6545) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Update megatron_gpt_model.py Signed-off-by: Abhinav Khattar <[email protected]> * scale changes for main Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Documentation for ASR-TTS models (#6594) (#6595) * Add docs about hybrid ASR-TTS models * Add docs about text-only datasets * Add docs about ASR-TTS checkpoints * Add docs about ASR-TTS configs and training * Clean up * ASR-TTS docs: add to api, fix imports * Clean up * Wrap optional import * Revert general ASR import --------- Signed-off-by: Vladimir Bataev <[email protected]> Co-authored-by: Vladimir Bataev <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * [TTS] Fix aligner nan loss in fp32 (#6435) * Fix nan loss in fp32 Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: hsiehjackson <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * Update SDP docs (#6485) (#6596) * add info about SDP e.g. processor classes in docs * add link to SDP docs in README * address code review comments and add SDP overview diagram * Fix spelling typo --------- Signed-off-by: Elena Rastorgueva <[email protected]> Co-authored-by: Elena Rastorgueva <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Bug/typo fixes (#6599) Signed-off-by: Igor Gitman <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Manual garbage collection with an interval (#6469) (#6482) * Manual garbage collection with an interval * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * use trainer.global_step for tracking the interval of GC --------- Signed-off-by: Sangkug Lym <[email protected]> Co-authored-by: Sangkug Lym <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Make tensor split contiguous (#6580) (#6593) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * [ASR] Fix for old models in change_attention_model (#6608) * fixes Signed-off-by: sam1373 <[email protected]> * done already Signed-off-by: sam1373 <[email protected]> --------- Signed-off-by: sam1373 <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Update manifest.py to use os.path for get_full_path (#6598) * Update manifest.py to use os.path for get_full_path Signed-off-by: He Huang (Steve) <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py to get rid of pathlib Signed-off-by: He Huang (Steve) <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py Signed-off-by: He Huang (Steve) <[email protected]> * Update manifest.py Signed-off-by: He Huang (Steve) <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: He Huang (Steve) <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Vahid Noroozi <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Cherry pick commits in #6601 to main (#6611) * fix write Signed-off-by: fayejf <[email protected]> * decoding ctc Signed-off-by: fayejf <[email protected]> * temp set rnnt decoding return_best_hypothesis to true Signed-off-by: fayejf <[email protected]> * add wer cal back to transcribe_speech as requested Signed-off-by: fayejf <[email protected]> * add wer cal back to speech_to_text_buffered_infer_rnnt as requested Signed-off-by: fayejf <[email protected]> * add wer cal back to speech_to_text_buffered_infer_ctc as requested Signed-off-by: fayejf <[email protected]> * style fix Signed-off-by: fayejf <[email protected]> * reflect change in asr_evaluator Signed-off-by: fayejf <[email protected]> * reflect som and vahid comment Signed-off-by: fayejf <[email protected]> * remove return_best_hy=true in transcribe_speech Signed-off-by: fayejf <[email protected]> * no text skip Signed-off-by: fayejf <[email protected]> * revert partial Signed-off-by: fayejf <[email protected]> --------- Signed-off-by: fayejf <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Create dummy iters to satisy len checks (#6600) (#6603) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * add GPT eval mode fix for interleaved to main (#6610) Signed-off-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Fix batch size reconf for T5 FT for multi-validation (#6582) (#6588) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Not doing CastToFloat by default (#6524) (#6563) * Not doing CastToFloat by default * Added docustring * Dummy commit --------- Signed-off-by: Boris Fomitchev <[email protected]> Co-authored-by: Boris Fomitchev <[email protected]> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Turn autocast off when precision is fp32 (#6576) * Turn autocast off when precision is fp32 (#6554) * Turn autocast off when precision is fp32 Signed-off-by: Abhinav Khattar <[email protected]> * address review Signed-off-by: Abhinav Khattar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: Abhinav Khattar <[email protected]> * merge Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> * correct auto-merge Signed-off-by: Abhinav Khattar <[email protected]> * correct auto-merge Signed-off-by: Abhinav Khattar <[email protected]> * add to GPT SFT Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * update core commit hash in readme (#6622) (#6623) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * add hat image to docs (#6619) (#6621) Signed-off-by: andrusenkoau <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Allow indices exchange via distributed (#6618) (#6624) Signed-off-by: Mikołaj Błaż <[email protected]> Co-authored-by: mikolajblaz <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Offline and streaming inference support for hybrid model (#6570) * streaming buffered for hybrid + ctc Signed-off-by: fayejf <[email protected]> * change default model_stride in eval.yaml Signed-off-by: fayejf <[email protected]> * add fc model_stride Signed-off-by: fayejf <[email protected]> * small fix Signed-off-by: fayejf <[email protected]> * check whether model and decoding match Signed-off-by: fayejf <[email protected]> * small fix Signed-off-by: fayejf <[email protected]> * streaming buffered for hybrid + rnnt Signed-off-by: fayejf <[email protected]> * style fix Signed-off-by: fayejf <[email protected]> * fix yaml Signed-off-by: fayejf <[email protected]> * reflect comment wip Signed-off-by: fayejf <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: fayejf <[email protected]> * refactor and verified Signed-off-by: fayejf <[email protected]> * add get_full_path to buffered Signed-off-by: fayejf <[email protected]> * small fix Signed-off-by: fayejf <[email protected]> * add RNNTDecodingConfig Signed-off-by: fayejf <[email protected]> * model name & instruction of changing decoding Signed-off-by: fayejf <[email protected]> --------- Signed-off-by: fayejf <[email protected]> Signed-off-by: fayejf <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * Patch decoding for PC models (#6630) (#6631) * Patch decoding logic for PC models * Patch decoding logic for PC models --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Fix wer.py where 'errors' variable was not set (#6633) (#6634) Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings Signed-off-by: He Huang (Steve) <[email protected]> Co-authored-by: He Huang (Steve) <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Restore GPT support for interleaved pipeline parallelism (#6528) (#6613) * Restore logic for data-parallel communication with pipeline parallelism in GPT * Support dynamic attention masks in GPT * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Debug typos * Debug data iterator caching with interleaved pipeline parallelism Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Megatron-LM commit * Distinguish between list of data iterators and data iterator that is a list * Create dummy iters to satisy len checks * Kludge while waiting for Megatron-LM update * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set transformers offline to avoid rate limiting --------- Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Eric Harper <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Signed-off-by: ericharper <[email protected]> Signed-off-by: Tim Moon <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Add FA Signed-off-by: hsiehjackson <[email protected]> * Fix XPOS Signed-off-by: hsiehjackson <[email protected]> * Add warning Signed-off-by: hsiehjackson <[email protected]> * Fix bugs Signed-off-by: hsiehjackson <[email protected]> * Fix attention Signed-off-by: hsiehjackson <[email protected]> * Fix comment Signed-off-by: hsiehjackson <[email protected]> * Fix cast dtype Signed-off-by: hsiehjackson <[email protected]> * Undo xpos Signed-off-by: hsiehjackson <[email protected]> * bugfix (#6636) Signed-off-by: fayejf <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Disable interctc tests (#6638) Signed-off-by: Igor Gitman <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Add megatron_core to requirements (#6639) (#6640) * add megatron_core to requirements * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ericharper <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * Remove from jenkins (#6642) * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <[email protected]> * remove from jenkins Signed-off-by: ericharper <[email protected]> --------- Signed-off-by: ericharper <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove dup Signed-off-by: ericharper <[email protected]> --------- Signed-off-by: ericharper <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * sft model can use this script for eval (#6637) * sft model can use this script for eval Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * please fix me Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor Signed-off-by: arendu <[email protected]> --------- Signed-off-by: arendu <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * [TTS] Fix TTS audio preprocessing bugs (#6628) Signed-off-by: Ryan <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Move black parameters to pyproject.toml (#6647) Signed-off-by: Vladimir Bataev <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * ASR-TTS Models: Support hybrid RNNT-CTC, improve docs. (#6620) * ASR-TTS: support hybrid RNNT-CTC models * Do not warn on optional import * Explain adding options to config * Fix import guard docs * Add docs for ConcatDataset * Add explanation for sampling parameters * Initial docs for the enhancer model * Fix use_start_end_token parameter usage --------- Signed-off-by: Vladimir Bataev <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * fix conversion and eval (#6648) * fix conversion and eval Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: arendu <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * Confidence ensembles implementation (#6614) * Working version to train conf model + save ensemble class Signed-off-by: Igor Gitman <[email protected]> * Working version Signed-off-by: Igor Gitman <[email protected]> * Remove copy of transcribe_speech.py Signed-off-by: Igor Gitman <[email protected]> * Move models parameter to config Signed-off-by: Igor Gitman <[email protected]> * Add explicit parameters to transcribe Signed-off-by: Igor Gitman <[email protected]> * Small cleanups Signed-off-by: Igor Gitman <[email protected]> * Add temperature and integration tests Signed-off-by: Igor Gitman <[email protected]> * Add more tests Signed-off-by: Igor Gitman <[email protected]> * Add pc removal config Signed-off-by: Igor Gitman <[email protected]> * Cleanup Signed-off-by: Igor Gitman <[email protected]> * Fix typo Signed-off-by: Igor Gitman <[email protected]> * Address review comments Signed-off-by: Igor Gitman <[email protected]> --------- Signed-off-by: Igor Gitman <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Patch memory used for NeMo Megatron models (#6615) * Patch memory used for NeMo Megatron models Signed-off-by: smajumdar <[email protected]> * Cleanup the dtype of embeddings Signed-off-by: smajumdar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor util function for parsing precision Signed-off-by: smajumdar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor util function for parsing precision Signed-off-by: smajumdar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Try patch for Megatron O2 Signed-off-by: smajumdar <[email protected]> * Refactor to incorporate megatron amp 02 state Signed-off-by: smajumdar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor to incorporate megatron amp 02 state Signed-off-by: smajumdar <[email protected]> * Correct indent Signed-off-by: smajumdar <[email protected]> * Correct utils import Signed-off-by: smajumdar <[email protected]> --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * handle artifacts when path is dir (#6658) Signed-off-by: arendu <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * remove upgrading setuptools in reinstall.sh (#6659) Signed-off-by: Xuesong Yang <[email protected]> Co-authored-by: fayejf <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * merge lora weights into base model (#6597) * merge lora weights into base model Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * typo fix Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor update Signed-off-by: arendu <[email protected]> * update copyright Signed-off-by: arendu <[email protected]> * eval needs to know the PEFT class Signed-off-by: arendu <[email protected]> * add target class in training script so that we can use it in eval Signed-off-by: arendu <[email protected]> * update Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update to work for tp1 Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set restore model path Signed-off-by: arendu <[email protected]> * peft can be none Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updated merge script so that eval works easily Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * eval with peft or sft model Signed-off-by: arendu <[email protected]> * keep sentences in jsonl format Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * convert sft using correct classpath Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updated to force sft yaml to have the correct target Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updated docs Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix conversion and eval Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: arendu <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <[email protected]> * upgrade to 23.04 (#6660) Signed-off-by: ericharper <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Merge r1.18.0 bugfixes and doc updates to main (#6655) * update branch Signed-off-by: ericharper <[email protected]> * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <[email protected]> * remove from jenkins Signed-off-by: ericharper <[email protected]> --------- Signed-off-by: ericharper <[email protected]> * remove dup Signed-off-by: ericharper <[email protected]> * update branch Signed-off-by: ericharper <[email protected]> * [TTS] reformat NeMo versions in the tts logging messages to avoid batch process them when upgrading NeMo versions. Signed-off-by: Xuesong Yang <[email protected]> --------- Signed-off-by: ericharper <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Confidence ensembles: fix issues and add tuning functionality (#6657) * Implement compute confidence to properly handle blanks Signed-off-by: Igor Gitman <[email protected]> * Implement proper confidence for transducers Signed-off-by: Igor Gitman <[email protected]> * Implement tuning logic Signed-off-by: Igor Gitman <[email protected]> * Add tests for confidence tuning Signed-off-by: Igor Gitman <[email protected]> * Remove unused imports Signed-off-by: Igor Gitman <[email protected]> * Add types/docs Signed-off-by: Igor Gitman <[email protected]> * Add comment about the main conf compute loop Signed-off-by: Igor Gitman <[email protected]> --------- Signed-off-by: Igor Gitman <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * [TTS] Implement new TextToSpeech dataset (#6575) * [TTS] Implement new TextToSpeech dataset Signed-off-by: Ryan <[email protected]> * [TTS] Add unit tests Signed-off-by: Ryan <[email protected]> * [TTS] Fix defaulting of use_log_energy Signed-off-by: Ryan <[email protected]> * [TTS] Fix TTS export test Signed-off-by: Ryan <[email protected]> --------- Signed-off-by: Ryan <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Dialogue dataset (#6654) * chatbot interface Signed-off-by: Yi Dong <[email protected]> * latest gradio Signed-off-by: Yi Dong <[email protected]> * default greedy Signed-off-by: Yi Dong <[email protected]> * better chatbot Signed-off-by: Yi Dong <[email protected]> * handle preamble Signed-off-by: Yi Dong <[email protected]> * added chatbot training capablity Signed-off-by: Yi Dong <[email protected]> * added chatbot ui Signed-off-by: Yi Dong <[email protected]> * remove debug code Signed-off-by: Yi Dong <[email protected]> * default human Signed-off-by: Yi Dong <[email protected]> * use special token for roles Signed-off-by: Yi Dong <[email protected]> * special tokens Signed-off-by: Yi Dong <[email protected]> * fix name Signed-off-by: Yi Dong <[email protected]> * new chat dataset Signed-off-by: Yi Dong <[email protected]> * fix the system token Signed-off-by: Yi Dong <[email protected]> * upgrade gradio Signed-off-by: Yi Dong <[email protected]> * save the chat history Signed-off-by: Yi Dong <[email protected]> * update ui Signed-off-by: root <[email protected]> * update chat interface Signed-off-by: Yi Dong <[email protected]> * handles canonical form Signed-off-by: Yi Dong <[email protected]> * new sft chatbot Signed-off-by: Yi Dong <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * change format Signed-off-by: Yi Dong <[email protected]> * check extra_id in the tokenizer Signed-off-by: Yi Dong <[email protected]> * added vocab property check Signed-off-by: Yi Dong <[email protected]> * added missing file Signed-off-by: Yi Dong <[email protected]> --------- Signed-off-by: Yi Dong <[email protected]> Signed-off-by: root <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sandeep Subramanian <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Add support for RNNT/hybrid models to partial transcribe (#6609) * Add support for RNNT/hybrid models to partial transcribe Signed-off-by: He Huang (Steve) <[email protected]> * Update transcribe_utils.py Signed-off-by: He Huang (Steve) <[email protected]> * Update transcribe_speech.py Signed-off-by: He Huang (Steve) <[email protected]> * Update transcr…

* peft eval directly from ckpt (#6785) * update to load from ckpt Signed-off-by: arendu <[email protected]> * update Signed-off-by: arendu <[email protected]> * load ckpt peft model Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update style Signed-off-by: arendu <[email protected]> --------- Signed-off-by: arendu <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add Frame-VAD examples and utils (#6463) * add model, dataset, necessary utils and tests Signed-off-by: stevehuang52 <[email protected]> * fix tarred data Signed-off-by: stevehuang52 <[email protected]> * fix typo Signed-off-by: stevehuang52 <[email protected]> * add fvad examples and update utils Signed-off-by: stevehuang52 <[email protected]> * add copyright Signed-off-by: stevehuang52 <[email protected]> * refactor and add tests Signed-off-by: stevehuang52 <[email protected]> * update dataset Signed-off-by: stevehuang52 <[email protected]> * update test Signed-off-by: stevehuang52 <[email protected]> * refactor Signed-off-by: stevehuang52 <[email protected]> * refactor Signed-off-by: stevehuang52 <[email protected]> * fix typos Signed-off-by: stevehuang52 <[email protected]> --------- Signed-off-by: stevehuang52 <[email protected]> Co-authored-by: fayejf <[email protected]> Co-authored-by: Taejin Park <[email protected]> * [TTS][zh] refine hardcoded lowercase for ASCII letters. (#6781) Signed-off-by: Xuesong Yang <[email protected]> * Spellchecking ASR customization model (#6179) * bug fixes Signed-off-by: Alexandra Antonova <[email protected]> * fix bugs, add preparation and evaluation scripts, add readme Signed-off-by: Alexandra Antonova <[email protected]> * small fixes Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add real coverage calculation, small fixes, more debug information Signed-off-by: Alexandra Antonova <[email protected]> * add option to pass a filelist and output folder - to handle inference from multiple input files Signed-off-by: Alexandra Antonova <[email protected]> * added preprocessing for yago wikipedia articles - finding yago entities and their subphrases Signed-off-by: Alexandra Antonova <[email protected]> * yago wiki preprocessing, sampling, pseudonormalization Signed-off-by: Alexandra Antonova <[email protected]> * more scripts for preparation of training examples Signed-off-by: Alexandra Antonova <[email protected]> * bug fixes Signed-off-by: Alexandra Antonova <[email protected]> * add some alphabet checks Signed-off-by: Alexandra Antonova <[email protected]> * add bert on subwords, concatenate it to bert on characters Signed-off-by: Alexandra Antonova <[email protected]> * add calculation of character_pos_to_subword_pos Signed-off-by: Alexandra Antonova <[email protected]> * bug fix Signed-off-by: Alexandra Antonova <[email protected]> * bug fix Signed-off-by: Alexandra Antonova <[email protected]> * pdb Signed-off-by: Alexandra Antonova <[email protected]> * tensor join bug fix Signed-off-by: Alexandra Antonova <[email protected]> * double hidden_size in classifier Signed-off-by: Alexandra Antonova <[email protected]> * pdb Signed-off-by: Alexandra Antonova <[email protected]> * default index value 0 instead of -1 because index cannot be negative Signed-off-by: Alexandra Antonova <[email protected]> * pad index value 0 instead of -1 because index cannot be negative Signed-off-by: Alexandra Antonova <[email protected]> * remove pdb Signed-off-by: Alexandra Antonova <[email protected]> * fix bugs, add creation of tarred dataset Signed-off-by: Alexandra Antonova <[email protected]> * add possibility to change sequence len at inference Signed-off-by: Alexandra Antonova <[email protected]> * change sampling of dummy candidates at inference, add candidate info file Signed-off-by: Alexandra Antonova <[email protected]> * fix import Signed-off-by: Alexandra Antonova <[email protected]> * fix bug Signed-off-by: Alexandra Antonova <[email protected]> * update transcription now uses info Signed-off-by: Alexandra Antonova <[email protected]> * write path Signed-off-by: Alexandra Antonova <[email protected]> * 1. add tarred dataset support(untested). 2. fix bug with ban_ngrams in indexing Signed-off-by: Alexandra Antonova <[email protected]> * skip short_sent if no real candidates Signed-off-by: Alexandra Antonova <[email protected]> * fix import Signed-off-by: Alexandra Antonova <[email protected]> * add braceexpand Signed-off-by: Alexandra Antonova <[email protected]> * fixes Signed-off-by: Alexandra Antonova <[email protected]> * fix bug Signed-off-by: Alexandra Antonova <[email protected]> * fix bug Signed-off-by: Alexandra Antonova <[email protected]> * fix bug in np.ones Signed-off-by: Alexandra Antonova <[email protected]> * fix bug in collate Signed-off-by: Alexandra Antonova <[email protected]> * change tensor type to long because of error in torch.gather Signed-off-by: Alexandra Antonova <[email protected]> * fix for empty spans tensor Signed-off-by: Alexandra Antonova <[email protected]> * same fixes in _collate_fn for tarred dataset Signed-off-by: Alexandra Antonova <[email protected]> * fix bug from previous commit Signed-off-by: Alexandra Antonova <[email protected]> * change int types to be shorter to minimize tar size Signed-off-by: Alexandra Antonova <[email protected]> * refactoring of datasets and inference Signed-off-by: Alexandra Antonova <[email protected]> * bug fix Signed-off-by: Alexandra Antonova <[email protected]> * bug fix Signed-off-by: Alexandra Antonova <[email protected]> * bug fix Signed-off-by: Alexandra Antonova <[email protected]> * tar by 100k examples, small fixes Signed-off-by: Alexandra Antonova <[email protected]> * small fixes, add analytics script Signed-off-by: Alexandra Antonova <[email protected]> * Add functions for dynamic programming comparison to get best path by ngrams Signed-off-by: Alexandra Antonova <[email protected]> * fixes Signed-off-by: Alexandra Antonova <[email protected]> * small fix Signed-off-by: Alexandra Antonova <[email protected]> * fixes to support testing on SPGISpeech Signed-off-by: Alexandra Antonova <[email protected]> * add preprocessing for userlibri Signed-off-by: Alexandra Antonova <[email protected]> * some refactoring Signed-off-by: Alexandra Antonova <[email protected]> * some refactoring Signed-off-by: Alexandra Antonova <[email protected]> * move some functions to utils to reuse from other project Signed-off-by: Alexandra Antonova <[email protected]> * move some functions to utils to reuse from other project Signed-off-by: Alexandra Antonova <[email protected]> * move some functions to utils to reuse from other project Signed-off-by: Alexandra Antonova <[email protected]> * small refactoring before pr. Add bash-scripts reproducing evaluation Signed-off-by: Alexandra Antonova <[email protected]> * style fix Signed-off-by: Alexandra Antonova <[email protected]> * small fixes in inference Signed-off-by: Alexandra Antonova <[email protected]> * bug fix - didn't move window on last symbol Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bug - shuffle was before truncation of sorted candidates Signed-off-by: Alexandra Antonova <[email protected]> * refactoring, fix some bugs Signed-off-by: Alexandra Antonova <[email protected]> * variour fixes. Add word_indices at inference Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add candidate positions Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Move data preparation and evaluation to other repo Signed-off-by: Alexandra Antonova <[email protected]> * add infer_reproduce_paper. Refactoring Signed-off-by: Alexandra Antonova <[email protected]> * refactor inference using fragment indices Signed-off-by: Alexandra Antonova <[email protected]> * add some helper functions Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bug with parameters order Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bugs Signed-off-by: Alexandra Antonova <[email protected]> * refactoring, fix bug Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add multiple variants of adjusting start/end positions Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * more fixes Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add unit tests, other fixes Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: Alexandra Antonova <[email protected]> * fix CodeQl warnings Signed-off-by: Alexandra Antonova <[email protected]> * bug fixes Signed-off-by: Alexandra Antonova <[email protected]> * fix bugs, add preparation and evaluation scripts, add readme Signed-off-by: Alexandra Antonova <[email protected]> * small fixes Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add real coverage calculation, small fixes, more debug information Signed-off-by: Alexandra Antonova <[email protected]> * add option to pass a filelist and output folder - to handle inference from multiple input files Signed-off-by: Alexandra Antonova <[email protected]> * added preprocessing for yago wikipedia articles - finding yago entities and their subphrases Signed-off-by: Alexandra Antonova <[email protected]> * yago wiki preprocessing, sampling, pseudonormalization Signed-off-by: Alexandra Antonova <[email protected]> * more scripts for preparation of training examples Signed-off-by: Alexandra Antonova <[email protected]> * bug fixes Signed-off-by: Alexandra Antonova <[email protected]> * add some alphabet checks Signed-off-by: Alexandra Antonova <[email protected]> * add bert on subwords, concatenate it to bert on characters Signed-off-by: Alexandra Antonova <[email protected]> * add calculation of character_pos_to_subword_pos Signed-off-by: Alexandra Antonova <[email protected]> * bug fix Signed-off-by: Alexandra Antonova <[email protected]> * bug fix Signed-off-by: Alexandra Antonova <[email protected]> * pdb Signed-off-by: Alexandra Antonova <[email protected]> * tensor join bug fix Signed-off-by: Alexandra Antonova <[email protected]> * double hidden_size in classifier Signed-off-by: Alexandra Antonova <[email protected]> * pdb Signed-off-by: Alexandra Antonova <[email protected]> * default index value 0 instead of -1 because index cannot be negative Signed-off-by: Alexandra Antonova <[email protected]> * pad index value 0 instead of -1 because index cannot be negative Signed-off-by: Alexandra Antonova <[email protected]> * remove pdb Signed-off-by: Alexandra Antonova <[email protected]> * fix bugs, add creation of tarred dataset Signed-off-by: Alexandra Antonova <[email protected]> * add possibility to change sequence len at inference Signed-off-by: Alexandra Antonova <[email protected]> * change sampling of dummy candidates at inference, add candidate info file Signed-off-by: Alexandra Antonova <[email protected]> * fix import Signed-off-by: Alexandra Antonova <[email protected]> * fix bug Signed-off-by: Alexandra Antonova <[email protected]> * update transcription now uses info Signed-off-by: Alexandra Antonova <[email protected]> * write path Signed-off-by: Alexandra Antonova <[email protected]> * 1. add tarred dataset support(untested). 2. fix bug with ban_ngrams in indexing Signed-off-by: Alexandra Antonova <[email protected]> * skip short_sent if no real candidates Signed-off-by: Alexandra Antonova <[email protected]> * fix import Signed-off-by: Alexandra Antonova <[email protected]> * add braceexpand Signed-off-by: Alexandra Antonova <[email protected]> * fixes Signed-off-by: Alexandra Antonova <[email protected]> * fix bug Signed-off-by: Alexandra Antonova <[email protected]> * fix bug Signed-off-by: Alexandra Antonova <[email protected]> * fix bug in np.ones Signed-off-by: Alexandra Antonova <[email protected]> * fix bug in collate Signed-off-by: Alexandra Antonova <[email protected]> * change tensor type to long because of error in torch.gather Signed-off-by: Alexandra Antonova <[email protected]> * fix for empty spans tensor Signed-off-by: Alexandra Antonova <[email protected]> * same fixes in _collate_fn for tarred dataset Signed-off-by: Alexandra Antonova <[email protected]> * fix bug from previous commit Signed-off-by: Alexandra Antonova <[email protected]> * change int types to be shorter to minimize tar size Signed-off-by: Alexandra Antonova <[email protected]> * refactoring of datasets and inference Signed-off-by: Alexandra Antonova <[email protected]> * bug fix Signed-off-by: Alexandra Antonova <[email protected]> * bug fix Signed-off-by: Alexandra Antonova <[email protected]> * bug fix Signed-off-by: Alexandra Antonova <[email protected]> * tar by 100k examples, small fixes Signed-off-by: Alexandra Antonova <[email protected]> * small fixes, add analytics script Signed-off-by: Alexandra Antonova <[email protected]> * Add functions for dynamic programming comparison to get best path by ngrams Signed-off-by: Alexandra Antonova <[email protected]> * fixes Signed-off-by: Alexandra Antonova <[email protected]> * small fix Signed-off-by: Alexandra Antonova <[email protected]> * fixes to support testing on SPGISpeech Signed-off-by: Alexandra Antonova <[email protected]> * add preprocessing for userlibri Signed-off-by: Alexandra Antonova <[email protected]> * some refactoring Signed-off-by: Alexandra Antonova <[email protected]> * some refactoring Signed-off-by: Alexandra Antonova <[email protected]> * move some functions to utils to reuse from other project Signed-off-by: Alexandra Antonova <[email protected]> * move some functions to utils to reuse from other project Signed-off-by: Alexandra Antonova <[email protected]> * move some functions to utils to reuse from other project Signed-off-by: Alexandra Antonova <[email protected]> * small refactoring before pr. Add bash-scripts reproducing evaluation Signed-off-by: Alexandra Antonova <[email protected]> * style fix Signed-off-by: Alexandra Antonova <[email protected]> * small fixes in inference Signed-off-by: Alexandra Antonova <[email protected]> * bug fix - didn't move window on last symbol Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bug - shuffle was before truncation of sorted candidates Signed-off-by: Alexandra Antonova <[email protected]> * refactoring, fix some bugs Signed-off-by: Alexandra Antonova <[email protected]> * variour fixes. Add word_indices at inference Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add candidate positions Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Move data preparation and evaluation to other repo Signed-off-by: Alexandra Antonova <[email protected]> * add infer_reproduce_paper. Refactoring Signed-off-by: Alexandra Antonova <[email protected]> * refactor inference using fragment indices Signed-off-by: Alexandra Antonova <[email protected]> * add some helper functions Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bug with parameters order Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bugs Signed-off-by: Alexandra Antonova <[email protected]> * refactoring, fix bug Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add multiple variants of adjusting start/end positions Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * more fixes Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add unit tests, other fixes Signed-off-by: Alexandra Antonova <[email protected]> * fix Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix CodeQl warnings Signed-off-by: Alexandra Antonova <[email protected]> * add script for full inference pipeline, refactoring Signed-off-by: Alexandra Antonova <[email protected]> * add tutorial Signed-off-by: Alexandra Antonova <[email protected]> * take example data from HuggingFace Signed-off-by: Alexandra Antonova <[email protected]> * add docs Signed-off-by: Alexandra Antonova <[email protected]> * fix comment Signed-off-by: Alexandra Antonova <[email protected]> * fix bug Signed-off-by: Alexandra Antonova <[email protected]> * small fixes for PR Signed-off-by: Alexandra Antonova <[email protected]> * add some more tests Signed-off-by: Alexandra Antonova <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * try to fix tests adding with_downloads Signed-off-by: Alexandra Antonova <[email protected]> * skip tests with tokenizer download Signed-off-by: Alexandra Antonova <[email protected]> --------- Signed-off-by: Alexandra Antonova <[email protected]> Signed-off-by: Alexandra Antonova <[email protected]> Co-authored-by: Alexandra Antonova <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] Implement new vocoder dataset (#6670) * [TTS] Implement new vocoder dataset Signed-off-by: Ryan <[email protected]> * [TTS] Redo config structure, minor fixes Signed-off-by: Ryan <[email protected]> * [TTS] Fix alignment logging Signed-off-by: Ryan <[email protected]> * [TTS] Fix script usage example Signed-off-by: Ryan <[email protected]> * [TTS] Fixed epoch LR scheduling Signed-off-by: Ryan <[email protected]> * [TTS] Support .nemo checkpoint in FP callback Signed-off-by: Ryan <[email protected]> * [TTS] Remove align interpolator Signed-off-by: Ryan <[email protected]> * [TTS] Remove HiFi-GAN defaults list interpolation Signed-off-by: Ryan <[email protected]> * [TTS] Rename weighted_sample_steps to weighted_sampling_steps_per_epoch Signed-off-by: Ryan <[email protected]> --------- Signed-off-by: Ryan <[email protected]> * GPT inference long context (#6687) * deb infer Signed-off-by: Evelina <[email protected]> * deb infer Signed-off-by: Evelina <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * clean up Signed-off-by: Evelina <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * dont do maxlen trunc for non abs pos emb Signed-off-by: Evelina <[email protected]> * dont do maxlen trunc for non abs pos emb Signed-off-by: Evelina <[email protected]> * convert for training only Signed-off-by: Evelina <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add eval test, add save .nemo for sft model Signed-off-by: Evelina <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * jenkins format fix Signed-off-by: Evelina <[email protected]> * update jenkins Signed-off-by: Evelina <[email protected]> * update jenkins Signed-off-by: Evelina <[email protected]> * fix jenkins Signed-off-by: Evelina <[email protected]> * remove test, ci timeout Signed-off-by: Evelina <[email protected]> * fix for m_gpt_eval.py Signed-off-by: Evelina <[email protected]> * jenkins test Signed-off-by: Evelina <[email protected]> * fix gpt_eval with sft model Signed-off-by: Evelina <[email protected]> * revert jenkins Signed-off-by: Evelina <[email protected]> * keep float conversion for model.generate() Signed-off-by: Evelina <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix inference dtype Signed-off-by: Evelina <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Evelina <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * TDT model pull request (#6536) * TDT model pull request, initial draft Signed-off-by: Hainan Xu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * TDT PR WIP Signed-off-by: Hainan Xu <[email protected]> * TDT PR WIP Signed-off-by: Hainan Xu <[email protected]> * TDT PR WIP Signed-off-by: Hainan Xu <[email protected]> * TDT WIP Signed-off-by: Hainan Xu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * TDT WIP Signed-off-by: Hainan Xu <[email protected]> * TDT WIP Signed-off-by: Hainan Xu <[email protected]> * TDT WIP Signed-off-by: Hainan Xu <[email protected]> * TDT WIP Signed-off-by: Hainan Xu <[email protected]> * TDT WIP Signed-off-by: Hainan Xu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * TDT WIP Signed-off-by: Hainan Xu <[email protected]> * TDT WIP Signed-off-by: Hainan Xu <[email protected]> * TDT WIP Signed-off-by: Hainan Xu <[email protected]> * TDT WIP Signed-off-by: Hainan Xu <[email protected]> * TDT WIP Signed-off-by: Hainan Xu <[email protected]> * addressed some review comments, part1 Signed-off-by: Hainan Xu <[email protected]> * addressed some review comments, part1, one line fix Signed-off-by: Hainan Xu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add tests for comparing TDT alphas with pytorch VS kernel computation Signed-off-by: Hainan Xu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add tests for comparing multiblank alphas with pytorch VS kernel computation Signed-off-by: Hainan Xu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add tests for fixed case computation for TDT Signed-off-by: Hainan Xu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add more comments for greedy-batch decoding for TDT Signed-off-by: Hainan Xu <[email protected]> * include config for TDT model with stateless decoders Signed-off-by: Hainan Xu <[email protected]> * add reference to TDT in Readme Signed-off-by: Hainan Xu <[email protected]> * slight modification of config file comments Signed-off-by: Hainan Xu <[email protected]> * addressed more comments Signed-off-by: Hainan Xu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * more detailed comments for tdt kernel Signed-off-by: Hainan Xu <[email protected]> * one line fix Signed-off-by: Hainan Xu <[email protected]> * fixed small bug that results in test fails for rnnt_decoding Signed-off-by: Hainan Xu <[email protected]> * fixed small bug that results in test fails for rnnt_decoding Signed-off-by: Hainan Xu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed small bug that results in test fails for rnnt_decoding Signed-off-by: Hainan Xu <[email protected]> * remove unused import Signed-off-by: Hainan Xu <[email protected]> --------- Signed-off-by: Hainan Xu <[email protected]> Co-authored-by: Hainan Xu <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix get_parameters when using main params optimizer (#6764) (#6787) * fix get param * change name --------- Signed-off-by: ericharper <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Lddl bert (#6761) (#6790) * initial POC for LDDL Bert * Finish LDDL POC * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * address comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix merge head * resolving merge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add support for val/test loaders * change to new LDDL class + add winding * fix logging level * fix winding * test fix * fixes to winding * add file system * add prepemption optimizations * more logging * more prints * better logging * asfsf * add barrier * removing prints * working with mb lddl loader * final changes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update requirements file with LDDL * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert adding to requirements --------- Signed-off-by: wdykas <[email protected]> Co-authored-by: wdykas <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> * Fix check (#6798) (#6800) Signed-off-by: MaximumEntropy <[email protected]> Co-authored-by: Sandeep Subramanian <[email protected]> * Fix validation with drop_last=False (#6704) Signed-off-by: Mikołaj Błaż <[email protected]> Co-authored-by: Eric Harper <[email protected]> * SDE unt lvl comparison (#6669) Added a visual utterance-level comparison of two ASR models Signed-off-by: George <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Debug Transformer Engine FP8 support with Megatron-core infrastructure (#6791) * Construct FP8 amax reduction group Signed-off-by: Tim Moon <[email protected]> * Update Megatron-core version in CI Signed-off-by: Tim Moon <[email protected]> --------- Signed-off-by: Tim Moon <[email protected]> Co-authored-by: Tim Moon <[email protected]> Co-authored-by: Tim Moon <[email protected]> * Lora/PEFT training script CI test (#6664) * new lora test Signed-off-by: arendu <[email protected]> * updates Signed-off-by: arendu <[email protected]> * check for chat Signed-off-by: arendu <[email protected]> * update Signed-off-by: arendu <[email protected]> * update Signed-off-by: arendu <[email protected]> * small train set Signed-off-by: arendu <[email protected]> * update Signed-off-by: arendu <[email protected]> * precision change Signed-off-by: arendu <[email protected]> * fixed typo in paths Signed-off-by: arendu <[email protected]> * full data with limit val batches Signed-off-by: arendu <[email protected]> * tp2 instead of pp2 Signed-off-by: arendu <[email protected]> * tp2 instead of pp2 Signed-off-by: arendu <[email protected]> --------- Signed-off-by: arendu <[email protected]> Signed-off-by: Adi Renduchintala <[email protected]> * change branch to main, small fix (#6803) Signed-off-by: Alexandra Antonova <[email protected]> * add call to p2p overlap (#6779) (#6786) * add call to p2p overlap * update Jenkins for test --------- Signed-off-by: Abhinav Khattar <[email protected]> Signed-off-by: Eric Harper <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Co-authored-by: Eric Harper <[email protected]> * fixed decor to show messages only when the wrapped object is called. (#6793) Signed-off-by: Xuesong Yang <[email protected]> * Bug fix for reset_sequence_parallel_args (#6802) (#6805) Signed-off-by: Markel Sanz Ausin <[email protected]> Co-authored-by: Markel Sanz Ausin <[email protected]> * text_generation_utils memory reduction if no logprob needed (#6773) * repro for gpt eval mp mem issue Signed-off-by: Yang Zhang <[email protected]> * add print statements for memory allocation Signed-off-by: Yang Zhang <[email protected]> * adjusted hot fix that prevents softmax on the entire output embedding,now memory bottlenecked by attention softmax which needs to be solved with FA or long attention Signed-off-by: Yang Zhang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * using compute_logprob to configure inference Signed-off-by: Yang Zhang <[email protected]> * enable compute logprob for peft Signed-off-by: Yang Zhang <[email protected]> * remove print statements Signed-off-by: Yang Zhang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix ci Signed-off-by: Yang Zhang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * added docstrings Signed-off-by: Yang Zhang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add missing config Signed-off-by: Yang Zhang <[email protected]> * remove truncate prompt length feature Signed-off-by: Yang Zhang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tensor before all gather needs to be contiguous Signed-off-by: Yang Zhang <[email protected]> --------- Signed-off-by: Yang Zhang <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Evelina <[email protected]> Co-authored-by: Sandeep Subramanian <[email protected]> * Fixed bug in MaskedSpecAug that overestimates samples. (#6775) Signed-off-by: tbartley94 <[email protected]> * update core version (#6817) (#6819) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * lora pp2 (#6818) Signed-off-by: arendu <[email protected]> * Add optional index mapping dir in mmap text datasets (#6683) If datasets are stored on a read-only medium, index files cannot be created into adjacent files and an alternative directory must be specified for index mapping files. This commit adds an optional `index_mapping_dir` to the constructors. Unit tests are also added. [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Update path formatting for relative paths Signed-off-by: Greg Heinrich <[email protected]> * Add inference kv cache support for transformer TE path (#6627) * Add kv cache support for transformer TE path Signed-off-by: Yen-Shi Wang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mark get_data_parallel_group as WAR Signed-off-by: Yen-Shi Wang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Initialize process group for FP8 training Signed-off-by: Tim Moon <[email protected]> * Update Megatron GPT eval script for non-FP8 path Signed-off-by: Yen-Shi Wang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Yen-Shi Wang <[email protected]> Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Yen-Shi Wang <[email protected]> Co-authored-by: Yen-Shi Wang <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Tim Moon <[email protected]> Co-authored-by: Tim Moon <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Support large inputs to Conformer and Fast Conformer (#6556) * initial commit Signed-off-by: Dima Rekesh <[email protected]> * typos Signed-off-by: Dima Rekesh <[email protected]> * tweaks to padding Signed-off-by: Dima Rekesh <[email protected]> * comments Signed-off-by: Dima Rekesh <[email protected]> * attempt at first working version Signed-off-by: Dima Rekesh <[email protected]> * typos and fixed p calculation Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removing merge artifacts Signed-off-by: Dima Rekesh <[email protected]> * typo Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removing unnessary imports Signed-off-by: Dima Rekesh <[email protected]> * if batch split succeeded no need to conv again Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adding channel wise split Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adding reference to pytorch issue 80020 Signed-off-by: Dima Rekesh <[email protected]> * removing time chunking methods Signed-off-by: Dima Rekesh <[email protected]> * accounting for the actual self._stride value Signed-off-by: Dima Rekesh <[email protected]> * limiting the fix to dw_striding subsampling Signed-off-by: Dima Rekesh <[email protected]> * renamed methods Signed-off-by: Dima Rekesh <[email protected]> * one more accounting for the actual self._stride value Signed-off-by: Dima Rekesh <[email protected]> * support for causal convs Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * option to set conv chunking size manually * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixing imports * subsampling test Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * rename variable Signed-off-by: Dima Rekesh <[email protected]> * imports in test Signed-off-by: Dima Rekesh <[email protected]> * more runtime checks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * a more careful test Signed-off-by: Dima Rekesh <[email protected]> * bug in causal Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix in causal Signed-off-by: Dima Rekesh <[email protected]> * change_conv_chunking_factor methods Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * renamed methods Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * disabling chunking by default Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * typo Signed-off-by: Dima Rekesh <[email protected]> * changing default chunking to auto Signed-off-by: Dima Rekesh <[email protected]> * only split if needed Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * only split if needed Signed-off-by: Dima Rekesh <[email protected]> --------- Signed-off-by: Dima Rekesh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * sharded_manifests updated docs (#6833) Signed-off-by: Dima Rekesh <[email protected]> * added fc-xl, xxl and titanet-s models (#6832) Signed-off-by: Nithin Rao Koluguri <nithinraok> Co-authored-by: Nithin Rao Koluguri <nithinraok> * add reference to our paper (#6821) * add reference to our paper Signed-off-by: Alexandra Antonova <[email protected]> * add paper reference to docs Signed-off-by: Alexandra Antonova <[email protected]> --------- Signed-off-by: Alexandra Antonova <[email protected]> * Upperbound Numpy to < 1.24 (#6829) Signed-off-by: smajumdar <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Multi-lookahead cache-aware streaming models (#6711) * added methods. Signed-off-by: Vahid <[email protected]> * added methods. Signed-off-by: Vahid <[email protected]> * added initial code. Signed-off-by: Vahid <[email protected]> * added initial code. Signed-off-by: Vahid <[email protected]> * added initial code. Signed-off-by: Vahid <[email protected]> * added config files. Signed-off-by: Vahid <[email protected]> * fixed bugs. Signed-off-by: Vahid <[email protected]> * updated confs. Signed-off-by: Vahid <[email protected]> * updated confs. Signed-off-by: Vahid <[email protected]> * updated confs. Signed-off-by: Vahid <[email protected]> * updated confs. Signed-off-by: Vahid <[email protected]> * improved f.conv1d Signed-off-by: Vahid <[email protected]> * pulled from main. Signed-off-by: Vahid <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * pulled from main. Signed-off-by: Vahid <[email protected]> * added postpostnorm. Signed-off-by: Vahid <[email protected]> * fixed the target continiouse bug. Signed-off-by: Vahid <[email protected]> * added dw_striding causal. Signed-off-by: Vahid <[email protected]> * added print for debugging. Signed-off-by: Vahid <[email protected]> * added print for debugging. Signed-off-by: Vahid <[email protected]> * fixed causal convolutions. Signed-off-by: Vahid <[email protected]> * added _midnorm. Signed-off-by: Vahid <[email protected]> * fixed transcribe. Signed-off-by: Vahid <[email protected]> * cleaned code. Signed-off-by: Vahid <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * moved back configs. Signed-off-by: Vahid <[email protected]> * moved back configs. Signed-off-by: Vahid <[email protected]> * updated fast emit for FC models. Signed-off-by: Vahid <[email protected]> * updated fast emit for FC models. Signed-off-by: Vahid <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed bug. Signed-off-by: Vahid <[email protected]> * fixed bug and addressed comments. Signed-off-by: Vahid <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed configs. Signed-off-by: Vahid <[email protected]> * fixed configs. Signed-off-by: Vahid <[email protected]> * dropped the test. Signed-off-by: Vahid <[email protected]> --------- Signed-off-by: Vahid <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * added changes to ramp up bs (#6799) * rampup bs changes Signed-off-by: dimapihtar <[email protected]> * rampup bs changes Signed-off-by: dimapihtar <[email protected]> * fixed styling Signed-off-by: dimapihtar <[email protected]> * fix bug Signed-off-by: Dmytro Pykhtar <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Fix typo in core.rst (#6838) Signed-off-by: Dounx <[email protected]> * add back ptuning pp2 test (#6394) Signed-off-by: arendu <[email protected]> * t5 lora tuning (#6612) * t5 lora Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * eval lora t5 Signed-off-by: arendu <[email protected]> * adjust differernt lora dims Signed-off-by: arendu <[email protected]> * minor changes Signed-off-by: David Mosallanezhad <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * bugfix for state_dict Signed-off-by: David Mosallanezhad <[email protected]> --------- Signed-off-by: arendu <[email protected]> Signed-off-by: David Mosallanezhad <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: David Mosallanezhad <[email protected]> Co-authored-by: David <[email protected]> * NFA updates (#6695) * update V_NEGATIVE_NUM constant to make better use of torch.float32 range Signed-off-by: Elena Rastorgueva <[email protected]> * adjust backpointers dtype if U_max too large Signed-off-by: Elena Rastorgueva <[email protected]> * Remove print statements Signed-off-by: Elena Rastorgueva <[email protected]> * Remove need for user to specify model_downsample_factor Signed-off-by: Elena Rastorgueva <[email protected]> * change model.cfg.sample_rate to model.cfg.preprocessor.sample_rate Signed-off-by: Elena Rastorgueva <[email protected]> * add check to make sure that window_stride is in model.cfg.preprocessor Signed-off-by: Elena Rastorgueva <[email protected]> * reduce memory consumption of backpointers by making them relative instead of absolute Signed-off-by: Elena Rastorgueva <[email protected]> * update librosa.get_duration() 'filename' param to 'path' Signed-off-by: Elena Rastorgueva <[email protected]> * Do not throw error if 'text' or 'pred_text' are empty and make sure CTM filepaths in the output manifest are null Signed-off-by: Elena Rastorgueva <[email protected]> * preprocess input text by removing any duplicate spaces and converting any newlines to spaces Signed-off-by: Elena Rastorgueva <[email protected]> * Use Utterance dataclass instead of dictionaries for keeping track of token/word/segment alignments Signed-off-by: Elena Rastorgueva <[email protected]> * refactor so can save alignments as ctm and ass format files Signed-off-by: Elena Rastorgueva <[email protected]> * fix bugs for saving character based ASS files and for using pred_text to do alignment Signed-off-by: Elena Rastorgueva <[email protected]> * Make token level .ass file use tokens with recovered capitalization Signed-off-by: Elena Rastorgueva <[email protected]> * Do not try to generate alignment files if text or pred text is empty, or if number of tokens is too large for T Signed-off-by: Elena Rastorgueva <[email protected]> * rename output manifest file to say '_with_output_file_paths.json' Signed-off-by: Elena Rastorgueva <[email protected]> * add flag to resegment ass subtitle file to fill available text space Signed-off-by: Elena Rastorgueva <[email protected]> * Fix bug in resegmentation code Signed-off-by: Elena Rastorgueva <[email protected]> * Fix bug which skipped some utterances if batch_size more than 1 Signed-off-by: Elena Rastorgueva <[email protected]> * reduce memory requirements by doing torch.gather on a slice of the log probs when they are needed Signed-off-by: Elena Rastorgueva <[email protected]> * reduce memory requirements by not saving whole v_matrix Signed-off-by: Elena Rastorgueva <[email protected]> * remove any extra spaces in pred_text Signed-off-by: Elena Rastorgueva <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unused list pred_text_all_lines Signed-off-by: Elena Rastorgueva <[email protected]> * support using hybrid Transducer-CTC models for alignment Signed-off-by: Elena Rastorgueva <[email protected]> * fix typo - add brackets to torch.cuda.is_available() Signed-off-by: Elena Rastorgueva <[email protected]> * make sure token case restoration will work if superscript or subscript num is in text Signed-off-by: Elena Rastorgueva <[email protected]> * remove any BOM from input text Signed-off-by: Elena Rastorgueva <[email protected]> * pick out 1st hypotheses if there is a tuple of them Signed-off-by: Elena Rastorgueva <[email protected]> * Remove print statement Signed-off-by: Elena Rastorgueva <[email protected]> * add detail to error message if fail to recover capitalization of tokens Signed-off-by: Elena Rastorgueva <[email protected]> * add flag use_local_attention Signed-off-by: Elena Rastorgueva <[email protected]> * rename additional_ctm_grouping_separator -> additional_segment_grouping_separator Signed-off-by: Elena Rastorgueva <[email protected]> * update description of additional_segment_grouping_separator Signed-off-by: Elena Rastorgueva <[email protected]> * add simple docstring to get_utt_obj function Signed-off-by: Elena Rastorgueva <[email protected]> * Make docstring for add_t_start_end_to_utt_obj Signed-off-by: Elena Rastorgueva <[email protected]> * update docstrings for add_t_start_end_to_utt_obj and get_batch_variables Signed-off-by: Elena Rastorgueva <[email protected]> * update README and comments in align.py Signed-off-by: Elena Rastorgueva <[email protected]> * change 'ground truth' -> 'reference text' in documentation Signed-off-by: Elena Rastorgueva <[email protected]> * add header Signed-off-by: Elena Rastorgueva <[email protected]> * add comments to get_utt_obj function Signed-off-by: Elena Rastorgueva <[email protected]> * move constants so they are after imports Signed-off-by: Elena Rastorgueva <[email protected]> * add file description for make_ass_files Signed-off-by: Elena Rastorgueva <[email protected]> * get rid of Utterance object's S attribute, and correct tests so they pass now Signed-off-by: Elena Rastorgueva <[email protected]> * remove some unused variables Signed-off-by: Elena Rastorgueva <[email protected]> * remove unused variable model from functions saving output files Signed-off-by: Elena Rastorgueva <[email protected]> * remove unused var minimum_timestamp_duration from make_ass_files functions and return utt_obj Signed-off-by: Elena Rastorgueva <[email protected]> * move minimum_timestamp_duration param to CTMFileConfig Signed-off-by: Elena Rastorgueva <[email protected]> * remove unused enumerate and unused import Signed-off-by: Elena Rastorgueva <[email protected]> * switch reading duration from librosa to soundfile to avoid filename/path deprecation message Signed-off-by: Elena Rastorgueva <[email protected]> --------- Signed-off-by: Elena Rastorgueva <[email protected]> Signed-off-by: Elena Rastorgueva <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Added rouge monitoring support for T5 (#6737) * Added rouge monitoring support for t5 Signed-off-by: Matvei Novikov <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Matvei Novikov <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * GPT extrapolatable position embedding (xpos/sandwich/alibi/kerple) and Flash Attention (#6666) * move to nvidia megatron repo (#6465) (#6475) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Megatron KERPLE positional embeddings (#6478) (#6480) * [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- * [TTS] FastPitch speaker encoder (#6417) * Add initial codes * Remove wemb * Fix import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss * Add ConditionalInput * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test * Add initial codes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix import error * Add initial codes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix dataset error * Remove reference speaker embedding * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove SV encoder * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix length type * Fix append * Move error msg * Add look-up into speaker encoder * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add valueerror msg * Move lookup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unused * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error * Rebase and Fix error * Fix spk encoder * Rename n_speakers * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error --------- * Sharded manifests for tarred datasets (#6395) * testing sharded manifests * compatibility * proper fixes * adding flag tot convert_to_tarred_audio_dataset * shard_manifests conf param * propagating the shard_manifests param * propagating the shard_manifests param * distributed checks * typo * typo * fixes * fixes * fixes * fixes * fixes * fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py * reversing manifest shards flag * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests * excluding manifests from webdataset url expansion * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore * explicit use of UTF-8 for manifest i/o --------- * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. --------- * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support * added tests for rampup batch size * fixed the typos * added assertions * changed assertion rules * deleted unused imports * changed tests for rampup batch size * updated rampup batch size tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling * rampup batch size tests changes --------- * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default * updated README with the new agg_tokenizer_manifest flag * fixed typo in scripts/speech_recognition/code_switching/README.md * changed agg_tokenizer_manifest to is_lid_manifest --------- * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script * add return encoding embedding option * update script * add rnnt and hat ngram decoding script * add some parameters * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig * replace return_encoder_embeddings parameter * generalization of scipt behavior * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter * remove return_encoder_embeddings parameter * add manual encoder_embeddings calculation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 * fix rescoring description --------- * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- * fix replace_bos_with_pad not found (#6443) (#6450) * reduce workers on NMT CI (#6472) (#6474) * 1. Added KERPLE positional embeddings to encoder-decoder. * 1. Added a missing file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: Jim O’Regan <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Kunal Dhawan <[email protected]> Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Cheng-Ping Hsieh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> Co-authored-by: Jim O’Regan <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: Mostafa Ghorbandoost <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Fix an invalid link in get_data.py of ljspeech (#6456) Usage of the link in line 63 leads to downloading a html file not a tsv file, so we need to change it to a raw link. Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * 1. Added external index sample. (#6462) (#6483) Signed-off-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Update README to add core installation (#6488) (#6489) * update README for megatron-core * fix --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Fix cache aware hybrid bugs (#6466) (#6484) Signed-off-by: hsiehjackson <[email protected]> * Fix typos (#6494) (#6495) Signed-off-by: smajumdar <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Add disclaimer about dataset for ASR (#6496) Signed-off-by: smajumdar <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * fix (#6502) datastore_path_to_webdataset_url(p) if is_datastore_path(p) and is_tarred_path(p) else p NameError: name 'is_tarred_path' is not defined Co-authored-by: George <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * fix broken links r1.18.0 (#6501) (#6504) * fix broken links * fix broken links --------- Signed-off-by: Evelina <[email protected]> Co-authored-by: Evelina <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * [TTS] Create functions for TTS preprocessing without dataloader (#6317) * [TTS] Create functions for TTS preprocessing without dataloader Signed-off-by: Ryan <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * Cache aware streaming nfa (#6209) * add cache aware streaming to nemo aligner Signed-off-by: Slyne Deng <[email protected]> Signed-off-by: hsiehjackson <[email protected]> * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator (#6499) * [BugFix] _get_batch_preds() is forced to keep logits in decoder timestamps generators Signed-off-by: Taejin Park <[email protected]> * Ingnore keep_logits boolean in FrameASRBatchLogits Signed-off-by: Taejin Park <[email protected]> --------- Signed-off-by: Taejin Park <[email protected]> Co-authored-by: Jagadeesh Balam <[email protected]> Signed-off-by: hsiehjackson <[email protected]> …

* [TTS] bugfix for missing configs. (#4725) Signed-off-by: Xuesong Yang <[email protected]> * docs typo fix Signed-off-by: Oleksii Kuchaiev <[email protected]> * Fix pynini install in TTS tutorials (#4729) Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> * Fix ASR notebooks (#4738) Signed-off-by: smajumdar <[email protected]> Signed-off-by: smajumdar <[email protected]> * Multilingual VAD model (#4734) * add ngc link Signed-off-by: fayejf <[email protected]> * add tuned VAD config on ASR data Signed-off-by: fayejf <[email protected]> * yaml note Signed-off-by: fayejf <[email protected]> * update vad asr notebook with mVAD Signed-off-by: fayejf <[email protected]> * update vad infer config comment Signed-off-by: fayejf <[email protected]> * fix Signed-off-by: fayejf <[email protected]> * mvad sd config for ch109 Signed-off-by: fayejf <[email protected]> * update sd readme Signed-off-by: fayejf <[email protected]> * add new mVAD model to doc Signed-off-by: fayejf <[email protected]> * style fix Signed-off-by: fayejf <[email protected]> * update sd tutorial with mVAD Signed-off-by: fayejf <[email protected]> * typo fix Signed-off-by: fayejf <[email protected]> Signed-off-by: fayejf <[email protected]> * publish pretrained itn t5 model for English (#4748) Signed-off-by: Alexandra Antonova <[email protected]> Signed-off-by: Alexandra Antonova <[email protected]> Co-authored-by: Alexandra Antonova <[email protected]> * Updated docs and doc paths (#4754) * Updated docs and doc paths Signed-off-by: Virginia Adams <[email protected]> * Update Multitask_Prompt_and_PTuning.ipynb * Update README.rst * Changed branch name to use single quotes Signed-off-by: Virginia Adams <[email protected]> Signed-off-by: Virginia Adams <[email protected]> * fix bug relating to ddp strategy in joint intent slot classification tutorial (#4762) * [TTS] updated config with a German IPA phoneme tokenizer (#4756) * [TTS] added a German IPA phoneme tokenizer * [TTS][ASR] enabled customized arguments for trimming the leading and trailing silence. * [TTS] disabled spline interpolation for beta-binomial distribution. Let it generate align prior and save to disks. Use a new phoneme tokenizer. * [TTS] use consistent spline interpolation with fastpitch checkpoint when generating mel-spectrograms for hifigan finetune. Signed-off-by: Xuesong Yang <[email protected]> * Update r1.11 to new heteronyms list (#4745) * Update configs to new heteronyms list * Remove old heteronyms list, add alt 'merchandise' pron to CMUdict * Update remaining references to old heteronyms list Signed-off-by: Jocelyn Huang <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> * [TTS] Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints (#4763) Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> * [TTS] Add single male speaker German FastPitch and HiFiGAN NGC checkpoints (#4770) Signed-off-by: Xuesong Yang <[email protected]> * Update CMUdict with more recent 0.7b entries (#4768) Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> * Install pynini in docker container (#4733) Signed-off-by: Vladimir Bataev <[email protected]> Signed-off-by: Vladimir Bataev <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Fix tutorial formatting (#4778) Signed-off-by: Jocelyn Huang <[email protected]> * [TTS] deprecated old scripts for ljspeech. (#4780) * deprecated old scripts for ljspeech. * removed relevent function calls in TTS docs. Signed-off-by: Xuesong Yang <[email protected]> * update branch and typos (#4788) Signed-off-by: ericharper <[email protected]> Signed-off-by: ericharper <[email protected]> * Adding support for models trained with full context for cache-aware streaming. (#4687) * added support for models trained with full context. Signed-off-by: Vahid <[email protected]> * fixed style. Signed-off-by: Vahid <[email protected]> * dropped seq_range Signed-off-by: Vahid <[email protected]> * fixed indexing in caching methods. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * updated docs. Signed-off-by: Vahid <[email protected]> * addressed comments. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * change frame-wise to cache-aware. Signed-off-by: Vahid <[email protected]> * change frame-wise to cache-aware. Signed-off-by: Vahid <[email protected]> * change frame-wise to cache-aware. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> Signed-off-by: Vahid <[email protected]> * Update megatron encoder decoder model to support py37 for colab (#4791) * [ASR] Add pretrained ASR models for Croatian (#4682) * [ASR] Add pretrained ASR models for Croatian Signed-off-by: Ante Jukić <[email protected]> * Fix style for import Signed-off-by: Ante Jukić <[email protected]> Signed-off-by: Ante Jukić <[email protected]> Co-authored-by: Ante Jukić <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> * added/fixed export for Megatron models (#4712) * added/fixed export for Megatron models Signed-off-by: David Mosallanezhad <[email protected]> * fixed style Signed-off-by: David Mosallanezhad <[email protected]> * fixed FusedScaleMaskSoftmax in BioMegatron Signed-off-by: David Mosallanezhad <[email protected]> * included comments Signed-off-by: David Mosallanezhad <[email protected]> Signed-off-by: David Mosallanezhad <[email protected]> Co-authored-by: David Mosallanezhad <[email protected]> Co-authored-by: Eric Harper <[email protected]> * update branch for qa notebook Signed-off-by: ericharper <[email protected]> * Fix initializing weights from ptl ckpt with exclude (#4807) Signed-off-by: sam1373 <[email protected]> Signed-off-by: sam1373 <[email protected]> * Fix index error from addition of voiced_mask and p_voiced (#4811) Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> * T5 prompt learning fixes (#4771) * RPE, hidden size and config fixes Signed-off-by: MaximumEntropy <[email protected]> * Update to reflect new config names Signed-off-by: MaximumEntropy <[email protected]> * Sentencepiece fixes Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Fix finetuning Signed-off-by: MaximumEntropy <[email protected]> * Add encoder seq len to gpt Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Add finetune eval script Signed-off-by: MaximumEntropy <[email protected]> * Fix name Signed-off-by: MaximumEntropy <[email protected]> * Update Jenkinsfile Signed-off-by: MaximumEntropy <[email protected]> * Update config Signed-off-by: MaximumEntropy <[email protected]> * Fix CI test Signed-off-by: MaximumEntropy <[email protected]> * Update check Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Backward compat Signed-off-by: MaximumEntropy <[email protected]> * Update CI test Signed-off-by: MaximumEntropy <[email protected]> * Split rank for Enc-Dec models Signed-off-by: MaximumEntropy <[email protected]> * Address comments Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Co-authored-by: Virginia Adams <[email protected]> * G2P docs (#4841) * g2p docs added Signed-off-by: ekmb <[email protected]> * fix references Signed-off-by: ekmb <[email protected]> * address review feedback Signed-off-by: ekmb <[email protected]> Signed-off-by: ekmb <[email protected]> * Fix providing glue in seq2seq eval (#4843) * Fix providing glue in seq2seq eval Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> * Updated inference code and squad scripts (#4835) * Updated inference code and squad scripts Signed-off-by: Virginia Adams <[email protected]> * Reverted GPT & T5 inference files back to use NLPDDPlugin Signed-off-by: Virginia Adams <[email protected]> * Overwrite frozen LM to use fused adam Signed-off-by: Virginia Adams <[email protected]> * Added padded vocab size Signed-off-by: Virginia Adams <[email protected]> * Fixed val check interval value Signed-off-by: Virginia Adams <[email protected]> * Python format fix Signed-off-by: Virginia Adams <[email protected]> * Make t5 prompt learning preds write to file Signed-off-by: Virginia Adams <[email protected]> * Added back dp=1 check Signed-off-by: Virginia Adams <[email protected]> Signed-off-by: Virginia Adams <[email protected]> Co-authored-by: Sandeep Subramanian <[email protected]> * Update README.rst * Fix uppercasing mismatch for IPA heteronyms (#4860) Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> * Set the number of workers to 0 for validation and test sets in all enc-dec models (#4790) * Set workers to 0 for validation and test Signed-off-by: MaximumEntropy <[email protected]> * Revert pin memory Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Co-authored-by: Sean Naren <[email protected]> * Fix mha (#4866) * fix bug in mha forward function related to cache update return type Signed-off-by: Yang Zhang <[email protected]> * fix lgtm Signed-off-by: Yang Zhang <[email protected]> Signed-off-by: Yang Zhang <[email protected]> Co-authored-by: Sean Naren <[email protected]> * ipa bug fix (#4871) Signed-off-by: ekmb <[email protected]> Signed-off-by: ekmb <[email protected]> * Fix Megatron NMT consumed samples and ckpt_to_nemo split rank (#4884) * Fix nmt and ckpt_to_nemo Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> * added utf8 encoding (#4892) Signed-off-by: Virginia Adams <[email protected]> Signed-off-by: Virginia Adams <[email protected]> * 1. Applying the same patch to r1.11.0 (#4894) Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Micha Livne <[email protected]> * Update tutorials.rst (#4897) * update readme with apex commit Signed-off-by: ericharper <[email protected]> * Add support for Apex distributed Adam optimizer with GPT-3 (#4487) * Add support for Apex distributed Adam optimizer with GPT-3 Signed-off-by: Tim Moon <[email protected]> * Fix bug in grad clipping with dist Adam Grad norm was computed over all params, not respecting model parallelism. Signed-off-by: Tim Moon <[email protected]> * Fix bug with DDP initialization Signed-off-by: Tim Moon <[email protected]> * Make distopt dependent on megatron_amp_o2 Signed-off-by: Tim Moon <[email protected]> * Fix code formatting Signed-off-by: Tim Moon <[email protected]> * Handle dist Adam in optimizer unit tests Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Tim Moon <[email protected]> Co-authored-by: Eric Harper <[email protected]> * update readme Signed-off-by: ericharper <[email protected]> * update readme Signed-off-by: ericharper <[email protected]> * latent model support * 1. Debugging. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. * update branch Signed-off-by: ericharper <[email protected]> * fix replace_bos_with_pad not found (#6443) Signed-off-by: Abhinav Khattar <[email protected]> * Support Swiglu in TP PP Conversion (#6437) * Support Swiglu in TP PP Conversion Signed-off-by: smajumdar <[email protected]> * Guard activation Signed-off-by: smajumdar <[email protected]> * Guard activation Signed-off-by: smajumdar <[email protected]> --------- Signed-off-by: smajumdar <[email protected]> * BERT pre-training mp fork to spawn (#6442) * change bert fork to spawn Signed-off-by: Abhinav Khattar <[email protected]> * num_workers=0 fix Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> * Meagtron encoder decoder fix for empty validation outputs (#6459) * 1. Meagtron encoder decoder fix for empty validation outputs. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. --------- Signed-off-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> * Added/updated new Conformer configs (#6426) * updated conf files. Signed-off-by: Vahid <[email protected]> * added confs. Signed-off-by: Vahid <[email protected]> * moved longconformer confs. Signed-off-by: Vahid <[email protected]> * updated readme. Signed-off-by: Vahid <[email protected]> * updated readme. Signed-off-by: Vahid <[email protected]> * updated batch sizes and added fastconformer ctc streaming configs. Signed-off-by: Vahid <[email protected]> * updated batch sizes. Signed-off-by: Vahid <[email protected]> * added hybrid support. Signed-off-by: Vahid <[email protected]> * added hybrid support. Signed-off-by: Vahid <[email protected]> --------- Signed-off-by: Vahid <[email protected]> * reduce workers on NMT CI (#6472) Signed-off-by: Abhinav Khattar <[email protected]> * move to nvidia megatron repo (#6465) Signed-off-by: Abhinav Khattar <[email protected]> * Megatron KERPLE positional embeddings (#6478) * [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- Signed-off-by: hsiehjackson <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. Signed-off-by: Xuesong Yang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Xuesong Yang <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] FastPitch speaker encoder (#6417) * Add initial codes Signed-off-by: hsiehjackson <[email protected]> * Remove wemb Signed-off-by: hsiehjackson <[email protected]> * Fix import Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss Signed-off-by: hsiehjackson <[email protected]> * Add ConditionalInput Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test Signed-off-by: hsiehjackson <[email protected]> * Add initial codes Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix import error Signed-off-by: hsiehjackson <[email protected]> * Add initial codes Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix dataset error Signed-off-by: hsiehjackson <[email protected]> * Remove reference speaker embedding Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Remove SV encoder Signed-off-by: hsiehjackson <[email protected]> * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix length type Signed-off-by: hsiehjackson <[email protected]> * Fix append Signed-off-by: hsiehjackson <[email protected]> * Move error msg Signed-off-by: hsiehjackson <[email protected]> * Add look-up into speaker encoder Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Add valueerror msg Signed-off-by: hsiehjackson <[email protected]> * Move lookup Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Remove unused Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix error Signed-off-by: hsiehjackson <[email protected]> * Rebase and Fix error Signed-off-by: hsiehjackson <[email protected]> * Fix spk encoder Signed-off-by: hsiehjackson <[email protected]> * Rename n_speakers Signed-off-by: hsiehjackson <[email protected]> * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error Signed-off-by: hsiehjackson <[email protected]> --------- Signed-off-by: hsiehjackson <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Sharded manifests for tarred datasets (#6395) * testing sharded manifests Signed-off-by: Dima Rekesh <[email protected]> * compatibility Signed-off-by: Dima Rekesh <[email protected]> * proper fixes Signed-off-by: Dima Rekesh <[email protected]> * adding flag tot convert_to_tarred_audio_dataset Signed-off-by: Dima Rekesh <[email protected]> * shard_manifests conf param Signed-off-by: Dima Rekesh <[email protected]> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <[email protected]> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <[email protected]> * distributed checks Signed-off-by: Dima Rekesh <[email protected]> * typo Signed-off-by: Dima Rekesh <[email protected]> * typo Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py Signed-off-by: Dima Rekesh <[email protected]> * reversing manifest shards flag Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests Signed-off-by: Dima Rekesh <[email protected]> * excluding manifests from webdataset url expansion Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore Signed-off-by: Dima Rekesh <[email protected]> * explicit use of UTF-8 for manifest i/o Signed-off-by: Dima Rekesh <[email protected]> --------- Signed-off-by: Dima Rekesh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) Signed-off-by: Jim O’Regan <[email protected]> * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. Signed-off-by: Mostafa Ghorbandoost <[email protected]> * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. Signed-off-by: Mostafa Ghorbandoost <[email protected]> --------- Signed-off-by: Mostafa Ghorbandoost <[email protected]> * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support Signed-off-by: Dmytro Pykhtar <[email protected]> * added tests for rampup batch size Signed-off-by: Dmytro Pykhtar <[email protected]> * fixed the typos Signed-off-by: Dmytro Pykhtar <[email protected]> * added assertions Signed-off-by: Dmytro Pykhtar <[email protected]> * changed assertion rules Signed-off-by: Dmytro Pykhtar <[email protected]> * deleted unused imports Signed-off-by: Dmytro Pykhtar <[email protected]> * changed tests for rampup batch size Signed-off-by: Dmytro Pykhtar <[email protected]> * updated rampup batch size tests Signed-off-by: Dmytro Pykhtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling Signed-off-by: Dmytro Pykhtar <[email protected]> * rampup batch size tests changes Signed-off-by: Dmytro Pykhtar <[email protected]> --------- Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- Signed-off-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default Signed-off-by: Kunal Dhawan <[email protected]> * updated README with the new agg_tokenizer_manifest flag Signed-off-by: Kunal Dhawan <[email protected]> * fixed typo in scripts/speech_recognition/code_switching/README.md Signed-off-by: Kunal Dhawan <[email protected]> * changed agg_tokenizer_manifest to is_lid_manifest Signed-off-by: Kunal Dhawan <[email protected]> --------- Signed-off-by: Kunal Dhawan <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script Signed-off-by: andrusenkoau <[email protected]> * add return encoding embedding option Signed-off-by: andrusenkoau <[email protected]> * update script Signed-off-by: andrusenkoau <[email protected]> * add rnnt and hat ngram decoding script Signed-off-by: andrusenkoau <[email protected]> * add some parameters Signed-off-by: andrusenkoau <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig Signed-off-by: andrusenkoau <[email protected]> * replace return_encoder_embeddings parameter Signed-off-by: andrusenkoau <[email protected]> * generalization of scipt behavior Signed-off-by: andrusenkoau <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <[email protected]> * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <[email protected]> * add manual encoder_embeddings calculation Signed-off-by: andrusenkoau <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 Signed-off-by: Andrei Andrusenko <[email protected]> * fix rescoring description Signed-off-by: Andrei Andrusenko <[email protected]> --------- Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * fix replace_bos_with_pad not found (#6443) (#6450) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * reduce workers on NMT CI (#6472) (#6474) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * 1. Added KERPLE positional embeddings to encoder-decoder. Signed-off-by: Micha Livne <[email protected]> * 1. Added a missing file. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: Jim O’Regan <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Kunal Dhawan <[email protected]> Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Cheng-Ping Hsieh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> Co-authored-by: Jim O’Regan <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: Mostafa Ghorbandoost <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * 1. Added external index sample. (#6462) Signed-off-by: Micha Livne <[email protected]> * Fix cache aware hybrid bugs (#6466) * Update README to add core installation (#6488) * update README for megatron-core Signed-off-by: Abhinav Khattar <[email protected]> * fix Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> * Fix typos (#6494) Signed-off-by: smajumdar <[email protected]> * fix broken links r1.18.0 (#6501) * fix broken links Signed-off-by: Evelina <[email protected]> * fix broken links Signed-off-by: Evelina <[email protected]> --------- Signed-off-by: Evelina <[email protected]> * 1. Fixed gaussian hidden transform. Signed-off-by: Micha Livne <[email protected]> * 1. Finished updating hidden loss for MIM. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix custom forward_torch_softmax (#6512) Signed-off-by: Abhinav Khattar <[email protected]> * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamp… (#6500) * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator r1.18.0 Signed-off-by: Taejin Park <[email protected]> * ignore keep_logits in FrameBatchASRLogits Signed-off-by: Taejin Park <[email protected]> --------- Signed-off-by: Taejin Park <[email protected]> * [TTS] fixed broken path. (#6514) Signed-off-by: Xuesong Yang <[email protected]> * 1. Added a hiddens module. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix typos (#6523) (#6539) * Fix typos Signed-off-by: smajumdar <[email protected]> * Fix typos Signed-off-by: smajumdar <[email protected]> --------- Signed-off-by: smajumdar <[email protected]> (cherry picked from commit 5468077f5127be1a4c88065de2544f4268b9a6e4) * added back the fast emit section to the configs. (#6540) * added back the fast emit section to the configs. Signed-off-by: Vahid <[email protected]> * added back the fast emit section to the configs. Signed-off-by: Vahid <[email protected]> --------- Signed-off-by: Vahid <[email protected]> * Fix fp16 (#6543) Signed-off-by: MaximumEntropy <[email protected]> * fix (#6529) Signed-off-by: Abhinav Khattar <[email protected]> * pass .scale instead of scaler object to core (#6545) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Change Megatron Enc Dec model to use persistent_workers (#6548) * persistent workers Signed-off-by: Abhinav Khattar <[email protected]> * fix Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) * Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by Signed-off-by: KunalDhawan <[email protected]> * updated ASR docs with the fastconformer hybrid checkpoints Signed-off-by: KunalDhawan <[email protected]> * added the fastconformer RNNT and CTC models Signed-off-by: KunalDhawan <[email protected]> --------- Signed-off-by: KunalDhawan <[email protected]> * Add scores for FastConformer models (#6557) Signed-off-by: smajumdar <[email protected]> * Patch transcribe and support offline transcribe for hybrid model (#6550) Signed-off-by: fayejf <[email protected]> * Not doing CastToFloat by default (#6524) * Not doing CastToFloat by default Signed-off-by: Boris Fomitchev <[email protected]> * Added docustring Signed-off-by: Boris Fomitchev <[email protected]> * Dummy commit Signed-off-by: Boris Fomitchev <[email protected]> --------- Signed-off-by: Boris Fomitchev <[email protected]> * temp rtd fix (#6568) Signed-off-by: Abhinav Khattar <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py for speedup (#6565) * Update manifest.py Re-order the checks for faster processing audio filepaths that are already absolute paths Signed-off-by: He Huang (Steve) <[email protected]> * Update manifest.py Signed-off-by: He Huang (Steve) <[email protected]> --------- Signed-off-by: He Huang (Steve) <[email protected]> Co-authored-by: Vahid Noroozi <[email protected]> * Turn autocast off when precision is fp32 (#6554) * Turn autocast off when precision is fp32 Signed-off-by: Abhinav Khattar <[email protected]> * address review Signed-off-by: Abhinav Khattar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: Abhinav Khattar <[email protected]> * merge Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> * More streaming conformer export fixes (#6567) Signed-off-by: Greg Clark <[email protected]> Co-authored-by: Vahid Noroozi <[email protected]> * Fix batch size reconf for T5 FT for multi-validation (#6582) Signed-off-by: Abhinav Khattar <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Updated Megatron LM encoder/decoder to use cfg for hiddens. Signed-off-by: Micha Livne <[email protected]> * 1. Added support to register externalhidden loss / transforms. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make tensor split contiguous (#6580) Signed-off-by: Abhinav Khattar <[email protected]> * Patches from main to r1.18.0 for Virtual Parallel (#6592) * Add interleaved pp support (#6498) * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <[email protected]> * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <[email protected]> * Switch to megatron core Signed-off-by: smajumdar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 892987169ef277f328e15b71a5a0c9bd961c8ee7) * Add patches for Virtual Parallel conversion (#6589) * Add patches for Virtual Parllel conversion Signed-off-by: smajumdar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 1d813a372ab51688e3af6395d905a4c0366ffd23) * Documentation for ASR-TTS models (#6594) * Add docs about hybrid ASR-TTS models Signed-off-by: Vladimir Bataev <[email protected]> * Add docs about text-only datasets Signed-off-by: Vladimir Bataev <[email protected]> * Add docs about ASR-TTS checkpoints Signed-off-by: Vladimir Bataev <[email protected]> * Add docs about ASR-TTS configs and training Signed-off-by: Vladimir Bataev <[email protected]> * Clean up Signed-off-by: Vladimir Bataev <[email protected]> * ASR-TTS docs: add to api, fix imports Signed-off-by: Vladimir Bataev <[email protected]> * Clean up Signed-off-by: Vladimir Bataev <[email protected]> * Wrap optional import Signed-off-by: Vladimir Bataev <[email protected]> * Revert general ASR import Signed-off-by: Vladimir Bataev <[email protected]> --------- Signed-off-by: Vladimir Bataev <[email protected]> * Update SDP docs (#6485) * add info about SDP e.g. processor classes in docs Signed-off-by: Elena Rastorgueva <[email protected]> * add link to SDP docs in README Signed-off-by: Elena Rastorgueva <[email protected]> * address code review comments and add SDP overview diagram Signed-off-by: Elena Rastorgueva <[email protected]> * Fix spelling typo Signed-off-by: Elena Rastorgueva <[email protected]> --------- Signed-off-by: Elena Rastorgueva <[email protected]> * Create dummy iters to satisy len checks (#6600) Signed-off-by: Abhinav Khattar <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * Restore GPT support for interleaved pipeline parallelism (#6528) * Restore logic for data-parallel communication with pipeline parallelism in GPT Signed-off-by: Tim Moon <[email protected]> * Support dynamic attention masks in GPT Signed-off-by: Tim Moon <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Debug typos Signed-off-by: Tim Moon <[email protected]> * Debug data iterator caching with interleaved pipeline parallelism Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples. Signed-off-by: Tim Moon <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Megatron-LM commit Signed-off-by: Tim Moon <[email protected]> * Distinguish between list of data iterators and data iterator that is a list Signed-off-by: Tim Moon <[email protected]> * Create dummy iters to satisy len checks Signed-off-by: Abhinav Khattar <[email protected]> * Kludge while waiting for Megatron-LM update Signed-off-by: Tim Moon <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set transformers offline to avoid rate limiting Signed-off-by: ericharper <[email protected]> --------- Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Eric Harper <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Signed-off-by: ericharper <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * Patch transcribe_util for steaming mode and add wer calculation back to inference scripts (#6601) * fix write Signed-off-by: fayejf <[email protected]> * decoding ctc Signed-off-by: fayejf <[email protected]> * temp set rnnt decoding return_best_hypothesis to true Signed-off-by: fayejf <[email protected]> * add wer cal back to transcribe_speech as requested Signed-off-by: fayejf <[email protected]> * add wer cal back to speech_to_text_buffered_infer_rnnt as requested Signed-off-by: fayejf <[email protected]> * add wer cal back to speech_to_text_buffered_infer_ctc as requested Signed-off-by: fayejf <[email protected]> * style fix Signed-off-by: fayejf <[email protected]> * reflect change in asr_evaluator Signed-off-by: fayejf <[email protected]> * reflect som and vahid comment Signed-off-by: fayejf <[email protected]> * remove return_best_hy=true in transcribe_speech Signed-off-by: fayejf <[email protected]> * no text skip Signed-off-by: fayejf <[email protected]> --------- Signed-off-by: fayejf <[email protected]> * 1. Added example conf YAML. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support in tensor_parallel. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add hat image to docs (#6619) Signed-off-by: andrusenkoau <[email protected]> * update core commit hash in readme (#6622) Signed-off-by: Abhinav Khattar <[email protected]> * Patch decoding for PC models (#6630) * Patch decoding logic for PC models Signed-off-by: smajumdar <[email protected]> * Patch decoding logic for PC models Signed-off-by: smajumdar <[email protected]> --------- Signed-off-by: smajumdar <[email protected]> * Fix wer.py where 'errors' variable was not set (#6633) Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings Signed-off-by: He Huang (Steve) <[email protected]> * fix att_context_size bug for older models. (#6635) Signed-off-by: Vahid <[email protected]> * Add megatron_core to requirements (#6639) * add megatron_core to requirements Signed-off-by: ericharper <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ericharper <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <[email protected]> * remove from jenkins Signed-off-by: ericharper <[email protected]> --------- Signed-off-by: ericharper <[email protected]> * remove dup (#6643) Signed-off-by: ericharper <[email protected]> * 1. Fixed config to use names, and added better error messages. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support to pass extra data to hiddens for loss computation. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Working on passing extra data to hiddnes. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Fixed support in loading .nemo without hiddnes module. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Improved and fixed logging of validation and testing. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed training logging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed logging of hidden loss. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Fixed logging names. 2. Added logging to hiddens and tokens loss. Signed-off-by: Micha Livne <[email protected]> * 1. Fixed conflicts. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> --------- Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Oleksii Kuchaiev <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: fayejf <[email protected]> Signed-off-by: Alexandra Antonova <[email protected]> Signed-off-by: Virginia Adams <[email protected]> Signed-off-by: Vladimir Bataev <[email protected]> Signed-off-by: ericharper <[email protected]> Signed-off-by: Vahid <[email protected]> Signed-off-by: Ante Jukić <[email protected]> Signed-off-by: David Mosallanezhad <[email protected]> Signed-off-by: sam1373 <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: ekmb <[email protected]> Signed-off-by: Yang Zhang <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: Jim O’Regan <[email protected]> Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Kunal Dhawan <[email protected]> Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Signed-off-by: Evelina <[email protected]> Signed-off-by: Taejin Park <[email protected]> Signed-off-by: KunalDhawan <[email protected]> Signed-off-by: Boris Fomitchev <[email protected]> Signed-off-by: He Huang (Steve) <[email protected]> Signed-off-by: Greg Clark <[email protected]> Signed-off-by: Elena Rastorgueva <[email protected]> Signed-off-by: Eric Harper <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Oleksii Kuchaiev <[email protected]> Co-authored-by: Jocelyn <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: fayejf <[email protected]> Co-authored-by: bene-ges <[email protected]> Co-authored-by: Alexandra Antonova <[email protected]> Co-authored-by: Virginia Adams <[email protected]> Co-authored-by: Zhilin Wang <[email protected]> Co-authored-by: Vladimir Bataev <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Vahid Noroozi <[email protected]> Co-authored-by: anteju <[email protected]> Co-authored-by: Ante Jukić <[email protected]> Co-authored-by: David <[email protected]> Co-authored-by: David Mosallanezhad <[email protected]> Co-authored-by: Samuel Kriman <[email protected]> Co-authored-by: Sandeep Subramanian <[email protected]> Co-authored-by: Evelina <[email protected]> Co-authored-by: Sean Naren <[email protected]> Co-authored-by: Yang Zhang <[email protected]> Co-authored-by: Sean Naren <[email protected]> Co-authored-by: Tim Moon <[email protected]> Co-authored-by: Neha Tadimeti <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abhinav Khattar <[email protected]> Co-authored-by: Cheng-Ping Hsieh <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> Co-authored-by: Jim O’Regan <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Mostafa Ghorbandoost <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Co-authored-by: Taejin Park <[email protected]> Co-authored-by: Boris Fomitchev <[email protected]> Co-authored-by: He Huang (Steve) <[email protected]> Co-authored-by: Greg Clark <[email protected]> Co-authored-by: Elena Rastorgueva <[email protected]>

* [TTS] bugfix for missing configs. (#4725) Signed-off-by: Xuesong Yang <[email protected]> * docs typo fix Signed-off-by: Oleksii Kuchaiev <[email protected]> * Fix pynini install in TTS tutorials (#4729) Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> * Fix ASR notebooks (#4738) Signed-off-by: smajumdar <[email protected]> Signed-off-by: smajumdar <[email protected]> * Multilingual VAD model (#4734) * add ngc link Signed-off-by: fayejf <[email protected]> * add tuned VAD config on ASR data Signed-off-by: fayejf <[email protected]> * yaml note Signed-off-by: fayejf <[email protected]> * update vad asr notebook with mVAD Signed-off-by: fayejf <[email protected]> * update vad infer config comment Signed-off-by: fayejf <[email protected]> * fix Signed-off-by: fayejf <[email protected]> * mvad sd config for ch109 Signed-off-by: fayejf <[email protected]> * update sd readme Signed-off-by: fayejf <[email protected]> * add new mVAD model to doc Signed-off-by: fayejf <[email protected]> * style fix Signed-off-by: fayejf <[email protected]> * update sd tutorial with mVAD Signed-off-by: fayejf <[email protected]> * typo fix Signed-off-by: fayejf <[email protected]> Signed-off-by: fayejf <[email protected]> * publish pretrained itn t5 model for English (#4748) Signed-off-by: Alexandra Antonova <[email protected]> Signed-off-by: Alexandra Antonova <[email protected]> Co-authored-by: Alexandra Antonova <[email protected]> * Updated docs and doc paths (#4754) * Updated docs and doc paths Signed-off-by: Virginia Adams <[email protected]> * Update Multitask_Prompt_and_PTuning.ipynb * Update README.rst * Changed branch name to use single quotes Signed-off-by: Virginia Adams <[email protected]> Signed-off-by: Virginia Adams <[email protected]> * fix bug relating to ddp strategy in joint intent slot classification tutorial (#4762) * [TTS] updated config with a German IPA phoneme tokenizer (#4756) * [TTS] added a German IPA phoneme tokenizer * [TTS][ASR] enabled customized arguments for trimming the leading and trailing silence. * [TTS] disabled spline interpolation for beta-binomial distribution. Let it generate align prior and save to disks. Use a new phoneme tokenizer. * [TTS] use consistent spline interpolation with fastpitch checkpoint when generating mel-spectrograms for hifigan finetune. Signed-off-by: Xuesong Yang <[email protected]> * Update r1.11 to new heteronyms list (#4745) * Update configs to new heteronyms list * Remove old heteronyms list, add alt 'merchandise' pron to CMUdict * Update remaining references to old heteronyms list Signed-off-by: Jocelyn Huang <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> * [TTS] Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints (#4763) Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> * [TTS] Add single male speaker German FastPitch and HiFiGAN NGC checkpoints (#4770) Signed-off-by: Xuesong Yang <[email protected]> * Update CMUdict with more recent 0.7b entries (#4768) Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> * Install pynini in docker container (#4733) Signed-off-by: Vladimir Bataev <[email protected]> Signed-off-by: Vladimir Bataev <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Fix tutorial formatting (#4778) Signed-off-by: Jocelyn Huang <[email protected]> * [TTS] deprecated old scripts for ljspeech. (#4780) * deprecated old scripts for ljspeech. * removed relevent function calls in TTS docs. Signed-off-by: Xuesong Yang <[email protected]> * update branch and typos (#4788) Signed-off-by: ericharper <[email protected]> Signed-off-by: ericharper <[email protected]> * Adding support for models trained with full context for cache-aware streaming. (#4687) * added support for models trained with full context. Signed-off-by: Vahid <[email protected]> * fixed style. Signed-off-by: Vahid <[email protected]> * dropped seq_range Signed-off-by: Vahid <[email protected]> * fixed indexing in caching methods. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * updated docs. Signed-off-by: Vahid <[email protected]> * addressed comments. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * change frame-wise to cache-aware. Signed-off-by: Vahid <[email protected]> * change frame-wise to cache-aware. Signed-off-by: Vahid <[email protected]> * change frame-wise to cache-aware. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> Signed-off-by: Vahid <[email protected]> * Update megatron encoder decoder model to support py37 for colab (#4791) * [ASR] Add pretrained ASR models for Croatian (#4682) * [ASR] Add pretrained ASR models for Croatian Signed-off-by: Ante Jukić <[email protected]> * Fix style for import Signed-off-by: Ante Jukić <[email protected]> Signed-off-by: Ante Jukić <[email protected]> Co-authored-by: Ante Jukić <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> * added/fixed export for Megatron models (#4712) * added/fixed export for Megatron models Signed-off-by: David Mosallanezhad <[email protected]> * fixed style Signed-off-by: David Mosallanezhad <[email protected]> * fixed FusedScaleMaskSoftmax in BioMegatron Signed-off-by: David Mosallanezhad <[email protected]> * included comments Signed-off-by: David Mosallanezhad <[email protected]> Signed-off-by: David Mosallanezhad <[email protected]> Co-authored-by: David Mosallanezhad <[email protected]> Co-authored-by: Eric Harper <[email protected]> * update branch for qa notebook Signed-off-by: ericharper <[email protected]> * Fix initializing weights from ptl ckpt with exclude (#4807) Signed-off-by: sam1373 <[email protected]> Signed-off-by: sam1373 <[email protected]> * Fix index error from addition of voiced_mask and p_voiced (#4811) Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> * T5 prompt learning fixes (#4771) * RPE, hidden size and config fixes Signed-off-by: MaximumEntropy <[email protected]> * Update to reflect new config names Signed-off-by: MaximumEntropy <[email protected]> * Sentencepiece fixes Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Fix finetuning Signed-off-by: MaximumEntropy <[email protected]> * Add encoder seq len to gpt Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Add finetune eval script Signed-off-by: MaximumEntropy <[email protected]> * Fix name Signed-off-by: MaximumEntropy <[email protected]> * Update Jenkinsfile Signed-off-by: MaximumEntropy <[email protected]> * Update config Signed-off-by: MaximumEntropy <[email protected]> * Fix CI test Signed-off-by: MaximumEntropy <[email protected]> * Update check Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Backward compat Signed-off-by: MaximumEntropy <[email protected]> * Update CI test Signed-off-by: MaximumEntropy <[email protected]> * Split rank for Enc-Dec models Signed-off-by: MaximumEntropy <[email protected]> * Address comments Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Co-authored-by: Virginia Adams <[email protected]> * G2P docs (#4841) * g2p docs added Signed-off-by: ekmb <[email protected]> * fix references Signed-off-by: ekmb <[email protected]> * address review feedback Signed-off-by: ekmb <[email protected]> Signed-off-by: ekmb <[email protected]> * Fix providing glue in seq2seq eval (#4843) * Fix providing glue in seq2seq eval Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> * Updated inference code and squad scripts (#4835) * Updated inference code and squad scripts Signed-off-by: Virginia Adams <[email protected]> * Reverted GPT & T5 inference files back to use NLPDDPlugin Signed-off-by: Virginia Adams <[email protected]> * Overwrite frozen LM to use fused adam Signed-off-by: Virginia Adams <[email protected]> * Added padded vocab size Signed-off-by: Virginia Adams <[email protected]> * Fixed val check interval value Signed-off-by: Virginia Adams <[email protected]> * Python format fix Signed-off-by: Virginia Adams <[email protected]> * Make t5 prompt learning preds write to file Signed-off-by: Virginia Adams <[email protected]> * Added back dp=1 check Signed-off-by: Virginia Adams <[email protected]> Signed-off-by: Virginia Adams <[email protected]> Co-authored-by: Sandeep Subramanian <[email protected]> * Update README.rst * Fix uppercasing mismatch for IPA heteronyms (#4860) Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> * Set the number of workers to 0 for validation and test sets in all enc-dec models (#4790) * Set workers to 0 for validation and test Signed-off-by: MaximumEntropy <[email protected]> * Revert pin memory Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Co-authored-by: Sean Naren <[email protected]> * Fix mha (#4866) * fix bug in mha forward function related to cache update return type Signed-off-by: Yang Zhang <[email protected]> * fix lgtm Signed-off-by: Yang Zhang <[email protected]> Signed-off-by: Yang Zhang <[email protected]> Co-authored-by: Sean Naren <[email protected]> * ipa bug fix (#4871) Signed-off-by: ekmb <[email protected]> Signed-off-by: ekmb <[email protected]> * Fix Megatron NMT consumed samples and ckpt_to_nemo split rank (#4884) * Fix nmt and ckpt_to_nemo Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> * added utf8 encoding (#4892) Signed-off-by: Virginia Adams <[email protected]> Signed-off-by: Virginia Adams <[email protected]> * 1. Applying the same patch to r1.11.0 (#4894) Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Micha Livne <[email protected]> * Update tutorials.rst (#4897) * update readme with apex commit Signed-off-by: ericharper <[email protected]> * Add support for Apex distributed Adam optimizer with GPT-3 (#4487) * Add support for Apex distributed Adam optimizer with GPT-3 Signed-off-by: Tim Moon <[email protected]> * Fix bug in grad clipping with dist Adam Grad norm was computed over all params, not respecting model parallelism. Signed-off-by: Tim Moon <[email protected]> * Fix bug with DDP initialization Signed-off-by: Tim Moon <[email protected]> * Make distopt dependent on megatron_amp_o2 Signed-off-by: Tim Moon <[email protected]> * Fix code formatting Signed-off-by: Tim Moon <[email protected]> * Handle dist Adam in optimizer unit tests Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Tim Moon <[email protected]> Co-authored-by: Eric Harper <[email protected]> * update readme Signed-off-by: ericharper <[email protected]> * update readme Signed-off-by: ericharper <[email protected]> * latent model support * 1. Debugging. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. * update branch Signed-off-by: ericharper <[email protected]> * fix replace_bos_with_pad not found (#6443) Signed-off-by: Abhinav Khattar <[email protected]> * Support Swiglu in TP PP Conversion (#6437) * Support Swiglu in TP PP Conversion Signed-off-by: smajumdar <[email protected]> * Guard activation Signed-off-by: smajumdar <[email protected]> * Guard activation Signed-off-by: smajumdar <[email protected]> --------- Signed-off-by: smajumdar <[email protected]> * BERT pre-training mp fork to spawn (#6442) * change bert fork to spawn Signed-off-by: Abhinav Khattar <[email protected]> * num_workers=0 fix Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> * Meagtron encoder decoder fix for empty validation outputs (#6459) * 1. Meagtron encoder decoder fix for empty validation outputs. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. --------- Signed-off-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> * Added/updated new Conformer configs (#6426) * updated conf files. Signed-off-by: Vahid <[email protected]> * added confs. Signed-off-by: Vahid <[email protected]> * moved longconformer confs. Signed-off-by: Vahid <[email protected]> * updated readme. Signed-off-by: Vahid <[email protected]> * updated readme. Signed-off-by: Vahid <[email protected]> * updated batch sizes and added fastconformer ctc streaming configs. Signed-off-by: Vahid <[email protected]> * updated batch sizes. Signed-off-by: Vahid <[email protected]> * added hybrid support. Signed-off-by: Vahid <[email protected]> * added hybrid support. Signed-off-by: Vahid <[email protected]> --------- Signed-off-by: Vahid <[email protected]> * reduce workers on NMT CI (#6472) Signed-off-by: Abhinav Khattar <[email protected]> * move to nvidia megatron repo (#6465) Signed-off-by: Abhinav Khattar <[email protected]> * Megatron KERPLE positional embeddings (#6478) * [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- Signed-off-by: hsiehjackson <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. Signed-off-by: Xuesong Yang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Xuesong Yang <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] FastPitch speaker encoder (#6417) * Add initial codes Signed-off-by: hsiehjackson <[email protected]> * Remove wemb Signed-off-by: hsiehjackson <[email protected]> * Fix import Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss Signed-off-by: hsiehjackson <[email protected]> * Add ConditionalInput Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test Signed-off-by: hsiehjackson <[email protected]> * Add initial codes Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix import error Signed-off-by: hsiehjackson <[email protected]> * Add initial codes Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix dataset error Signed-off-by: hsiehjackson <[email protected]> * Remove reference speaker embedding Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Remove SV encoder Signed-off-by: hsiehjackson <[email protected]> * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix length type Signed-off-by: hsiehjackson <[email protected]> * Fix append Signed-off-by: hsiehjackson <[email protected]> * Move error msg Signed-off-by: hsiehjackson <[email protected]> * Add look-up into speaker encoder Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Add valueerror msg Signed-off-by: hsiehjackson <[email protected]> * Move lookup Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Remove unused Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix error Signed-off-by: hsiehjackson <[email protected]> * Rebase and Fix error Signed-off-by: hsiehjackson <[email protected]> * Fix spk encoder Signed-off-by: hsiehjackson <[email protected]> * Rename n_speakers Signed-off-by: hsiehjackson <[email protected]> * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error Signed-off-by: hsiehjackson <[email protected]> --------- Signed-off-by: hsiehjackson <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Sharded manifests for tarred datasets (#6395) * testing sharded manifests Signed-off-by: Dima Rekesh <[email protected]> * compatibility Signed-off-by: Dima Rekesh <[email protected]> * proper fixes Signed-off-by: Dima Rekesh <[email protected]> * adding flag tot convert_to_tarred_audio_dataset Signed-off-by: Dima Rekesh <[email protected]> * shard_manifests conf param Signed-off-by: Dima Rekesh <[email protected]> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <[email protected]> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <[email protected]> * distributed checks Signed-off-by: Dima Rekesh <[email protected]> * typo Signed-off-by: Dima Rekesh <[email protected]> * typo Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py Signed-off-by: Dima Rekesh <[email protected]> * reversing manifest shards flag Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests Signed-off-by: Dima Rekesh <[email protected]> * excluding manifests from webdataset url expansion Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore Signed-off-by: Dima Rekesh <[email protected]> * explicit use of UTF-8 for manifest i/o Signed-off-by: Dima Rekesh <[email protected]> --------- Signed-off-by: Dima Rekesh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) Signed-off-by: Jim O’Regan <[email protected]> * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. Signed-off-by: Mostafa Ghorbandoost <[email protected]> * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. Signed-off-by: Mostafa Ghorbandoost <[email protected]> --------- Signed-off-by: Mostafa Ghorbandoost <[email protected]> * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support Signed-off-by: Dmytro Pykhtar <[email protected]> * added tests for rampup batch size Signed-off-by: Dmytro Pykhtar <[email protected]> * fixed the typos Signed-off-by: Dmytro Pykhtar <[email protected]> * added assertions Signed-off-by: Dmytro Pykhtar <[email protected]> * changed assertion rules Signed-off-by: Dmytro Pykhtar <[email protected]> * deleted unused imports Signed-off-by: Dmytro Pykhtar <[email protected]> * changed tests for rampup batch size Signed-off-by: Dmytro Pykhtar <[email protected]> * updated rampup batch size tests Signed-off-by: Dmytro Pykhtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling Signed-off-by: Dmytro Pykhtar <[email protected]> * rampup batch size tests changes Signed-off-by: Dmytro Pykhtar <[email protected]> --------- Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- Signed-off-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default Signed-off-by: Kunal Dhawan <[email protected]> * updated README with the new agg_tokenizer_manifest flag Signed-off-by: Kunal Dhawan <[email protected]> * fixed typo in scripts/speech_recognition/code_switching/README.md Signed-off-by: Kunal Dhawan <[email protected]> * changed agg_tokenizer_manifest to is_lid_manifest Signed-off-by: Kunal Dhawan <[email protected]> --------- Signed-off-by: Kunal Dhawan <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script Signed-off-by: andrusenkoau <[email protected]> * add return encoding embedding option Signed-off-by: andrusenkoau <[email protected]> * update script Signed-off-by: andrusenkoau <[email protected]> * add rnnt and hat ngram decoding script Signed-off-by: andrusenkoau <[email protected]> * add some parameters Signed-off-by: andrusenkoau <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig Signed-off-by: andrusenkoau <[email protected]> * replace return_encoder_embeddings parameter Signed-off-by: andrusenkoau <[email protected]> * generalization of scipt behavior Signed-off-by: andrusenkoau <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <[email protected]> * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <[email protected]> * add manual encoder_embeddings calculation Signed-off-by: andrusenkoau <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 Signed-off-by: Andrei Andrusenko <[email protected]> * fix rescoring description Signed-off-by: Andrei Andrusenko <[email protected]> --------- Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * fix replace_bos_with_pad not found (#6443) (#6450) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * reduce workers on NMT CI (#6472) (#6474) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * 1. Added KERPLE positional embeddings to encoder-decoder. Signed-off-by: Micha Livne <[email protected]> * 1. Added a missing file. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: Jim O’Regan <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Kunal Dhawan <[email protected]> Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Cheng-Ping Hsieh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> Co-authored-by: Jim O’Regan <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: Mostafa Ghorbandoost <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * 1. Added external index sample. (#6462) Signed-off-by: Micha Livne <[email protected]> * Fix cache aware hybrid bugs (#6466) * Update README to add core installation (#6488) * update README for megatron-core Signed-off-by: Abhinav Khattar <[email protected]> * fix Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> * Fix typos (#6494) Signed-off-by: smajumdar <[email protected]> * fix broken links r1.18.0 (#6501) * fix broken links Signed-off-by: Evelina <[email protected]> * fix broken links Signed-off-by: Evelina <[email protected]> --------- Signed-off-by: Evelina <[email protected]> * 1. Fixed gaussian hidden transform. Signed-off-by: Micha Livne <[email protected]> * 1. Finished updating hidden loss for MIM. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix custom forward_torch_softmax (#6512) Signed-off-by: Abhinav Khattar <[email protected]> * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamp… (#6500) * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator r1.18.0 Signed-off-by: Taejin Park <[email protected]> * ignore keep_logits in FrameBatchASRLogits Signed-off-by: Taejin Park <[email protected]> --------- Signed-off-by: Taejin Park <[email protected]> * [TTS] fixed broken path. (#6514) Signed-off-by: Xuesong Yang <[email protected]> * 1. Added a hiddens module. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix typos (#6523) (#6539) * Fix typos Signed-off-by: smajumdar <[email protected]> * Fix typos Signed-off-by: smajumdar <[email protected]> --------- Signed-off-by: smajumdar <[email protected]> (cherry picked from commit 5468077f5127be1a4c88065de2544f4268b9a6e4) * added back the fast emit section to the configs. (#6540) * added back the fast emit section to the configs. Signed-off-by: Vahid <[email protected]> * added back the fast emit section to the configs. Signed-off-by: Vahid <[email protected]> --------- Signed-off-by: Vahid <[email protected]> * Fix fp16 (#6543) Signed-off-by: MaximumEntropy <[email protected]> * fix (#6529) Signed-off-by: Abhinav Khattar <[email protected]> * pass .scale instead of scaler object to core (#6545) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Change Megatron Enc Dec model to use persistent_workers (#6548) * persistent workers Signed-off-by: Abhinav Khattar <[email protected]> * fix Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) * Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by Signed-off-by: KunalDhawan <[email protected]> * updated ASR docs with the fastconformer hybrid checkpoints Signed-off-by: KunalDhawan <[email protected]> * added the fastconformer RNNT and CTC models Signed-off-by: KunalDhawan <[email protected]> --------- Signed-off-by: KunalDhawan <[email protected]> * Add scores for FastConformer models (#6557) Signed-off-by: smajumdar <[email protected]> * Patch transcribe and support offline transcribe for hybrid model (#6550) Signed-off-by: fayejf <[email protected]> * Not doing CastToFloat by default (#6524) * Not doing CastToFloat by default Signed-off-by: Boris Fomitchev <[email protected]> * Added docustring Signed-off-by: Boris Fomitchev <[email protected]> * Dummy commit Signed-off-by: Boris Fomitchev <[email protected]> --------- Signed-off-by: Boris Fomitchev <[email protected]> * temp rtd fix (#6568) Signed-off-by: Abhinav Khattar <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py for speedup (#6565) * Update manifest.py Re-order the checks for faster processing audio filepaths that are already absolute paths Signed-off-by: He Huang (Steve) <[email protected]> * Update manifest.py Signed-off-by: He Huang (Steve) <[email protected]> --------- Signed-off-by: He Huang (Steve) <[email protected]> Co-authored-by: Vahid Noroozi <[email protected]> * Turn autocast off when precision is fp32 (#6554) * Turn autocast off when precision is fp32 Signed-off-by: Abhinav Khattar <[email protected]> * address review Signed-off-by: Abhinav Khattar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: Abhinav Khattar <[email protected]> * merge Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> * More streaming conformer export fixes (#6567) Signed-off-by: Greg Clark <[email protected]> Co-authored-by: Vahid Noroozi <[email protected]> * Fix batch size reconf for T5 FT for multi-validation (#6582) Signed-off-by: Abhinav Khattar <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Updated Megatron LM encoder/decoder to use cfg for hiddens. Signed-off-by: Micha Livne <[email protected]> * 1. Added support to register externalhidden loss / transforms. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make tensor split contiguous (#6580) Signed-off-by: Abhinav Khattar <[email protected]> * Patches from main to r1.18.0 for Virtual Parallel (#6592) * Add interleaved pp support (#6498) * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <[email protected]> * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <[email protected]> * Switch to megatron core Signed-off-by: smajumdar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 892987169ef277f328e15b71a5a0c9bd961c8ee7) * Add patches for Virtual Parallel conversion (#6589) * Add patches for Virtual Parllel conversion Signed-off-by: smajumdar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 1d813a372ab51688e3af6395d905a4c0366ffd23) * Documentation for ASR-TTS models (#6594) * Add docs about hybrid ASR-TTS models Signed-off-by: Vladimir Bataev <[email protected]> * Add docs about text-only datasets Signed-off-by: Vladimir Bataev <[email protected]> * Add docs about ASR-TTS checkpoints Signed-off-by: Vladimir Bataev <[email protected]> * Add docs about ASR-TTS configs and training Signed-off-by: Vladimir Bataev <[email protected]> * Clean up Signed-off-by: Vladimir Bataev <[email protected]> * ASR-TTS docs: add to api, fix imports Signed-off-by: Vladimir Bataev <[email protected]> * Clean up Signed-off-by: Vladimir Bataev <[email protected]> * Wrap optional import Signed-off-by: Vladimir Bataev <[email protected]> * Revert general ASR import Signed-off-by: Vladimir Bataev <[email protected]> --------- Signed-off-by: Vladimir Bataev <[email protected]> * Update SDP docs (#6485) * add info about SDP e.g. processor classes in docs Signed-off-by: Elena Rastorgueva <[email protected]> * add link to SDP docs in README Signed-off-by: Elena Rastorgueva <[email protected]> * address code review comments and add SDP overview diagram Signed-off-by: Elena Rastorgueva <[email protected]> * Fix spelling typo Signed-off-by: Elena Rastorgueva <[email protected]> --------- Signed-off-by: Elena Rastorgueva <[email protected]> * Create dummy iters to satisy len checks (#6600) Signed-off-by: Abhinav Khattar <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * Restore GPT support for interleaved pipeline parallelism (#6528) * Restore logic for data-parallel communication with pipeline parallelism in GPT Signed-off-by: Tim Moon <[email protected]> * Support dynamic attention masks in GPT Signed-off-by: Tim Moon <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Debug typos Signed-off-by: Tim Moon <[email protected]> * Debug data iterator caching with interleaved pipeline parallelism Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples. Signed-off-by: Tim Moon <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Megatron-LM commit Signed-off-by: Tim Moon <[email protected]> * Distinguish between list of data iterators and data iterator that is a list Signed-off-by: Tim Moon <[email protected]> * Create dummy iters to satisy len checks Signed-off-by: Abhinav Khattar <[email protected]> * Kludge while waiting for Megatron-LM update Signed-off-by: Tim Moon <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set transformers offline to avoid rate limiting Signed-off-by: ericharper <[email protected]> --------- Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Eric Harper <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Signed-off-by: ericharper <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * Patch transcribe_util for steaming mode and add wer calculation back to inference scripts (#6601) * fix write Signed-off-by: fayejf <[email protected]> * decoding ctc Signed-off-by: fayejf <[email protected]> * temp set rnnt decoding return_best_hypothesis to true Signed-off-by: fayejf <[email protected]> * add wer cal back to transcribe_speech as requested Signed-off-by: fayejf <[email protected]> * add wer cal back to speech_to_text_buffered_infer_rnnt as requested Signed-off-by: fayejf <[email protected]> * add wer cal back to speech_to_text_buffered_infer_ctc as requested Signed-off-by: fayejf <[email protected]> * style fix Signed-off-by: fayejf <[email protected]> * reflect change in asr_evaluator Signed-off-by: fayejf <[email protected]> * reflect som and vahid comment Signed-off-by: fayejf <[email protected]> * remove return_best_hy=true in transcribe_speech Signed-off-by: fayejf <[email protected]> * no text skip Signed-off-by: fayejf <[email protected]> --------- Signed-off-by: fayejf <[email protected]> * 1. Added example conf YAML. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support in tensor_parallel. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add hat image to docs (#6619) Signed-off-by: andrusenkoau <[email protected]> * update core commit hash in readme (#6622) Signed-off-by: Abhinav Khattar <[email protected]> * Patch decoding for PC models (#6630) * Patch decoding logic for PC models Signed-off-by: smajumdar <[email protected]> * Patch decoding logic for PC models Signed-off-by: smajumdar <[email protected]> --------- Signed-off-by: smajumdar <[email protected]> * Fix wer.py where 'errors' variable was not set (#6633) Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings Signed-off-by: He Huang (Steve) <[email protected]> * fix att_context_size bug for older models. (#6635) Signed-off-by: Vahid <[email protected]> * Add megatron_core to requirements (#6639) * add megatron_core to requirements Signed-off-by: ericharper <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ericharper <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <[email protected]> * remove from jenkins Signed-off-by: ericharper <[email protected]> --------- Signed-off-by: ericharper <[email protected]> * remove dup (#6643) Signed-off-by: ericharper <[email protected]> * 1. Fixed config to use names, and added better error messages. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support to pass extra data to hiddens for loss computation. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Working on passing extra data to hiddnes. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Fixed support in loading .nemo without hiddnes module. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Improved and fixed logging of validation and testing. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed training logging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed logging of hidden loss. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Fixed logging names. 2. Added logging to hiddens and tokens loss. Signed-off-by: Micha Livne <[email protected]> * 1. Fixed conflicts. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> --------- Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Oleksii Kuchaiev <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: fayejf <[email protected]> Signed-off-by: Alexandra Antonova <[email protected]> Signed-off-by: Virginia Adams <[email protected]> Signed-off-by: Vladimir Bataev <[email protected]> Signed-off-by: ericharper <[email protected]> Signed-off-by: Vahid <[email protected]> Signed-off-by: Ante Jukić <[email protected]> Signed-off-by: David Mosallanezhad <[email protected]> Signed-off-by: sam1373 <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: ekmb <[email protected]> Signed-off-by: Yang Zhang <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: Jim O’Regan <[email protected]> Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Kunal Dhawan <[email protected]> Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Signed-off-by: Evelina <[email protected]> Signed-off-by: Taejin Park <[email protected]> Signed-off-by: KunalDhawan <[email protected]> Signed-off-by: Boris Fomitchev <[email protected]> Signed-off-by: He Huang (Steve) <[email protected]> Signed-off-by: Greg Clark <[email protected]> Signed-off-by: Elena Rastorgueva <[email protected]> Signed-off-by: Eric Harper <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Oleksii Kuchaiev <[email protected]> Co-authored-by: Jocelyn <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: fayejf <[email protected]> Co-authored-by: bene-ges <[email protected]> Co-authored-by: Alexandra Antonova <[email protected]> Co-authored-by: Virginia Adams <[email protected]> Co-authored-by: Zhilin Wang <[email protected]> Co-authored-by: Vladimir Bataev <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Vahid Noroozi <[email protected]> Co-authored-by: anteju <[email protected]> Co-authored-by: Ante Jukić <[email protected]> Co-authored-by: David <[email protected]> Co-authored-by: David Mosallanezhad <[email protected]> Co-authored-by: Samuel Kriman <[email protected]> Co-authored-by: Sandeep Subramanian <[email protected]> Co-authored-by: Evelina <[email protected]> Co-authored-by: Sean Naren <[email protected]> Co-authored-by: Yang Zhang <[email protected]> Co-authored-by: Sean Naren <[email protected]> Co-authored-by: Tim Moon <[email protected]> Co-authored-by: Neha Tadimeti <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abhinav Khattar <[email protected]> Co-authored-by: Cheng-Ping Hsieh <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> Co-authored-by: Jim O’Regan <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Mostafa Ghorbandoost <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Co-authored-by: Taejin Park <[email protected]> Co-authored-by: Boris Fomitchev <[email protected]> Co-authored-by: He Huang (Steve) <[email protected]> Co-authored-by: Greg Clark <[email protected]> Co-authored-by: Elena Rastorgueva <[email protected]> Signed-off-by: dorotat <[email protected]>

* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- * [TTS] FastPitch speaker encoder (#6417) * Add initial codes * Remove wemb * Fix import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss * Add ConditionalInput * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test * Add initial codes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix import error * Add initial codes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix dataset error * Remove reference speaker embedding * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove SV encoder * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix length type * Fix append * Move error msg * Add look-up into speaker encoder * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add valueerror msg * Move lookup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unused * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error * Rebase and Fix error * Fix spk encoder * Rename n_speakers * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error --------- * Sharded manifests for tarred datasets (#6395) * testing sharded manifests * compatibility * proper fixes * adding flag tot convert_to_tarred_audio_dataset * shard_manifests conf param * propagating the shard_manifests param * propagating the shard_manifests param * distributed checks * typo * typo * fixes * fixes * fixes * fixes * fixes * fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py * reversing manifest shards flag * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests * excluding manifests from webdataset url expansion * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore * explicit use of UTF-8 for manifest i/o --------- * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. --------- * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support * added tests for rampup batch size * fixed the typos * added assertions * changed assertion rules * deleted unused imports * changed tests for rampup batch size * updated rampup batch size tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling * rampup batch size tests changes --------- * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default * updated README with the new agg_tokenizer_manifest flag * fixed typo in scripts/speech_recognition/code_switching/README.md * changed agg_tokenizer_manifest to is_lid_manifest --------- * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script * add return encoding embedding option * update script * add rnnt and hat ngram decoding script * add some parameters * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig * replace return_encoder_embeddings parameter * generalization of scipt behavior * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter * remove return_encoder_embeddings parameter * add manual encoder_embeddings calculation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 * fix rescoring description --------- * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- * fix replace_bos_with_pad not found (#6443) (#6450) * reduce workers on NMT CI (#6472) (#6474) * 1. Added KERPLE positional embeddings to encoder-decoder. * 1. Added a missing file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: Jim O’Regan <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Kunal Dhawan <[email protected]> Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Cheng-Ping Hsieh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> Co-authored-by: Jim O’Regan <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: Mostafa Ghorbandoost <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]>

* [TTS] bugfix for missing configs. (#4725) Signed-off-by: Xuesong Yang <[email protected]> * docs typo fix Signed-off-by: Oleksii Kuchaiev <[email protected]> * Fix pynini install in TTS tutorials (#4729) Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> * Fix ASR notebooks (#4738) Signed-off-by: smajumdar <[email protected]> Signed-off-by: smajumdar <[email protected]> * Multilingual VAD model (#4734) * add ngc link Signed-off-by: fayejf <[email protected]> * add tuned VAD config on ASR data Signed-off-by: fayejf <[email protected]> * yaml note Signed-off-by: fayejf <[email protected]> * update vad asr notebook with mVAD Signed-off-by: fayejf <[email protected]> * update vad infer config comment Signed-off-by: fayejf <[email protected]> * fix Signed-off-by: fayejf <[email protected]> * mvad sd config for ch109 Signed-off-by: fayejf <[email protected]> * update sd readme Signed-off-by: fayejf <[email protected]> * add new mVAD model to doc Signed-off-by: fayejf <[email protected]> * style fix Signed-off-by: fayejf <[email protected]> * update sd tutorial with mVAD Signed-off-by: fayejf <[email protected]> * typo fix Signed-off-by: fayejf <[email protected]> Signed-off-by: fayejf <[email protected]> * publish pretrained itn t5 model for English (#4748) Signed-off-by: Alexandra Antonova <[email protected]> Signed-off-by: Alexandra Antonova <[email protected]> Co-authored-by: Alexandra Antonova <[email protected]> * Updated docs and doc paths (#4754) * Updated docs and doc paths Signed-off-by: Virginia Adams <[email protected]> * Update Multitask_Prompt_and_PTuning.ipynb * Update README.rst * Changed branch name to use single quotes Signed-off-by: Virginia Adams <[email protected]> Signed-off-by: Virginia Adams <[email protected]> * fix bug relating to ddp strategy in joint intent slot classification tutorial (#4762) * [TTS] updated config with a German IPA phoneme tokenizer (#4756) * [TTS] added a German IPA phoneme tokenizer * [TTS][ASR] enabled customized arguments for trimming the leading and trailing silence. * [TTS] disabled spline interpolation for beta-binomial distribution. Let it generate align prior and save to disks. Use a new phoneme tokenizer. * [TTS] use consistent spline interpolation with fastpitch checkpoint when generating mel-spectrograms for hifigan finetune. Signed-off-by: Xuesong Yang <[email protected]> * Update r1.11 to new heteronyms list (#4745) * Update configs to new heteronyms list * Remove old heteronyms list, add alt 'merchandise' pron to CMUdict * Update remaining references to old heteronyms list Signed-off-by: Jocelyn Huang <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> * [TTS] Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints (#4763) Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> * [TTS] Add single male speaker German FastPitch and HiFiGAN NGC checkpoints (#4770) Signed-off-by: Xuesong Yang <[email protected]> * Update CMUdict with more recent 0.7b entries (#4768) Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> * Install pynini in docker container (#4733) Signed-off-by: Vladimir Bataev <[email protected]> Signed-off-by: Vladimir Bataev <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Fix tutorial formatting (#4778) Signed-off-by: Jocelyn Huang <[email protected]> * [TTS] deprecated old scripts for ljspeech. (#4780) * deprecated old scripts for ljspeech. * removed relevent function calls in TTS docs. Signed-off-by: Xuesong Yang <[email protected]> * update branch and typos (#4788) Signed-off-by: ericharper <[email protected]> Signed-off-by: ericharper <[email protected]> * Adding support for models trained with full context for cache-aware streaming. (#4687) * added support for models trained with full context. Signed-off-by: Vahid <[email protected]> * fixed style. Signed-off-by: Vahid <[email protected]> * dropped seq_range Signed-off-by: Vahid <[email protected]> * fixed indexing in caching methods. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * updated docs. Signed-off-by: Vahid <[email protected]> * addressed comments. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> * change frame-wise to cache-aware. Signed-off-by: Vahid <[email protected]> * change frame-wise to cache-aware. Signed-off-by: Vahid <[email protected]> * change frame-wise to cache-aware. Signed-off-by: Vahid <[email protected]> * fixed code style. Signed-off-by: Vahid <[email protected]> Signed-off-by: Vahid <[email protected]> * Update megatron encoder decoder model to support py37 for colab (#4791) * [ASR] Add pretrained ASR models for Croatian (#4682) * [ASR] Add pretrained ASR models for Croatian Signed-off-by: Ante Jukić <[email protected]> * Fix style for import Signed-off-by: Ante Jukić <[email protected]> Signed-off-by: Ante Jukić <[email protected]> Co-authored-by: Ante Jukić <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> * added/fixed export for Megatron models (#4712) * added/fixed export for Megatron models Signed-off-by: David Mosallanezhad <[email protected]> * fixed style Signed-off-by: David Mosallanezhad <[email protected]> * fixed FusedScaleMaskSoftmax in BioMegatron Signed-off-by: David Mosallanezhad <[email protected]> * included comments Signed-off-by: David Mosallanezhad <[email protected]> Signed-off-by: David Mosallanezhad <[email protected]> Co-authored-by: David Mosallanezhad <[email protected]> Co-authored-by: Eric Harper <[email protected]> * update branch for qa notebook Signed-off-by: ericharper <[email protected]> * Fix initializing weights from ptl ckpt with exclude (#4807) Signed-off-by: sam1373 <[email protected]> Signed-off-by: sam1373 <[email protected]> * Fix index error from addition of voiced_mask and p_voiced (#4811) Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> * T5 prompt learning fixes (#4771) * RPE, hidden size and config fixes Signed-off-by: MaximumEntropy <[email protected]> * Update to reflect new config names Signed-off-by: MaximumEntropy <[email protected]> * Sentencepiece fixes Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Fix finetuning Signed-off-by: MaximumEntropy <[email protected]> * Add encoder seq len to gpt Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Add finetune eval script Signed-off-by: MaximumEntropy <[email protected]> * Fix name Signed-off-by: MaximumEntropy <[email protected]> * Update Jenkinsfile Signed-off-by: MaximumEntropy <[email protected]> * Update config Signed-off-by: MaximumEntropy <[email protected]> * Fix CI test Signed-off-by: MaximumEntropy <[email protected]> * Update check Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Backward compat Signed-off-by: MaximumEntropy <[email protected]> * Update CI test Signed-off-by: MaximumEntropy <[email protected]> * Split rank for Enc-Dec models Signed-off-by: MaximumEntropy <[email protected]> * Address comments Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Co-authored-by: Virginia Adams <[email protected]> * G2P docs (#4841) * g2p docs added Signed-off-by: ekmb <[email protected]> * fix references Signed-off-by: ekmb <[email protected]> * address review feedback Signed-off-by: ekmb <[email protected]> Signed-off-by: ekmb <[email protected]> * Fix providing glue in seq2seq eval (#4843) * Fix providing glue in seq2seq eval Signed-off-by: MaximumEntropy <[email protected]> * Fix Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> * Updated inference code and squad scripts (#4835) * Updated inference code and squad scripts Signed-off-by: Virginia Adams <[email protected]> * Reverted GPT & T5 inference files back to use NLPDDPlugin Signed-off-by: Virginia Adams <[email protected]> * Overwrite frozen LM to use fused adam Signed-off-by: Virginia Adams <[email protected]> * Added padded vocab size Signed-off-by: Virginia Adams <[email protected]> * Fixed val check interval value Signed-off-by: Virginia Adams <[email protected]> * Python format fix Signed-off-by: Virginia Adams <[email protected]> * Make t5 prompt learning preds write to file Signed-off-by: Virginia Adams <[email protected]> * Added back dp=1 check Signed-off-by: Virginia Adams <[email protected]> Signed-off-by: Virginia Adams <[email protected]> Co-authored-by: Sandeep Subramanian <[email protected]> * Update README.rst * Fix uppercasing mismatch for IPA heteronyms (#4860) Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> * Set the number of workers to 0 for validation and test sets in all enc-dec models (#4790) * Set workers to 0 for validation and test Signed-off-by: MaximumEntropy <[email protected]> * Revert pin memory Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Co-authored-by: Sean Naren <[email protected]> * Fix mha (#4866) * fix bug in mha forward function related to cache update return type Signed-off-by: Yang Zhang <[email protected]> * fix lgtm Signed-off-by: Yang Zhang <[email protected]> Signed-off-by: Yang Zhang <[email protected]> Co-authored-by: Sean Naren <[email protected]> * ipa bug fix (#4871) Signed-off-by: ekmb <[email protected]> Signed-off-by: ekmb <[email protected]> * Fix Megatron NMT consumed samples and ckpt_to_nemo split rank (#4884) * Fix nmt and ckpt_to_nemo Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> * added utf8 encoding (#4892) Signed-off-by: Virginia Adams <[email protected]> Signed-off-by: Virginia Adams <[email protected]> * 1. Applying the same patch to r1.11.0 (#4894) Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Micha Livne <[email protected]> * Update tutorials.rst (#4897) * update readme with apex commit Signed-off-by: ericharper <[email protected]> * Add support for Apex distributed Adam optimizer with GPT-3 (#4487) * Add support for Apex distributed Adam optimizer with GPT-3 Signed-off-by: Tim Moon <[email protected]> * Fix bug in grad clipping with dist Adam Grad norm was computed over all params, not respecting model parallelism. Signed-off-by: Tim Moon <[email protected]> * Fix bug with DDP initialization Signed-off-by: Tim Moon <[email protected]> * Make distopt dependent on megatron_amp_o2 Signed-off-by: Tim Moon <[email protected]> * Fix code formatting Signed-off-by: Tim Moon <[email protected]> * Handle dist Adam in optimizer unit tests Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Tim Moon <[email protected]> Co-authored-by: Eric Harper <[email protected]> * update readme Signed-off-by: ericharper <[email protected]> * update readme Signed-off-by: ericharper <[email protected]> * latent model support * 1. Debugging. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. * update branch Signed-off-by: ericharper <[email protected]> * fix replace_bos_with_pad not found (#6443) Signed-off-by: Abhinav Khattar <[email protected]> * Support Swiglu in TP PP Conversion (#6437) * Support Swiglu in TP PP Conversion Signed-off-by: smajumdar <[email protected]> * Guard activation Signed-off-by: smajumdar <[email protected]> * Guard activation Signed-off-by: smajumdar <[email protected]> --------- Signed-off-by: smajumdar <[email protected]> * BERT pre-training mp fork to spawn (#6442) * change bert fork to spawn Signed-off-by: Abhinav Khattar <[email protected]> * num_workers=0 fix Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> * Meagtron encoder decoder fix for empty validation outputs (#6459) * 1. Meagtron encoder decoder fix for empty validation outputs. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. --------- Signed-off-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> * Added/updated new Conformer configs (#6426) * updated conf files. Signed-off-by: Vahid <[email protected]> * added confs. Signed-off-by: Vahid <[email protected]> * moved longconformer confs. Signed-off-by: Vahid <[email protected]> * updated readme. Signed-off-by: Vahid <[email protected]> * updated readme. Signed-off-by: Vahid <[email protected]> * updated batch sizes and added fastconformer ctc streaming configs. Signed-off-by: Vahid <[email protected]> * updated batch sizes. Signed-off-by: Vahid <[email protected]> * added hybrid support. Signed-off-by: Vahid <[email protected]> * added hybrid support. Signed-off-by: Vahid <[email protected]> --------- Signed-off-by: Vahid <[email protected]> * reduce workers on NMT CI (#6472) Signed-off-by: Abhinav Khattar <[email protected]> * move to nvidia megatron repo (#6465) Signed-off-by: Abhinav Khattar <[email protected]> * Megatron KERPLE positional embeddings (#6478) * [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- Signed-off-by: hsiehjackson <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. Signed-off-by: Xuesong Yang <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Xuesong Yang <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] FastPitch speaker encoder (#6417) * Add initial codes Signed-off-by: hsiehjackson <[email protected]> * Remove wemb Signed-off-by: hsiehjackson <[email protected]> * Fix import Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss Signed-off-by: hsiehjackson <[email protected]> * Add ConditionalInput Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test Signed-off-by: hsiehjackson <[email protected]> * Add initial codes Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix import error Signed-off-by: hsiehjackson <[email protected]> * Add initial codes Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix dataset error Signed-off-by: hsiehjackson <[email protected]> * Remove reference speaker embedding Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Remove SV encoder Signed-off-by: hsiehjackson <[email protected]> * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix length type Signed-off-by: hsiehjackson <[email protected]> * Fix append Signed-off-by: hsiehjackson <[email protected]> * Move error msg Signed-off-by: hsiehjackson <[email protected]> * Add look-up into speaker encoder Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Add valueerror msg Signed-off-by: hsiehjackson <[email protected]> * Move lookup Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Remove unused Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <[email protected]> * Fix error Signed-off-by: hsiehjackson <[email protected]> * Rebase and Fix error Signed-off-by: hsiehjackson <[email protected]> * Fix spk encoder Signed-off-by: hsiehjackson <[email protected]> * Rename n_speakers Signed-off-by: hsiehjackson <[email protected]> * Follow comments Signed-off-by: hsiehjackson <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error Signed-off-by: hsiehjackson <[email protected]> --------- Signed-off-by: hsiehjackson <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Sharded manifests for tarred datasets (#6395) * testing sharded manifests Signed-off-by: Dima Rekesh <[email protected]> * compatibility Signed-off-by: Dima Rekesh <[email protected]> * proper fixes Signed-off-by: Dima Rekesh <[email protected]> * adding flag tot convert_to_tarred_audio_dataset Signed-off-by: Dima Rekesh <[email protected]> * shard_manifests conf param Signed-off-by: Dima Rekesh <[email protected]> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <[email protected]> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <[email protected]> * distributed checks Signed-off-by: Dima Rekesh <[email protected]> * typo Signed-off-by: Dima Rekesh <[email protected]> * typo Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * fixes Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py Signed-off-by: Dima Rekesh <[email protected]> * reversing manifest shards flag Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests Signed-off-by: Dima Rekesh <[email protected]> * excluding manifests from webdataset url expansion Signed-off-by: Dima Rekesh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore Signed-off-by: Dima Rekesh <[email protected]> * explicit use of UTF-8 for manifest i/o Signed-off-by: Dima Rekesh <[email protected]> --------- Signed-off-by: Dima Rekesh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) Signed-off-by: Jim O’Regan <[email protected]> * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. Signed-off-by: Mostafa Ghorbandoost <[email protected]> * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. Signed-off-by: Mostafa Ghorbandoost <[email protected]> --------- Signed-off-by: Mostafa Ghorbandoost <[email protected]> * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support Signed-off-by: Dmytro Pykhtar <[email protected]> * added tests for rampup batch size Signed-off-by: Dmytro Pykhtar <[email protected]> * fixed the typos Signed-off-by: Dmytro Pykhtar <[email protected]> * added assertions Signed-off-by: Dmytro Pykhtar <[email protected]> * changed assertion rules Signed-off-by: Dmytro Pykhtar <[email protected]> * deleted unused imports Signed-off-by: Dmytro Pykhtar <[email protected]> * changed tests for rampup batch size Signed-off-by: Dmytro Pykhtar <[email protected]> * updated rampup batch size tests Signed-off-by: Dmytro Pykhtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling Signed-off-by: Dmytro Pykhtar <[email protected]> * rampup batch size tests changes Signed-off-by: Dmytro Pykhtar <[email protected]> --------- Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- Signed-off-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Micha Livne <[email protected]> * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default Signed-off-by: Kunal Dhawan <[email protected]> * updated README with the new agg_tokenizer_manifest flag Signed-off-by: Kunal Dhawan <[email protected]> * fixed typo in scripts/speech_recognition/code_switching/README.md Signed-off-by: Kunal Dhawan <[email protected]> * changed agg_tokenizer_manifest to is_lid_manifest Signed-off-by: Kunal Dhawan <[email protected]> --------- Signed-off-by: Kunal Dhawan <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script Signed-off-by: andrusenkoau <[email protected]> * add return encoding embedding option Signed-off-by: andrusenkoau <[email protected]> * update script Signed-off-by: andrusenkoau <[email protected]> * add rnnt and hat ngram decoding script Signed-off-by: andrusenkoau <[email protected]> * add some parameters Signed-off-by: andrusenkoau <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig Signed-off-by: andrusenkoau <[email protected]> * replace return_encoder_embeddings parameter Signed-off-by: andrusenkoau <[email protected]> * generalization of scipt behavior Signed-off-by: andrusenkoau <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <[email protected]> * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <[email protected]> * add manual encoder_embeddings calculation Signed-off-by: andrusenkoau <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 Signed-off-by: Andrei Andrusenko <[email protected]> * fix rescoring description Signed-off-by: Andrei Andrusenko <[email protected]> --------- Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * fix replace_bos_with_pad not found (#6443) (#6450) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * reduce workers on NMT CI (#6472) (#6474) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * 1. Added KERPLE positional embeddings to encoder-decoder. Signed-off-by: Micha Livne <[email protected]> * 1. Added a missing file. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: Jim O’Regan <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Kunal Dhawan <[email protected]> Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Cheng-Ping Hsieh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> Co-authored-by: Jim O’Regan <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: Mostafa Ghorbandoost <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Micha Livne <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * 1. Added external index sample. (#6462) Signed-off-by: Micha Livne <[email protected]> * Fix cache aware hybrid bugs (#6466) * Update README to add core installation (#6488) * update README for megatron-core Signed-off-by: Abhinav Khattar <[email protected]> * fix Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> * Fix typos (#6494) Signed-off-by: smajumdar <[email protected]> * fix broken links r1.18.0 (#6501) * fix broken links Signed-off-by: Evelina <[email protected]> * fix broken links Signed-off-by: Evelina <[email protected]> --------- Signed-off-by: Evelina <[email protected]> * 1. Fixed gaussian hidden transform. Signed-off-by: Micha Livne <[email protected]> * 1. Finished updating hidden loss for MIM. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix custom forward_torch_softmax (#6512) Signed-off-by: Abhinav Khattar <[email protected]> * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamp… (#6500) * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator r1.18.0 Signed-off-by: Taejin Park <[email protected]> * ignore keep_logits in FrameBatchASRLogits Signed-off-by: Taejin Park <[email protected]> --------- Signed-off-by: Taejin Park <[email protected]> * [TTS] fixed broken path. (#6514) Signed-off-by: Xuesong Yang <[email protected]> * 1. Added a hiddens module. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix typos (#6523) (#6539) * Fix typos Signed-off-by: smajumdar <[email protected]> * Fix typos Signed-off-by: smajumdar <[email protected]> --------- Signed-off-by: smajumdar <[email protected]> (cherry picked from commit 7fa47bfcd359f1ea80ad056d8bb2d78f97c6206a) * added back the fast emit section to the configs. (#6540) * added back the fast emit section to the configs. Signed-off-by: Vahid <[email protected]> * added back the fast emit section to the configs. Signed-off-by: Vahid <[email protected]> --------- Signed-off-by: Vahid <[email protected]> * Fix fp16 (#6543) Signed-off-by: MaximumEntropy <[email protected]> * fix (#6529) Signed-off-by: Abhinav Khattar <[email protected]> * pass .scale instead of scaler object to core (#6545) Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Change Megatron Enc Dec model to use persistent_workers (#6548) * persistent workers Signed-off-by: Abhinav Khattar <[email protected]> * fix Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) * Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by Signed-off-by: KunalDhawan <[email protected]> * updated ASR docs with the fastconformer hybrid checkpoints Signed-off-by: KunalDhawan <[email protected]> * added the fastconformer RNNT and CTC models Signed-off-by: KunalDhawan <[email protected]> --------- Signed-off-by: KunalDhawan <[email protected]> * Add scores for FastConformer models (#6557) Signed-off-by: smajumdar <[email protected]> * Patch transcribe and support offline transcribe for hybrid model (#6550) Signed-off-by: fayejf <[email protected]> * Not doing CastToFloat by default (#6524) * Not doing CastToFloat by default Signed-off-by: Boris Fomitchev <[email protected]> * Added docustring Signed-off-by: Boris Fomitchev <[email protected]> * Dummy commit Signed-off-by: Boris Fomitchev <[email protected]> --------- Signed-off-by: Boris Fomitchev <[email protected]> * temp rtd fix (#6568) Signed-off-by: Abhinav Khattar <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py for speedup (#6565) * Update manifest.py Re-order the checks for faster processing audio filepaths that are already absolute paths Signed-off-by: He Huang (Steve) <[email protected]> * Update manifest.py Signed-off-by: He Huang (Steve) <[email protected]> --------- Signed-off-by: He Huang (Steve) <[email protected]> Co-authored-by: Vahid Noroozi <[email protected]> * Turn autocast off when precision is fp32 (#6554) * Turn autocast off when precision is fp32 Signed-off-by: Abhinav Khattar <[email protected]> * address review Signed-off-by: Abhinav Khattar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: Abhinav Khattar <[email protected]> * merge Signed-off-by: Abhinav Khattar <[email protected]> --------- Signed-off-by: Abhinav Khattar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> * More streaming conformer export fixes (#6567) Signed-off-by: Greg Clark <[email protected]> Co-authored-by: Vahid Noroozi <[email protected]> * Fix batch size reconf for T5 FT for multi-validation (#6582) Signed-off-by: Abhinav Khattar <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Updated Megatron LM encoder/decoder to use cfg for hiddens. Signed-off-by: Micha Livne <[email protected]> * 1. Added support to register externalhidden loss / transforms. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make tensor split contiguous (#6580) Signed-off-by: Abhinav Khattar <[email protected]> * Patches from main to r1.18.0 for Virtual Parallel (#6592) * Add interleaved pp support (#6498) * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <[email protected]> * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <[email protected]> * Switch to megatron core Signed-off-by: smajumdar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit fa9ab5b41301c9d92c97bf37894f9c94de6b91b5) * Add patches for Virtual Parallel conversion (#6589) * Add patches for Virtual Parllel conversion Signed-off-by: smajumdar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 4d639ef36b10d8536db02bc47ea30f4acbc061bb) * Documentation for ASR-TTS models (#6594) * Add docs about hybrid ASR-TTS models Signed-off-by: Vladimir Bataev <[email protected]> * Add docs about text-only datasets Signed-off-by: Vladimir Bataev <[email protected]> * Add docs about ASR-TTS checkpoints Signed-off-by: Vladimir Bataev <[email protected]> * Add docs about ASR-TTS configs and training Signed-off-by: Vladimir Bataev <[email protected]> * Clean up Signed-off-by: Vladimir Bataev <[email protected]> * ASR-TTS docs: add to api, fix imports Signed-off-by: Vladimir Bataev <[email protected]> * Clean up Signed-off-by: Vladimir Bataev <[email protected]> * Wrap optional import Signed-off-by: Vladimir Bataev <[email protected]> * Revert general ASR import Signed-off-by: Vladimir Bataev <[email protected]> --------- Signed-off-by: Vladimir Bataev <[email protected]> * Update SDP docs (#6485) * add info about SDP e.g. processor classes in docs Signed-off-by: Elena Rastorgueva <[email protected]> * add link to SDP docs in README Signed-off-by: Elena Rastorgueva <[email protected]> * address code review comments and add SDP overview diagram Signed-off-by: Elena Rastorgueva <[email protected]> * Fix spelling typo Signed-off-by: Elena Rastorgueva <[email protected]> --------- Signed-off-by: Elena Rastorgueva <[email protected]> * Create dummy iters to satisy len checks (#6600) Signed-off-by: Abhinav Khattar <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * Restore GPT support for interleaved pipeline parallelism (#6528) * Restore logic for data-parallel communication with pipeline parallelism in GPT Signed-off-by: Tim Moon <[email protected]> * Support dynamic attention masks in GPT Signed-off-by: Tim Moon <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Debug typos Signed-off-by: Tim Moon <[email protected]> * Debug data iterator caching with interleaved pipeline parallelism Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples. Signed-off-by: Tim Moon <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Megatron-LM commit Signed-off-by: Tim Moon <[email protected]> * Distinguish between list of data iterators and data iterator that is a list Signed-off-by: Tim Moon <[email protected]> * Create dummy iters to satisy len checks Signed-off-by: Abhinav Khattar <[email protected]> * Kludge while waiting for Megatron-LM update Signed-off-by: Tim Moon <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set transformers offline to avoid rate limiting Signed-off-by: ericharper <[email protected]> --------- Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Eric Harper <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Signed-off-by: ericharper <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Abhinav Khattar <[email protected]> * Patch transcribe_util for steaming mode and add wer calculation back to inference scripts (#6601) * fix write Signed-off-by: fayejf <[email protected]> * decoding ctc Signed-off-by: fayejf <[email protected]> * temp set rnnt decoding return_best_hypothesis to true Signed-off-by: fayejf <[email protected]> * add wer cal back to transcribe_speech as requested Signed-off-by: fayejf <[email protected]> * add wer cal back to speech_to_text_buffered_infer_rnnt as requested Signed-off-by: fayejf <[email protected]> * add wer cal back to speech_to_text_buffered_infer_ctc as requested Signed-off-by: fayejf <[email protected]> * style fix Signed-off-by: fayejf <[email protected]> * reflect change in asr_evaluator Signed-off-by: fayejf <[email protected]> * reflect som and vahid comment Signed-off-by: fayejf <[email protected]> * remove return_best_hy=true in transcribe_speech Signed-off-by: fayejf <[email protected]> * no text skip Signed-off-by: fayejf <[email protected]> --------- Signed-off-by: fayejf <[email protected]> * 1. Added example conf YAML. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support in tensor_parallel. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add hat image to docs (#6619) Signed-off-by: andrusenkoau <[email protected]> * update core commit hash in readme (#6622) Signed-off-by: Abhinav Khattar <[email protected]> * Patch decoding for PC models (#6630) * Patch decoding logic for PC models Signed-off-by: smajumdar <[email protected]> * Patch decoding logic for PC models Signed-off-by: smajumdar <[email protected]> --------- Signed-off-by: smajumdar <[email protected]> * Fix wer.py where 'errors' variable was not set (#6633) Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings Signed-off-by: He Huang (Steve) <[email protected]> * fix att_context_size bug for older models. (#6635) Signed-off-by: Vahid <[email protected]> * Add megatron_core to requirements (#6639) * add megatron_core to requirements Signed-off-by: ericharper <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ericharper <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <[email protected]> * remove from jenkins Signed-off-by: ericharper <[email protected]> --------- Signed-off-by: ericharper <[email protected]> * remove dup (#6643) Signed-off-by: ericharper <[email protected]> * 1. Fixed config to use names, and added better error messages. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support to pass extra data to hiddens for loss computation. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Working on passing extra data to hiddnes. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Fixed support in loading .nemo without hiddnes module. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Improved and fixed logging of validation and testing. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed training logging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed logging of hidden loss. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Fixed logging names. 2. Added logging to hiddens and tokens loss. Signed-off-by: Micha Livne <[email protected]> * 1. Fixed conflicts. Signed-off-by: Micha Livne <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> * 1. Debugging. Signed-off-by: Micha Livne <[email protected]> --------- Signed-off-by: Xuesong Yang <[email protected]> Signed-off-by: Oleksii Kuchaiev <[email protected]> Signed-off-by: Jocelyn Huang <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: fayejf <[email protected]> Signed-off-by: Alexandra Antonova <[email protected]> Signed-off-by: Virginia Adams <[email protected]> Signed-off-by: Vladimir Bataev <[email protected]> Signed-off-by: ericharper <[email protected]> Signed-off-by: Vahid <[email protected]> Signed-off-by: Ante Jukić <[email protected]> Signed-off-by: David Mosallanezhad <[email protected]> Signed-off-by: sam1373 <[email protected]> Signed-off-by: MaximumEntropy <[email protected]> Signed-off-by: ekmb <[email protected]> Signed-off-by: Yang Zhang <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Abhinav Khattar <[email protected]> Signed-off-by: smajumdar <[email protected]> Signed-off-by: Micha Livne <[email protected]> Signed-off-by: hsiehjackson <[email protected]> Signed-off-by: Dima Rekesh <[email protected]> Signed-off-by: Jim O’Regan <[email protected]> Signed-off-by: Mostafa Ghorbandoost <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: Kunal Dhawan <[email protected]> Signed-off-by: andrusenkoau <[email protected]> Signed-off-by: Andrei Andrusenko <[email protected]> Signed-off-by: Evelina <[email protected]> Signed-off-by: Taejin Park <[email protected]> Signed-off-by: KunalDhawan <[email protected]> Signed-off-by: Boris Fomitchev <[email protected]> Signed-off-by: He Huang (Steve) <[email protected]> Signed-off-by: Greg Clark <[email protected]> Signed-off-by: Elena Rastorgueva <[email protected]> Signed-off-by: Eric Harper <[email protected]> Co-authored-by: Xuesong Yang <[email protected]> Co-authored-by: Oleksii Kuchaiev <[email protected]> Co-authored-by: Jocelyn <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: fayejf <[email protected]> Co-authored-by: bene-ges <[email protected]> Co-authored-by: Alexandra Antonova <[email protected]> Co-authored-by: Virginia Adams <[email protected]> Co-authored-by: Zhilin Wang <[email protected]> Co-authored-by: Vladimir Bataev <[email protected]> Co-authored-by: Nithin Rao <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Vahid Noroozi <[email protected]> Co-authored-by: anteju <[email protected]> Co-authored-by: Ante Jukić <[email protected]> Co-authored-by: David <[email protected]> Co-authored-by: David Mosallanezhad <[email protected]> Co-authored-by: Samuel Kriman <[email protected]> Co-authored-by: Sandeep Subramanian <[email protected]> Co-authored-by: Evelina <[email protected]> Co-authored-by: Sean Naren <[email protected]> Co-authored-by: Yang Zhang <[email protected]> Co-authored-by: Sean Naren <[email protected]> Co-authored-by: Tim Moon <[email protected]> Co-authored-by: Neha Tadimeti <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abhinav Khattar <[email protected]> Co-authored-by: Cheng-Ping Hsieh <[email protected]> Co-authored-by: Dima Rekesh <[email protected]> Co-authored-by: Jim O’Regan <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Mostafa Ghorbandoost <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]> Co-authored-by: Kunal Dhawan <[email protected]> Co-authored-by: Andrei Andrusenko <[email protected]> Co-authored-by: Taejin Park <[email protected]> Co-authored-by: Boris Fomitchev <[email protected]> Co-authored-by: He Huang (Steve) <[email protected]> Co-authored-by: Greg Clark <[email protected]> Co-authored-by: Elena Rastorgueva <[email protected]>

added rampup batch size support

158d10b

Signed-off-by: Dmytro Pykhtar <[email protected]>

dimapihtar requested review from ericharper and okuchaiev April 13, 2023 15:36

github-actions bot added the NLP label Apr 13, 2023

added tests for rampup batch size

c4f3f6e

Signed-off-by: Dmytro Pykhtar <[email protected]>

ericharper reviewed Apr 14, 2023

View reviewed changes

dimapihtar and others added 4 commits April 15, 2023 13:53

fixed the typos

67b7cfc

Signed-off-by: Dmytro Pykhtar <[email protected]>

added assertions

b6a306f

Signed-off-by: Dmytro Pykhtar <[email protected]>

changed assertion rules

81e23e6

Signed-off-by: Dmytro Pykhtar <[email protected]>

Merge branch 'main' into batch_size_rampup

e0681e5

Signed-off-by: Dmytro Pykhtar <[email protected]>

github-advanced-security bot found potential problems Apr 18, 2023

View reviewed changes

tests/collections/nlp/test_rampup_batch_size.py Fixed Show fixed Hide fixed

tests/collections/nlp/test_rampup_batch_size.py Fixed Show fixed Hide fixed

dimapihtar added 2 commits April 18, 2023 05:44

deleted unused imports

1472083

Signed-off-by: Dmytro Pykhtar <[email protected]>

changed tests for rampup batch size

e8b05de

Signed-off-by: Dmytro Pykhtar <[email protected]>

dimapihtar requested a review from ericharper April 18, 2023 14:59

dimapihtar and others added 5 commits April 18, 2023 10:41

updated rampup batch size tests

9b8237f

Signed-off-by: Dmytro Pykhtar <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

94a5389

for more information, see https://pre-commit.ci

fixed styling

3fffcd1

Signed-off-by: Dmytro Pykhtar <[email protected]>

Merge branch 'batch_size_rampup' of https://github.com/NVIDIA/NeMo in…

a80ab50

…to batch_size_rampup

Merge branch 'main' into batch_size_rampup

998b179

ericharper previously approved these changes Apr 18, 2023

View reviewed changes

github-advanced-security bot found potential problems Apr 18, 2023

View reviewed changes

dimapihtar added 2 commits April 18, 2023 17:01

rampup batch size tests changes

9bd543a

Signed-off-by: Dmytro Pykhtar <[email protected]>

Merge branch 'batch_size_rampup' of https://github.com/NVIDIA/NeMo in…

0cc62f7

…to batch_size_rampup

dimapihtar dismissed ericharper’s stale review via 0cc62f7 April 19, 2023 00:03

dimapihtar added 2 commits April 19, 2023 03:04

Merge branch 'main' into batch_size_rampup

6b6de4c

Merge branch 'main' into batch_size_rampup

f3a0215

dimapihtar requested a review from ericharper April 19, 2023 15:47

Merge branch 'main' into batch_size_rampup

d0b57e0

ericharper approved these changes Apr 20, 2023

View reviewed changes

dimapihtar merged commit 9e72326 into main Apr 20, 2023

dimapihtar deleted the batch_size_rampup branch April 20, 2023 17:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add rampup batch size support for Megatron GPT #6424

add rampup batch size support for Megatron GPT #6424

dimapihtar commented Apr 13, 2023 •

edited

Loading

ericharper commented Apr 13, 2023

ericharper Apr 14, 2023

ericharper left a comment

ericharper left a comment

add rampup batch size support for Megatron GPT #6424

add rampup batch size support for Megatron GPT #6424

Conversation

dimapihtar commented Apr 13, 2023 • edited Loading

What does this PR do ?

Changelog

Usage

Before your PR is "Ready for review"

Who can review?

Additional Information

ericharper commented Apr 13, 2023

ericharper Apr 14, 2023

Choose a reason for hiding this comment

ericharper left a comment

Choose a reason for hiding this comment

ericharper left a comment

Choose a reason for hiding this comment

dimapihtar commented Apr 13, 2023 •

edited

Loading