Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator #6499

Merged
merged 5 commits into from
Apr 28, 2023

Conversation

tango4j
Copy link
Collaborator

@tango4j tango4j commented Apr 26, 2023

What does this PR do ?

The argument of _get_batch_preds in FrameBatchASRLogits class has not been updated when FrameBatchASR is updated.
Now, this PR force _get_batch_preds() to keep logits in decoder timestamps generators.

Collection: [Note which collection this PR will affect]
ASR

Changelog

  • Added force_keep_logits for _get_batch_preds function in class FrameBatchASRLogits in decoder_timestamp_utils.py
  • Changed class name FrameBatchASR_Logits to FrameBatchASRLogits

Usage

  • You can potentially add a usage example below
python offline_diar_with_asr_infer.py \
    diarizer.manifest_filepath=<path to manifest file> \
    diarizer.out_dir='demo_asr_output' \
    diarizer.speaker_embeddings.model_path=<pretrained modelname or path to .nemo> \
    diarizer.asr.model_path=<pretrained modelname or path to .nemo> \
    diarizer.asr.parameters.asr_based_vad=True \
    diarizer.speaker_embeddings.parameters.save_embeddings=False

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

ASR contributors

Additional Information

This is for main, and there will be another PR for r1.18

@github-actions github-actions bot added the ASR label Apr 26, 2023
@tango4j tango4j changed the title [BugFix] _get_batch_preds() is forced to keep logits in decoder time… [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator Apr 26, 2023
@tango4j tango4j marked this pull request as ready for review April 27, 2023 01:42
Copy link
Collaborator

@jbalam-nv jbalam-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tango4j tango4j merged commit bdfb950 into main Apr 28, 2023
8 checks passed
@tango4j tango4j deleted the fix/get_batch_preds branch April 28, 2023 16:39
hsiehjackson pushed a commit to hsiehjackson/NeMo that referenced this pull request Jun 2, 2023
…s generator (NVIDIA#6499)

* [BugFix] _get_batch_preds() is forced to keep logits in  decoder timestamps generators

Signed-off-by: Taejin Park <[email protected]>

* Ingnore keep_logits boolean in FrameASRBatchLogits

Signed-off-by: Taejin Park <[email protected]>

---------

Signed-off-by: Taejin Park <[email protected]>
Co-authored-by: Jagadeesh Balam <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>
ekmb added a commit that referenced this pull request Jun 12, 2023
…d Flash Attention (#6666)

* move to nvidia megatron repo (#6465) (#6475)

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Megatron KERPLE positional embeddings (#6478) (#6480)

* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

---------




* [TTS] whitelist broken path fix. (#6412)

* [TTS] whitelist broken path fix.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------




* [TTS] FastPitch speaker encoder (#6417)

* Add initial codes



* Remove wemb



* Fix import



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss



* Add ConditionalInput



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix import error



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix dataset error



* Remove reference speaker embedding



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove SV encoder



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix length type



* Fix append



* Move error msg



* Add look-up into speaker encoder



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Add valueerror msg



* Move lookup



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove unused



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix error



* Rebase and Fix error



* Fix spk encoder



* Rename n_speakers



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error



---------




* Sharded manifests for tarred datasets (#6395)

* testing sharded manifests



* compatibility



* proper fixes



* adding flag tot convert_to_tarred_audio_dataset



* shard_manifests conf param



* propagating the shard_manifests param



* propagating the shard_manifests param



* distributed checks



* typo



* typo



* fixes



* fixes



* fixes



* fixes



* fixes



* fixes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py



* reversing manifest shards flag



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests



* excluding manifests from webdataset url expansion



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore



* explicit use of UTF-8 for manifest i/o



---------




* Update wfst_text_normalization.rst (#6374)

Add Hungarian (incoming in NeMo-text-processing)



* Support Swiglu in TP PP Conversion (#6437) (#6451)

* Support Swiglu in TP PP Conversion



* Guard activation



* Guard activation



---------




* Update NeMo_TTS_Primer.ipynb (#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.



* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.



---------



* add rampup batch size support for Megatron GPT (#6424)

* added rampup batch size support



* added tests for rampup batch size



* fixed the typos



* added assertions



* changed assertion rules



* deleted unused imports



* changed tests for rampup batch size



* updated rampup batch size tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling



* rampup batch size tests changes



---------







* Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.



* 1. Debugging.

---------





* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default



* updated README with the new agg_tokenizer_manifest flag



* fixed typo in scripts/speech_recognition/code_switching/README.md



* changed agg_tokenizer_manifest to is_lid_manifest



---------




* Added/updated new Conformer configs (#6426) (#6467)

* Update script for ngram rnnt and hat beam search decoding (#6370)

* add rnnt ngram beamsearch script



* add return encoding embedding option



* update script



* add rnnt and hat ngram decoding script



* add some parameters



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig



* replace return_encoder_embeddings parameter



* generalization of scipt behavior



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter



* remove return_encoder_embeddings parameter



* add manual encoder_embeddings calculation



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8



* fix rescoring description



---------






* BERT pre-training mp fork to spawn (#6442) (#6454)

* change bert fork to spawn



* num_workers=0 fix



---------




* fix replace_bos_with_pad not found (#6443) (#6450)




* reduce workers on NMT CI (#6472) (#6474)




* 1. Added KERPLE positional embeddings to encoder-decoder.



* 1. Added a missing file.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.



* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <[email protected]>
Signed-off-by: Xuesong Yang <[email protected]>
Signed-off-by: Dima Rekesh <[email protected]>
Signed-off-by: Jim O’Regan <[email protected]>
Signed-off-by: smajumdar <[email protected]>
Signed-off-by: Mostafa Ghorbandoost <[email protected]>
Signed-off-by: Dmytro Pykhtar <[email protected]>
Signed-off-by: Dmytro Pykhtar <[email protected]>
Signed-off-by: Micha Livne <[email protected]>
Signed-off-by: Kunal Dhawan <[email protected]>
Signed-off-by: andrusenkoau <[email protected]>
Signed-off-by: Andrei Andrusenko <[email protected]>
Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Micha Livne <[email protected]>
Co-authored-by: Cheng-Ping Hsieh <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <[email protected]>
Co-authored-by: Dima Rekesh <[email protected]>
Co-authored-by: Jim O’Regan <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <[email protected]>
Co-authored-by: Mostafa Ghorbandoost <[email protected]>
Co-authored-by: Dmytro Pykhtar <[email protected]>
Co-authored-by: Dmytro Pykhtar <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Co-authored-by: Micha Livne <[email protected]>
Co-authored-by: Kunal Dhawan <[email protected]>
Co-authored-by: Andrei Andrusenko <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Fix an invalid link in get_data.py of ljspeech (#6456)

Usage of the link in line 63 leads to downloading a html file not a tsv file, so we need to change it to a raw link.

Signed-off-by: Mostafa Ghorbandoost <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* 1. Added external index sample. (#6462) (#6483)

Signed-off-by: Micha Livne <[email protected]>
Co-authored-by: Micha Livne <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Update README to add core installation (#6488) (#6489)

* update README for megatron-core



* fix



---------

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Fix cache aware hybrid bugs (#6466) (#6484)

Signed-off-by: hsiehjackson <[email protected]>

* Fix typos (#6494) (#6495)

Signed-off-by: smajumdar <[email protected]>
Co-authored-by: Somshubra Majumdar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Add disclaimer about dataset for ASR (#6496)

Signed-off-by: smajumdar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* fix (#6502)

datastore_path_to_webdataset_url(p) if is_datastore_path(p) and is_tarred_path(p) else p
NameError: name 'is_tarred_path' is not defined

Co-authored-by: George <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* fix broken links r1.18.0 (#6501) (#6504)

* fix broken links



* fix broken links



---------

Signed-off-by: Evelina <[email protected]>
Co-authored-by: Evelina <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* [TTS] Create functions for TTS preprocessing without dataloader (#6317)

* [TTS] Create functions for TTS preprocessing without dataloader

Signed-off-by: Ryan <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Cache aware streaming nfa (#6209)

* add cache aware streaming to nemo aligner

Signed-off-by: Slyne Deng <[email protected]>

Signed-off-by: hsiehjackson <[email protected]>

* [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator (#6499)

* [BugFix] _get_batch_preds() is forced to keep logits in  decoder timestamps generators

Signed-off-by: Taejin Park <[email protected]>

* Ingnore keep_logits boolean in FrameASRBatchLogits

Signed-off-by: Taejin Park <[email protected]>

---------

Signed-off-by: Taejin Park <[email protected]>
Co-authored-by: Jagadeesh Balam <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* [TTS] Fix FastPitch energy code (#6511)

Signed-off-by: Ryan <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* fix custom forward_torch_softmax (#6512) (#6517)

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* [TTS] fixed broken path. (#6514) (#6518)

Signed-off-by: Xuesong Yang <[email protected]>
Co-authored-by: Xuesong Yang <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Fix normalization of impulse response in ImpulsePerturbation (#6505)

Signed-off-by: Ante Jukić <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Add interleaved pp support (#6498)

* Add support for Virtual Pipeline Parallel conversion

Signed-off-by: smajumdar <[email protected]>

* Add support for Virtual Pipeline Parallel conversion

Signed-off-by: smajumdar <[email protected]>

* Switch to megatron core

Signed-off-by: smajumdar <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* Fix typos (#6523)

* Fix typos

Signed-off-by: smajumdar <[email protected]>

* Fix typos

Signed-off-by: smajumdar <[email protected]>

---------

Signed-off-by: smajumdar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* New noise_norm perturbation based on Riva work (#6445)

* Initial commit for new noise_norm perturbation

Signed-off-by: Daniel Egert <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Minor fix to random seed in perturb

Signed-off-by: Daniel Egert <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated code to reflect feedback

Signed-off-by: Daniel Egert <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updates for feedback given by code reviewers

Signed-off-by: Daniel Egert <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updates in response to PR feedback

Signed-off-by: Daniel Egert <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added comment about ref_mic being None

Signed-off-by: Daniel Egert <[email protected]>

* Updated perturb to use inspect module

Signed-off-by: Daniel Egert <[email protected]>

---------

Signed-off-by: Daniel Egert <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* [TTS] Add script for computing feature stats (#6508)

* [TTS] Add script for computing feature stats

Signed-off-by: Ryan <[email protected]>

* [TTS] Add overwrite config

Signed-off-by: Ryan <[email protected]>

---------

Signed-off-by: Ryan <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Add Frame-VAD model and datasets (#6441)

* add model, dataset, necessary utils and tests

Signed-off-by: stevehuang52 <[email protected]>

* fix tarred data

Signed-off-by: stevehuang52 <[email protected]>

* fix typo

Signed-off-by: stevehuang52 <[email protected]>

* update docstring

Signed-off-by: stevehuang52 <[email protected]>

* update doc

Signed-off-by: stevehuang52 <[email protected]>

* update doc

Signed-off-by: stevehuang52 <[email protected]>

* update pretrained model info

Signed-off-by: stevehuang52 <[email protected]>

---------

Signed-off-by: stevehuang52 <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Support dynamic length batches with GPT SFT (#6510)

* Support synamic length with GPT SFT

Signed-off-by: Abhinav Khattar <[email protected]>

* make branch functional

Signed-off-by: Abhinav Khattar <[email protected]>

---------

Signed-off-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* added back the fast emit section to the configs. (#6540) (#6542)

* added back the fast emit section to the configs.



* added back the fast emit section to the configs.



---------

Signed-off-by: Vahid <[email protected]>
Co-authored-by: Vahid Noroozi <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* removing unnessary avoid_bfloat16_autocast_context (#6481)

Signed-off-by: Dima Rekesh <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* FC models in menu (#6473)

* FC models in menu

Signed-off-by: Dima Rekesh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Dima Rekesh <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* [TTS] Add tutorials for FastPitch TTS speaker adaptation with adapters (#6431)

* Add tts adapter tutorial

Signed-off-by: hsiehjackson <[email protected]>

* Update main tutorial

Signed-off-by: hsiehjackson <[email protected]>

* Add tts adapter tutorial

Signed-off-by: hsiehjackson <[email protected]>

* Update main tutorial

Signed-off-by: hsiehjackson <[email protected]>

* Update tutorial

Signed-off-by: hsiehjackson <[email protected]>

* Follow comments

Signed-off-by: hsiehjackson <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments

Signed-off-by: hsiehjackson <[email protected]>

* Fix load .nemo error

Signed-off-by: hsiehjackson <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Support multi-speaker fine-tune

Signed-off-by: hsiehjackson <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments

Signed-off-by: hsiehjackson <[email protected]>

* Use .nemo

Signed-off-by: hsiehjackson <[email protected]>

* Follow Comments

Signed-off-by: hsiehjackson <[email protected]>

* Fix bug

Signed-off-by: hsiehjackson <[email protected]>

* Fix bug

Signed-off-by: hsiehjackson <[email protected]>

* Fix bug

Signed-off-by: hsiehjackson <[email protected]>

* Add precomputed speaker emb

Signed-off-by: hsiehjackson <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix space

Signed-off-by: hsiehjackson <[email protected]>

* Remove repeated argument

Signed-off-by: hsiehjackson <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* optional batch size

Signed-off-by: hsiehjackson <[email protected]>

* Fix comments in notebook

Signed-off-by: hsiehjackson <[email protected]>

---------

Signed-off-by: hsiehjackson <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* [TTS] Create initial TTS dataset feature processors (#6507)

Signed-off-by: Ryan <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* fix (#6529) (#6546)

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) (#6553)

* Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by



* updated ASR docs with the fastconformer hybrid checkpoints



* added the fastconformer RNNT and CTC models



---------

Signed-off-by: KunalDhawan <[email protected]>
Co-authored-by: Kunal Dhawan <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Add scores for FastConformer models (#6557) (#6558)

Signed-off-by: smajumdar <[email protected]>
Co-authored-by: Somshubra Majumdar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Fix fp16 (#6543) (#6544)

Signed-off-by: MaximumEntropy <[email protected]>
Co-authored-by: Sandeep Subramanian <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Patch transcribe and support offline transcribe for hybrid model (#6550) (#6559)

Signed-off-by: fayejf <[email protected]>
Co-authored-by: fayejf <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Fix notebook bad json (#6561)

Signed-off-by: smajumdar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Change Megatron Enc Dec model to use persistent_workers (#6548) (#6552)

* persistent workers



* fix



---------

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Make KenLM with PC for AggregateTokenizer and merge it (#6081)

* do_lowercase, rm_punctuation

Signed-off-by: Nikolay Karpov <[email protected]>

* support beam_strategy = beam

Signed-off-by: Nikolay Karpov <[email protected]>

* black

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix config and^Cunctuation capitalization

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rm math

Signed-off-by: Nikolay Karpov <[email protected]>

* update kenlm

Signed-off-by: Nikolay Karpov <[email protected]>

* black

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add opengrm

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* mv install_beamsearch_decoders

Signed-off-by: Nikolay Karpov <[email protected]>

* punctuation_to_preserve

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Only tikenizer opion

Signed-off-by: Nikolay Karpov <[email protected]>

* Black

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* DEFAULT_TOKEN_OFFSET

Signed-off-by: Nikolay Karpov <[email protected]>

* aggregate_tokenizer

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* install kenlm with more than 5gram

Signed-off-by: Nikolay Karpov <[email protected]>

* install_beamsearch_decoders

Signed-off-by: Nikolay Karpov <[email protected]>

* ngram_bin_path kenlm_bin_path

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* black

Signed-off-by: Nikolay Karpov <[email protected]>

* fix greedy PC bug

Signed-off-by: Nikolay Karpov <[email protected]>

* move global params

Signed-off-by: Nikolay Karpov <[email protected]>

* fix description and perplexity

Signed-off-by: Nikolay Karpov <[email protected]>

* fix description

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* NEMO_PATH

Signed-off-by: Nikolay Karpov <[email protected]>

* nemo:23.01

Signed-off-by: Nikolay Karpov <[email protected]>

* License

Signed-off-by: Nikolay Karpov <[email protected]>

* description

Signed-off-by: Nikolay Karpov <[email protected]>

* isinstance

Signed-off-by: Nikolay Karpov <[email protected]>

* refactor kenlm stdin

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* black

Signed-off-by: Nikolay Karpov <[email protected]>

* add cmd arg

Signed-off-by: Nikolay Karpov <[email protected]>

* use new iter_files

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* EncDecHybridRNNTCTCModel

Signed-off-by: Nikolay Karpov <[email protected]>

* punctuation

Signed-off-by: Nikolay Karpov <[email protected]>

* train_kenlm args

Signed-off-by: Nikolay Karpov <[email protected]>

* add docstrings

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add ngram_merge docs

Signed-off-by: Nikolay Karpov <[email protected]>

* ngram_prune

Signed-off-by: Nikolay Karpov <[email protected]>

* rename to ngram_merge

Signed-off-by: Nikolay Karpov <[email protected]>

* rename to ngram

Signed-off-by: Nikolay Karpov <[email protected]>

* add comments

Signed-off-by: Nikolay Karpov <[email protected]>

* Ngram

Signed-off-by: Nikolay Karpov <[email protected]>

* nemo_model_file

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* install_opengrm_ngram

Signed-off-by: Nikolay Karpov <[email protected]>

* install opengrm

Signed-off-by: Nikolay Karpov <[email protected]>

* rename to install_opengrm.sh

Signed-off-by: Nikolay Karpov <[email protected]>

* rm extra import

Signed-off-by: Nikolay Karpov <[email protected]>

* train_paths

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* text_processing

Signed-off-by: Nikolay Karpov <[email protected]>

* fix ngram_bin_path

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* DECODERS_PATH

Signed-off-by: Nikolay Karpov <[email protected]>

* farcompile

Signed-off-by: Nikolay Karpov <[email protected]>

* rm text processing

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* text_processing

Signed-off-by: Nikolay Karpov <[email protected]>

* AggregateTokenizer.DummyTokenizer

Signed-off-by: Nikolay Karpov <[email protected]>

* comments

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* TextProcessingConfig

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* typo

Signed-off-by: Nikolay Karpov <[email protected]>

* doc

Signed-off-by: Nikolay Karpov <[email protected]>

* types

Signed-off-by: Nikolay Karpov <[email protected]>

* nemo_model_file

Signed-off-by: Nikolay Karpov <[email protected]>

* rm assert

Signed-off-by: Nikolay Karpov <[email protected]>

* import kenlm_utils

Signed-off-by: Nikolay Karpov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* return None

Signed-off-by: Nikolay Karpov <[email protected]>

* Copyright

Signed-off-by: Nikolay Karpov <[email protected]>

* 2022

Signed-off-by: Nikolay Karpov <[email protected]>

* 2023

Signed-off-by: Nikolay Karpov <[email protected]>

---------

Signed-off-by: Nikolay Karpov <[email protected]>
Signed-off-by: Nikolay Karpov <[email protected]>
Co-authored-by: Nikolay Karpov <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* fix for running on 1 GPU.

Signed-off-by: hsiehjackson <[email protected]>

* temp rtd fix (#6568) (#6569)

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* [TTS] Add script for mapping speaker names to indices (#6509)

Signed-off-by: Ryan <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* whitespace (#6574)

Signed-off-by: Nikolay Karpov <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Update manifest.py for speedup (#6565) (#6573)

* Update manifest.py

Re-order the checks for faster processing audio filepaths that are already absolute paths



* Update manifest.py



---------

Signed-off-by: He Huang (Steve) <[email protected]>
Co-authored-by: He Huang (Steve) <[email protected]>
Co-authored-by: Vahid Noroozi <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* More streaming conformer export fixes (#6567) (#6578)

Signed-off-by: Greg Clark <[email protected]>
Co-authored-by: Greg Clark <[email protected]>
Co-authored-by: Vahid Noroozi <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* user selected max_seq_len should be less than model's max_seq_len (#6333) (#6386)

* user selection should not break model max limit



* eval max seq length



---------

Signed-off-by: arendu <[email protected]>
Signed-off-by: Adi Renduchintala <[email protected]>
Co-authored-by: Adi Renduchintala <[email protected]>
Co-authored-by: Sandeep Subramanian <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Framework for PEFT via mixins  (#6391)

* init commit ptuning via mixin

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates

Signed-off-by: arendu <[email protected]>

* gpt ptuning places virtual tokens on the left only

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* encoder input modified when pre_process is true

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* optimizer group and state dict updates

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adapter ptuning working for pp>1

Signed-off-by: arendu <[email protected]>

* adapter defaults

Signed-off-by: arendu <[email protected]>

* adapter ptuining config defaults

Signed-off-by: arendu <[email protected]>

* training works

Signed-off-by: arendu <[email protected]>

* loading and saving adapter only params during training

Signed-off-by: arendu <[email protected]>

* added checks and comments

Signed-off-by: arendu <[email protected]>

* clean up

Signed-off-by: arendu <[email protected]>

* checks for grad is None before calling all_reduce

Signed-off-by: arendu <[email protected]>

* load adapter .nemo file working

Signed-off-by: arendu <[email protected]>

* resume training for adapters

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* peft tuning

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor

Signed-off-by: arendu <[email protected]>

* file not needed

Signed-off-by: arendu <[email protected]>

* undo prompt learning dataset changes

Signed-off-by: arendu <[email protected]>

* undo updates to gpt prompt learning model

Signed-off-by: arendu <[email protected]>

* naming updates

Signed-off-by: arendu <[email protected]>

* decoding

Signed-off-by: arendu <[email protected]>

* predict_step in gpt_sft_model

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed inference from tuning config

Signed-off-by: arendu <[email protected]>

* no test in peft training

Signed-off-by: arendu <[email protected]>

* answer only loss and correct defaults for val_loss

Signed-off-by: arendu <[email protected]>

* hybrid adapters and ptuning

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* eval working..

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* prepending tokens for ptuning

Signed-off-by: arendu <[email protected]>

* cleaned up eval config

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean up

Signed-off-by: arendu <[email protected]>

* update

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* default prompt template

Signed-off-by: arendu <[email protected]>

* Lora added

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Support synamic length with GPT SFT

Signed-off-by: Abhinav Khattar <[email protected]>

* make branch functional

Signed-off-by: Abhinav Khattar <[email protected]>

* defaults to max_pad_length=False in GPT SFT dataset

Signed-off-by: arendu <[email protected]>

* adapter parallel_adapters to support Lora

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added early stopping by default

Signed-off-by: arendu <[email protected]>

* eval script for peft and eval config. bug fixes in predict step and added out_features to t5 adapter config

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docs

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* better defaults

Signed-off-by: arendu <[email protected]>

* updates

Signed-off-by: arendu <[email protected]>

* update

Signed-off-by: arendu <[email protected]>

* docs

Signed-off-by: arendu <[email protected]>

---------

Signed-off-by: arendu <[email protected]>
Signed-off-by: Abhinav Khattar <[email protected]>
Signed-off-by: Adi Renduchintala <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* cache and reuse inputs (#6422) (#6452)

Co-authored-by: Sangkug Lym <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Add patches for Virtual Parallel conversion (#6589)

* Add patches for Virtual Parllel conversion

Signed-off-by: smajumdar <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* Pass `.scale` instead of scaler object to core (#6551)

* pass .scale instead of scaler object to core (#6545)

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Eric Harper <[email protected]>

* Update megatron_gpt_model.py

Signed-off-by: Abhinav Khattar <[email protected]>

* scale changes for main

Signed-off-by: Abhinav Khattar <[email protected]>

---------

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Documentation for ASR-TTS models (#6594) (#6595)

* Add docs about hybrid ASR-TTS models



* Add docs about text-only datasets



* Add docs about ASR-TTS checkpoints



* Add docs about ASR-TTS configs and training



* Clean up



* ASR-TTS docs: add to api, fix imports



* Clean up



* Wrap optional import



* Revert general ASR import



---------

Signed-off-by: Vladimir Bataev <[email protected]>
Co-authored-by: Vladimir Bataev <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* [TTS] Fix aligner nan loss in fp32 (#6435)

* Fix nan loss in fp32

Signed-off-by: hsiehjackson <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: hsiehjackson <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* Update SDP docs (#6485) (#6596)

* add info about SDP e.g. processor classes in docs



* add link to SDP docs in README



* address code review comments and add SDP overview diagram



* Fix spelling typo



---------

Signed-off-by: Elena Rastorgueva <[email protected]>
Co-authored-by: Elena Rastorgueva <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Bug/typo fixes (#6599)

Signed-off-by: Igor Gitman <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Manual garbage collection with an interval (#6469) (#6482)

* Manual garbage collection with an interval



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use trainer.global_step for tracking the interval of GC



---------

Signed-off-by: Sangkug Lym <[email protected]>
Co-authored-by: Sangkug Lym <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Make tensor split contiguous (#6580) (#6593)

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* [ASR] Fix for old models in change_attention_model (#6608)

* fixes

Signed-off-by: sam1373 <[email protected]>

* done already

Signed-off-by: sam1373 <[email protected]>

---------

Signed-off-by: sam1373 <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Update manifest.py to use os.path for get_full_path (#6598)

* Update manifest.py to use os.path for get_full_path

Signed-off-by: He Huang (Steve) <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update manifest.py to get rid of pathlib

Signed-off-by: He Huang (Steve) <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update manifest.py

Signed-off-by: He Huang (Steve) <[email protected]>

* Update manifest.py

Signed-off-by: He Huang (Steve) <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: He Huang (Steve) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Vahid Noroozi <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Cherry pick commits in #6601 to main (#6611)

* fix write

Signed-off-by: fayejf <[email protected]>

* decoding ctc

Signed-off-by: fayejf <[email protected]>

* temp set rnnt decoding return_best_hypothesis to true

Signed-off-by: fayejf <[email protected]>

* add wer cal back to transcribe_speech as requested

Signed-off-by: fayejf <[email protected]>

* add wer cal back to speech_to_text_buffered_infer_rnnt  as requested

Signed-off-by: fayejf <[email protected]>

* add wer cal back to speech_to_text_buffered_infer_ctc as requested

Signed-off-by: fayejf <[email protected]>

* style fix

Signed-off-by: fayejf <[email protected]>

* reflect change in asr_evaluator

Signed-off-by: fayejf <[email protected]>

* reflect som and vahid comment

Signed-off-by: fayejf <[email protected]>

* remove return_best_hy=true in transcribe_speech

Signed-off-by: fayejf <[email protected]>

* no text skip

Signed-off-by: fayejf <[email protected]>

* revert partial

Signed-off-by: fayejf <[email protected]>

---------

Signed-off-by: fayejf <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Create dummy iters to satisy len checks (#6600) (#6603)

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* add GPT eval mode fix for interleaved to main (#6610)

Signed-off-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Fix batch size reconf for T5 FT for multi-validation (#6582) (#6588)

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Not doing CastToFloat by default (#6524) (#6563)

* Not doing CastToFloat by default



* Added docustring



* Dummy commit



---------

Signed-off-by: Boris Fomitchev <[email protected]>
Co-authored-by: Boris Fomitchev <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Turn autocast off when precision is fp32 (#6576)

* Turn autocast off when precision is fp32 (#6554)

* Turn autocast off when precision is fp32

Signed-off-by: Abhinav Khattar <[email protected]>

* address review

Signed-off-by: Abhinav Khattar <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes

Signed-off-by: Abhinav Khattar <[email protected]>

* merge

Signed-off-by: Abhinav Khattar <[email protected]>

---------

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <[email protected]>

* correct auto-merge

Signed-off-by: Abhinav Khattar <[email protected]>

* correct auto-merge

Signed-off-by: Abhinav Khattar <[email protected]>

* add to GPT SFT

Signed-off-by: Abhinav Khattar <[email protected]>

---------

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* update core commit hash in readme (#6622) (#6623)

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* add hat image to docs (#6619) (#6621)

Signed-off-by: andrusenkoau <[email protected]>
Co-authored-by: Andrei Andrusenko <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Allow indices exchange via distributed (#6618) (#6624)

Signed-off-by: Mikołaj Błaż <[email protected]>
Co-authored-by: mikolajblaz <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Offline and streaming inference support for hybrid model (#6570)

* streaming buffered for hybrid + ctc

Signed-off-by: fayejf <[email protected]>

* change default model_stride in eval.yaml

Signed-off-by: fayejf <[email protected]>

* add fc model_stride

Signed-off-by: fayejf <[email protected]>

* small fix

Signed-off-by: fayejf <[email protected]>

* check whether model and decoding match

Signed-off-by: fayejf <[email protected]>

* small fix

Signed-off-by: fayejf <[email protected]>

* streaming buffered for hybrid + rnnt

Signed-off-by: fayejf <[email protected]>

* style fix

Signed-off-by: fayejf <[email protected]>

* fix yaml

Signed-off-by: fayejf <[email protected]>

* reflect comment wip

Signed-off-by: fayejf <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix

Signed-off-by: fayejf <[email protected]>

* refactor and verified

Signed-off-by: fayejf <[email protected]>

* add get_full_path to buffered

Signed-off-by: fayejf <[email protected]>

* small fix

Signed-off-by: fayejf <[email protected]>

* add RNNTDecodingConfig

Signed-off-by: fayejf <[email protected]>

* model name & instruction of changing decoding

Signed-off-by: fayejf <[email protected]>

---------

Signed-off-by: fayejf <[email protected]>
Signed-off-by: fayejf <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* Patch decoding for PC models (#6630) (#6631)

* Patch decoding logic for PC models



* Patch decoding logic for PC models



---------

Signed-off-by: smajumdar <[email protected]>
Co-authored-by: Somshubra Majumdar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Fix wer.py where 'errors' variable was not set (#6633) (#6634)

Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings

Signed-off-by: He Huang (Steve) <[email protected]>
Co-authored-by: He Huang (Steve) <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Restore GPT support for interleaved pipeline parallelism (#6528) (#6613)

* Restore logic for data-parallel communication with pipeline parallelism in GPT



* Support dynamic attention masks in GPT



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Debug typos



* Debug data iterator caching with interleaved pipeline parallelism

Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Megatron-LM commit



* Distinguish between list of data iterators and data iterator that is a list



* Create dummy iters to satisy len checks



* Kludge while waiting for Megatron-LM update



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set transformers offline to avoid rate limiting



---------

Signed-off-by: Tim Moon <[email protected]>
Signed-off-by: Eric Harper <[email protected]>
Signed-off-by: Abhinav Khattar <[email protected]>
Signed-off-by: ericharper <[email protected]>
Signed-off-by: Tim Moon <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Add FA

Signed-off-by: hsiehjackson <[email protected]>

* Fix XPOS

Signed-off-by: hsiehjackson <[email protected]>

* Add warning

Signed-off-by: hsiehjackson <[email protected]>

* Fix bugs

Signed-off-by: hsiehjackson <[email protected]>

* Fix attention

Signed-off-by: hsiehjackson <[email protected]>

* Fix comment

Signed-off-by: hsiehjackson <[email protected]>

* Fix cast dtype

Signed-off-by: hsiehjackson <[email protected]>

* Undo xpos

Signed-off-by: hsiehjackson <[email protected]>

* bugfix (#6636)

Signed-off-by: fayejf <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Disable interctc tests (#6638)

Signed-off-by: Igor Gitman <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Add megatron_core to requirements (#6639) (#6640)

* add megatron_core to requirements



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: ericharper <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* Remove from jenkins (#6642)

* Remove from jenkins (#6641)

* add megatron_core to requirements

Signed-off-by: ericharper <[email protected]>

* remove from jenkins

Signed-off-by: ericharper <[email protected]>

---------

Signed-off-by: ericharper <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove dup

Signed-off-by: ericharper <[email protected]>

---------

Signed-off-by: ericharper <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* sft model can use this script for eval (#6637)

* sft model can use this script for eval

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* please fix me

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor

Signed-off-by: arendu <[email protected]>

---------

Signed-off-by: arendu <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* [TTS] Fix TTS audio preprocessing bugs (#6628)

Signed-off-by: Ryan <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Move black parameters to pyproject.toml (#6647)

Signed-off-by: Vladimir Bataev <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* ASR-TTS Models: Support hybrid RNNT-CTC, improve docs. (#6620)

* ASR-TTS: support hybrid RNNT-CTC models
* Do not warn on optional import
* Explain adding options to config
* Fix import guard docs
* Add docs for ConcatDataset
* Add explanation for sampling parameters
* Initial docs for the enhancer model
* Fix use_start_end_token parameter usage

---------

Signed-off-by: Vladimir Bataev <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* fix conversion and eval (#6648)

* fix conversion and eval

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: arendu <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* Confidence ensembles implementation (#6614)

* Working version to train conf model + save ensemble class

Signed-off-by: Igor Gitman <[email protected]>

* Working version

Signed-off-by: Igor Gitman <[email protected]>

* Remove copy of transcribe_speech.py

Signed-off-by: Igor Gitman <[email protected]>

* Move models parameter to config

Signed-off-by: Igor Gitman <[email protected]>

* Add explicit parameters to transcribe

Signed-off-by: Igor Gitman <[email protected]>

* Small cleanups

Signed-off-by: Igor Gitman <[email protected]>

* Add temperature and integration tests

Signed-off-by: Igor Gitman <[email protected]>

* Add more tests

Signed-off-by: Igor Gitman <[email protected]>

* Add pc removal config

Signed-off-by: Igor Gitman <[email protected]>

* Cleanup

Signed-off-by: Igor Gitman <[email protected]>

* Fix typo

Signed-off-by: Igor Gitman <[email protected]>

* Address review comments

Signed-off-by: Igor Gitman <[email protected]>

---------

Signed-off-by: Igor Gitman <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Patch memory used for NeMo Megatron models (#6615)

* Patch memory used for NeMo Megatron models

Signed-off-by: smajumdar <[email protected]>

* Cleanup the dtype of embeddings

Signed-off-by: smajumdar <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor util function for parsing precision

Signed-off-by: smajumdar <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor util function for parsing precision

Signed-off-by: smajumdar <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Try patch for Megatron O2

Signed-off-by: smajumdar <[email protected]>

* Refactor to incorporate megatron amp 02 state

Signed-off-by: smajumdar <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor to incorporate megatron amp 02 state

Signed-off-by: smajumdar <[email protected]>

* Correct indent

Signed-off-by: smajumdar <[email protected]>

* Correct utils import

Signed-off-by: smajumdar <[email protected]>

---------

Signed-off-by: smajumdar <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* handle artifacts when path is dir (#6658)

Signed-off-by: arendu <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* remove upgrading setuptools in reinstall.sh (#6659)

Signed-off-by: Xuesong Yang <[email protected]>
Co-authored-by: fayejf <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* merge lora weights into base model (#6597)

* merge lora weights into base model

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* typo fix

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor update

Signed-off-by: arendu <[email protected]>

* update copyright

Signed-off-by: arendu <[email protected]>

* eval needs to know the PEFT class

Signed-off-by: arendu <[email protected]>

* add target class in training script so that we can use it in eval

Signed-off-by: arendu <[email protected]>

* update

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update to work for tp1

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set restore model path

Signed-off-by: arendu <[email protected]>

* peft can be none

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated merge script so that eval works easily

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* eval with peft or sft model

Signed-off-by: arendu <[email protected]>

* keep sentences in jsonl format

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* convert sft using correct classpath

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated to force sft yaml to have the correct target

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated docs

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix conversion and eval

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: arendu <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <[email protected]>

* upgrade to 23.04 (#6660)

Signed-off-by: ericharper <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Merge r1.18.0 bugfixes and doc updates to main (#6655)

* update branch

Signed-off-by: ericharper <[email protected]>

* Remove from jenkins (#6641)

* add megatron_core to requirements

Signed-off-by: ericharper <[email protected]>

* remove from jenkins

Signed-off-by: ericharper <[email protected]>

---------

Signed-off-by: ericharper <[email protected]>

* remove dup

Signed-off-by: ericharper <[email protected]>

* update branch

Signed-off-by: ericharper <[email protected]>

* [TTS] reformat NeMo versions in the tts logging messages to avoid batch process them when upgrading NeMo versions.

Signed-off-by: Xuesong Yang <[email protected]>

---------

Signed-off-by: ericharper <[email protected]>
Signed-off-by: Xuesong Yang <[email protected]>
Co-authored-by: Xuesong Yang <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Confidence ensembles: fix issues and add tuning functionality (#6657)

* Implement compute confidence to properly handle blanks

Signed-off-by: Igor Gitman <[email protected]>

* Implement proper confidence for transducers

Signed-off-by: Igor Gitman <[email protected]>

* Implement tuning logic

Signed-off-by: Igor Gitman <[email protected]>

* Add tests for confidence tuning

Signed-off-by: Igor Gitman <[email protected]>

* Remove unused imports

Signed-off-by: Igor Gitman <[email protected]>

* Add types/docs

Signed-off-by: Igor Gitman <[email protected]>

* Add comment about the main conf compute loop

Signed-off-by: Igor Gitman <[email protected]>

---------

Signed-off-by: Igor Gitman <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* [TTS] Implement new TextToSpeech dataset (#6575)

* [TTS] Implement new TextToSpeech dataset

Signed-off-by: Ryan <[email protected]>

* [TTS] Add unit tests

Signed-off-by: Ryan <[email protected]>

* [TTS] Fix defaulting of use_log_energy

Signed-off-by: Ryan <[email protected]>

* [TTS] Fix TTS export test

Signed-off-by: Ryan <[email protected]>

---------

Signed-off-by: Ryan <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Dialogue dataset  (#6654)

* chatbot interface

Signed-off-by: Yi Dong <[email protected]>

* latest gradio

Signed-off-by: Yi Dong <[email protected]>

* default greedy

Signed-off-by: Yi Dong <[email protected]>

* better chatbot

Signed-off-by: Yi Dong <[email protected]>

* handle preamble

Signed-off-by: Yi Dong <[email protected]>

* added chatbot training capablity

Signed-off-by: Yi Dong <[email protected]>

* added chatbot ui

Signed-off-by: Yi Dong <[email protected]>

* remove debug code

Signed-off-by: Yi Dong <[email protected]>

* default human

Signed-off-by: Yi Dong <[email protected]>

* use special token for roles

Signed-off-by: Yi Dong <[email protected]>

* special tokens

Signed-off-by: Yi Dong <[email protected]>

* fix name

Signed-off-by: Yi Dong <[email protected]>

* new chat dataset

Signed-off-by: Yi Dong <[email protected]>

* fix the system token

Signed-off-by: Yi Dong <[email protected]>

* upgrade gradio

Signed-off-by: Yi Dong <[email protected]>

* save the chat history

Signed-off-by: Yi Dong <[email protected]>

* update ui

Signed-off-by: root <[email protected]>

* update chat interface

Signed-off-by: Yi Dong <[email protected]>

* handles canonical form

Signed-off-by: Yi Dong <[email protected]>

* new sft chatbot

Signed-off-by: Yi Dong <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change format

Signed-off-by: Yi Dong <[email protected]>

* check extra_id in the tokenizer

Signed-off-by: Yi Dong <[email protected]>

* added vocab property check

Signed-off-by: Yi Dong <[email protected]>

* added missing file

Signed-off-by: Yi Dong <[email protected]>

---------

Signed-off-by: Yi Dong <[email protected]>
Signed-off-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sandeep Subramanian <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Add support for RNNT/hybrid models to partial transcribe (#6609)

* Add support for RNNT/hybrid models to partial transcribe

Signed-off-by: He Huang (Steve) <[email protected]>

* Update transcribe_utils.py

Signed-off-by: He Huang (Steve) <[email protected]>

* Update transcribe_speech.py

Signed-off-by: He Huang (Steve) <[email protected]>

* Update transcr…
KunalDhawan added a commit that referenced this pull request Jul 20, 2023
* peft eval directly from ckpt (#6785)

* update to load from ckpt

Signed-off-by: arendu <[email protected]>

* update

Signed-off-by: arendu <[email protected]>

* load ckpt peft model

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update style

Signed-off-by: arendu <[email protected]>

---------

Signed-off-by: arendu <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add Frame-VAD examples and utils (#6463)

* add model, dataset, necessary utils and tests

Signed-off-by: stevehuang52 <[email protected]>

* fix tarred data

Signed-off-by: stevehuang52 <[email protected]>

* fix typo

Signed-off-by: stevehuang52 <[email protected]>

* add fvad examples and update utils

Signed-off-by: stevehuang52 <[email protected]>

* add copyright

Signed-off-by: stevehuang52 <[email protected]>

* refactor and add tests

Signed-off-by: stevehuang52 <[email protected]>

* update dataset

Signed-off-by: stevehuang52 <[email protected]>

* update test

Signed-off-by: stevehuang52 <[email protected]>

* refactor

Signed-off-by: stevehuang52 <[email protected]>

* refactor

Signed-off-by: stevehuang52 <[email protected]>

* fix typos

Signed-off-by: stevehuang52 <[email protected]>

---------

Signed-off-by: stevehuang52 <[email protected]>
Co-authored-by: fayejf <[email protected]>
Co-authored-by: Taejin Park <[email protected]>

* [TTS][zh] refine hardcoded lowercase for ASCII letters. (#6781)

Signed-off-by: Xuesong Yang <[email protected]>

* Spellchecking ASR customization model (#6179)

* bug fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bugs, add preparation and evaluation scripts, add readme

Signed-off-by: Alexandra Antonova <[email protected]>

* small fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add real coverage calculation, small fixes, more debug information

Signed-off-by: Alexandra Antonova <[email protected]>

* add option to pass a filelist and output folder - to handle inference from multiple input files

Signed-off-by: Alexandra Antonova <[email protected]>

* added preprocessing for yago wikipedia articles - finding yago entities and their subphrases

Signed-off-by: Alexandra Antonova <[email protected]>

* yago wiki preprocessing, sampling, pseudonormalization

Signed-off-by: Alexandra Antonova <[email protected]>

* more scripts for preparation of training examples

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* add some alphabet checks

Signed-off-by: Alexandra Antonova <[email protected]>

* add bert on subwords, concatenate it to bert on characters

Signed-off-by: Alexandra Antonova <[email protected]>

* add calculation of character_pos_to_subword_pos

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fix

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fix

Signed-off-by: Alexandra Antonova <[email protected]>

* pdb

Signed-off-by: Alexandra Antonova <[email protected]>

* tensor join bug fix

Signed-off-by: Alexandra Antonova <[email protected]>

* double hidden_size in classifier

Signed-off-by: Alexandra Antonova <[email protected]>

* pdb

Signed-off-by: Alexandra Antonova <[email protected]>

* default index value 0 instead of -1 because index cannot be negative

Signed-off-by: Alexandra Antonova <[email protected]>

* pad index value 0 instead of -1 because index cannot be negative

Signed-off-by: Alexandra Antonova <[email protected]>

* remove pdb

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bugs, add creation of tarred dataset

Signed-off-by: Alexandra Antonova <[email protected]>

* add possibility to change sequence len at inference

Signed-off-by: Alexandra Antonova <[email protected]>

* change sampling of dummy candidates at inference, add candidate info file

Signed-off-by: Alexandra Antonova <[email protected]>

* fix import

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug

Signed-off-by: Alexandra Antonova <[email protected]>

* update transcription now uses info

Signed-off-by: Alexandra Antonova <[email protected]>

* write path

Signed-off-by: Alexandra Antonova <[email protected]>

* 1. add tarred dataset support(untested). 2. fix bug with ban_ngrams in indexing

Signed-off-by: Alexandra Antonova <[email protected]>

* skip short_sent if no real candidates

Signed-off-by: Alexandra Antonova <[email protected]>

* fix import

Signed-off-by: Alexandra Antonova <[email protected]>

* add braceexpand

Signed-off-by: Alexandra Antonova <[email protected]>

* fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug in np.ones

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug in collate

Signed-off-by: Alexandra Antonova <[email protected]>

* change tensor type to long because of error in torch.gather

Signed-off-by: Alexandra Antonova <[email protected]>

* fix for empty spans tensor

Signed-off-by: Alexandra Antonova <[email protected]>

* same fixes in _collate_fn for tarred dataset

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug from previous commit

Signed-off-by: Alexandra Antonova <[email protected]>

* change int types to be shorter to minimize tar size

Signed-off-by: Alexandra Antonova <[email protected]>

* refactoring of datasets and inference

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fix

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fix

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fix

Signed-off-by: Alexandra Antonova <[email protected]>

* tar by 100k examples, small fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* small fixes, add analytics script

Signed-off-by: Alexandra Antonova <[email protected]>

* Add functions for dynamic programming comparison to get best path by ngrams

Signed-off-by: Alexandra Antonova <[email protected]>

* fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* small fix

Signed-off-by: Alexandra Antonova <[email protected]>

* fixes to support testing on SPGISpeech

Signed-off-by: Alexandra Antonova <[email protected]>

* add preprocessing for userlibri

Signed-off-by: Alexandra Antonova <[email protected]>

* some refactoring

Signed-off-by: Alexandra Antonova <[email protected]>

* some refactoring

Signed-off-by: Alexandra Antonova <[email protected]>

* move some functions to utils to reuse from other project

Signed-off-by: Alexandra Antonova <[email protected]>

* move some functions to utils to reuse from other project

Signed-off-by: Alexandra Antonova <[email protected]>

* move some functions to utils to reuse from other project

Signed-off-by: Alexandra Antonova <[email protected]>

* small refactoring before pr. Add bash-scripts reproducing evaluation

Signed-off-by: Alexandra Antonova <[email protected]>

* style fix

Signed-off-by: Alexandra Antonova <[email protected]>

* small fixes in inference

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fix - didn't move window on last symbol

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug - shuffle was before truncation of sorted candidates

Signed-off-by: Alexandra Antonova <[email protected]>

* refactoring, fix some bugs

Signed-off-by: Alexandra Antonova <[email protected]>

* variour fixes. Add word_indices at inference

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add candidate positions

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Move data preparation and evaluation to other repo

Signed-off-by: Alexandra Antonova <[email protected]>

* add infer_reproduce_paper. Refactoring

Signed-off-by: Alexandra Antonova <[email protected]>

* refactor inference using fragment indices

Signed-off-by: Alexandra Antonova <[email protected]>

* add some helper functions

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug with parameters order

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bugs

Signed-off-by: Alexandra Antonova <[email protected]>

* refactoring, fix bug

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add multiple variants of adjusting start/end positions

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* more fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add unit tests, other fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix

Signed-off-by: Alexandra Antonova <[email protected]>

* fix CodeQl warnings

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bugs, add preparation and evaluation scripts, add readme

Signed-off-by: Alexandra Antonova <[email protected]>

* small fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add real coverage calculation, small fixes, more debug information

Signed-off-by: Alexandra Antonova <[email protected]>

* add option to pass a filelist and output folder - to handle inference from multiple input files

Signed-off-by: Alexandra Antonova <[email protected]>

* added preprocessing for yago wikipedia articles - finding yago entities and their subphrases

Signed-off-by: Alexandra Antonova <[email protected]>

* yago wiki preprocessing, sampling, pseudonormalization

Signed-off-by: Alexandra Antonova <[email protected]>

* more scripts for preparation of training examples

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* add some alphabet checks

Signed-off-by: Alexandra Antonova <[email protected]>

* add bert on subwords, concatenate it to bert on characters

Signed-off-by: Alexandra Antonova <[email protected]>

* add calculation of character_pos_to_subword_pos

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fix

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fix

Signed-off-by: Alexandra Antonova <[email protected]>

* pdb

Signed-off-by: Alexandra Antonova <[email protected]>

* tensor join bug fix

Signed-off-by: Alexandra Antonova <[email protected]>

* double hidden_size in classifier

Signed-off-by: Alexandra Antonova <[email protected]>

* pdb

Signed-off-by: Alexandra Antonova <[email protected]>

* default index value 0 instead of -1 because index cannot be negative

Signed-off-by: Alexandra Antonova <[email protected]>

* pad index value 0 instead of -1 because index cannot be negative

Signed-off-by: Alexandra Antonova <[email protected]>

* remove pdb

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bugs, add creation of tarred dataset

Signed-off-by: Alexandra Antonova <[email protected]>

* add possibility to change sequence len at inference

Signed-off-by: Alexandra Antonova <[email protected]>

* change sampling of dummy candidates at inference, add candidate info file

Signed-off-by: Alexandra Antonova <[email protected]>

* fix import

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug

Signed-off-by: Alexandra Antonova <[email protected]>

* update transcription now uses info

Signed-off-by: Alexandra Antonova <[email protected]>

* write path

Signed-off-by: Alexandra Antonova <[email protected]>

* 1. add tarred dataset support(untested). 2. fix bug with ban_ngrams in indexing

Signed-off-by: Alexandra Antonova <[email protected]>

* skip short_sent if no real candidates

Signed-off-by: Alexandra Antonova <[email protected]>

* fix import

Signed-off-by: Alexandra Antonova <[email protected]>

* add braceexpand

Signed-off-by: Alexandra Antonova <[email protected]>

* fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug in np.ones

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug in collate

Signed-off-by: Alexandra Antonova <[email protected]>

* change tensor type to long because of error in torch.gather

Signed-off-by: Alexandra Antonova <[email protected]>

* fix for empty spans tensor

Signed-off-by: Alexandra Antonova <[email protected]>

* same fixes in _collate_fn for tarred dataset

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug from previous commit

Signed-off-by: Alexandra Antonova <[email protected]>

* change int types to be shorter to minimize tar size

Signed-off-by: Alexandra Antonova <[email protected]>

* refactoring of datasets and inference

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fix

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fix

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fix

Signed-off-by: Alexandra Antonova <[email protected]>

* tar by 100k examples, small fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* small fixes, add analytics script

Signed-off-by: Alexandra Antonova <[email protected]>

* Add functions for dynamic programming comparison to get best path by ngrams

Signed-off-by: Alexandra Antonova <[email protected]>

* fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* small fix

Signed-off-by: Alexandra Antonova <[email protected]>

* fixes to support testing on SPGISpeech

Signed-off-by: Alexandra Antonova <[email protected]>

* add preprocessing for userlibri

Signed-off-by: Alexandra Antonova <[email protected]>

* some refactoring

Signed-off-by: Alexandra Antonova <[email protected]>

* some refactoring

Signed-off-by: Alexandra Antonova <[email protected]>

* move some functions to utils to reuse from other project

Signed-off-by: Alexandra Antonova <[email protected]>

* move some functions to utils to reuse from other project

Signed-off-by: Alexandra Antonova <[email protected]>

* move some functions to utils to reuse from other project

Signed-off-by: Alexandra Antonova <[email protected]>

* small refactoring before pr. Add bash-scripts reproducing evaluation

Signed-off-by: Alexandra Antonova <[email protected]>

* style fix

Signed-off-by: Alexandra Antonova <[email protected]>

* small fixes in inference

Signed-off-by: Alexandra Antonova <[email protected]>

* bug fix - didn't move window on last symbol

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug - shuffle was before truncation of sorted candidates

Signed-off-by: Alexandra Antonova <[email protected]>

* refactoring, fix some bugs

Signed-off-by: Alexandra Antonova <[email protected]>

* variour fixes. Add word_indices at inference

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add candidate positions

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Move data preparation and evaluation to other repo

Signed-off-by: Alexandra Antonova <[email protected]>

* add infer_reproduce_paper. Refactoring

Signed-off-by: Alexandra Antonova <[email protected]>

* refactor inference using fragment indices

Signed-off-by: Alexandra Antonova <[email protected]>

* add some helper functions

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug with parameters order

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bugs

Signed-off-by: Alexandra Antonova <[email protected]>

* refactoring, fix bug

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add multiple variants of adjusting start/end positions

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* more fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add unit tests, other fixes

Signed-off-by: Alexandra Antonova <[email protected]>

* fix

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix CodeQl warnings

Signed-off-by: Alexandra Antonova <[email protected]>

* add script for full inference pipeline, refactoring

Signed-off-by: Alexandra Antonova <[email protected]>

* add tutorial

Signed-off-by: Alexandra Antonova <[email protected]>

* take example data from HuggingFace

Signed-off-by: Alexandra Antonova <[email protected]>

* add docs

Signed-off-by: Alexandra Antonova <[email protected]>

* fix comment

Signed-off-by: Alexandra Antonova <[email protected]>

* fix bug

Signed-off-by: Alexandra Antonova <[email protected]>

* small fixes for PR

Signed-off-by: Alexandra Antonova <[email protected]>

* add some more tests

Signed-off-by: Alexandra Antonova <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* try to fix tests adding with_downloads

Signed-off-by: Alexandra Antonova <[email protected]>

* skip tests with tokenizer download

Signed-off-by: Alexandra Antonova <[email protected]>

---------

Signed-off-by: Alexandra Antonova <[email protected]>
Signed-off-by: Alexandra Antonova <[email protected]>
Co-authored-by: Alexandra Antonova <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] Implement new vocoder dataset (#6670)

* [TTS] Implement new vocoder dataset

Signed-off-by: Ryan <[email protected]>

* [TTS] Redo config structure, minor fixes

Signed-off-by: Ryan <[email protected]>

* [TTS] Fix alignment logging

Signed-off-by: Ryan <[email protected]>

* [TTS] Fix script usage example

Signed-off-by: Ryan <[email protected]>

* [TTS] Fixed epoch LR scheduling

Signed-off-by: Ryan <[email protected]>

* [TTS] Support .nemo checkpoint in FP callback

Signed-off-by: Ryan <[email protected]>

* [TTS] Remove align interpolator

Signed-off-by: Ryan <[email protected]>

* [TTS] Remove HiFi-GAN defaults list interpolation

Signed-off-by: Ryan <[email protected]>

* [TTS] Rename weighted_sample_steps to weighted_sampling_steps_per_epoch

Signed-off-by: Ryan <[email protected]>

---------

Signed-off-by: Ryan <[email protected]>

* GPT inference long context (#6687)

* deb infer

Signed-off-by: Evelina <[email protected]>

* deb infer

Signed-off-by: Evelina <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean up

Signed-off-by: Evelina <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* dont do maxlen trunc for non abs pos emb

Signed-off-by: Evelina <[email protected]>

* dont do maxlen trunc for non abs pos emb

Signed-off-by: Evelina <[email protected]>

* convert for training only

Signed-off-by: Evelina <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add eval test, add save .nemo for sft model

Signed-off-by: Evelina <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* jenkins format fix

Signed-off-by: Evelina <[email protected]>

* update jenkins

Signed-off-by: Evelina <[email protected]>

* update jenkins

Signed-off-by: Evelina <[email protected]>

* fix jenkins

Signed-off-by: Evelina <[email protected]>

* remove test, ci timeout

Signed-off-by: Evelina <[email protected]>

* fix for m_gpt_eval.py

Signed-off-by: Evelina <[email protected]>

* jenkins test

Signed-off-by: Evelina <[email protected]>

* fix gpt_eval with sft model

Signed-off-by: Evelina <[email protected]>

* revert jenkins

Signed-off-by: Evelina <[email protected]>

* keep float conversion for model.generate()

Signed-off-by: Evelina <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix inference dtype

Signed-off-by: Evelina <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Evelina <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* TDT model pull request (#6536)

* TDT model pull request, initial draft

Signed-off-by: Hainan Xu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* TDT PR WIP

Signed-off-by: Hainan Xu <[email protected]>

* TDT PR WIP

Signed-off-by: Hainan Xu <[email protected]>

* TDT PR WIP

Signed-off-by: Hainan Xu <[email protected]>

* TDT WIP

Signed-off-by: Hainan Xu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* TDT WIP

Signed-off-by: Hainan Xu <[email protected]>

* TDT WIP

Signed-off-by: Hainan Xu <[email protected]>

* TDT WIP

Signed-off-by: Hainan Xu <[email protected]>

* TDT WIP

Signed-off-by: Hainan Xu <[email protected]>

* TDT WIP

Signed-off-by: Hainan Xu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* TDT WIP

Signed-off-by: Hainan Xu <[email protected]>

* TDT WIP

Signed-off-by: Hainan Xu <[email protected]>

* TDT WIP

Signed-off-by: Hainan Xu <[email protected]>

* TDT WIP

Signed-off-by: Hainan Xu <[email protected]>

* TDT WIP

Signed-off-by: Hainan Xu <[email protected]>

* addressed some review comments, part1

Signed-off-by: Hainan Xu <[email protected]>

* addressed some review comments, part1, one line fix

Signed-off-by: Hainan Xu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add tests for comparing TDT alphas with pytorch VS kernel computation

Signed-off-by: Hainan Xu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add tests for comparing multiblank alphas with pytorch VS kernel computation

Signed-off-by: Hainan Xu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add tests for fixed case computation for TDT

Signed-off-by: Hainan Xu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add more comments for greedy-batch decoding for TDT

Signed-off-by: Hainan Xu <[email protected]>

* include config for TDT model with stateless decoders

Signed-off-by: Hainan Xu <[email protected]>

* add reference to TDT in Readme

Signed-off-by: Hainan Xu <[email protected]>

* slight modification of config file comments

Signed-off-by: Hainan Xu <[email protected]>

* addressed more comments

Signed-off-by: Hainan Xu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* more detailed comments for tdt kernel

Signed-off-by: Hainan Xu <[email protected]>

* one line fix

Signed-off-by: Hainan Xu <[email protected]>

* fixed small bug that results in test fails for rnnt_decoding

Signed-off-by: Hainan Xu <[email protected]>

* fixed small bug that results in test fails for rnnt_decoding

Signed-off-by: Hainan Xu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed small bug that results in test fails for rnnt_decoding

Signed-off-by: Hainan Xu <[email protected]>

* remove unused import

Signed-off-by: Hainan Xu <[email protected]>

---------

Signed-off-by: Hainan Xu <[email protected]>
Co-authored-by: Hainan Xu <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix get_parameters when using main params optimizer (#6764) (#6787)

* fix get param



* change name



---------

Signed-off-by: ericharper <[email protected]>
Co-authored-by: Eric Harper <[email protected]>

* Lddl bert (#6761) (#6790)

* initial POC for LDDL Bert

* Finish LDDL POC

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* address comments

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix merge head

* resolving merge

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add support for  val/test loaders

* change to new LDDL class + add winding

* fix logging level

* fix winding

* test fix

* fixes to winding

* add file system

* add prepemption optimizations

* more logging

* more prints

* better logging

* asfsf

* add barrier

* removing prints

* working with mb lddl loader

* final changes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update requirements file with LDDL



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* revert adding to requirements

---------

Signed-off-by: wdykas <[email protected]>
Co-authored-by: wdykas <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <[email protected]>

* Fix check (#6798) (#6800)

Signed-off-by: MaximumEntropy <[email protected]>
Co-authored-by: Sandeep Subramanian <[email protected]>

* Fix validation with drop_last=False (#6704)

Signed-off-by: Mikołaj Błaż <[email protected]>
Co-authored-by: Eric Harper <[email protected]>

* SDE unt lvl comparison (#6669)

Added a visual utterance-level comparison of two ASR models

Signed-off-by: George <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Debug Transformer Engine FP8 support with Megatron-core infrastructure (#6791)

* Construct FP8 amax reduction group

Signed-off-by: Tim Moon <[email protected]>

* Update Megatron-core version in CI

Signed-off-by: Tim Moon <[email protected]>

---------

Signed-off-by: Tim Moon <[email protected]>
Co-authored-by: Tim Moon <[email protected]>
Co-authored-by: Tim Moon <[email protected]>

* Lora/PEFT training script CI test (#6664)

* new lora test

Signed-off-by: arendu <[email protected]>

* updates

Signed-off-by: arendu <[email protected]>

* check for chat

Signed-off-by: arendu <[email protected]>

* update

Signed-off-by: arendu <[email protected]>

* update

Signed-off-by: arendu <[email protected]>

* small train set

Signed-off-by: arendu <[email protected]>

* update

Signed-off-by: arendu <[email protected]>

* precision change

Signed-off-by: arendu <[email protected]>

* fixed typo in paths

Signed-off-by: arendu <[email protected]>

* full data with limit val batches

Signed-off-by: arendu <[email protected]>

* tp2 instead of pp2

Signed-off-by: arendu <[email protected]>

* tp2 instead of pp2

Signed-off-by: arendu <[email protected]>

---------

Signed-off-by: arendu <[email protected]>
Signed-off-by: Adi Renduchintala <[email protected]>

* change branch to main, small fix (#6803)

Signed-off-by: Alexandra Antonova <[email protected]>

* add call to p2p overlap (#6779) (#6786)

* add call to p2p overlap



* update Jenkins for test



---------

Signed-off-by: Abhinav Khattar <[email protected]>
Signed-off-by: Eric Harper <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Co-authored-by: Eric Harper <[email protected]>

* fixed  decor to show messages only when the wrapped object is called. (#6793)

Signed-off-by: Xuesong Yang <[email protected]>

* Bug fix for reset_sequence_parallel_args (#6802) (#6805)

Signed-off-by: Markel Sanz Ausin <[email protected]>
Co-authored-by: Markel Sanz Ausin <[email protected]>

* text_generation_utils memory reduction if no logprob needed (#6773)

* repro for gpt eval mp mem issue

Signed-off-by: Yang Zhang <[email protected]>

* add print statements for memory allocation

Signed-off-by: Yang Zhang <[email protected]>

* adjusted hot fix that prevents softmax on the entire output embedding,now memory bottlenecked by attention softmax which needs to be solved with FA or long attention

Signed-off-by: Yang Zhang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* using compute_logprob to configure inference

Signed-off-by: Yang Zhang <[email protected]>

* enable compute logprob for peft

Signed-off-by: Yang Zhang <[email protected]>

* remove print statements

Signed-off-by: Yang Zhang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix ci

Signed-off-by: Yang Zhang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added docstrings

Signed-off-by: Yang Zhang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add missing config

Signed-off-by: Yang Zhang <[email protected]>

* remove truncate prompt length feature

Signed-off-by: Yang Zhang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tensor before all gather needs to be contiguous

Signed-off-by: Yang Zhang <[email protected]>

---------

Signed-off-by: Yang Zhang <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Evelina <[email protected]>
Co-authored-by: Sandeep Subramanian <[email protected]>

* Fixed bug in MaskedSpecAug that overestimates samples. (#6775)

Signed-off-by: tbartley94 <[email protected]>

* update core version (#6817) (#6819)

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>

* lora pp2 (#6818)

Signed-off-by: arendu <[email protected]>

* Add optional index mapping dir in mmap text datasets (#6683)

If datasets are stored on a read-only medium, index files
cannot be created into adjacent files and an
alternative directory must be specified for index
mapping files.

This commit adds an optional `index_mapping_dir` to
the constructors.
Unit tests are also added.



[pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Update path formatting for relative paths

Signed-off-by: Greg Heinrich <[email protected]>

* Add inference kv cache support for transformer TE path (#6627)

* Add kv cache support for transformer TE path

Signed-off-by: Yen-Shi Wang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Mark get_data_parallel_group as WAR

Signed-off-by: Yen-Shi Wang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Initialize process group for FP8 training

Signed-off-by: Tim Moon <[email protected]>

* Update Megatron GPT eval script for non-FP8 path

Signed-off-by: Yen-Shi Wang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Yen-Shi Wang <[email protected]>
Signed-off-by: Tim Moon <[email protected]>
Signed-off-by: Yen-Shi Wang <[email protected]>
Co-authored-by: Yen-Shi Wang <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Tim Moon <[email protected]>
Co-authored-by: Tim Moon <[email protected]>
Co-authored-by: Eric Harper <[email protected]>

* Support large inputs to Conformer and Fast Conformer (#6556)

* initial commit

Signed-off-by: Dima Rekesh <[email protected]>

* typos

Signed-off-by: Dima Rekesh <[email protected]>

* tweaks to padding

Signed-off-by: Dima Rekesh <[email protected]>

* comments

Signed-off-by: Dima Rekesh <[email protected]>

* attempt at first working version

Signed-off-by: Dima Rekesh <[email protected]>

* typos and fixed p calculation

Signed-off-by: Dima Rekesh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removing merge artifacts

Signed-off-by: Dima Rekesh <[email protected]>

* typo

Signed-off-by: Dima Rekesh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removing unnessary imports

Signed-off-by: Dima Rekesh <[email protected]>

* if batch split succeeded no need to conv again

Signed-off-by: Dima Rekesh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adding channel wise split

Signed-off-by: Dima Rekesh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adding reference to pytorch issue 80020

Signed-off-by: Dima Rekesh <[email protected]>

* removing time chunking methods

Signed-off-by: Dima Rekesh <[email protected]>

* accounting for the actual self._stride value

Signed-off-by: Dima Rekesh <[email protected]>

* limiting the fix to dw_striding subsampling

Signed-off-by: Dima Rekesh <[email protected]>

* renamed methods

Signed-off-by: Dima Rekesh <[email protected]>

* one more accounting for the actual self._stride value

Signed-off-by: Dima Rekesh <[email protected]>

* support for causal convs

Signed-off-by: Dima Rekesh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* option to set conv chunking size manually

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixing imports

* subsampling test

Signed-off-by: Dima Rekesh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rename variable

Signed-off-by: Dima Rekesh <[email protected]>

* imports in test

Signed-off-by: Dima Rekesh <[email protected]>

* more runtime checks

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* a more careful test

Signed-off-by: Dima Rekesh <[email protected]>

* bug in causal

Signed-off-by: Dima Rekesh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix in causal

Signed-off-by: Dima Rekesh <[email protected]>

* change_conv_chunking_factor methods

Signed-off-by: Dima Rekesh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* renamed methods

Signed-off-by: Dima Rekesh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* disabling chunking by default

Signed-off-by: Dima Rekesh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* typo

Signed-off-by: Dima Rekesh <[email protected]>

* changing default chunking to auto

Signed-off-by: Dima Rekesh <[email protected]>

* only split if needed

Signed-off-by: Dima Rekesh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* only split if needed

Signed-off-by: Dima Rekesh <[email protected]>

---------

Signed-off-by: Dima Rekesh <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* sharded_manifests updated docs (#6833)

Signed-off-by: Dima Rekesh <[email protected]>

* added fc-xl, xxl and titanet-s models (#6832)

Signed-off-by: Nithin Rao Koluguri <nithinraok>
Co-authored-by: Nithin Rao Koluguri <nithinraok>

* add reference to our paper (#6821)

* add reference to our paper

Signed-off-by: Alexandra Antonova <[email protected]>

* add paper reference to docs

Signed-off-by: Alexandra Antonova <[email protected]>

---------

Signed-off-by: Alexandra Antonova <[email protected]>

* Upperbound Numpy to < 1.24 (#6829)

Signed-off-by: smajumdar <[email protected]>
Co-authored-by: Eric Harper <[email protected]>

* Multi-lookahead cache-aware streaming models (#6711)

* added methods.

Signed-off-by: Vahid <[email protected]>

* added methods.

Signed-off-by: Vahid <[email protected]>

* added initial code.

Signed-off-by: Vahid <[email protected]>

* added initial code.

Signed-off-by: Vahid <[email protected]>

* added initial code.

Signed-off-by: Vahid <[email protected]>

* added config files.

Signed-off-by: Vahid <[email protected]>

* fixed bugs.

Signed-off-by: Vahid <[email protected]>

* updated confs.

Signed-off-by: Vahid <[email protected]>

* updated confs.

Signed-off-by: Vahid <[email protected]>

* updated confs.

Signed-off-by: Vahid <[email protected]>

* updated confs.

Signed-off-by: Vahid <[email protected]>

* improved f.conv1d

Signed-off-by: Vahid <[email protected]>

* pulled from main.

Signed-off-by: Vahid <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* pulled from main.

Signed-off-by: Vahid <[email protected]>

* added postpostnorm.

Signed-off-by: Vahid <[email protected]>

* fixed the target continiouse bug.

Signed-off-by: Vahid <[email protected]>

* added dw_striding causal.

Signed-off-by: Vahid <[email protected]>

* added print for debugging.

Signed-off-by: Vahid <[email protected]>

* added print for debugging.

Signed-off-by: Vahid <[email protected]>

* fixed causal convolutions.

Signed-off-by: Vahid <[email protected]>

* added _midnorm.

Signed-off-by: Vahid <[email protected]>

* fixed transcribe.

Signed-off-by: Vahid <[email protected]>

* cleaned code.

Signed-off-by: Vahid <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* moved back configs.

Signed-off-by: Vahid <[email protected]>

* moved back configs.

Signed-off-by: Vahid <[email protected]>

* updated fast emit for FC models.

Signed-off-by: Vahid <[email protected]>

* updated fast emit for FC models.

Signed-off-by: Vahid <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed bug.

Signed-off-by: Vahid <[email protected]>

* fixed bug and addressed comments.

Signed-off-by: Vahid <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed configs.

Signed-off-by: Vahid <[email protected]>

* fixed configs.

Signed-off-by: Vahid <[email protected]>

* dropped the test.

Signed-off-by: Vahid <[email protected]>

---------

Signed-off-by: Vahid <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* added changes to ramp up bs  (#6799)

* rampup bs changes

Signed-off-by: dimapihtar <[email protected]>

* rampup bs changes

Signed-off-by: dimapihtar <[email protected]>

* fixed styling

Signed-off-by: dimapihtar <[email protected]>

* fix bug

Signed-off-by: Dmytro Pykhtar <[email protected]>

---------

Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: Dmytro Pykhtar <[email protected]>
Co-authored-by: Eric Harper <[email protected]>

* Fix typo in core.rst (#6838)

Signed-off-by: Dounx <[email protected]>

* add back ptuning pp2 test (#6394)

Signed-off-by: arendu <[email protected]>

* t5 lora tuning (#6612)

* t5 lora

Signed-off-by: arendu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* eval lora t5

Signed-off-by: arendu <[email protected]>

* adjust differernt lora dims

Signed-off-by: arendu <[email protected]>

* minor changes

Signed-off-by: David Mosallanezhad <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* bugfix for state_dict

Signed-off-by: David Mosallanezhad <[email protected]>

---------

Signed-off-by: arendu <[email protected]>
Signed-off-by: David Mosallanezhad <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David Mosallanezhad <[email protected]>
Co-authored-by: David <[email protected]>

* NFA updates (#6695)

* update V_NEGATIVE_NUM constant to make better use of torch.float32 range

Signed-off-by: Elena Rastorgueva <[email protected]>

* adjust backpointers dtype if U_max too large

Signed-off-by: Elena Rastorgueva <[email protected]>

* Remove print statements

Signed-off-by: Elena Rastorgueva <[email protected]>

* Remove need for user to specify model_downsample_factor

Signed-off-by: Elena Rastorgueva <[email protected]>

* change model.cfg.sample_rate to model.cfg.preprocessor.sample_rate

Signed-off-by: Elena Rastorgueva <[email protected]>

* add check to make sure that window_stride is in model.cfg.preprocessor

Signed-off-by: Elena Rastorgueva <[email protected]>

* reduce memory consumption of backpointers by making them relative instead of absolute

Signed-off-by: Elena Rastorgueva <[email protected]>

* update librosa.get_duration() 'filename' param to 'path'

Signed-off-by: Elena Rastorgueva <[email protected]>

* Do not throw error if 'text' or 'pred_text' are empty and make sure CTM filepaths in the output manifest are null

Signed-off-by: Elena Rastorgueva <[email protected]>

* preprocess input text by removing any duplicate spaces and converting any newlines to spaces

Signed-off-by: Elena Rastorgueva <[email protected]>

* Use Utterance dataclass instead of dictionaries for keeping track of token/word/segment alignments

Signed-off-by: Elena Rastorgueva <[email protected]>

* refactor so can save alignments as ctm and ass format files

Signed-off-by: Elena Rastorgueva <[email protected]>

* fix bugs for saving character based ASS files and for using pred_text to do alignment

Signed-off-by: Elena Rastorgueva <[email protected]>

* Make token level .ass file use tokens with recovered capitalization

Signed-off-by: Elena Rastorgueva <[email protected]>

* Do not try to generate alignment files if text or pred text is empty, or if number of tokens is too large for T

Signed-off-by: Elena Rastorgueva <[email protected]>

* rename output manifest file to say '_with_output_file_paths.json'

Signed-off-by: Elena Rastorgueva <[email protected]>

* add flag to resegment ass subtitle file to fill available text space

Signed-off-by: Elena Rastorgueva <[email protected]>

* Fix bug in resegmentation code

Signed-off-by: Elena Rastorgueva <[email protected]>

* Fix bug which skipped some utterances if batch_size more than 1

Signed-off-by: Elena Rastorgueva <[email protected]>

* reduce memory requirements by doing torch.gather on a slice of the log probs when they are needed

Signed-off-by: Elena Rastorgueva <[email protected]>

* reduce memory requirements by not saving whole v_matrix

Signed-off-by: Elena Rastorgueva <[email protected]>

* remove any extra spaces in pred_text

Signed-off-by: Elena Rastorgueva <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove unused list pred_text_all_lines

Signed-off-by: Elena Rastorgueva <[email protected]>

* support using hybrid Transducer-CTC models for alignment

Signed-off-by: Elena Rastorgueva <[email protected]>

* fix typo - add brackets to torch.cuda.is_available()

Signed-off-by: Elena Rastorgueva <[email protected]>

* make sure token case restoration will work if superscript or subscript num is in text

Signed-off-by: Elena Rastorgueva <[email protected]>

* remove any BOM from input text

Signed-off-by: Elena Rastorgueva <[email protected]>

* pick out 1st hypotheses if there is a tuple of them

Signed-off-by: Elena Rastorgueva <[email protected]>

* Remove print statement

Signed-off-by: Elena Rastorgueva <[email protected]>

* add detail to error message if fail to recover capitalization of tokens

Signed-off-by: Elena Rastorgueva <[email protected]>

* add flag use_local_attention

Signed-off-by: Elena Rastorgueva <[email protected]>

* rename additional_ctm_grouping_separator -> additional_segment_grouping_separator

Signed-off-by: Elena Rastorgueva <[email protected]>

* update description of additional_segment_grouping_separator

Signed-off-by: Elena Rastorgueva <[email protected]>

* add simple docstring to get_utt_obj function

Signed-off-by: Elena Rastorgueva <[email protected]>

* Make docstring for add_t_start_end_to_utt_obj

Signed-off-by: Elena Rastorgueva <[email protected]>

* update docstrings for add_t_start_end_to_utt_obj and get_batch_variables

Signed-off-by: Elena Rastorgueva <[email protected]>

* update README and comments in align.py

Signed-off-by: Elena Rastorgueva <[email protected]>

* change 'ground truth' -> 'reference text' in documentation

Signed-off-by: Elena Rastorgueva <[email protected]>

* add header

Signed-off-by: Elena Rastorgueva <[email protected]>

* add comments to get_utt_obj function

Signed-off-by: Elena Rastorgueva <[email protected]>

* move constants so they are after imports

Signed-off-by: Elena Rastorgueva <[email protected]>

* add file description for make_ass_files

Signed-off-by: Elena Rastorgueva <[email protected]>

* get rid of Utterance object's S attribute, and correct tests so they pass now

Signed-off-by: Elena Rastorgueva <[email protected]>

* remove some unused variables

Signed-off-by: Elena Rastorgueva <[email protected]>

* remove unused variable model from functions saving output files

Signed-off-by: Elena Rastorgueva <[email protected]>

* remove unused var minimum_timestamp_duration from make_ass_files functions and return utt_obj

Signed-off-by: Elena Rastorgueva <[email protected]>

* move minimum_timestamp_duration param to CTMFileConfig

Signed-off-by: Elena Rastorgueva <[email protected]>

* remove unused enumerate and unused import

Signed-off-by: Elena Rastorgueva <[email protected]>

* switch reading duration from librosa to soundfile to avoid filename/path deprecation message

Signed-off-by: Elena Rastorgueva <[email protected]>

---------

Signed-off-by: Elena Rastorgueva <[email protected]>
Signed-off-by: Elena Rastorgueva <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Added rouge monitoring support for T5 (#6737)

* Added rouge monitoring support for t5

Signed-off-by: Matvei Novikov <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Matvei Novikov <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* GPT extrapolatable position embedding (xpos/sandwich/alibi/kerple) and Flash Attention (#6666)

* move to nvidia megatron repo (#6465) (#6475)

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Megatron KERPLE positional embeddings (#6478) (#6480)

* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

---------




* [TTS] whitelist broken path fix. (#6412)

* [TTS] whitelist broken path fix.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------




* [TTS] FastPitch speaker encoder (#6417)

* Add initial codes



* Remove wemb



* Fix import



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss



* Add ConditionalInput



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix import error



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix dataset error



* Remove reference speaker embedding



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove SV encoder



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix length type



* Fix append



* Move error msg



* Add look-up into speaker encoder



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Add valueerror msg



* Move lookup



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove unused



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix error



* Rebase and Fix error



* Fix spk encoder



* Rename n_speakers



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error



---------




* Sharded manifests for tarred datasets (#6395)

* testing sharded manifests



* compatibility



* proper fixes



* adding flag tot convert_to_tarred_audio_dataset



* shard_manifests conf param



* propagating the shard_manifests param



* propagating the shard_manifests param



* distributed checks



* typo



* typo



* fixes



* fixes



* fixes



* fixes



* fixes



* fixes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py



* reversing manifest shards flag



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests



* excluding manifests from webdataset url expansion



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore



* explicit use of UTF-8 for manifest i/o



---------




* Update wfst_text_normalization.rst (#6374)

Add Hungarian (incoming in NeMo-text-processing)



* Support Swiglu in TP PP Conversion (#6437) (#6451)

* Support Swiglu in TP PP Conversion



* Guard activation



* Guard activation



---------




* Update NeMo_TTS_Primer.ipynb (#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.



* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.



---------



* add rampup batch size support for Megatron GPT (#6424)

* added rampup batch size support



* added tests for rampup batch size



* fixed the typos



* added assertions



* changed assertion rules



* deleted unused imports



* changed tests for rampup batch size



* updated rampup batch size tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling



* rampup batch size tests changes



---------







* Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.



* 1. Debugging.

---------





* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default



* updated README with the new agg_tokenizer_manifest flag



* fixed typo in scripts/speech_recognition/code_switching/README.md



* changed agg_tokenizer_manifest to is_lid_manifest



---------




* Added/updated new Conformer configs (#6426) (#6467)

* Update script for ngram rnnt and hat beam search decoding (#6370)

* add rnnt ngram beamsearch script



* add return encoding embedding option



* update script



* add rnnt and hat ngram decoding script



* add some parameters



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig



* replace return_encoder_embeddings parameter



* generalization of scipt behavior



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter



* remove return_encoder_embeddings parameter



* add manual encoder_embeddings calculation



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8



* fix rescoring description



---------






* BERT pre-training mp fork to spawn (#6442) (#6454)

* change bert fork to spawn



* num_workers=0 fix



---------




* fix replace_bos_with_pad not found (#6443) (#6450)




* reduce workers on NMT CI (#6472) (#6474)




* 1. Added KERPLE positional embeddings to encoder-decoder.



* 1. Added a missing file.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.



* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <[email protected]>
Signed-off-by: Xuesong Yang <[email protected]>
Signed-off-by: Dima Rekesh <[email protected]>
Signed-off-by: Jim O’Regan <[email protected]>
Signed-off-by: smajumdar <[email protected]>
Signed-off-by: Mostafa Ghorbandoost <[email protected]>
Signed-off-by: Dmytro Pykhtar <[email protected]>
Signed-off-by: Dmytro Pykhtar <[email protected]>
Signed-off-by: Micha Livne <[email protected]>
Signed-off-by: Kunal Dhawan <[email protected]>
Signed-off-by: andrusenkoau <[email protected]>
Signed-off-by: Andrei Andrusenko <[email protected]>
Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Micha Livne <[email protected]>
Co-authored-by: Cheng-Ping Hsieh <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <[email protected]>
Co-authored-by: Dima Rekesh <[email protected]>
Co-authored-by: Jim O’Regan <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <[email protected]>
Co-authored-by: Mostafa Ghorbandoost <[email protected]>
Co-authored-by: Dmytro Pykhtar <[email protected]>
Co-authored-by: Dmytro Pykhtar <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Co-authored-by: Micha Livne <[email protected]>
Co-authored-by: Kunal Dhawan <[email protected]>
Co-authored-by: Andrei Andrusenko <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Fix an invalid link in get_data.py of ljspeech (#6456)

Usage of the link in line 63 leads to downloading a html file not a tsv file, so we need to change it to a raw link.

Signed-off-by: Mostafa Ghorbandoost <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* 1. Added external index sample. (#6462) (#6483)

Signed-off-by: Micha Livne <[email protected]>
Co-authored-by: Micha Livne <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Update README to add core installation (#6488) (#6489)

* update README for megatron-core



* fix



---------

Signed-off-by: Abhinav Khattar <[email protected]>
Co-authored-by: Abhinav Khattar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Fix cache aware hybrid bugs (#6466) (#6484)

Signed-off-by: hsiehjackson <[email protected]>

* Fix typos (#6494) (#6495)

Signed-off-by: smajumdar <[email protected]>
Co-authored-by: Somshubra Majumdar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Add disclaimer about dataset for ASR (#6496)

Signed-off-by: smajumdar <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* fix (#6502)

datastore_path_to_webdataset_url(p) if is_datastore_path(p) and is_tarred_path(p) else p
NameError: name 'is_tarred_path' is not defined

Co-authored-by: George <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* fix broken links r1.18.0 (#6501) (#6504)

* fix broken links



* fix broken links



---------

Signed-off-by: Evelina <[email protected]>
Co-authored-by: Evelina <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* [TTS] Create functions for TTS preprocessing without dataloader (#6317)

* [TTS] Create functions for TTS preprocessing without dataloader

Signed-off-by: Ryan <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

* Cache aware streaming nfa (#6209)

* add cache aware streaming to nemo aligner

Signed-off-by: Slyne Deng <[email protected]>

Signed-off-by: hsiehjackson <[email protected]>

* [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator (#6499)

* [BugFix] _get_batch_preds() is forced to keep logits in  decoder timestamps generators

Signed-off-by: Taejin Park <[email protected]>

* Ingnore keep_logits boolean in FrameASRBatchLogits

Signed-off-by: Taejin Park <[email protected]>

---------

Signed-off-by: Taejin Park <[email protected]>
Co-authored-by: Jagadeesh Balam <[email protected]>
Signed-off-by: hsiehjackson <[email protected]>

…
yaoyu-33 pushed a commit that referenced this pull request Oct 16, 2023
…s generator (#6499)

* [BugFix] _get_batch_preds() is forced to keep logits in  decoder timestamps generators

Signed-off-by: Taejin Park <[email protected]>

* Ingnore keep_logits boolean in FrameASRBatchLogits

Signed-off-by: Taejin Park <[email protected]>

---------

Signed-off-by: Taejin Park <[email protected]>
Co-authored-by: Jagadeesh Balam <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants