NVIDIA Neural Modules 1.19.0

Highlights

NeMo ASR

• Sharded Manifests for Tarred Datasets #6395
• Frame-VAD model + datasets support #6441
• Noise Norm Perturbation #6445
• Code Switched Dataset with IID Sampling #6448

NeMo TTS

• Speaker adaptation for FastPitch #6416, #6417

NeMo Megatron

• Batch size rampup #6424
• Unify dataset and model classes for all PEFT #6391
• LoRA for GPT #6391
• Convert interleaved pipeline model to non-interleaved #6498
• Dialog Dataset for SFT #6654
• Dynamic length batches for GPT SFT #6510
• Merge LoRA weights into base model #6597

Container

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:23.04

Detailed Changelogs

ASR

Changelog

• Sharded manifests for tarred datasets by @bmwshop :: PR: #6395
• Update script for ngram rnnt and hat beam search decoding by @andrusenkoau :: PR: #6370
• Add disclaimer about dataset for ASR by @titu1994 :: PR: #6496
• New noise_norm perturbation based on Riva work by @trias702 :: PR: #6445
• Add Frame-VAD model and datasets by @stevehuang52 :: PR: #6441
• removing unnecessary avoid_bfloat16_autocast_context by @bmwshop :: PR: #6481
• FC models in menu by @bmwshop :: PR: #6473
• Separate punctuation by whitespace by @karpnv :: PR: #6574
• Cherry pick commits in #6601 to main by @fayejf :: PR: #6611
• Offline and streaming inference support for hybrid model by @fayejf :: PR: #6570
• Disable interctc tests by @Kipok :: PR: #6638
• ASR-TTS Models: Support hybrid RNNT-CTC, improve docs. by @artbataev :: PR: #6620
• Confidence ensembles implementation by @Kipok :: PR: #6614
• Confidence ensembles: fix issues and add tuning functionality by @Kipok :: PR: #6657
• Add support for RNNT/hybrid models to partial transcribe by @stevehuang52 :: PR: #6609
• eval_beamsearch_ngram.py with hybrid ctc by @karpnv :: PR: #6656

TTS

Changelog

• [TTS] FastPitch adapter fine-tune and conditional layer normalization by @hsiehjackson :: PR: #6416
• [TTS] whitelist broken path fix. by @XuesongYang :: PR: #6412
• [TTS] FastPitch speaker encoder by @hsiehjackson :: PR: #6417
• Update NeMo_TTS_Primer.ipynb by @pythinker :: PR: #6436
• [TTS] Create functions for TTS preprocessing without dataloader by @rlangman :: PR: #6317
• [TTS] Fix FastPitch energy code by @rlangman :: PR: #6511
• [TTS] Add script for computing feature stats by @rlangman :: PR: #6508
• [TTS] Add tutorials for FastPitch TTS speaker adaptation with adapters by @hsiehjackson :: PR: #6431
• [TTS] Create initial TTS dataset feature processors by @rlangman :: PR: #6507
• [TTS] Add script for mapping speaker names to indices by @rlangman :: PR: #6509
• [TTS] Implement new TextToSpeech dataset by @rlangman :: PR: #6575

NLP / NMT

Changelog

• Add patches for Virtual Parallel conversion by @titu1994 :: PR: #6589
• Update wfst_text_normalization.rst by @jimregan :: PR: #6374
• add rampup batch size support for Megatron GPT by @dimapihtar :: PR: #6424
• Add interleaved pp support by @titu1994 :: PR: #6498
• Support dynamic length batches with GPT SFT by @aklife97 :: PR: #6510
• Framework for PEFT via mixins by @arendu :: PR: #6391
• Add GPT eval mode fix for interleaved to main (#6449) by @aklife97 :: PR: #6610
• sft model can use this script for eval by @arendu :: PR: #6637
• Patch memory used for NeMo Megatron models by @titu1994 :: PR: #6615
• merge lora weights into base model by @arendu :: PR: #6597
• Dialogue dataset by @yidong72 :: PR: #6654
• check for first or last stage by @ericharper :: PR: #6708
• A few small typo fixes by @Kipok :: PR: #6599
• Lddl bert by @wdykas :: PR: #6761
• Debug Transformer Engine FP8 support with Megatron-core infrastructure by @timmoon10 :: PR: #6740
• Tensor-parallel communication overlap with userbuffer backend by @erhoo82 :: PR: #6780
• Add ub communicator initialization to validation step by @erhoo82 :: PR: #6807
• Add trainer.validate example for GPT by @ericharper :: PR: #6794
• Add API docs for NeMo Megatron by @ericharper :: PR: #6850
• Apply garbage collection interval to validation steps by @erhoo82 :: PR: #6870

Bugfixes

Changelog

• [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator by @tango4j :: PR: #6499
• small bugfix for asr_evaluator by @fayejf :: PR: #6636
• fix bucketing bug issue for picking new bucket by @nithinraok :: PR: #6663
• [TTS] Fix TTS audio preprocessing bugs by @rlangman :: PR: #6628
• Fix a bug, use _ceil_to_nearest instead as _round_to_nearest is not d… by @BestJuly :: PR: #6681
• Bug fix to restore act ckpt by @markelsanz14 :: PR: #6753
• Bug fix to reset sequence parallelism by @markelsanz14 :: PR: #6756
• Bug fix for reset_sequence_parallel_args by @markelsanz14 :: PR: #6802
• Fix adapter tutorial r1.19.0 by @hsiehjackson :: PR: #6776
• Fix error appearing when using tar datasets by @Jorjeous :: PR: #6502
• Fix normalization of impulse response in ImpulsePerturbation by @anteju :: PR: #6505
• Fix typos by @titu1994 :: PR: #6523
• Fix notebook bad json by @titu1994 :: PR: #6561
• [ASR] Fix for old models in change_attention_model by @sam1373 :: PR: #6608
• Fix k2 installation in Docker with CUDA 12 by @artbataev :: PR: #6707
• Tutorial fixes by @titu1994 :: PR: #6717
• Vp fixes by @titu1994 :: PR: #6738
• [TTS] Fix aligner nan loss in fp32 by @hsiehjackson :: PR: #6435
• fix conversion and eval by @arendu :: PR: #6648
• Fix checkpointed forward and add test for full activation checkpointing by @aklife97 :: PR: #6744
• add call to p2p overlap by @aklife97 :: PR: #6779
• Fix get_parameters when using main params optimizer by @ericharper :: PR: #6764
• Fix GPTDataset Assert by @MaximumEntropy :: PR: #6798
• fix notebook error by @yidong72 :: PR: #6840
• final fix of notebook by @yidong72 :: PR: #6842

General Improvements

Changelog

• Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format by @KunalDhawan :: PR: #6448
• Fix an invalid link in get_data.py of ljspeech by @pythinker :: PR: #6456
• Update manifest.py to use os.path for get_full_path by @stevehuang52 :: PR: #6598
• Cherry pick commits in #6528 to main by @timmoon10 :: PR: #6613
• Move black parameters to pyproject.toml by @artbataev :: PR: #6647
• handle artifacts when path is an extracted dir by @arendu :: PR: #6658
• remove upgrading setuptools in reinstall.sh by @XuesongYang :: PR: #6659
• Upgrade to PyTorch 23.04 Container by @ericharper :: PR: #6660
• Fix fastpitch test nightly by @hsiehjackson :: PR: #6742
• Fix Links for tutorials by @titu1994 :: PR: #6777
• Update core version in Jenkinsfile by @aklife97 :: PR: #6817
• Update mcore requirement to 0.2.0 by @ericharper :: PR: #6875