Add PP support in NeVA along with few bug fixes #11170
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
…ya/add_llama_vlm_hf
…/add_llama_vlm_hf
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
# Conflicts: # nemo/collections/multimodal/data/energon/base.py # nemo/collections/vlm/__init__.py # nemo/collections/vlm/mllama/data/mock.py # nemo/collections/vlm/mllama/model/base.py # nemo/collections/vlm/neva/data/__init__.py # nemo/collections/vlm/neva/data/conversation.py # nemo/collections/vlm/neva/data/lazy.py # nemo/collections/vlm/peft/lora.py # nemo/collections/vlm/recipes/__init__.py
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
Signed-off-by: yaoyu-33 <[email protected]>
|
beep boop 🤖: 🚨 The following files must be fixed before merge! Your code was analyzed with PyLint. The following annotations have been identified: Thank you for improving NeMo's documentation! |
|
beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base. Your code was analyzed with PyLint. The following annotations have been identified: Thank you for improving NeMo's documentation! |
|
[🤖]: Hi @yaoyu-33 👋, We wanted to let you know that a CICD pipeline for this PR just finished successfully So it might be time to merge this PR or get some approvals I'm just a bot so I'll leave it you what to do next. //cc @pablo-garay @ko3n1g |
XuesongYang
pushed a commit
to paarthneekhara/NeMo
that referenced
this pull request
Jan 18, 2025
* evian3 update Signed-off-by: yaoyu-33 <[email protected]> * add encoder parallel default config Signed-off-by: yaoyu-33 <[email protected]> * add encoder parallel default config Signed-off-by: yaoyu-33 <[email protected]> * clean up Signed-off-by: yaoyu-33 <[email protected]> * add aspect ratio in model * support energon dataloader * some pp update Signed-off-by: yaoyu-33 <[email protected]> * fixes Signed-off-by: yaoyu-33 <[email protected]> * fix kv merging Signed-off-by: yaoyu-33 <[email protected]> * fix get_key_value_tensors Signed-off-by: yaoyu-33 <[email protected]> * rename files Signed-off-by: yaoyu-33 <[email protected]> * update to HF style position embedding Signed-off-by: yaoyu-33 <[email protected]> * fix energon dataloader and support batching * update forward args Signed-off-by: yaoyu-33 <[email protected]> * clean up and move to aspect_ratio_ids Signed-off-by: yaoyu-33 <[email protected]> * rename back to language.py Signed-off-by: yaoyu-33 <[email protected]> * fix loss function Signed-off-by: yaoyu-33 <[email protected]> * update and fix energon Signed-off-by: yaoyu-33 <[email protected]> * Add hf import * Fix type * Change config * update energon pretrain Signed-off-by: yaoyu-33 <[email protected]> * clean up * clean up * reformat Signed-off-by: yaoyu-33 <[email protected]> * update inference files for new code * update to instruct * update to instruct * update few names Signed-off-by: yaoyu-33 <[email protected]> * update generation Signed-off-by: yaoyu-33 <[email protected]> * fix importer embedding.weight * few fixes Signed-off-by: yaoyu-33 <[email protected]> * add hf script Signed-off-by: yaoyu-33 <[email protected]> * fix kv import * remove interleaved * fixes and updates Signed-off-by: yaoyu-33 <[email protected]> * lora fixes Signed-off-by: yaoyu-33 <[email protected]> * some code clean ups Signed-off-by: yaoyu-33 <[email protected]> * update training scripts Signed-off-by: yaoyu-33 <[email protected]> * refactors Signed-off-by: yaoyu-33 <[email protected]> * add LoRA finetuning * fixes and nemo update Signed-off-by: yaoyu-33 <[email protected]> * fix importer registering issue by adding 11B and 90B configs * update `decoder_seq_len` Signed-off-by: yaoyu-33 <[email protected]> * science vqa script Signed-off-by: yaoyu-33 <[email protected]> * clean up script name Signed-off-by: yaoyu-33 <[email protected]> * fix ckpt save serialization issue * fix predefined config classes * add num_chunks in input Signed-off-by: yaoyu-33 <[email protected]> * fix format Signed-off-by: yaoyu-33 <[email protected]> * update finetuning scripts for PEFT * add 11b recipe (need NVIDIA#10645 to test) * fix mask generation Signed-off-by: yaoyu-33 <[email protected]> * minor fix code style Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Support no image inference * add llama svqa eval * fix masking Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix generation Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * add 90b recipe and revise 11b recipe * Apply isort and black reformatting Signed-off-by: cuichenx <[email protected]> * clean up typing * add option to disable vision padding * Apply isort and black reformatting Signed-off-by: cuichenx <[email protected]> * base model finetuning (does not work yet) * Apply isort and black reformatting Signed-off-by: cuichenx <[email protected]> * fixed default conversation template config for MLLama * Update svqa * add multinode * bot happy * Apply isort and black reformatting Signed-off-by: cuichenx <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: artbataev <[email protected]> * Perf improvements. Mainly from XAttn mask calculation (NVIDIA#10901) * Perf improvements. Mainly from XAttn mask calculation * Apply isort and black reformatting Signed-off-by: parthmannan <[email protected]> --------- Signed-off-by: parthmannan <[email protected]> Co-authored-by: parthmannan <[email protected]> * fix existing issues Signed-off-by: yaoyu-33 <[email protected]> * fix scripts Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix lora * few fixes for non image support Signed-off-by: yaoyu-33 <[email protected]> * update masking gen Signed-off-by: yaoyu-33 <[email protected]> * update lazy dataset Signed-off-by: yaoyu-33 <[email protected]> * fix data sampler and loading issue Signed-off-by: yaoyu-33 <[email protected]> * Add vlm generation * Apply isort and black reformatting Signed-off-by: meatybobby <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * generation update Signed-off-by: yaoyu-33 <[email protected]> * update lazy dataset Signed-off-by: yaoyu-33 <[email protected]> * Fix _strategy_lib.py Signed-off-by: Yu Yao <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix warning Signed-off-by: yaoyu-33 <[email protected]> * hide vlm examples Signed-off-by: yaoyu-33 <[email protected]> * Revert "Add vlm generation" This reverts commit 4711c75 Signed-off-by: yaoyu-33 <[email protected]> * Fix VisionEncoder multi-batch bug * update mcore parallelism initialization Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Update megatron_init.py Signed-off-by: Yu Yao <[email protected]> * add encoder parallel default config Signed-off-by: yaoyu-33 <[email protected]> * Fix _strategy_lib.py Signed-off-by: Yu Yao <[email protected]> * llm.generate fixes (NVIDIA#10983) * fix context path, disable optimizer init, add tp Signed-off-by: HuiyingLi <[email protected]> * format Signed-off-by: HuiyingLi <[email protected]> * address comments, require user to provide trainer Signed-off-by: HuiyingLi <[email protected]> * minor fix Signed-off-by: HuiyingLi <[email protected]> * minor fixes Signed-off-by: HuiyingLi <[email protected]> --------- Signed-off-by: HuiyingLi <[email protected]> * use __dict__ in check (NVIDIA#11012) * check is_hf_model in leaf module Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * disable getattr alternative path Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * undo; Signed-off-by: Alexandros Koumparoulis <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]> * LoRA support for HF::AutoModelForCausalLM (NVIDIA#10982) * add LinearAdapter Signed-off-by: Alexandros Koumparoulis <[email protected]> * add hf lora example Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove unused imports Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * subclass mixin Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove stale imports Signed-off-by: Alexandros Koumparoulis <[email protected]> * undo Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix scale Signed-off-by: Alexandros Koumparoulis <[email protected]> * regex selector for peft Signed-off-by: Alexandros Koumparoulis <[email protected]> * move lora Signed-off-by: Alexandros Koumparoulis <[email protected]> * fmt Signed-off-by: Alexandros Koumparoulis <[email protected]> * hf_auto_model_for_causal_lm finetune recipe Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]> * Change default for always_save_context to True (NVIDIA#11014) Signed-off-by: Abhishree <[email protected]> Co-authored-by: Pablo Garay <[email protected]> * Add a build option to load_context (NVIDIA#10713) * Add a build option to load_context Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> * Adding test Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> * Trying to fix failing CPU test Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> * cherry-pick fix Signed-off-by: Alexandros Koumparoulis <[email protected]> --------- Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> * Fix pip install (NVIDIA#11026) * Move AutoTokenizer inline Signed-off-by: Marc Romeyn <[email protected]> * Move einops to common requirements Signed-off-by: Marc Romeyn <[email protected]> * Move AutoTokenizer import to top-level again in fine_tuning Signed-off-by: Marc Romeyn <[email protected]> * Move megatron init inside nemo.lightning Signed-off-by: Marc Romeyn <[email protected]> * Make megatron_lazy_init_context work when transformer-engine is not installed Signed-off-by: Marc Romeyn <[email protected]> * Only import get_nmt_tokenizer when needed Signed-off-by: Marc Romeyn <[email protected]> * Apply isort and black reformatting Signed-off-by: marcromeyn <[email protected]> --------- Signed-off-by: Marc Romeyn <[email protected]> Signed-off-by: marcromeyn <[email protected]> Co-authored-by: marcromeyn <[email protected]> * [WIP] Add docs for NEST SSL (NVIDIA#10804) * add docs Signed-off-by: stevehuang52 <[email protected]> * update doc and fix missing param Signed-off-by: stevehuang52 <[email protected]> --------- Signed-off-by: stevehuang52 <[email protected]> * Change dist ckpt defaults (NVIDIA#10913) * Enable ckpt features by default (async ckpt), ckpt every 15mins and reduce preemption time to 1min Signed-off-by: Shriya Palsamudram <[email protected]> * fix ssm tests Signed-off-by: Shriya Palsamudram <[email protected]> * Make note that ckpt_async_save is disabled for SSMs Signed-off-by: Shriya Palsamudram <[email protected]> * Enable async ckpt for SSMs with fix Signed-off-by: Shriya Palsamudram <[email protected]> * Disable async ckpt in the peft test as it is a known bug, add note. Signed-off-by: Shriya Palsamudram <[email protected]> * Fix failing unit tests Signed-off-by: Shriya Palsamudram <[email protected]> * Ashors/peft async ckpt (NVIDIA#11010) * [WIP] prototype for supporting async checkpointing with peft Signed-off-by: ashors1 <[email protected]> Signed-off-by: Shriya Palsamudram <[email protected]> * Enable async ckpt for the peft test Signed-off-by: Shriya Palsamudram <[email protected]> * Fix peft setup test Signed-off-by: Shriya Palsamudram <[email protected]> --------- Signed-off-by: Shriya Palsamudram <[email protected]> Signed-off-by: ashors1 <[email protected]> Co-authored-by: ataghibakhsh <[email protected]> * Akoumparouli/mixtral recipe fix r2.0.0 (NVIDIA#10994) * Mixtral TP8 EP1 Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]> * Fix _strategy_lib tests (NVIDIA#11033) * fix world size and don't mock Signed-off-by: Maanu Grover <[email protected]> * cleanup global state Signed-off-by: Maanu Grover <[email protected]> * check app state instead Signed-off-by: Maanu Grover <[email protected]> * fix syntax nemo logger test Signed-off-by: Maanu Grover <[email protected]> --------- Signed-off-by: Maanu Grover <[email protected]> * Update `BaseMegatronSampler` for compatibility with PTL's `_BatchProgress` (NVIDIA#11016) * Revert "[NeMo-UX] Use custom `BatchProgress` class which does not restore states (NVIDIA#10383)" This reverts commit b5798de. * make megatron sampler return the total number of batches in the dataset Signed-off-by: ashors1 <[email protected]> --------- Signed-off-by: ashors1 <[email protected]> * PTQ example for NeMo 2.0 (NVIDIA#10642) * initial commit Signed-off-by: Piotr Kaminski <[email protected]> * create Quantizer for NeMo 2.0 Signed-off-by: Piotr Kaminski <[email protected]> * refactor Signed-off-by: Piotr Kaminski <[email protected]> * Call quantize on an unwrapped mcore model Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * Add tests, adjust unwrapping Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * fix export Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * Apply isort and black reformatting Signed-off-by: artbataev <[email protected]> * Fix output_path argument for HF import Signed-off-by: Piotr Kamiński <[email protected]> * fix fabric ckpt loading Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * code review suggestions Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * remove unused import Signed-off-by: Piotr Kaminski <[email protected]> * use cnn dataset in github ci Signed-off-by: Piotr Kaminski <[email protected]> * applied code review Signed-off-by: Piotr Kaminski <[email protected]> * code review changes Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * simplify interface for data iterator Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * (partial) PP fix Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> --------- Signed-off-by: Piotr Kaminski <[email protected]> Signed-off-by: Laplasjan107 <[email protected]> Signed-off-by: Piotr Kamiński <[email protected]> Signed-off-by: artbataev <[email protected]> Co-authored-by: Piotr Kaminski <[email protected]> Co-authored-by: Laplasjan107 <[email protected]> Co-authored-by: artbataev <[email protected]> * TDT compute timestamps option and Extra Whitespace handling for SPE (NVIDIA#10875) * add token duration Signed-off-by: monica-sekoyan <[email protected]> * revert rnnt change Signed-off-by: monica-sekoyan <[email protected]> * add remove_extra_whitespaces arg to spe tokenizer Signed-off-by: monica-sekoyan <[email protected]> * add token duration retrieval Signed-off-by: monica-sekoyan <[email protected]> * add ignore_extra_whitespace to spe Signed-off-by: monica-sekoyan <[email protected]> * add compute_timestamp support for tdt Signed-off-by: monica-sekoyan <[email protected]> * fix config field name Signed-off-by: monica-sekoyan <[email protected]> * add refinement for tdt timestamps Signed-off-by: monica-sekoyan <[email protected]> * add segments timestamp support and refinement for ctc Signed-off-by: monica-sekoyan <[email protected]> * modify tests for ctc decoding timestamps Signed-off-by: monica-sekoyan <[email protected]> * add rnnt timestamp tests Signed-off-by: monica-sekoyan <[email protected]> * updated doc Signed-off-by: monica-sekoyan <[email protected]> * fix in test Signed-off-by: monica-sekoyan <[email protected]> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <[email protected]> * fix of unicode char Signed-off-by: monica-sekoyan <[email protected]> * fix rnnt_decoding test Signed-off-by: monica-sekoyan <[email protected]> * workaround for tesst tokenizer Signed-off-by: monica-sekoyan <[email protected]> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <[email protected]> * modify segments formation Signed-off-by: monica-sekoyan <[email protected]> * modify segments for ctc Signed-off-by: monica-sekoyan <[email protected]> * fix in ctc refinement Signed-off-by: monica-sekoyan <[email protected]> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <[email protected]> * minor changes Signed-off-by: monica-sekoyan <[email protected]> * reverse offset change Signed-off-by: monica-sekoyan <[email protected]> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <[email protected]> * warning mode=once Signed-off-by: monica-sekoyan <[email protected]> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <[email protected]> * make ignore_extrawhitespaces false Signed-off-by: monica-sekoyan <[email protected]> * minor changes Signed-off-by: monica-sekoyan <[email protected]> * adjust changes to the tests Signed-off-by: monica-sekoyan <[email protected]> * modify prompt_formatter tests Signed-off-by: monica-sekoyan <[email protected]> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <[email protected]> --------- Signed-off-by: monica-sekoyan <[email protected]> Signed-off-by: monica-sekoyan <[email protected]> Co-authored-by: monica-sekoyan <[email protected]> * Basic online dynamic FP8 quantization with vLLM (NVIDIA#10904) * Basic online dynamic quantization with vLLM Signed-off-by: Jan Lasek <[email protected]> * Apply isort and black reformatting Signed-off-by: janekl <[email protected]> * vllm 0.6.3 updates Signed-off-by: Jan Lasek <[email protected]> * Pass quantization param in deploy_vllm_triton.py script Signed-off-by: Jan Lasek <[email protected]> --------- Signed-off-by: Jan Lasek <[email protected]> Signed-off-by: janekl <[email protected]> Co-authored-by: janekl <[email protected]> * ci: Improve VM maintenance (NVIDIA#10758) * ci: Improve VM maintenance Signed-off-by: Oliver Koenig <[email protected]> * rename stuff Signed-off-by: Oliver Koenig <[email protected]> * title Signed-off-by: Oliver Koenig <[email protected]> * use team Signed-off-by: Oliver Koenig <[email protected]> * run on failure too Signed-off-by: Oliver Koenig <[email protected]> * fix Signed-off-by: Oliver Koenig <[email protected]> * yrdy Signed-off-by: Oliver Koenig <[email protected]> * f Signed-off-by: Oliver Koenig <[email protected]> * test Signed-off-by: Oliver Koenig <[email protected]> * fix Signed-off-by: Oliver Koenig <[email protected]> * f Signed-off-by: Oliver Koenig <[email protected]> * f Signed-off-by: Oliver Koenig <[email protected]> * f Signed-off-by: Oliver Koenig <[email protected]> --------- Signed-off-by: Oliver Koenig <[email protected]> * neva update Signed-off-by: yaoyu-33 <[email protected]> * Add comment for vision transpose * update megatron_init.py inside lightning Signed-off-by: yaoyu-33 <[email protected]> * Fix PP Signed-off-by: yaoyu-33 <[email protected]> * add examples Signed-off-by: yaoyu-33 <[email protected]> * fix test Signed-off-by: yaoyu-33 <[email protected]> * try fix test Signed-off-by: yaoyu-33 <[email protected]> * try fix test Signed-off-by: yaoyu-33 <[email protected]> * Fix megatron megatron_init.py dp Signed-off-by: Yu Yao <[email protected]> * Update lightning megatron_init.py dp Signed-off-by: Yu Yao <[email protected]> * make it possible to update pre_preprocess and post_process for llm, required in vlm Signed-off-by: yaoyu-33 <[email protected]> * Fixes for neva to run with PP Signed-off-by: yaoyu-33 <[email protected]> * Add mcore vit support, and checkpoint conversion Signed-off-by: yaoyu-33 <[email protected]> * fix checkpoint loading for epp Signed-off-by: yaoyu-33 <[email protected]> * update script Signed-off-by: yaoyu-33 <[email protected]> * rename llama to mllama folder name Signed-off-by: yaoyu-33 <[email protected]> * update to attention bias Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * added datamodule for llava-next * modified state dict transform * neva model changes to support llava-next * remove accidentally checked in files Signed-off-by: Yashaswi Karnati <[email protected]> * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * remove unused imports * added io_init to not save task_encoder and image_processor * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * added scripts for pretrain and finetune Signed-off-by: Yashaswi Karnati <[email protected]> * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * generation example * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * small change in llava next example * llava next end-end train * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * finetune changes * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * finetune debug changes * update dropout to 0 Signed-off-by: yaoyu-33 <[email protected]> * added example generation script * added doc strings, formating, remove debug statemens and unsued imports * remove example scripts * fix attention bias Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * remove disable_vision_padding since we now have a fix Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Update init for mllama Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Address comments Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix copyright title Signed-off-by: yaoyu-33 <[email protected]> * multiple fixes Signed-off-by: yaoyu-33 <[email protected]> * bug fix Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix code scan Signed-off-by: yaoyu-33 <[email protected]> * Fix for SP Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * update vision code Signed-off-by: yaoyu-33 <[email protected]> * revert attention bias changes until latest MLM code got merged Signed-off-by: yaoyu-33 <[email protected]> * fix warning Signed-off-by: yaoyu-33 <[email protected]> * Turn off system message check, as it's "" now Signed-off-by: yaoyu-33 <[email protected]> * Update layer spec and add siglip support Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * update pretrain script Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Fix scripts Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * add neva training recipes Signed-off-by: yaoyu-33 <[email protected]> * fix mllama mock ds Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix recipe Signed-off-by: yaoyu-33 <[email protected]> * fix pp Signed-off-by: yaoyu-33 <[email protected]> * scripts update Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * scripts update Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * update config api Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * few updates Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * update 70b Signed-off-by: yaoyu-33 <[email protected]> * hide examples for pr Signed-off-by: yaoyu-33 <[email protected]> * fix few issues Signed-off-by: yaoyu-33 <[email protected]> * add docstring layer spec Signed-off-by: yaoyu-33 <[email protected]> * add docstring to vit config Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix copyright Signed-off-by: yaoyu-33 <[email protected]> * fix Signed-off-by: yaoyu-33 <[email protected]> --------- Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: cuichenx <[email protected]> Signed-off-by: Chen Cui <[email protected]> Signed-off-by: artbataev <[email protected]> Signed-off-by: parthmannan <[email protected]> Signed-off-by: meatybobby <[email protected]> Signed-off-by: Yu Yao <[email protected]> Signed-off-by: HuiyingLi <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Signed-off-by: Abhishree <[email protected]> Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Marc Romeyn <[email protected]> Signed-off-by: marcromeyn <[email protected]> Signed-off-by: stevehuang52 <[email protected]> Signed-off-by: Shriya Palsamudram <[email protected]> Signed-off-by: ashors1 <[email protected]> Signed-off-by: Maanu Grover <[email protected]> Signed-off-by: Piotr Kaminski <[email protected]> Signed-off-by: Laplasjan107 <[email protected]> Signed-off-by: Piotr Kamiński <[email protected]> Signed-off-by: monica-sekoyan <[email protected]> Signed-off-by: monica-sekoyan <[email protected]> Signed-off-by: Jan Lasek <[email protected]> Signed-off-by: janekl <[email protected]> Signed-off-by: Oliver Koenig <[email protected]> Signed-off-by: Yashaswi Karnati <[email protected]> Signed-off-by: yashaswikarnati <[email protected]> Signed-off-by: Yashaswi Karnati <[email protected]> Co-authored-by: Chen Cui <[email protected]> Co-authored-by: Bobby Chen <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Co-authored-by: cuichenx <[email protected]> Co-authored-by: Yashaswi Karnati <[email protected]> Co-authored-by: artbataev <[email protected]> Co-authored-by: Parth Mannan <[email protected]> Co-authored-by: parthmannan <[email protected]> Co-authored-by: meatybobby <[email protected]> Co-authored-by: Huiying <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: akoumpa <[email protected]> Co-authored-by: Abhishree Thittenamane <[email protected]> Co-authored-by: Pablo Garay <[email protected]> Co-authored-by: Marc Romeyn <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: marcromeyn <[email protected]> Co-authored-by: He Huang (Steve) <[email protected]> Co-authored-by: Shriya Rishab <[email protected]> Co-authored-by: ataghibakhsh <[email protected]> Co-authored-by: Maanu Grover <[email protected]> Co-authored-by: Anna Shors <[email protected]> Co-authored-by: Piotr Kamiński <[email protected]> Co-authored-by: Piotr Kaminski <[email protected]> Co-authored-by: Laplasjan107 <[email protected]> Co-authored-by: monica-sekoyan <[email protected]> Co-authored-by: monica-sekoyan <[email protected]> Co-authored-by: Jan Lasek <[email protected]> Co-authored-by: janekl <[email protected]> Co-authored-by: oliver könig <[email protected]> Co-authored-by: Pablo Garay <[email protected]> Co-authored-by: ykarnati <[email protected]> Co-authored-by: Yashaswi Karnati <[email protected]> Co-authored-by: yashaswikarnati <[email protected]>
youngeunkwon0405
pushed a commit
to youngeunkwon0405/NeMo
that referenced
this pull request
Feb 10, 2025
* evian3 update Signed-off-by: yaoyu-33 <[email protected]> * add encoder parallel default config Signed-off-by: yaoyu-33 <[email protected]> * add encoder parallel default config Signed-off-by: yaoyu-33 <[email protected]> * clean up Signed-off-by: yaoyu-33 <[email protected]> * add aspect ratio in model * support energon dataloader * some pp update Signed-off-by: yaoyu-33 <[email protected]> * fixes Signed-off-by: yaoyu-33 <[email protected]> * fix kv merging Signed-off-by: yaoyu-33 <[email protected]> * fix get_key_value_tensors Signed-off-by: yaoyu-33 <[email protected]> * rename files Signed-off-by: yaoyu-33 <[email protected]> * update to HF style position embedding Signed-off-by: yaoyu-33 <[email protected]> * fix energon dataloader and support batching * update forward args Signed-off-by: yaoyu-33 <[email protected]> * clean up and move to aspect_ratio_ids Signed-off-by: yaoyu-33 <[email protected]> * rename back to language.py Signed-off-by: yaoyu-33 <[email protected]> * fix loss function Signed-off-by: yaoyu-33 <[email protected]> * update and fix energon Signed-off-by: yaoyu-33 <[email protected]> * Add hf import * Fix type * Change config * update energon pretrain Signed-off-by: yaoyu-33 <[email protected]> * clean up * clean up * reformat Signed-off-by: yaoyu-33 <[email protected]> * update inference files for new code * update to instruct * update to instruct * update few names Signed-off-by: yaoyu-33 <[email protected]> * update generation Signed-off-by: yaoyu-33 <[email protected]> * fix importer embedding.weight * few fixes Signed-off-by: yaoyu-33 <[email protected]> * add hf script Signed-off-by: yaoyu-33 <[email protected]> * fix kv import * remove interleaved * fixes and updates Signed-off-by: yaoyu-33 <[email protected]> * lora fixes Signed-off-by: yaoyu-33 <[email protected]> * some code clean ups Signed-off-by: yaoyu-33 <[email protected]> * update training scripts Signed-off-by: yaoyu-33 <[email protected]> * refactors Signed-off-by: yaoyu-33 <[email protected]> * add LoRA finetuning * fixes and nemo update Signed-off-by: yaoyu-33 <[email protected]> * fix importer registering issue by adding 11B and 90B configs * update `decoder_seq_len` Signed-off-by: yaoyu-33 <[email protected]> * science vqa script Signed-off-by: yaoyu-33 <[email protected]> * clean up script name Signed-off-by: yaoyu-33 <[email protected]> * fix ckpt save serialization issue * fix predefined config classes * add num_chunks in input Signed-off-by: yaoyu-33 <[email protected]> * fix format Signed-off-by: yaoyu-33 <[email protected]> * update finetuning scripts for PEFT * add 11b recipe (need NVIDIA#10645 to test) * fix mask generation Signed-off-by: yaoyu-33 <[email protected]> * minor fix code style Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Support no image inference * add llama svqa eval * fix masking Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix generation Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * add 90b recipe and revise 11b recipe * Apply isort and black reformatting Signed-off-by: cuichenx <[email protected]> * clean up typing * add option to disable vision padding * Apply isort and black reformatting Signed-off-by: cuichenx <[email protected]> * base model finetuning (does not work yet) * Apply isort and black reformatting Signed-off-by: cuichenx <[email protected]> * fixed default conversation template config for MLLama * Update svqa * add multinode * bot happy * Apply isort and black reformatting Signed-off-by: cuichenx <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: artbataev <[email protected]> * Perf improvements. Mainly from XAttn mask calculation (NVIDIA#10901) * Perf improvements. Mainly from XAttn mask calculation * Apply isort and black reformatting Signed-off-by: parthmannan <[email protected]> --------- Signed-off-by: parthmannan <[email protected]> Co-authored-by: parthmannan <[email protected]> * fix existing issues Signed-off-by: yaoyu-33 <[email protected]> * fix scripts Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix lora * few fixes for non image support Signed-off-by: yaoyu-33 <[email protected]> * update masking gen Signed-off-by: yaoyu-33 <[email protected]> * update lazy dataset Signed-off-by: yaoyu-33 <[email protected]> * fix data sampler and loading issue Signed-off-by: yaoyu-33 <[email protected]> * Add vlm generation * Apply isort and black reformatting Signed-off-by: meatybobby <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * generation update Signed-off-by: yaoyu-33 <[email protected]> * update lazy dataset Signed-off-by: yaoyu-33 <[email protected]> * Fix _strategy_lib.py Signed-off-by: Yu Yao <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix warning Signed-off-by: yaoyu-33 <[email protected]> * hide vlm examples Signed-off-by: yaoyu-33 <[email protected]> * Revert "Add vlm generation" This reverts commit 4711c75 Signed-off-by: yaoyu-33 <[email protected]> * Fix VisionEncoder multi-batch bug * update mcore parallelism initialization Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Update megatron_init.py Signed-off-by: Yu Yao <[email protected]> * add encoder parallel default config Signed-off-by: yaoyu-33 <[email protected]> * Fix _strategy_lib.py Signed-off-by: Yu Yao <[email protected]> * llm.generate fixes (NVIDIA#10983) * fix context path, disable optimizer init, add tp Signed-off-by: HuiyingLi <[email protected]> * format Signed-off-by: HuiyingLi <[email protected]> * address comments, require user to provide trainer Signed-off-by: HuiyingLi <[email protected]> * minor fix Signed-off-by: HuiyingLi <[email protected]> * minor fixes Signed-off-by: HuiyingLi <[email protected]> --------- Signed-off-by: HuiyingLi <[email protected]> * use __dict__ in check (NVIDIA#11012) * check is_hf_model in leaf module Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * disable getattr alternative path Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * undo; Signed-off-by: Alexandros Koumparoulis <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]> * LoRA support for HF::AutoModelForCausalLM (NVIDIA#10982) * add LinearAdapter Signed-off-by: Alexandros Koumparoulis <[email protected]> * add hf lora example Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove unused imports Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * subclass mixin Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove stale imports Signed-off-by: Alexandros Koumparoulis <[email protected]> * undo Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix scale Signed-off-by: Alexandros Koumparoulis <[email protected]> * regex selector for peft Signed-off-by: Alexandros Koumparoulis <[email protected]> * move lora Signed-off-by: Alexandros Koumparoulis <[email protected]> * fmt Signed-off-by: Alexandros Koumparoulis <[email protected]> * hf_auto_model_for_causal_lm finetune recipe Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]> * Change default for always_save_context to True (NVIDIA#11014) Signed-off-by: Abhishree <[email protected]> Co-authored-by: Pablo Garay <[email protected]> * Add a build option to load_context (NVIDIA#10713) * Add a build option to load_context Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> * Adding test Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> * Trying to fix failing CPU test Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> * cherry-pick fix Signed-off-by: Alexandros Koumparoulis <[email protected]> --------- Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> * Fix pip install (NVIDIA#11026) * Move AutoTokenizer inline Signed-off-by: Marc Romeyn <[email protected]> * Move einops to common requirements Signed-off-by: Marc Romeyn <[email protected]> * Move AutoTokenizer import to top-level again in fine_tuning Signed-off-by: Marc Romeyn <[email protected]> * Move megatron init inside nemo.lightning Signed-off-by: Marc Romeyn <[email protected]> * Make megatron_lazy_init_context work when transformer-engine is not installed Signed-off-by: Marc Romeyn <[email protected]> * Only import get_nmt_tokenizer when needed Signed-off-by: Marc Romeyn <[email protected]> * Apply isort and black reformatting Signed-off-by: marcromeyn <[email protected]> --------- Signed-off-by: Marc Romeyn <[email protected]> Signed-off-by: marcromeyn <[email protected]> Co-authored-by: marcromeyn <[email protected]> * [WIP] Add docs for NEST SSL (NVIDIA#10804) * add docs Signed-off-by: stevehuang52 <[email protected]> * update doc and fix missing param Signed-off-by: stevehuang52 <[email protected]> --------- Signed-off-by: stevehuang52 <[email protected]> * Change dist ckpt defaults (NVIDIA#10913) * Enable ckpt features by default (async ckpt), ckpt every 15mins and reduce preemption time to 1min Signed-off-by: Shriya Palsamudram <[email protected]> * fix ssm tests Signed-off-by: Shriya Palsamudram <[email protected]> * Make note that ckpt_async_save is disabled for SSMs Signed-off-by: Shriya Palsamudram <[email protected]> * Enable async ckpt for SSMs with fix Signed-off-by: Shriya Palsamudram <[email protected]> * Disable async ckpt in the peft test as it is a known bug, add note. Signed-off-by: Shriya Palsamudram <[email protected]> * Fix failing unit tests Signed-off-by: Shriya Palsamudram <[email protected]> * Ashors/peft async ckpt (NVIDIA#11010) * [WIP] prototype for supporting async checkpointing with peft Signed-off-by: ashors1 <[email protected]> Signed-off-by: Shriya Palsamudram <[email protected]> * Enable async ckpt for the peft test Signed-off-by: Shriya Palsamudram <[email protected]> * Fix peft setup test Signed-off-by: Shriya Palsamudram <[email protected]> --------- Signed-off-by: Shriya Palsamudram <[email protected]> Signed-off-by: ashors1 <[email protected]> Co-authored-by: ataghibakhsh <[email protected]> * Akoumparouli/mixtral recipe fix r2.0.0 (NVIDIA#10994) * Mixtral TP8 EP1 Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]> * Fix _strategy_lib tests (NVIDIA#11033) * fix world size and don't mock Signed-off-by: Maanu Grover <[email protected]> * cleanup global state Signed-off-by: Maanu Grover <[email protected]> * check app state instead Signed-off-by: Maanu Grover <[email protected]> * fix syntax nemo logger test Signed-off-by: Maanu Grover <[email protected]> --------- Signed-off-by: Maanu Grover <[email protected]> * Update `BaseMegatronSampler` for compatibility with PTL's `_BatchProgress` (NVIDIA#11016) * Revert "[NeMo-UX] Use custom `BatchProgress` class which does not restore states (NVIDIA#10383)" This reverts commit b5798de. * make megatron sampler return the total number of batches in the dataset Signed-off-by: ashors1 <[email protected]> --------- Signed-off-by: ashors1 <[email protected]> * PTQ example for NeMo 2.0 (NVIDIA#10642) * initial commit Signed-off-by: Piotr Kaminski <[email protected]> * create Quantizer for NeMo 2.0 Signed-off-by: Piotr Kaminski <[email protected]> * refactor Signed-off-by: Piotr Kaminski <[email protected]> * Call quantize on an unwrapped mcore model Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * Add tests, adjust unwrapping Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * fix export Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * Apply isort and black reformatting Signed-off-by: artbataev <[email protected]> * Fix output_path argument for HF import Signed-off-by: Piotr Kamiński <[email protected]> * fix fabric ckpt loading Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * code review suggestions Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * remove unused import Signed-off-by: Piotr Kaminski <[email protected]> * use cnn dataset in github ci Signed-off-by: Piotr Kaminski <[email protected]> * applied code review Signed-off-by: Piotr Kaminski <[email protected]> * code review changes Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * simplify interface for data iterator Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> * (partial) PP fix Signed-off-by: Piotr Kaminski <[email protected]> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <[email protected]> --------- Signed-off-by: Piotr Kaminski <[email protected]> Signed-off-by: Laplasjan107 <[email protected]> Signed-off-by: Piotr Kamiński <[email protected]> Signed-off-by: artbataev <[email protected]> Co-authored-by: Piotr Kaminski <[email protected]> Co-authored-by: Laplasjan107 <[email protected]> Co-authored-by: artbataev <[email protected]> * TDT compute timestamps option and Extra Whitespace handling for SPE (NVIDIA#10875) * add token duration Signed-off-by: monica-sekoyan <[email protected]> * revert rnnt change Signed-off-by: monica-sekoyan <[email protected]> * add remove_extra_whitespaces arg to spe tokenizer Signed-off-by: monica-sekoyan <[email protected]> * add token duration retrieval Signed-off-by: monica-sekoyan <[email protected]> * add ignore_extra_whitespace to spe Signed-off-by: monica-sekoyan <[email protected]> * add compute_timestamp support for tdt Signed-off-by: monica-sekoyan <[email protected]> * fix config field name Signed-off-by: monica-sekoyan <[email protected]> * add refinement for tdt timestamps Signed-off-by: monica-sekoyan <[email protected]> * add segments timestamp support and refinement for ctc Signed-off-by: monica-sekoyan <[email protected]> * modify tests for ctc decoding timestamps Signed-off-by: monica-sekoyan <[email protected]> * add rnnt timestamp tests Signed-off-by: monica-sekoyan <[email protected]> * updated doc Signed-off-by: monica-sekoyan <[email protected]> * fix in test Signed-off-by: monica-sekoyan <[email protected]> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <[email protected]> * fix of unicode char Signed-off-by: monica-sekoyan <[email protected]> * fix rnnt_decoding test Signed-off-by: monica-sekoyan <[email protected]> * workaround for tesst tokenizer Signed-off-by: monica-sekoyan <[email protected]> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <[email protected]> * modify segments formation Signed-off-by: monica-sekoyan <[email protected]> * modify segments for ctc Signed-off-by: monica-sekoyan <[email protected]> * fix in ctc refinement Signed-off-by: monica-sekoyan <[email protected]> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <[email protected]> * minor changes Signed-off-by: monica-sekoyan <[email protected]> * reverse offset change Signed-off-by: monica-sekoyan <[email protected]> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <[email protected]> * warning mode=once Signed-off-by: monica-sekoyan <[email protected]> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <[email protected]> * make ignore_extrawhitespaces false Signed-off-by: monica-sekoyan <[email protected]> * minor changes Signed-off-by: monica-sekoyan <[email protected]> * adjust changes to the tests Signed-off-by: monica-sekoyan <[email protected]> * modify prompt_formatter tests Signed-off-by: monica-sekoyan <[email protected]> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <[email protected]> --------- Signed-off-by: monica-sekoyan <[email protected]> Signed-off-by: monica-sekoyan <[email protected]> Co-authored-by: monica-sekoyan <[email protected]> * Basic online dynamic FP8 quantization with vLLM (NVIDIA#10904) * Basic online dynamic quantization with vLLM Signed-off-by: Jan Lasek <[email protected]> * Apply isort and black reformatting Signed-off-by: janekl <[email protected]> * vllm 0.6.3 updates Signed-off-by: Jan Lasek <[email protected]> * Pass quantization param in deploy_vllm_triton.py script Signed-off-by: Jan Lasek <[email protected]> --------- Signed-off-by: Jan Lasek <[email protected]> Signed-off-by: janekl <[email protected]> Co-authored-by: janekl <[email protected]> * ci: Improve VM maintenance (NVIDIA#10758) * ci: Improve VM maintenance Signed-off-by: Oliver Koenig <[email protected]> * rename stuff Signed-off-by: Oliver Koenig <[email protected]> * title Signed-off-by: Oliver Koenig <[email protected]> * use team Signed-off-by: Oliver Koenig <[email protected]> * run on failure too Signed-off-by: Oliver Koenig <[email protected]> * fix Signed-off-by: Oliver Koenig <[email protected]> * yrdy Signed-off-by: Oliver Koenig <[email protected]> * f Signed-off-by: Oliver Koenig <[email protected]> * test Signed-off-by: Oliver Koenig <[email protected]> * fix Signed-off-by: Oliver Koenig <[email protected]> * f Signed-off-by: Oliver Koenig <[email protected]> * f Signed-off-by: Oliver Koenig <[email protected]> * f Signed-off-by: Oliver Koenig <[email protected]> --------- Signed-off-by: Oliver Koenig <[email protected]> * neva update Signed-off-by: yaoyu-33 <[email protected]> * Add comment for vision transpose * update megatron_init.py inside lightning Signed-off-by: yaoyu-33 <[email protected]> * Fix PP Signed-off-by: yaoyu-33 <[email protected]> * add examples Signed-off-by: yaoyu-33 <[email protected]> * fix test Signed-off-by: yaoyu-33 <[email protected]> * try fix test Signed-off-by: yaoyu-33 <[email protected]> * try fix test Signed-off-by: yaoyu-33 <[email protected]> * Fix megatron megatron_init.py dp Signed-off-by: Yu Yao <[email protected]> * Update lightning megatron_init.py dp Signed-off-by: Yu Yao <[email protected]> * make it possible to update pre_preprocess and post_process for llm, required in vlm Signed-off-by: yaoyu-33 <[email protected]> * Fixes for neva to run with PP Signed-off-by: yaoyu-33 <[email protected]> * Add mcore vit support, and checkpoint conversion Signed-off-by: yaoyu-33 <[email protected]> * fix checkpoint loading for epp Signed-off-by: yaoyu-33 <[email protected]> * update script Signed-off-by: yaoyu-33 <[email protected]> * rename llama to mllama folder name Signed-off-by: yaoyu-33 <[email protected]> * update to attention bias Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * added datamodule for llava-next * modified state dict transform * neva model changes to support llava-next * remove accidentally checked in files Signed-off-by: Yashaswi Karnati <[email protected]> * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * remove unused imports * added io_init to not save task_encoder and image_processor * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * added scripts for pretrain and finetune Signed-off-by: Yashaswi Karnati <[email protected]> * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * generation example * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * small change in llava next example * llava next end-end train * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * finetune changes * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * finetune debug changes * update dropout to 0 Signed-off-by: yaoyu-33 <[email protected]> * added example generation script * added doc strings, formating, remove debug statemens and unsued imports * remove example scripts * fix attention bias Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * remove disable_vision_padding since we now have a fix Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Update init for mllama Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Address comments Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix copyright title Signed-off-by: yaoyu-33 <[email protected]> * multiple fixes Signed-off-by: yaoyu-33 <[email protected]> * bug fix Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix code scan Signed-off-by: yaoyu-33 <[email protected]> * Fix for SP Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * update vision code Signed-off-by: yaoyu-33 <[email protected]> * revert attention bias changes until latest MLM code got merged Signed-off-by: yaoyu-33 <[email protected]> * fix warning Signed-off-by: yaoyu-33 <[email protected]> * Turn off system message check, as it's "" now Signed-off-by: yaoyu-33 <[email protected]> * Update layer spec and add siglip support Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * update pretrain script Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * Fix scripts Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * add neva training recipes Signed-off-by: yaoyu-33 <[email protected]> * fix mllama mock ds Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix recipe Signed-off-by: yaoyu-33 <[email protected]> * fix pp Signed-off-by: yaoyu-33 <[email protected]> * scripts update Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * scripts update Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * update config api Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * few updates Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * update 70b Signed-off-by: yaoyu-33 <[email protected]> * hide examples for pr Signed-off-by: yaoyu-33 <[email protected]> * fix few issues Signed-off-by: yaoyu-33 <[email protected]> * add docstring layer spec Signed-off-by: yaoyu-33 <[email protected]> * add docstring to vit config Signed-off-by: yaoyu-33 <[email protected]> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <[email protected]> * fix copyright Signed-off-by: yaoyu-33 <[email protected]> * fix Signed-off-by: yaoyu-33 <[email protected]> --------- Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: yaoyu-33 <[email protected]> Signed-off-by: cuichenx <[email protected]> Signed-off-by: Chen Cui <[email protected]> Signed-off-by: artbataev <[email protected]> Signed-off-by: parthmannan <[email protected]> Signed-off-by: meatybobby <[email protected]> Signed-off-by: Yu Yao <[email protected]> Signed-off-by: HuiyingLi <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Signed-off-by: Abhishree <[email protected]> Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Marc Romeyn <[email protected]> Signed-off-by: marcromeyn <[email protected]> Signed-off-by: stevehuang52 <[email protected]> Signed-off-by: Shriya Palsamudram <[email protected]> Signed-off-by: ashors1 <[email protected]> Signed-off-by: Maanu Grover <[email protected]> Signed-off-by: Piotr Kaminski <[email protected]> Signed-off-by: Laplasjan107 <[email protected]> Signed-off-by: Piotr Kamiński <[email protected]> Signed-off-by: monica-sekoyan <[email protected]> Signed-off-by: monica-sekoyan <[email protected]> Signed-off-by: Jan Lasek <[email protected]> Signed-off-by: janekl <[email protected]> Signed-off-by: Oliver Koenig <[email protected]> Signed-off-by: Yashaswi Karnati <[email protected]> Signed-off-by: yashaswikarnati <[email protected]> Signed-off-by: Yashaswi Karnati <[email protected]> Co-authored-by: Chen Cui <[email protected]> Co-authored-by: Bobby Chen <[email protected]> Co-authored-by: yaoyu-33 <[email protected]> Co-authored-by: cuichenx <[email protected]> Co-authored-by: Yashaswi Karnati <[email protected]> Co-authored-by: artbataev <[email protected]> Co-authored-by: Parth Mannan <[email protected]> Co-authored-by: parthmannan <[email protected]> Co-authored-by: meatybobby <[email protected]> Co-authored-by: Huiying <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: akoumpa <[email protected]> Co-authored-by: Abhishree Thittenamane <[email protected]> Co-authored-by: Pablo Garay <[email protected]> Co-authored-by: Marc Romeyn <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: marcromeyn <[email protected]> Co-authored-by: He Huang (Steve) <[email protected]> Co-authored-by: Shriya Rishab <[email protected]> Co-authored-by: ataghibakhsh <[email protected]> Co-authored-by: Maanu Grover <[email protected]> Co-authored-by: Anna Shors <[email protected]> Co-authored-by: Piotr Kamiński <[email protected]> Co-authored-by: Piotr Kaminski <[email protected]> Co-authored-by: Laplasjan107 <[email protected]> Co-authored-by: monica-sekoyan <[email protected]> Co-authored-by: monica-sekoyan <[email protected]> Co-authored-by: Jan Lasek <[email protected]> Co-authored-by: janekl <[email protected]> Co-authored-by: oliver könig <[email protected]> Co-authored-by: Pablo Garay <[email protected]> Co-authored-by: ykarnati <[email protected]> Co-authored-by: Yashaswi Karnati <[email protected]> Co-authored-by: yashaswikarnati <[email protected]> Signed-off-by: Youngeun Kwon <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Add support for enabling PP in neva training. 2 Options are supplied: (1) vit share 1st pp stage with llm, encoder_pp=0. (2) vit takes entire 1st pp stage with llm, encoder_pp=1.
Collection: [multimodal]
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information