Support Audio Extraction from MP4 Video for Nemotron Nano VL#35539

Merged

vllm-bot merged 40 commits intovllm-project:mainfrom

netanel-haber:feature/support-audio-from-mp4-video

Mar 4, 2026

Commits on Mar 2, 2026

add support for transformers parakeet audio encoder for nemotron-nano-v3 avlm

netanel-haber
authored and
Andrii Skliar
committed
move ParakeetConfig and ExtractorConfig to vllm/transformers_utils/configs/parakeet.py

netanel-haber
authored and
Andrii Skliar
committed
the model starts at least

netanel-haber
authored and
Andrii Skliar
committed
support bs>1?

netanel-haber
authored and
Andrii Skliar
committed
support bs>1

netanel-haber
authored and
Andrii Skliar
committed
encase with <so_start> and <so_end>

netanel-haber
authored and
Andrii Skliar
committed
don't truncate hop_length remainder, pad to 0.1s instead

netanel-haber
authored and
Andrii Skliar
committed
allow configuring static and dynamic resolution per request, via max_num_tiles, for passing images as video frames

netanel-haber
authored and
Andrii Skliar
committed
initial implementation

Andrii Skliar
authored and
Andrii Skliar
committed
Refactor audio extraction from video bytes to use PyAV instead of librosa, improving performance and compatibility with CUDA. Cleaned up related code and removed unused parameters. Updated .gitigno…

Andrii Skliar
authored and
Andrii Skliar
committed
Enhance multimodal processing by adding support for `mm_processor_kwargs` in various components, allowing for more flexible video handling. Updated `.gitignore` to remove exclusions for asset files…
Andrii
committed
Ensure audio extractor is not None in NanoNemotronVLMultiModalProcessor and remove redundant assertion in NanoNemotronVLDummyInputsBuilder for improved clarity and consistency.
Andrii
committed
Enhance error handling for audio extraction and video loading in multimodal processing. Added checks for empty video bytes in `extract_audio_from_video_bytes` and `NanoNemotronVLMultiModalProcessor…

root
authored and
Andrii Skliar
committed
Refactor video loading methods in `MediaConnector` and `VideoMediaIO` for improved readability. Removed redundant assertions and streamlined parameter handling for video bytes. Updated `load_bytes`…
Andrii
committed
Refactor audio handling in by removing placeholder imports and directly importing necessary libraries. Simplify method calls by eliminating redundant parameters. Clean up by removing unnecessary…

Andrii Skliar
authored and
Andrii Skliar
committed
Enhance prompt handling in by adding a check for prompt type. If the prompt is not a string, it is decoded using the tokenizer to ensure compatibility with audio processing. This improves flexibil…

root
authored and
Andrii Skliar
committed
Refactor code for improved readability by consolidating multi-line statements into single lines across various modules. This includes adjustments in , , , , , and to enhance clarity and maintainab…

root
authored and
Andrii Skliar
committed
add av to audio setup

Andrii Skliar
authored and
Andrii Skliar
committed
Reverted unnecessary changes; Fixed video loading
Andrii
committed
Add get_data_parser method to NanoNemotronVLProcessingInfo and update data parser usage in NanoNemotronVLMultiModalProcessor
Andrii
committed
Refactor chat message parsing by updating content format type annotations for consistency and improving readability in AsyncMultiModalContentParser. Consolidated multi-line statements into single l…
Andrii
committed
Refactor video handling in AsyncMultiModalContentParser and NanoNemotronVLMultiModalProcessor. Updated method signatures to improve clarity and consistency in parameter usage. Enhanced MediaConnect…

Andrii Skliar
authored and
Andrii Skliar
committed
Refactor conditional statements in NanoNemotronVLMultiModalProcessor for improved readability. Consolidated multi-line conditions into single lines to enhance clarity in the handling of audio and v…
Andrii
committed
Update vllm/model_executor/models/nano_nemotron_vl.py

authored and
Andrii Skliar
committed
comment clarifying parakeet.py is a dependency of nano_nemotron_vl.py

netanel-haber
authored and
Andrii Skliar
committed
Enhance error handling for audio extraction and video loading in multimodal processing. Added checks for empty video bytes in `extract_audio_from_video_bytes` and `NanoNemotronVLMultiModalProcessor…

root
authored and
Andrii Skliar
committed
Refactor code style in `nano_nemotron_vl.py` for improved readability. Adjusted import statements and formatted conditional checks for audio extraction from video, ensuring consistent indentation a…
Andrii
committed
Add NemotronVLVideoBackend for enhanced video loading and improve audio extraction error handling
Andrii
committed
[Attention] FA4 integration (#32974)

authored and
Andrii Skliar
committed
[Misc] Cleanup useless `current_platform` import (#35715)

wangxiyuan
authored and
Andrii Skliar
committed
[Feat] Add NemotronHNanoVLV2 model configuration and update video metadata handling

Andrii Skliar
authored and
Andrii Skliar
committed
Format w/ pre-commit
Andrii
committed
[Refactor] Simplify video fetching in MultiModalContentParser and MediaConnector

root
authored and
Andrii Skliar
committed
run pre-commit formatting
Andrii Skliar
committed
[Refactor] Update video metadata handling in NanoNemotronVLMultiModalProcessor
Andrii Skliar
committed
[Refactor] Simplify MultiModalItemTracker initialization and update video loading method
Andrii Skliar
committed
undo unnecessary changes
Andrii Skliar
committed

Commits on Mar 3, 2026

[Refactor] Revise NemotronHNanoVLV2Config and update video backend handling
Andrii
committed
Optimize imports
Andrii Skliar
committed

Commits on Mar 4, 2026

Merge branch 'main' into feature/support-audio-from-mp4-video
askliar
authored