Add Nemotron VLM support in video captioning - #1160

Merged

weijiac0619 merged 22 commits into

NVIDIA-NeMo:mainfrom

ronjer30:feature/nemotron-vlm

Apr 1, 2026

Commits on Jan 28, 2026

Add NemotronH VLM support for video captioning
ronjer30
committed
ce0a6af
Refactor caption generation and preparation to remove qwen_llm_input in favor of a unified llm_inputs structure. Update related tests and examples to reflect this change
ronjer30
committed
8bf8f34
Updated to use vLLM V1 engine for NemotronH Nano V2 model
ronjer30
committed
5efda69
Reverted flash-attn back to 2.8.3 to work with latest upstream NemotronH VL fixes
ronjer30
committed
eea9248
Enhance NemotronH VLM support for video captioning using HF model instead of local weights, introducing multiple model variants (BF16, FP8, NVFP4) and updating the prompt formatter. Remove the depr…
ronjer30
committed
78eb6b5
Fix unit tests and address review comments
ronjer30
committed
bea3f95
Fix linting errors: remove trailing whitespace and add newline at EOF
ronjer30
committed
7ecab77
Improve dtype check in prompt_formatter to handle all floating types
ronjer30
committed
ccca9d7

Commits on Feb 27, 2026

patch
weijiac0619
committed
76c365b
clean
weijiac0619
committed
1c37443
Merge pull request #1 from ronjer30/fix/cosmos-embed1-transformers-compat
ronjer30
authored
a5d84b7

Commits on Mar 24, 2026

nemotron preprocessing
weijiac0619
committed
f60dab8
resolved conflicts
weijiac0619
committed
b8306cd
stage2 fallback for qwen
weijiac0619
committed
d1064a4

Commits on Apr 1, 2026

Merge branch 'main' into feature/nemotron-vlm
weijiac0619
authored
05bdd95
clean up
weijiac0619
committed
9ec33e8
resolved conflicts
weijiac0619
committed
d308896
clean
weijiac0619
committed
e13ece3
Merge branch 'main' into feature/nemotron-vlm
weijiac0619
authored
d8ea9ee
comments
weijiac0619
committed
c44c94d
cap transformers <5.0
weijiac0619
committed
c9a2c1c
Merge branch 'main' into feature/nemotron-vlm
weijiac0619
authored
85bcb78