Skip to content

Add Nemotron VLM support in video captioning - #1160

Merged
weijiac0619 merged 22 commits into
NVIDIA-NeMo:mainfrom
ronjer30:feature/nemotron-vlm
Apr 1, 2026
Merged

Add Nemotron VLM support in video captioning#1160
weijiac0619 merged 22 commits into
NVIDIA-NeMo:mainfrom
ronjer30:feature/nemotron-vlm

Conversation

@ronjer30

@ronjer30 ronjer30 commented Oct 2, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Adds support for the Nemotron Nano 12B V2 VLM

Usage

  1. Download model checkpoints
  2. Run
python tutorials/video/getting-started/video_split_clip_example.py \
  --video-dir </path/to/videos_directory/> \
  --output-clip-path ./outputs \
  --generate-captions \
  --captioning-algorithm nemotron \
  --nemotronh-vl-model-path </path/to/checkpoints/> \
  --captioning-batch-size 8 \
  --no-generate-embeddings

Checklist

  • I am familiar with the Contributing Guide.
  • New or Existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot

copy-pr-bot Bot commented Oct 2, 2025

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

suiyoubi
suiyoubi reviewed Oct 2, 2025
View reviewed changes
Comment thread nemo_curator/stages/video/caption/caption_preparation.py Outdated
suiyoubi
suiyoubi reviewed Oct 2, 2025
View reviewed changes
Comment thread nemo_curator/tasks/video.py Outdated
suiyoubi
suiyoubi reviewed Oct 2, 2025
View reviewed changes
Comment thread pyproject.toml Outdated
suiyoubi
suiyoubi reviewed Oct 2, 2025
View reviewed changes
Comment thread tutorials/video/getting-started/video_split_clip_example.py Outdated
@ronjer30

ronjer30 commented Oct 7, 2025

Copy link
Copy Markdown
Contributor Author

Thanks @suiyoubi, I've addressed the issues, added tests and updated existing ones as well.

@ronjer30
ronjer30 force-pushed the feature/nemotron-vlm branch from 2d81c50 to b63e9d2 Compare January 9, 2026 18:01
weijiac0619 and others added 3 commits February 27, 2026 03:00
@weijiac0619
patch
76c365b 
Signed-off-by: weijiac <weijiac@NVIDIA.com>
@weijiac0619
clean
1c37443 
Signed-off-by: weijiac <weijiac@NVIDIA.com>
@ronjer30
Merge pull request #1 from ronjer30/fix/cosmos-embed1-transformers-co…
a5d84b7 
…mpat

Fix Cosmos-Embed1 compatibility with transformers 4.56+ / 5.x
weijiac0619
weijiac0619 reviewed Mar 13, 2026
View reviewed changes
Comment thread nemo_curator/models/nemotron_h_vl.py
weijiac0619
weijiac0619 reviewed Mar 13, 2026
View reviewed changes
Comment thread tutorials/video/getting-started/video_split_clip_example.py
@weijiac0619

weijiac0619 commented Mar 13, 2026

Copy link
Copy Markdown
Contributor

When running with --captioning-algorithm nemotron-bf16, vLLM's nano_nemotron_vl.py accesses config.dtype directly during model initialization:
self.vision_model = self.get_vit_model_from_radio_config(config).to(config.dtype)
However, the pinned model revision (5d250e2e) of NVIDIA-Nemotron-Nano-12B-v2-VL-BF16 never sets self.dtype in NemotronHConfig.init, causing:

AttributeError: 'NemotronHConfig' object has no attribute 'dtype'

@weijiac0619

weijiac0619 commented Mar 13, 2026

Copy link
Copy Markdown
Contributor
  1. we may want to force --captioning-model-does-preprocess to be true when using nemotron model.
  2. When using nemotron, the video frames are first resized to dimensions divisible by 28(from Qwen2.5-VL's patch granularity), then vLLM resizes them again to 512×512. we may want to avoid the intermediate resizing.

@weijiac0619
nemotron preprocessing
f60dab8 
Signed-off-by: weijiac <weijiac@NVIDIA.com>
@weijiac0619
resolved conflicts
b8306cd 
Signed-off-by: weijiac <weijiac@NVIDIA.com>
@weijiac0619
stage2 fallback for qwen
d1064a4 
Signed-off-by: weijiac <weijiac@NVIDIA.com>
suiyoubi
suiyoubi reviewed Mar 25, 2026
View reviewed changes
Comment thread nemo_curator/models/nemotron_h_vl.py Outdated
msg = "vllm is required for NemotronHVL but is not installed. Please install vllm: pip install vllm"
raise ImportError(msg)

os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

@suiyoubi suiyoubi Mar 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please explain why we need this ?

@weijiac0619 weijiac0619 Mar 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The env var is not needed on my 48GB GPU. i believe it's a safeguard for smaller GPUs where the 12B BF16 model can trigger OOM. we can move this to docs rather than hardcoding it here.

@suiyoubi suiyoubi Mar 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isee. Thanks for explaining. I would prefer to have the min requirement in the docs rather than the safeguard here

suiyoubi
suiyoubi reviewed Mar 25, 2026
View reviewed changes
Comment thread nemo_curator/models/cosmos_embed1.py Outdated
}


_HEAD_MASK_VECTOR_DIM: Final[int] = 1

@suiyoubi suiyoubi Mar 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need all the changes to the cosmos_embed1.py files ?

@weijiac0619 weijiac0619 Mar 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, they're all needed for trust_remote_code=True to work with newer transformers versions. If the file bulk is a concern, they can be moved to a separate _cosmos_embed1_compat.py.

@weijiac0619 weijiac0619 Mar 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

discussed with @suiyoubi offline. the patches are not needed with latest cosmos model. will clean this up

weijiac0619 and others added 4 commits March 31, 2026 19:59
@weijiac0619
Merge branch 'main' into feature/nemotron-vlm
05bdd95
@weijiac0619
clean up
9ec33e8 
Signed-off-by: weijiac <weijiac@NVIDIA.com>
@weijiac0619
resolved conflicts
d308896 
Signed-off-by: weijiac <weijiac@NVIDIA.com>
@weijiac0619
clean
e13ece3 
Signed-off-by: weijiac <weijiac@NVIDIA.com>
@weijiac0619

weijiac0619 commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test e13ece3

suiyoubi
suiyoubi approved these changes Apr 1, 2026
View reviewed changes

@suiyoubi suiyoubi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks

@suiyoubi

suiyoubi commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test d8ea9ee

thomasdhc
thomasdhc reviewed Apr 1, 2026
View reviewed changes
Comment thread pyproject.toml Outdated
"torchaudio",
# Nemotron VL support
"transformers>=4.56.0",
"vllm>=0.13; (platform_machine == 'x86_64' and platform_system != 'Darwin')",

@thomasdhc thomasdhc Apr 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like vllm already has its own dependency group, and video_cuda12 already inherits it via nemo_curator[vllm]. Can the requirement here be consolidated?

@weijiac0619 weijiac0619 Apr 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch, removed

thomasdhc
thomasdhc reviewed Apr 1, 2026
View reviewed changes
Comment thread pyproject.toml Outdated
"torch<=2.9.1",
"torchaudio",
# Nemotron VL support
"transformers>=4.56.0",

@thomasdhc thomasdhc Apr 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current transformers version is being overwritten via override-dependency: "transformers<=4.55.2", # Else Cosmos Embed imports fail. Is this ok?

@weijiac0619 weijiac0619 Apr 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch. removed that override-dependency since the cosmos patch was gone.

@weijiac0619
comments
c44c94d 
Signed-off-by: weijiac <weijiac@NVIDIA.com>
@weijiac0619

weijiac0619 commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test c44c94d

@weijiac0619
cap transformers <5.0
c9a2c1c 
Signed-off-by: weijiac <weijiac@NVIDIA.com>
@weijiac0619

weijiac0619 commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test c9a2c1c

@weijiac0619

weijiac0619 commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test 85bcb78

@lbliii lbliii mentioned this pull request Apr 3, 2026
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@weijiac0619 weijiac0619 weijiac0619 left review comments
@thomasdhc thomasdhc thomasdhc approved these changes
@suiyoubi suiyoubi suiyoubi approved these changes
@abhinavg4 abhinavg4 Awaiting requested review from abhinavg4 abhinavg4 is a code owner
+1 more reviewer
@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ronjer30 @weijiac0619 @suiyoubi @thomasdhc