Add VideoChat-Flash (OpenGVLab) language model support

[anilmartha]

Summary

Adds support for OpenGVLab/VideoChat-Flash-Qwen2_5-7B_InternVideo2-1B, a video-language model whose architecture is VideoChatFlashQwenForCausalLM. The language backbone is standard Qwen2.5-7B (flat config, standard weight keys, 2D RoPE, rope_theta=1e6, 28 layers / 28 heads / 4 KV heads / hidden=3584). Because it does not use MRoPE, the new builder inherits from QwenModel directly rather than Qwen25VLTextModel.

Changes

Python builder (src/python/py/models/)

• builders/qwen.py — New VideoChatFlashQwenModel subclass of QwenModel.
  • Overrides make_genai_config() to bypass the model's custom remote code (which imports av, cv2, decord, imageio even for config-only loads) by writing a temporary Qwen2Config-based config.json and patching genai_config.json["model"]["type"] back to videochat_flash_qwen.
  • Overrides load_weights() to load the LM backbone directly via Qwen2ForCausalLM.from_pretrained(), avoiding video deps.
  • Sets model_type = "videochat_flash_qwen" and hf_remote = False.
• builders/__init__.py — Exports VideoChatFlashQwenModel.
• builder.py —
  • Detects VideoChatFlashQwenForCausalLM by peeking at raw config.json (local dir or via hf_hub_download) before calling AutoConfig, then loads as Qwen2Config to avoid pulling in video libraries.
  • Routes VideoChatFlashQwenForCausalLM → VideoChatFlashQwenModel and defaults exclude_embeds=True (the text decoder receives inputs_embeds from the embedding merger). exclude_embeds is now opt-in (guarded with not in extra_options) so standalone input_ids export still works.

C++ runtime (src/)

• models/videochat_flash_processor.{h,cpp} (new) — VideoChatFlashProcessor runs ORT-Extensions image preprocessing, transposes HWC→CHW, reshapes to [1, num_frames, C, H, W], casts pixel values to the session input dtype, builds input_ids by replacing each <|vision_start|>…<|vision_end|> block with a fixed number of <|image_pad|> tokens, and emits image_grid_thw so GetImageFeatureBatchSize can determine num_images after the pixel_values name remap.
• models/model.cpp — Registers "videochat_flash_qwen" → VideoChatFlashProcessor in MultiModalProcessor.
• models/model_type.h — Adds "videochat_flash_qwen" to the IsVLM() list (size 6 → 7).
• config.{h,cpp} — Adds Vision::num_visual_tokens (default 0) — fixed visual tokens per image; 0 falls back to computing from image_grid_thw. Required (>0) by the new processor.

Files changed

File	Change
src/config.cpp	Parse num_visual_tokens from vision config
src/config.h	Add num_visual_tokens field on Vision
src/models/model.cpp	Register VideoChatFlashProcessor
src/models/model.h	Header / license update
src/models/model_type.h	Add videochat_flash_qwen to VLM list
src/models/videochat_flash_processor.cpp	New — image processor
src/models/videochat_flash_processor.h	New — header
src/python/py/models/builder.py	Detect VCF arch, bypass remote-code video deps, route to new builder
src/python/py/models/builders/__init__.py	Export VideoChatFlashQwenModel
src/python/py/models/builders/qwen.py	Add VideoChatFlashQwenModel

Validation

• Text-only export of the Qwen2.5-7B backbone via python -m onnxruntime_genai.models.builder -m OpenGVLab/VideoChat-Flash-Qwen2_5-7B_InternVideo2-1B -o <dir> -p fp32 -e cpu --extra_options exclude_embeds=false succeeds without av/cv2/decord/imageio installed.
• Standalone inference on the exported vcf-oga-fp32-standalone model produces correct answers across 4 batched QA prompts (e.g. Paris, 56, ONNX Runtime description) using generator.append_tokens() on the Qwen2.5-7B backbone.

Test plan

• Build OGA with the new processor compiled in and confirm IsVLM("videochat_flash_qwen") returns true.
• Export the text decoder with exclude_embeds=false and run text-only inference end-to-end.
• Confirm genai_config.json contains "type": "videochat_flash_qwen" (full VLM) or "type": "qwen2" (text-only standalone).
• Image-input smoke test once vision.onnx and embedding.onnx land.

[Vishal Jain (VishalX)]

Olive Recipe: microsoft/olive-recipes#406

[anilmartha]

@microsoft-github-policy-service agree company="AMD"

[Copilot]

Pull request overview

Adds first-class support for the OpenGVLab VideoChat-Flash Qwen-based video-language model by extending the Python model builder to export its (text) decoder without pulling video dependencies, and extending the C++ runtime to recognize/run the new multimodal model type.

Changes:

• Python: detect VideoChatFlashQwenForCausalLM early from raw config.json, route to a new VideoChatFlashQwenModel, and avoid trust_remote_code video-library imports.
• C++: introduce VideoChatFlashProcessor and register videochat_flash_qwen as a VLM model type/processor.
• Config: add and parse vision.num_visual_tokens used by the new processor for fixed visual-token padding.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/python/py/models/builders/qwen.py	Adds VideoChatFlashQwenModel subclass to export the LM backbone while bypassing remote-code video deps.
src/python/py/models/builders/init.py	Exports the new builder class from the builders package.
src/python/py/models/builder.py	Adds config “peek” detection for VideoChat-Flash and routes to the new builder with default exclude_embeds=True.
src/models/videochat_flash_processor.h	Declares VideoChatFlashProcessor (new multimodal processor).
src/models/videochat_flash_processor.cpp	Implements preprocessing/token padding + emits image_grid_thw for batching inference.
src/models/model.h	Updates header/license comments.
src/models/model.cpp	Registers videochat_flash_qwen → VideoChatFlashProcessor in the multimodal processor factory.
src/models/model_type.h	Adds videochat_flash_qwen to the VLM model-type list.
src/config.h	Adds vision.num_visual_tokens to config schema.
src/config.cpp	Parses vision.num_visual_tokens from genai_config.json.

[anilmartha]

Hi kunal-vaishnavi
Could you please review it again?

[anilmartha]

Hi kunal-vaishnavi
I have addressed the review comments. Could you please take another look?