Add Gemma4 multimodal support (vision + audio)

[apsonawane]

Summary

Adds end-to-end support for Google Gemma 4 multimodal models in ORT GenAI, covering text-only (gemma4_text), vision-language, and any-to-any (vision + audio + text) variants.

Changes

Model registration

• Register gemma4_text as LLM and gemma4 as MMM (multi-modal model)
• MMM auto-detects speech support from speech.filename in genai_config — no separate gemma4_any_to_any type needed
• Register Gemma4MultiModalProcessor in the processor factory

Gemma4 multimodal processor (gemma4_multimodal_processor.cpp/h)

• Vision: Preprocesses images via Gemma4ImageTransform (onnxruntime-extensions), trims padded patches to actual count using num_soft_tokens from preprocessor, produces pixel_values + pixel_position_ids
• Audio: Extracts mel features via Gemma4LogMel, computes audio_sizes for the pipeline, generates input_features_mask (all-True for single-clip inference), and expands <|audio|> placeholder tokens in the prompt
• Prompt handling: Expands both <|image|> and <|audio|> tokens from the chat template into the correct number of soft tokens before encoding. Handles template-inserted tokens and auto-insertion when no template is available

KV cache — per-layer head_dim (kv_cache.cpp/h)

• Auto-detects varying head_dim across layers from ONNX session input shapes (Gemma4 uses 256 for sliding-window layers, 512 for global attention layers)
• Creates per-layer empty_pasts_ with correct head dimensions
• Handles layer_shapes_[i][2] == 0 (unconstrained) in Update to avoid zero-size allocation
• Updates layer_shapes_ sequence dimension for past_present_share_buffer mode

Position inputs — int64 support (position_inputs.cpp)

• WindowedPositionInputs now supports both int32_t and int64_t for position_ids and attention_mask
• Type-dispatching lambdas for all data access points (first window, subsequent windows, token generation)

Multi-modal pipeline (multi_modal.cpp/h)

• DecoderState: Optional decoder_input_ids_ for models requiring input_ids alongside inputs_embeds
• EmbeddingState: Handles empty audio_features tensor when embedding model requires it but no speech session exists (AllocateEmptyFeatures)
• SpeechState: Manages 3D→2D reshape of speech output (ReshapeFeatures) before passing to embedding model
• Pipeline: Conditional audio feature reshape and empty audio fallback based on num_audio_tokens_

MultiModalFeatures (multi_modal_features.cpp/h)

• AllocateEmptyFeatures() — pre-allocates empty tensor for optional inputs
• ReshapeFeatures() — in-place reshape with data copy and state pointer update
• batch_size <= 0 support — skip batch dimension for 3D model outputs

Config (config.h/cpp)

• Added pixel_position_ids to vision inputs
• Added audio_token_id and boa_token_id to model config
• Added PixelPositionIdsName default constant

Example script (common.py)

• Added {"type": "audio"} entries for Gemma-style structured content in get_user_content

Testing

Tested with Gemma4 E2B model exported via mobius:

• ✅ Text-only generation
• ✅ Image description (detailed landscape analysis)
• ✅ Audio transcription (Windows SAPI TTS → model correctly identifies speech content)
• ✅ Image-only with any-to-any config (empty audio_features handled)
• ✅ Mixed GQA + standard Attention with past_present_share_buffer=false
• ✅ Per-layer head_dim KV cache (256/512)
• ✅ int64 position_ids with WindowedPositionInputs

[Copilot]

Pull request overview

Adds end-to-end Gemma4 multimodal (vision + optional audio) support to ONNX Runtime GenAI, including preprocessing, pipeline/runtime updates, and model registration/config plumbing.

Changes:

• Register Gemma4 model types/processors and extend config to cover new inputs/IDs (e.g., pixel_position_ids, audio-related token IDs).
• Extend multimodal pipeline/runtime to support Gemma4 decoder requirements (input_ids alongside inputs_embeds), optional/empty audio features, and speech output reshaping.
• Enhance runtime primitives (KV cache per-layer head_dim, WindowedPositionInputs int64 support).

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
src/models/position_inputs.cpp	Allow WindowedPositionInputs to accept int64 position_ids/attention_mask in addition to int32.
src/models/multi_modal_features.h	Add APIs to allocate empty feature tensors and reshape features.
src/models/multi_modal_features.cpp	Implement empty allocation + reshape and tweak batch-dim behavior for 3D tensors.
src/models/multi_modal.h	Add optional decoder input_ids support (plus a new include).
src/models/multi_modal.cpp	Pipeline updates for optional decoder input_ids, empty-audio fallback, and speech output reshape (3D→2D).
src/models/model_type.h	Register gemma4_text as LLM and gemma4 as MMM.
src/models/model.h	Include Gemma4 processor header.
src/models/model.cpp	Enable MMM speech session based on speech.filename; register Gemma4 processor in factory.
src/models/kv_cache.h	Add empty_pasts_ for per-layer empty past tensors.
src/models/kv_cache.cpp	Auto-detect per-layer head_dim (Gemma4 256/512), handle unconstrained cache length, and sync per-layer shapes in shared-buffer mode.
src/models/gemma4_multimodal_processor.h	New Gemma4 multimodal processor interface.
src/models/gemma4_multimodal_processor.cpp	New preprocessing for Gemma4 vision (incl. patch trimming + pixel position ids) and audio (mel features + prompt expansion).
src/config.h	Add pixel_position_ids naming, plus new token ID fields (audio_token_id, boa_token_id).
src/config.cpp	Parse pixel_position_ids, audio_token_id, boa_token_id from config JSON.
examples/python/common.py	Extend example structured user content to include {"type": "audio"} entries.
cmake/deps.txt	Bump onnxruntime-extensions dependency SHA.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.