Support gemma-4-12B unified (encoder-free) multimodal model - #2286
Open
Justin Chu (justinchuby) wants to merge 3 commits into
Open
Support gemma-4-12B unified (encoder-free) multimodal model#2286Justin Chu (justinchuby) wants to merge 3 commits into
Justin Chu (justinchuby) wants to merge 3 commits into
Conversation
The gemma-4-12B "unified" variant is encoder-free: it consumes raw 48px merged pixel patches (patch_dim 6912) and raw 640-sample waveform frames directly, rather than the SigLIP image / Conformer log-mel audio contract of the standard gemma-4 (E2B/E4B) model. Register "gemma4_unified" as an MMM model type and route it to Gemma4MultiModalProcessor with a unified flag that adjusts two things: * Vision: no pixel-value trimming. The unified vision graph strips padding patches internally (position == -1), so the full padded (max_soft_tokens, 6912) pixel_values and position_ids are fed as-is instead of being trimmed to actual_soft_tokens * pooling^2. * Audio: each 640-sample frame is exactly one audio soft token, so the audio-token count equals the number of frames (no Conv2d stride-2 subsampling as in the Conformer speech encoder). Everything else (prompt image/audio token expansion, embedding fusion, per-layer-input decoder wiring) is shared with the standard gemma4 path. The paired preprocessing ops live in onnxruntime-extensions (Gemma4ImageTransform at patch_size=48/pooling=1, Gemma4UnifiedAudioFrames). Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds first-class ONNX Runtime GenAI support for the Gemma 4 “unified” (encoder-free) multimodal variant by registering a new model.type (gemma4_unified) and adjusting the existing Gemma4 multimodal processor logic to match the unified model’s vision/audio tensor contracts.
Changes:
- Recognize
gemma4_unifiedas a multimodal model type and register it in the multimodal processor factory. - Add a
unified_mode toGemma4MultiModalProcessorto:- skip vision tensor trimming (feed padded patch tensors as-is), and
- compute audio token counts without stride-2 subsampling (1 frame → 1 token).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/models/model.cpp |
Registers gemma4_unified to use the existing Gemma4MultiModalProcessor in the processor factory. |
src/models/model_type.h |
Adds gemma4_unified to the multimodal-model classification (IsMMM). |
src/models/gemma4_multimodal_processor.h |
Introduces a unified_ flag to switch behavior for the unified model variant. |
src/models/gemma4_multimodal_processor.cpp |
Implements unified-specific behavior (no vision trimming; audio token count equals frame count). |
| const int64_t patch_dim = (pv_dims == 3) ? pv_shape[2] : pv_shape[1]; | ||
|
|
||
| if (actual_patches < num_padded_patches) { | ||
| if (!unified_ && actual_patches < num_padded_patches) { |
| inline static bool IsMMM(const std::string& model_type) { | ||
| // Multi-modal model (MMM) | ||
| static constexpr std::array<std::string_view, 2> MMM = {"gemma4", "phi4mm"}; | ||
| static constexpr std::array<std::string_view, 3> MMM = {"gemma4", "gemma4_unified", "phi4mm"}; |
…est coverage * gemma4_multimodal_processor.cpp: rewrite the pixel-values comment to distinguish standard gemma4 (trim to actual teacher patches) from gemma4_unified (feed the full padded grid; the encoder-free graph strips padding via position_ids == -1). * Add test/python/create/create_dummy_gemma4_unified_models.py to derive a gemma4_unified fixture dir from the gemma4 fixtures with the unified I/O contract (pixel_values dim 6912, audio_embeds dim 640, model.type gemma4_unified, unified image/audio processor configs). * Add test/python/models/test_gemma4_unified_models.py covering processor creation and the unified vision/audio contracts (6912-dim pixel_values fed untrimmed; 640-dim audio frames with audio_sizes == frame count). Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Follow the onnxruntime-extensions consolidation: the unified audio config now uses the single Gemma4Audio op with type="raw_frames" instead of the separate Gemma4UnifiedAudioFrames kernel. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds native ORT GenAI support for the gemma-4-12B "unified" model. Unlike the standard
gemma4(E2B/E4B) model, the unified variant is encoder-free: it consumes raw 48px merged pixel patches (patch_dim = 6912) and raw 640-sample waveform frames directly, instead of the SigLIP image / Conformer log-mel audio contract.Before this change,
gemma4_unifiedwas not a recognized model type, so the only way to run it was to bypass the built-in processor entirely and feed pre-computed tensors viaGenerator.set_inputs(using the HuggingFace processor). This wires it into the nativeMultiModalProcessorpipeline.Changes
model_type.h: addgemma4_unifiedtoIsMMM.model.cpp: registergemma4_unified→Gemma4MultiModalProcessorin the processor factory.gemma4_multimodal_processor.{h,cpp}: add aunified_flag (set whenmodel.type == "gemma4_unified") that adjusts two things vs. the standard path:position == -1), so the full padded(max_soft_tokens, 6912)tensors are fed as-is rather than trimmed toactual_soft_tokens * pooling².num_audio_tokens = num_frames(no Conv2d stride-2 subsampling as in the Conformer speech encoder).All shared logic (image/audio prompt-token expansion, embedding fusion, per-layer-input decoder wiring) is unchanged.
Dependency
The paired native preprocessing ops live in onnxruntime-extensions (companion PR microsoft/onnxruntime-extensions#1091):
Gemma4ImageTransformconfigured atpatch_size=48, pooling_kernel_size=1(the unified 6912-dim merged patch is provably identical to a direct 48px patchify).Gemma4UnifiedAudioFrames(raw 640-sample framing).Testing
Built
libonnxruntime-genai.soviapython build.py --config Release --skip_tests --skip_wheel— the changed translation units compile cleanly. End-to-end generation parity against HuggingFace requires the gemma-4-12B weights and is intended to run in the model-enabled CI / on a GPU box.