Add Gemma4 image_processor.json generation with onnxruntime-extension… - #209
Merged
Conversation
…s transforms pipeline
Performance Comparison
|
|
The author of this PR, apsonawane, is not an activated member of this organization on Codecov. |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the ORT-GenAI export integration so Gemma4 models emit an onnxruntime-extensions–compatible image processor configuration and reference it from genai_config.json, preventing runtime failures in OrtxCreateProcessor.
Changes:
- Generate
image_processor.jsonforgemma4/gemma4_textusing aprocessor.transformspipeline (DecodeImage→Gemma4ImageTransform). - Keep non-Gemma4 models on the existing HuggingFace-style
processor_config.json. - Set
vision.config_filenametoimage_processor.jsonfor Gemma4 ingenai_config.json.
justinchuby
approved these changes
Apr 24, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the ORT GenAI auto-export integration so Gemma4 models generate an onnxruntime-extensions compatible image processor configuration (transforms pipeline) and reference it from genai_config.json, avoiding runtime crashes caused by missing processor.transforms.
Changes:
- Generate
image_processor.json(DecodeImage → Gemma4ImageTransform) forgemma4/gemma4_textinstead of HF-styleprocessor_config.json. - Update Gemma4
genai_config.jsonto setvision.config_filenametoimage_processor.json. - Adjust the Gemma4 genai-config test expectation accordingly.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
src/mobius/integrations/ort_genai/auto_export.py |
Adds Gemma4-specific processor JSON generation in ORT-Extensions format and updates genai config to reference it. |
src/mobius/integrations/ort_genai/auto_export_test.py |
Updates Gemma4 genai-config assertion for the new processor config filename. |
justinchuby
approved these changes
Apr 24, 2026
apsonawane
enabled auto-merge (squash)
April 24, 2026 19:28
justinchuby
pushed a commit
that referenced
this pull request
Apr 25, 2026
#209) ## Summary Updates the ort-genai runtime integration to generate the correct image processor config format for Gemma4 models. ## Problem The `OrtxCreateProcessor` API in onnxruntime-extensions requires a JSON config with a `processor.transforms` array defining an ordered pipeline of image processing operations. Previously, Gemma4 models generated a HuggingFace-style `processor_config.json` with raw config values (`image_size`, `patch_size`, etc.), which caused a runtime crash: RuntimeError: [json.exception.out_of_range.403] key 'transforms' not found ## Changes **`src/mobius/integrations/ort_genai/auto_export.py`** - `_write_processor_config()`: For `gemma4`/`gemma4_text` models, generates `image_processor.json` with the onnxruntime-extensions transforms pipeline format (`DecodeImage` → `Gemma4ImageTransform`) instead of the HF-style `processor_config.json`. Reads `patch_size`, `max_soft_tokens`, and `pooling_kernel_size` from the model config. - `_write_genai_config()`: Sets `vision.config_filename` to `"image_processor.json"` for Gemma4 models so the genai runtime points to the correct file. Non-Gemma4 models are unaffected and continue generating `processor_config.json`. ## Generated `image_processor.json` example ```json { "processor": { "name": "gemma_4_image_processing", "transforms": [ { "operation": { "name": "decode_image", "type": "DecodeImage", "attrs": { "color_space": "RGB" } } }, { "operation": { "name": "gemma4_image_transform", "type": "Gemma4ImageTransform", "attrs": { "patch_size": 16, "max_soft_tokens": 280, "pooling_kernel_size": 3 } } } ] } }
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates the ort-genai runtime integration to generate the correct image processor config format for Gemma4 models.
Problem
The
OrtxCreateProcessorAPI in onnxruntime-extensions requires a JSON config with aprocessor.transformsarray defining an ordered pipeline of image processing operations. Previously, Gemma4 models generated a HuggingFace-styleprocessor_config.jsonwith raw config values (image_size,patch_size, etc.), which caused a runtime crash:RuntimeError: [json.exception.out_of_range.403] key 'transforms' not found
Changes
src/mobius/integrations/ort_genai/auto_export.py_write_processor_config(): Forgemma4/gemma4_textmodels, generatesimage_processor.jsonwith the onnxruntime-extensions transforms pipeline format (DecodeImage→Gemma4ImageTransform) instead of the HF-styleprocessor_config.json. Readspatch_size,max_soft_tokens, andpooling_kernel_sizefrom the model config._write_genai_config(): Setsvision.config_filenameto"image_processor.json"for Gemma4 models so the genai runtime points to the correct file.Non-Gemma4 models are unaffected and continue generating
processor_config.json.Generated
image_processor.jsonexample{ "processor": { "name": "gemma_4_image_processing", "transforms": [ { "operation": { "name": "decode_image", "type": "DecodeImage", "attrs": { "color_space": "RGB" } } }, { "operation": { "name": "gemma4_image_transform", "type": "Gemma4ImageTransform", "attrs": { "patch_size": 16, "max_soft_tokens": 280, "pooling_kernel_size": 3 } } } ] } }