Skip to content

fix: genai config gaps - decoder path, image_processor.json, spatial_merge_size #204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
justinchuby merged 2 commits into from
Apr 23, 2026
Merged

fix: genai config gaps - decoder path, image_processor.json, spatial_merge_size #204

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e170a4d
fix: genai config gaps - decoder path, image_processor.json, spatial_…
justinchuby Apr 23, 2026
a370b9c
fix: keep processor_config.json naming convention for all VLMs
justinchuby Apr 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 20 additions & 3 deletions examples/gemma4/ort_genai/vlm/genai_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,18 @@
"vocab_size": 262144,
"context_length": 131072,
"bos_token_id": 2,
"eos_token_id": [1, 106],
"eos_token_id": [
1,
106
],
"pad_token_id": 0,
"image_token_id": 255999,
"decoder": {
"session_options": {
"log_id": "onnxruntime-genai",
"provider_options": []
},
"filename": "model.onnx",
"filename": "decoder/model.onnx",
"hidden_size": 1536,
"head_size": 256,
"num_attention_heads": 8,
Expand All @@ -37,12 +40,26 @@
"alignment": "right",
"slide_key_value_cache": true,
"slide_inputs": true,
"layers": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13]
"layers": [
0,
1,
2,
3,
5,
6,
7,
8,
10,
11,
12,
13
]
}
},
"vision": {
"filename": "vision.onnx",
"config_filename": "processor_config.json",
"spatial_merge_size": 2,
"session_options": {
"log_id": "onnxruntime-genai",
"provider_options": []
Expand Down
9 changes: 6 additions & 3 deletions src/mobius/integrations/ort_genai/auto_export.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,9 @@ def _write_processor_config(
"patch_size": getattr(vision, "patch_size", None) or 14,
}

path = os.path.join(output_dir, "processor_config.json")
proc_filename = "processor_config.json"

path = os.path.join(output_dir, proc_filename)
with open(path, "w", encoding="utf-8") as f:
json.dump(processor, f, indent=4)
return path
Expand Down Expand Up @@ -275,8 +277,9 @@ def _write_genai_config(
vision_kwargs["spatial_merge_size"] = None
vision_kwargs["config_filename"] = "vision_processor.json"
elif model_type in ("gemma4", "gemma4_text"):
vision_kwargs["spatial_merge_size"] = None
vision_kwargs["config_filename"] = "processor_config.json"
vision_cfg = getattr(config, "vision", None)
sms = getattr(vision_cfg, "spatial_merge_size", 2)
vision_kwargs["spatial_merge_size"] = sms

if vision_input_mapping is not None:
vision_kwargs["input_names"] = vision_input_mapping
Comment on lines 277 to 285

Copilot AI Apr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

genai_config.json can reference vision_processor.json when has_speech is true, but _write_processor_config() never writes that filename (it writes either processor_config.json or image_processor.json). This makes the generated artifacts inconsistent: ORT-GenAI will try to load a processor file that wasn't exported.

A concrete fix is to have processor-config writing share the same filename selection logic as genai-config generation (or write an additional copy under the referenced name when has_speech is set).

Copilot uses AI. Check for mistakes.
Expand Down
4 changes: 2 additions & 2 deletions src/mobius/integrations/ort_genai/auto_export_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -586,7 +586,7 @@ def test_gemma4_vision_inputs(self, tmp_path):
assert "pixel_values" in vision_inputs
assert "pixel_position_ids" in vision_inputs
assert "image_grid_thw" not in vision_inputs
assert "spatial_merge_size" not in data["model"]["vision"]
assert data["model"]["vision"]["spatial_merge_size"] == 2

def test_gemma4_decoder_has_input_ids_and_inputs_embeds(self, tmp_path):
"""Gemma4 decoder has both inputs_embeds and input_ids."""
Expand Down Expand Up @@ -779,7 +779,7 @@ def test_gemma4_genai_config_from_real_model(self, tmp_path):

# Config-level properties are still present
assert data["model"]["image_token_id"] == 255999
assert "spatial_merge_size" not in data["model"]["vision"]
assert data["model"]["vision"]["spatial_merge_size"] == 2
assert data["model"]["vision"]["config_filename"] == "processor_config.json"

def test_auto_export_produces_genai_config(self, tmp_path):
Expand Down
3 changes: 2 additions & 1 deletion src/mobius/integrations/ort_genai/genai_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,9 +353,10 @@ def generate(self) -> dict[str, Any]:
decoder_inputs = dict(self._decoder_inputs)
else:
decoder_inputs = _default_decoder_inputs(is_vlm=is_multimodal)
decoder_filename = "decoder/model.onnx" if is_multimodal else "model.onnx"
decoder: dict[str, Any] = {
"session_options": _make_session_options(self.ep),
"filename": "model.onnx",
"filename": decoder_filename,
"head_size": self.head_dim,
Comment on lines +356 to 360

Copilot AI Apr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For multimodal packages this hardcodes the decoder path to decoder/model.onnx, but not all multi-model exports save the decoder under a decoder/ subfolder. For example, Phi4MMMultiModalTask stores the decoder under the component key model, so ModelPackage.save() writes it to model/model.onnx. This will generate a genai_config that points at a non-existent file for those packages.

Consider making the decoder filename configurable (e.g., a decoder_filename parameter on GenaiConfigGenerator), and set it in write_ort_genai_config() based on which decoder component key is present in the ModelPackage (e.g., prefer decoder/model.onnx if "decoder" in pkg else model/model.onnx if "model" in pkg). Adding a regression test for the phi4mm export path would prevent this from reoccurring.

Copilot uses AI. Check for mistakes.
"hidden_size": self.hidden_size,
"inputs": decoder_inputs,
Expand Down
Loading