Gemma4 ONNX + CUDA support: all PRs tracker

[justinchuby]

Gemma4 CUDA inference — tracking issue

Performance (H200 GPU)

Config	tok/s	VRAM
F16 INT4 (Q4_K_M)	176.7	9 GB
F16 onnx-standard	161.9	12 GB
F16 default	157.8	12 GB
F16 cuda (GQA)	151.4	14 GB
F32 default	143.9	22 GB
Image CUDA	~240	—
Audio CPU	15.1	—
Tri-modal (image+audio+text)	90	~14 GB

Text generation: exact token match with HF PyTorch ✅

mobius PRs

• Add static KV cache (TensorScatter) support for Gemma4 #246
• Add bool mask vs float additive bias guidance to debugging-memcpy skill #272
• Add benchmark methodology to profiling skill #276
• Fix vision encoder for 26b/31b: respect use_clipped_linears config #277
• Enable GQA for KV-shared layers and fix dtype issues in vision and audio #279
• Add Gemma4 GGUF support with per-layer KV heads fix #234
• Add ONNX export and quantization skill #238
• Fix Gemma4 weight mapping for 26b MoE and 31b alternative attention #239
• Address PR #239 follow-up: GQA limit, GGUF k_eq_v, DRY expert rename, tests #240
• Add Foundry Local deployment skill #243
• Fix genai_config.json provider names to use short format #247
• Warn about safetensors CUDA alignment issue #248
• Add Gemma4 GenAI text generation example with benchmarking #249
• Fix BF16 vision encoder: keep OneHot values as float32 #252
• Fix Gemma4 scale-free V norm FP16 overflow on CUDA #253
• Lower opset 24→23 for non-default EPs to eliminate memcpy #254
• Add profiling-onnx-models skill #256
• Cast vision/audio inputs f32 to model dtype for GenAI compatibility #265
• [KV cache] support-past-present-share-buffer #268
• Gemma4 CUDA EP: float additive bias + hybrid GQA/Attention dispatch #269
• Reduce CPU Concat ops in Gemma4 audio encoder #270
• Document f32 input casting for vision/audio encoders in multimodal skill #271
• Improve arch diff dtype reporting and Gemma4 test coverage #273
• Fix Gemma4 GenAI regression: disable past_present_share_buffer for dual head_dim #274
• Keep vision/audio inputs f32 in single-model tasks for f16/bf16 builds #275
• Validate QDQ with gemma 4 #332

onnxruntime PRs

• Fix CUDA ReduceSum erroring out on empty tensors with explicit axes microsoft/onnxruntime#28353
• Fix CUTLASS FMHA BiasLoader alignment for unaligned kernel path microsoft/onnxruntime#28369
• Fix CUDA Attention dispatch: skip MEA when head_size != v_head_size in GQA microsoft/onnxruntime#28358
• Skip MEA in ONNX Attention for head_dim > 256 (CUTLASS shared memory limit) microsoft/onnxruntime#28383
• [GQA] Make present_key/present_value outputs optional and add Gemma4 support microsoft/onnxruntime#28242

onnxruntime-extensions PRs

• Fix Gemma4 image resize to use float-domain bicubic for HF parity microsoft/onnxruntime-extensions#1056

onnxruntime-extensions Issues

• Optimize bicubic resize with OpenMP + SIMD for large images microsoft/onnxruntime-extensions#1057

onnxruntime-genai PRs

• Fix multimodal CUDA pipeline: embedding output persistence causes shape mismatch microsoft/onnxruntime-genai#2123
• Add CUDA multimodal debugging skill microsoft/onnxruntime-genai#2129
• Fix Gemma4 pixel_values trimming microsoft/onnxruntime-genai#2135

Olive

• MobiusBuilder + cache: produce loadable ORT GenAI composite packages microsoft/Olive#2472
• Add Gemma 4 E2B recipes (MobiusBuilder + INT4) microsoft/olive-recipes#381

Issues

• Gemma4 static cache (TensorScatter) support #244
• Tutorial: Running Gemma4 on Foundry Local #245
• Gemma4 E2B-it CUDA performance analysis and ablation study #263
• CUDA Attention kernel crashes with mismatched Q/K head dimensions (head_size != v_head_size) microsoft/onnxruntime#28357 (resolved)
• ORT memory reuse aliases node outputs with externally-provided inputs, breaking dynamic shapes across runs microsoft/onnxruntime#28359
• ONNX Attention MEA errors out with cudaErrorInvalidValue for KV-shared layers with borrowed K/V microsoft/onnxruntime#28376
• CUDA EP: CUBLAS misaligned address crash during prefill with Gemma4 model microsoft/onnxruntime-genai#2120 (resolved)
• Add local_window_size attribute to Attention op for sliding window support onnx/onnx#7914

[justinchuby]

• Add Gemma 4 E2B recipes (MobiusBuilder + INT4) microsoft/olive-recipes#381
• feat: add MobiusModelBuilder Olive pass for mobius-backed ONNX export microsoft/Olive#2406

[justinchuby]

Profile update: CUDA EP is 2x faster in raw compute

GQA raw compute: 215 ms vs default EP 420 ms (2x faster). The 151 vs 157 tok/s gap is GenAI pipeline overhead, not kernel performance. See microsoft/onnxruntime#28376 for the MEA right-padding crash bug.

[justinchuby]

Correction: GQA head_dim=512 works fine

The CUDA EP crash was NOT from GQA head_dim=512 — it was from KV-shared Attention layer 15 (head_dim=256, borrowed K/V). PR microsoft/onnxruntime#28374 closed (wrong guard). See microsoft/onnxruntime#28376 (updated).

[justinchuby]

Session Update: Audio CUDA Working

All three modalities now work on CUDA:

• Text: 158 tok/s ✅
• Image: ~240 tok/s ✅
• Audio: 117-127 tok/s ✅

New fixes applied:

• PR Gemma4 CUDA EP: float additive bias + hybrid GQA/Attention dispatch #269: Added audio encoder SkipSimplifiedLayerNormalization fix (inline manual RMSNorm to prevent ORT fusion with 1D bias as skip)
• ORT (local): Equal opset 19 CUDA kernel registration
• GenAI PR #2128: CUDA multimodal debugging skill (11 issues documented)

Audio CUDA root cause:

1. speech.session_options.provider_options must be [{"cuda": {}}] — mobius genai_config already generates this correctly when --ep cuda is used
2. Audio encoder SkipSimplifiedLayerNormalization had 1D bias in skip position — fixed by inlining manual RMSNorm ops

Performance issue #263 updated with complete table.

[justinchuby]

New PRs

• PR Reduce CPU Concat ops in Gemma4 audio encoder #270: Reduce audio encoder CPU Concat ops (49→24, 51% reduction)
• GenAI PR #2129: CUDA multimodal debugging skill (skill-only, clean)

Tri-modal CUDA

Image+Audio+Text works simultaneously on CUDA at 90 tok/s.

[justinchuby]

FlashAttention head_dim 512: Dao-AILab/flash-attention#2427