diff --git a/.github/skills/debugging-vl-pipeline/SKILL.md b/.github/skills/debugging-vl-pipeline/SKILL.md
index fc7dad4b..be7cf4e0 100644
--- a/.github/skills/debugging-vl-pipeline/SKILL.md
+++ b/.github/skills/debugging-vl-pipeline/SKILL.md
@@ -1,10 +1,11 @@
 ---
 name: debugging-vl-pipeline
 description: >
-  How to debug vision-language (VL) model output issues in mobius.
-  Covers the systematic pipeline isolation methodology, common failure modes,
-  stage-by-stage comparison with HuggingFace, and numerical tolerance
-  expectations. Use this skill when ORT GenAI multimodal output is wrong,
+  How to debug vision-language (VL) and multimodal (vision + audio) model
+  output issues in mobius.  Covers the systematic pipeline isolation
+  methodology, common failure modes, stage-by-stage comparison with
+  HuggingFace, numerical tolerance expectations, and CUDA EP-specific
+  issues. Use this skill when ORT GenAI multimodal output is wrong,
   garbled, or doesn't match HuggingFace.
 ---
 
@@ -18,6 +19,8 @@ Use this skill when:
 - ONNX model logits diverge significantly from HuggingFace
 - The model generates text-only descriptions ignoring the image
 - Image features appear correct but decoder output is wrong
+- Audio transcription is garbled or wrong despite correct encoder output
+- CUDA EP crashes or produces different results than CPU
 
 ## Debugging methodology: isolate each stage
 
@@ -433,6 +436,88 @@ spatial_merge_size = getattr(vc, "spatial_merge_size", 2)
 temporal_patch_size = getattr(vc, "temporal_patch_size", 2)
 ```
 
+### 6. ClippableLinear divergence (Gemma4)
+
+**Symptoms:** Vision or audio encoder output has large max diff (> 1.0)
+against HuggingFace, even though weights are loaded correctly.
+
+**Root cause:** Gemma4 uses `Gemma4ClippableLinear` with learned finite
+input/output activation clamping for ALL linear layers in its vision and
+audio encoders. Using plain `Linear` misses the clamping.
+
+**Detection:** Check if the HuggingFace model uses `ClippableLinear`:
+```bash
+grep -n "ClippableLinear" transformers/models/<model>/modeling_<model>.py
+```
+
+**Fix:** Use `ClippableLinear` (from `mobius.components`) for all
+affected linear layers. For vision attention: q/k/v/o_proj. For MLP:
+pass `linear_class=ClippableLinear` to the MLP component.
+
+**Impact:**
+- Audio: max diff 52.68 → 0.0003 after fix
+- Vision: max diff 3.92 → 0.00007 after fix
+
+### 7. Missing audio boundary markers
+
+**Symptoms:** Audio transcription is garbled or completely wrong, even
+though the audio encoder output matches HuggingFace.
+
+**Root cause:** HuggingFace wraps audio placeholder tokens with boundary
+markers in `input_ids`:
+```
+<|audio> (256000) + N × <|audio|> (258881) + <audio|> (258883)
+```
+If boundary markers are missing, the model cannot distinguish audio
+regions from text, producing wrong output.
+
+**Fix:** Add boundary markers to `build_input_ids()` when constructing
+audio inputs, matching HuggingFace's token wrapping.
+
+### 8. CUDA EP: ORT Gather int32 overflow
+
+**Symptoms:** CUDA EP crashes or produces incorrect results for models
+with large embedding tables. CPU EP works correctly.
+
+**Root cause:** ORT CUDA `gather_impl.cu` uses `int32` for element
+offset computation: `input_index = idx * cols + col_offset`. For
+tensors with > 2^31 elements (e.g. Gemma4 per-layer embedding
+[262144, 8960] = 2.35B elements), this overflows.
+
+**ORT bug:** microsoft/onnxruntime#28107
+
+**Workaround:** Split large embeddings into smaller tables via
+`nn.ModuleList` so each individual Gather stays under the int32 limit.
+Use Slice instead of Gather for column-wise indexing on large tensors.
+
+### 9. CUDA EP: opset 24 kernel registration
+
+**Symptoms:** ORT CUDA EP fails to find kernels for standard ops
+(Squeeze, Reshape, etc.) even though they work on CPU.
+
+**Root cause:** ORT ≤1.24.x CUDA/TRT EPs don't register kernels for
+opset 24, even though the op semantics are unchanged from opset 23.
+
+**Fix:** Use the `ort_lower_opset_for_ep` feature flag (enabled by
+default) which lowers the declared opset import from 24 to 23 for
+non-CPU EPs. See `src/mobius/_flags.py` and
+`src/mobius/_testing/ort_inference.py`.
+
+### 10. Wrong audio feature extractor
+
+**Symptoms:** Audio model produces completely wrong output. Audio
+encoder features don't match HuggingFace at all.
+
+**Root cause:** Using `WhisperFeatureExtractor` instead of the
+model-specific feature extractor (e.g. `Gemma4AudioFeatureExtractor`).
+Different extractors produce different mel spectrograms.
+
+**Fix:** Always use the correct feature extractor for the model:
+```python
+from transformers import AutoFeatureExtractor
+feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
+```
+
 ## Reference files
 
 - **Integration tests:** `tests/integration_test.py`
@@ -440,7 +525,9 @@ temporal_patch_size = getattr(vc, "temporal_patch_size", 2)
 - **ORT GenAI tests:** `tests/ort_genai_test.py`
   (`TestOrtGenaiQwen25VL.test_multimodal_image_generation`)
 - **Example scripts:** `examples/qwen25_vl_ort_genai.py`,
-  `examples/qwen3_vl_ort_genai.py`
+  `examples/qwen3_vl_ort_genai.py`, `examples/gemma4_multimodal.py`
 - **genai_config reference:** `.github/skills/ort-genai-config/SKILL.md`
 - **ORT GenAI position_ids code (external):**
   `onnxruntime-genai/src/models/position_inputs.cpp:617-814`
+- **Feature flags:** `src/mobius/_flags.py`
+  (`ort_lower_opset_for_ep`, `ort_cuda_grouped_rmsnorm_workaround`)
diff --git a/.github/skills/multimodal-models/SKILL.md b/.github/skills/multimodal-models/SKILL.md
index 870d8fd7..3c64af5f 100644
--- a/.github/skills/multimodal-models/SKILL.md
+++ b/.github/skills/multimodal-models/SKILL.md
@@ -1,20 +1,21 @@
 ---
 name: multimodal-models
 description: >
-  How to add multimodal (vision + language) models to mobius.
+  How to add multimodal (vision + language + audio) models to mobius.
   Covers projector variants (Gemma3, MLP, Linear), the VisionModel encoder,
-  InputMixer, VisionLanguageTask, image token handling, and weight name
-  mappings. Use this skill when adding a model that processes both images and
-  text.
+  InputMixer, VisionLanguageTask, image/audio token handling, ClippableLinear,
+  and weight name mappings. Use this skill when adding a model that processes
+  images, audio, or both alongside text.
 ---
 
-# Skill: Multimodal (Vision + Language) Models
+# Skill: Multimodal (Vision + Language + Audio) Models
 
 ## When to use
 
 Use this skill when adding a model that processes both images and text — such
-as Gemma3, LLaVA, LLaVA-NeXT, Phi-3-Vision, PaliGemma, InternVL2, Pixtral,
-Idefics2/3, Molmo, Florence2, or Video-LLaVA.
+as Gemma3, Gemma4, LLaVA, LLaVA-NeXT, Phi-3-Vision, PaliGemma, InternVL2,
+Pixtral, Idefics2/3, Molmo, Florence2, or Video-LLaVA — or a model that
+also processes audio (e.g. Gemma4 with speech/audio inputs).
 
 ## Architecture overview
 
@@ -390,6 +391,90 @@ The vision pipeline is completely shared with Qwen3-VL — only the text
 decoder differs (hybrid DeltaNet + full attention). This means vision
 encoder bugs/fixes apply to both models equally.
 
+## Gemma4: vision + audio multimodal
+
+Gemma4 models (E2B, E4B, 26B-A4B, 31B) support **both vision and audio**
+inputs. The architecture has 4 sub-models: decoder, vision encoder, audio
+encoder, and embedding.
+
+### Architecture
+
+```
+pixel_values ──► [Vision Encoder] ──► image_features ──┐
+                                                        │
+audio_features ─► [Audio Encoder] ──► audio_features ──┤
+                                                        │
+input_ids ──────► [Embedding] ◄────────────────────────┘
+                       │
+                       ▼
+                 [Text Decoder] ──► logits
+```
+
+### ClippableLinear (critical for Gemma4)
+
+Gemma4's vision and audio encoders use `Gemma4ClippableLinear` — a
+`Linear` with learned finite input/output activation clamping:
+
+```python
+x = Clip(x, input_min, input_max)
+x = x @ weight.T [+ bias]
+x = Clip(x, output_min, output_max)
+```
+
+**This is the single most common source of Gemma4 divergence.** Using
+plain `Linear` instead of `ClippableLinear` causes:
+- Audio encoder max diff: 52.68 → 0.0003 after fix
+- Vision encoder max diff: 3.92 → 0.00007 after fix
+
+Vision encoder uses ClippableLinear for ALL linear layers:
+- Q/K/V/O projections in `Gemma4VisionSelfAttention`
+- gate/up/down projections in MLP (via `linear_class=ClippableLinear`)
+
+Audio encoder uses ClippableLinear for its linear layers as well.
+
+See `reusable-components` skill for full `ClippableLinear` API reference.
+
+### Audio boundary markers
+
+HuggingFace wraps audio tokens with boundary markers that must be present
+in `input_ids` for correct generation:
+
+```
+<|audio>  (256000) + N × <|audio|> (258881) + <audio|> (258883)
+```
+
+This parallels the image token pattern:
+```
+<|image>  (255999) + N × <|image|> (258880) + <image|> (258882)
+```
+
+**Missing audio boundary markers** cause garbled audio transcription output
+even when the audio encoder output is numerically correct.
+
+### Per-layer embeddings (CUDA ORT workaround)
+
+Gemma4 uses per-layer embedding: `embed_tokens_per_layer` with shape
+`[V, L*D]` where V=vocab_size, L=num_layers, D=per_layer_dim. For large
+models this creates a single Gather on a 2.35B-element tensor, which
+**overflows ORT's CUDA Gather kernel** (int32 offset computation in
+`gather_impl.cu`).
+
+**Workaround:** Split into L separate `Embedding([V, D])` tables via
+`nn.ModuleList`. In `preprocess_weights`, split the HF weight column-wise:
+```python
+for i in range(num_layers):
+    renamed[f"embed_tokens_per_layer.{i}.weight"] = value[:, i*D:(i+1)*D]
+```
+
+Use `Slice` instead of `Gather` for per-layer projection indexing to
+avoid the large-tensor issue entirely.
+
+### Audio feature extraction
+
+Gemma4 uses `Gemma4AudioFeatureExtractor` (not `WhisperFeatureExtractor`).
+Using the wrong feature extractor produces completely different mel features
+and the model fails silently (produces garbage transcription).
+
 ## Testing multimodal models
 
 ### Image token count
diff --git a/.github/skills/phi4mm-component-parity/SKILL.md b/.github/skills/phi4mm-component-parity/SKILL.md
index 6f5eec96..0b6b903c 100644
--- a/.github/skills/phi4mm-component-parity/SKILL.md
+++ b/.github/skills/phi4mm-component-parity/SKILL.md
@@ -5,9 +5,10 @@ description: >
   embedding/projector, text decoder) matches HuggingFace output. Covers pipeline
   isolation methodology, common failure modes from real debugging experience,
   step-by-step debugging process, and integration test patterns. Applicable to
-  any multimodal model with similar architecture (Phi4MM, future audio+vision
-  models). Use this skill when multimodal ONNX model output diverges from
-  HuggingFace, or when adding a new multimodal model with multiple encoders.
+  any multimodal model with similar architecture (Phi4MM, Gemma4, future
+  audio+vision models). Use this skill when multimodal ONNX model output
+  diverges from HuggingFace, or when adding a new multimodal model with
+  multiple encoders.
 ---
 
 # Skill: Multimodal Component Parity Debugging
@@ -371,6 +372,46 @@ the actual sequence includes fused image/audio tokens, the lengths diverge.
 **Fix:** Always derive `seq_len` from `inputs_embeds.shape[1]` when the
 model uses inputs_embeds as input.
 
+### 8. ClippableLinear not used in encoder (Gemma4)
+
+**Symptoms:** Encoder output has large numerical divergence (max diff > 1.0)
+from HuggingFace, despite all weights loading correctly.
+
+**Root cause:** Some HuggingFace models (e.g. Gemma4) use
+`ClippableLinear` — a linear layer with learned finite input/output
+activation clipping — for ALL linear layers in their vision and audio
+encoders (attention q/k/v/o projections AND MLP gate/up/down projections).
+Using plain `Linear` misses the clamping and causes divergence.
+
+**Detection:** Check HF source for `ClippableLinear`:
+```bash
+grep -n "ClippableLinear" transformers/models/<model>/modeling_<model>.py
+```
+
+**Fix:** Use `ClippableLinear` from `mobius.components`:
+- For attention: use `ClippableLinear` for q/k/v/o_proj
+- For MLP: pass `linear_class=ClippableLinear` parameter
+
+**Impact (Gemma4):**
+- Audio: max diff 52.68 → 0.0003
+- Vision: max diff 3.92 → 0.00007
+
+### 9. Missing audio/image boundary tokens
+
+**Symptoms:** Audio transcription or vision description is garbled, but
+encoder output matches HuggingFace.
+
+**Root cause:** HuggingFace wraps modality placeholder tokens with
+boundary markers:
+- Image: `<|image>` (open) + N × `<|image|>` (pad) + `<image|>` (close)
+- Audio: `<|audio>` (open) + N × `<|audio|>` (pad) + `<audio|>` (close)
+
+Missing boundary markers prevent the model from correctly identifying
+modality regions in the input sequence.
+
+**Fix:** Ensure `build_input_ids` wraps placeholder tokens with the
+correct open/close marker token IDs.
+
 ## Step-by-step debugging process
 
 ### Phase 1: Text-only baseline
@@ -562,11 +603,16 @@ image_sizes = np.array([[384, 384]], dtype=np.int64)
 - **Integration tests:** `tests/phi4mm_integration_test.py`,
   `tests/integration_test.py`
 - **VL debugging skill:** `.github/skills/debugging-vl-pipeline/SKILL.md`
-- **Model implementation:** `src/mobius/models/phi.py`
-- **Audio components:** `src/mobius/components/_audio.py`
+- **Model implementation:** `src/mobius/models/phi.py`,
+  `src/mobius/models/gemma4.py`
+- **Audio components:** `src/mobius/components/_audio.py`,
+  `src/mobius/components/_gemma4_audio.py`
 - **Vision components:** `src/mobius/components/_vision.py`
+- **ClippableLinear:** `src/mobius/components/_gemma4_audio.py`
 - **LoRA component:** `src/mobius/components/_lora.py`
 - **Weight loading:** `src/mobius/_weight_loading.py`
+- **Feature flags:** `src/mobius/_flags.py`
 - **ORT GenAI config skill:** `.github/skills/ort-genai-config/SKILL.md`
 - **Weight name alignment skill:**
   `.github/skills/weight-name-alignment/SKILL.md`
+- **Gemma4 example:** `examples/gemma4_multimodal.py`
diff --git a/.github/skills/quality-checklist/SKILL.md b/.github/skills/quality-checklist/SKILL.md
index d9a3d67e..3aad778d 100644
--- a/.github/skills/quality-checklist/SKILL.md
+++ b/.github/skills/quality-checklist/SKILL.md
@@ -87,6 +87,14 @@ before the PR is merged.
       `testdata/golden/<cat>/<model>_generation.json`
 - [ ] `python -m pytest tests/e2e_golden_test.py -m generation -k "<model>"` passes
 
+> **Speech-language models:** The golden generation script supports the
+> `speech-language` task type for models that process audio inputs (e.g.,
+> Gemma4). The `_generate_speech_language()` function in
+> `scripts/generate_golden.py` handles audio feature extraction and input
+> construction for these models. Ensure the correct feature extractor
+> (e.g. `Gemma4AudioFeatureExtractor`, not `WhisperFeatureExtractor`) is
+> auto-detected via `AutoFeatureExtractor.from_pretrained()`.
+
 > **Why L4/L5 matter:** Graph-build tests (L1) only verify ONNX graph
 > construction; they never execute the graph with real data.  A MatMul shape
 > mismatch that crashes at runtime, a wrong normalisation type, or a missing
@@ -113,6 +121,15 @@ python examples/<model>_text_generation.py --compare-hf --dtype bf16
 - [ ] `mobius build --model <hf-model-id> /tmp/out` completes without error
 - [ ] Output directory contains the expected ONNX files and `genai_config.json`
 
+### 8a. Multi-EP correctness (CUDA)
+
+- [ ] Model runs correctly with `--ep cuda` (or `--device cuda`)
+- [ ] CUDA results match CPU results (compare generation output)
+- [ ] No crashes from large tensor operations (see ORT Gather int32
+      overflow: microsoft/onnxruntime#28107)
+- [ ] `ort_lower_opset_for_ep` flag handles opset 24→23 lowering for
+      CUDA EP (enabled by default in `src/mobius/_flags.py`)
+
 ### 9. ORT GenAI runtime
 
 - [ ] Model can be loaded with `ort_genai.Model(output_dir)` without error
diff --git a/.github/skills/reusable-components/SKILL.md b/.github/skills/reusable-components/SKILL.md
index 80cbe8cb..bf1004a6 100644
--- a/.github/skills/reusable-components/SKILL.md
+++ b/.github/skills/reusable-components/SKILL.md
@@ -28,6 +28,7 @@ components/
 ├── _attention.py          # Multi-head / GQA attention with KV cache; Qwen35Attention (gated GQA)
 ├── _audio.py              # ConformerEncoder (NeMo subsampling, T5 bias, Conformer layers)
 ├── _common.py             # Embedding, Linear, LayerNorm, LayerNormNoAffine, GroupNorm, create_attention_bias
+├── _gemma4_audio.py       # Gemma4 audio encoder: ClippableLinear, ConvSubsampling, SlidingWindowAttention
 ├── _conv.py               # Conv2d (2D convolution with bias and groups)
 ├── _decoder.py            # DecoderLayer (pre-norm residual block)
 ├── _encoder.py            # BertEmbeddings, EncoderAttention, EncoderLayer
@@ -334,6 +335,42 @@ Linear(in_features, out_features, bias=False)
 # Uses MatMul (+ optional Add for bias)
 ```
 
+### ClippableLinear
+
+```python
+ClippableLinear(in_features, out_features, bias=False)
+# Linear with learned input/output activation clamping
+# Matches HuggingFace Gemma4ClippableLinear
+```
+
+Wraps a standard `Linear` with 4 learned scalar parameters:
+`input_min`, `input_max`, `output_min`, `output_max`.  Inputs are clamped
+before the linear projection and outputs are clamped after:
+
+```python
+x = Clip(x, input_min, input_max)
+x = MatMul(x, weight.T) [+ bias]
+x = Clip(x, output_min, output_max)
+```
+
+**Critical:** HuggingFace `Gemma4ClippableLinear` stores *finite* learned
+bounds (not ±inf).  Using plain `Linear` instead of `ClippableLinear` causes
+large numerical divergence in Gemma4 vision and audio encoders:
+
+- Audio encoder: max diff 52.68 → 0.0003 after fix
+- Vision encoder: max diff 3.92 → 0.00007 after fix
+
+The component is exported from the public API: `from mobius.components
+import ClippableLinear`.
+
+**Weight mapping:** HF stores `<prefix>.linear.weight` for the actual
+weight (the `.linear.` segment is stripped by `preprocess_weights`), and
+`<prefix>.input_min`, `<prefix>.input_max`, `<prefix>.output_min`,
+`<prefix>.output_max` as direct scalar buffers.
+
+The MLP component accepts a `linear_class` parameter, so you can pass
+`linear_class=ClippableLinear` to use it for all projections in the MLP.
+
 ### Embedding
 
 ```python
diff --git a/examples/gemma4_multimodal.py b/examples/gemma4_multimodal.py
index 1bf158c1..b18cd65d 100644
--- a/examples/gemma4_multimodal.py
+++ b/examples/gemma4_multimodal.py
@@ -100,6 +100,8 @@
 IMAGE_CLOSE_TOKEN_ID = 258882  # <image|>   — closing boundary marker after image tokens
 # AUDIO_TOKEN_ID: placeholder inserted N times (once per audio frame) in input_ids
 AUDIO_TOKEN_ID = 258881  # <|audio|>  — confirmed from google/gemma-4-E2B-it HF config
+AUDIO_OPEN_TOKEN_ID = 256000  # <|audio>   — opening boundary marker before audio tokens
+AUDIO_CLOSE_TOKEN_ID = 258883  # <audio|>   — closing boundary marker after audio tokens
 EOS_TOKEN_IDS = {1, 106}  # <eos> (1) and <turn|> (106, end-of-turn marker)
 
 # Gemma 4 SigLIP vision encoder: default output length from vc.default_output_length.
@@ -111,10 +113,6 @@
 # stride 2 each → total time reduction factor of 4.
 AUDIO_SUBSAMPLING_FACTOR = 4
 
-# Mel spectrogram parameters for the Gemma 4 audio encoder input
-AUDIO_SAMPLE_RATE = 16_000
-AUDIO_N_MELS = 128  # Gemma 4 uses 128-dim mel (vs Whisper's 80)
-
 
 # ---------------------------------------------------------------------------
 # Input preprocessing — one function per ONNX session
@@ -150,53 +148,40 @@ def prepare_vision_feeds(
 
 
 def prepare_audio_feeds(
+    processor,
     audio_path: str,
-    sample_rate: int = AUDIO_SAMPLE_RATE,
-    n_mels: int = AUDIO_N_MELS,
 ) -> dict[str, np.ndarray]:
     """Prepare feeds for the **audio** session.
 
-    Loads the audio file, resamples to 16 kHz if needed, computes a
-    128-dim log-mel spectrogram, and transposes to ``(1, time, n_mels)``
-    layout expected by the Conformer encoder.
+    Loads the audio file and uses the Gemma 4 processor's built-in
+    ``Gemma4AudioFeatureExtractor`` to compute the 128-dim log-mel
+    spectrogram in ``(1, time, n_mels)`` layout expected by the
+    Conformer encoder.
 
     Args:
+        processor: ``AutoProcessor`` loaded for the Gemma 4 model.
         audio_path: Path to an audio file (WAV, FLAC, MP3, etc.).
-        sample_rate: Target sample rate (16 000 Hz for Gemma 4).
-        n_mels: Number of mel filterbank bins (128 for Gemma 4).
 
     Returns:
         ``{"input_features": float32[1, T, n_mels]}``
     """
-    import scipy.signal
     import soundfile as sf
 
     raw, sr = sf.read(audio_path, always_2d=True)  # [frames, channels]
     # Average channels to mono
     audio_np = raw.mean(axis=1).astype(np.float32)
-    if sr != sample_rate:
-        # Resample using scipy to avoid torchaudio/torchcodec dependency issues
-        num_samples = int(len(audio_np) * sample_rate / sr)
-        audio_np = scipy.signal.resample(audio_np, num_samples).astype(np.float32)
-
-    # Compute log-mel spectrogram using the HuggingFace feature extractor.
-    # Gemma 4 uses the same interface as WhisperFeatureExtractor.
-    feature_extractor = transformers.WhisperFeatureExtractor(
-        feature_size=n_mels,
-        sampling_rate=sample_rate,
-        # Use a wider window than Whisper to match the Conformer receptive field
-        hop_length=160,
-        chunk_length=30,
-    )
-    out = feature_extractor(
-        audio_np,
-        sampling_rate=sample_rate,
+
+    # Use the processor's Gemma4AudioFeatureExtractor for correct mel computation.
+    # The feature extractor handles resampling internally.
+    fe = processor.feature_extractor
+    out = fe(
+        [audio_np],
+        sampling_rate=sr,
         return_tensors="np",
         padding=False,
     )
-    # out["input_features"]: [1, n_mels, time_frames]
-    # Transpose to [1, time_frames, n_mels] for the Conformer encoder
-    audio_features = out["input_features"].astype(np.float32).transpose(0, 2, 1)
+    # out["input_features"]: [1, T, n_mels]  (already in correct layout)
+    audio_features = out["input_features"].astype(np.float32)
     return {"input_features": audio_features}  # [1, T, n_mels]
 
 
@@ -323,7 +308,12 @@ def build_input_ids(
         close_marker = np.array([[IMAGE_CLOSE_TOKEN_ID]], dtype=np.int64)
         modality_parts.extend([open_marker, soft_tokens, close_marker])
     if num_audio_tokens > 0:
-        modality_parts.append(np.full((1, num_audio_tokens), AUDIO_TOKEN_ID, dtype=np.int64))
+        # Wrap audio soft tokens with boundary markers, matching HF processor layout:
+        # <|audio>(256000) + Nx<|audio|>(258881) + <audio|>(258883)
+        audio_open = np.array([[AUDIO_OPEN_TOKEN_ID]], dtype=np.int64)
+        audio_soft = np.full((1, num_audio_tokens), AUDIO_TOKEN_ID, dtype=np.int64)
+        audio_close = np.array([[AUDIO_CLOSE_TOKEN_ID]], dtype=np.int64)
+        modality_parts.extend([audio_open, audio_soft, audio_close])
     modality_ids = np.concatenate(modality_parts, axis=1)
 
     # Find insertion point: right after the user header "<|turn>user\n"
@@ -589,6 +579,7 @@ def demo_audio(
     embedding_session: OnnxModelSession,
     decoder_session: OnnxModelSession,
     tokenizer,
+    processor,
     config,
     audio_path: str,
     prompt: str = "Transcribe the following audio.",
@@ -605,7 +596,7 @@ def demo_audio(
     # Step 1: Encode audio through the Conformer encoder.
     # Input:  audio_features [1, T, n_mels]  (mel spectrogram)
     # Output: audio_features [1, T', hidden_size]  (T' = T / subsampling_factor)
-    audio_out = audio_session.run(prepare_audio_feeds(audio_path))
+    audio_out = audio_session.run(prepare_audio_feeds(processor, audio_path))
     audio_features: np.ndarray = audio_out["audio_features"]
     if audio_features.ndim == 3:
         audio_features = audio_features[0]  # [T', hidden_size]
@@ -657,7 +648,7 @@ def demo_vision_audio(
         image_features = image_features[0]
 
     # Step 2: Encode audio
-    audio_out = audio_session.run(prepare_audio_feeds(audio_path))
+    audio_out = audio_session.run(prepare_audio_feeds(processor, audio_path))
     audio_features: np.ndarray = audio_out["audio_features"]
     if audio_features.ndim == 3:
         audio_features = audio_features[0]
@@ -734,6 +725,80 @@ def _hf_generate_vision(
     return processor.decode(out[0][prompt_len:], skip_special_tokens=True)
 
 
+def _hf_generate_audio(
+    model_id: str, audio_path: str, prompt: str, max_new_tokens: int
+) -> str:
+    """Run audio generation with HuggingFace PyTorch and return the output."""
+    import soundfile as sf
+    import torch
+    from transformers import AutoProcessor, Gemma4ForConditionalGeneration
+
+    processor = AutoProcessor.from_pretrained(model_id)
+    model = Gemma4ForConditionalGeneration.from_pretrained(model_id, torch_dtype=torch.float32)
+    model.eval()
+
+    raw, sr = sf.read(audio_path, always_2d=True)
+    audio_np = raw.mean(axis=1).astype(np.float32)
+
+    messages = [
+        {
+            "role": "user",
+            "content": [{"type": "audio"}, {"type": "text", "text": prompt}],
+        }
+    ]
+    text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
+    inputs = processor(text=text, audio=audio_np, sampling_rate=sr, return_tensors="pt")
+    with torch.no_grad():
+        out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
+    prompt_len = inputs["input_ids"].shape[1]
+    return processor.decode(out[0][prompt_len:], skip_special_tokens=True)
+
+
+def _hf_generate_vision_audio(
+    model_id: str,
+    image_path: str,
+    audio_path: str,
+    prompt: str,
+    max_new_tokens: int,
+) -> str:
+    """Run combined vision + audio generation with HuggingFace PyTorch."""
+    import soundfile as sf
+    import torch
+    from PIL import Image
+    from transformers import AutoProcessor, Gemma4ForConditionalGeneration
+
+    processor = AutoProcessor.from_pretrained(model_id)
+    model = Gemma4ForConditionalGeneration.from_pretrained(model_id, torch_dtype=torch.float32)
+    model.eval()
+
+    image = Image.open(image_path).convert("RGB")
+    raw, sr = sf.read(audio_path, always_2d=True)
+    audio_np = raw.mean(axis=1).astype(np.float32)
+
+    messages = [
+        {
+            "role": "user",
+            "content": [
+                {"type": "image"},
+                {"type": "audio"},
+                {"type": "text", "text": prompt},
+            ],
+        }
+    ]
+    text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
+    inputs = processor(
+        text=text,
+        images=image,
+        audio=audio_np,
+        sampling_rate=sr,
+        return_tensors="pt",
+    )
+    with torch.no_grad():
+        out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
+    prompt_len = inputs["input_ids"].shape[1]
+    return processor.decode(out[0][prompt_len:], skip_special_tokens=True)
+
+
 def _print_side_by_side(label: str, onnx_out: str, hf_out: str) -> None:
     """Print ONNX and HuggingFace outputs side by side for easy comparison."""
     divider = "─" * 64
@@ -751,21 +816,30 @@ def run_compare_hf(
     model_id: str,
     onnx_outputs: dict[str, str],
     has_image: bool,
+    has_audio: bool,
     image_path: str,
+    audio_path: str,
     max_new_tokens: int,
     text_prompt: str,
     vision_prompt: str,
+    audio_prompt: str,
+    vision_audio_prompt: str,
 ) -> None:
     """Run HF PyTorch inference for each completed ONNX demo and compare outputs.
 
     Args:
         model_id: HuggingFace model ID.
-        onnx_outputs: Mapping of mode name (``"text"``, ``"vision"``) to ONNX text output.
+        onnx_outputs: Mapping of mode name (``"text"``, ``"vision"``,
+            ``"audio"``, ``"vision-audio"``) to ONNX text output.
         has_image: Whether the image asset is available.
+        has_audio: Whether audio is available (file exists and model has audio).
         image_path: Path to the image file.
+        audio_path: Path to the audio file.
         max_new_tokens: Max tokens for generation.
         text_prompt: Prompt used for text demo.
         vision_prompt: Prompt used for vision demo.
+        audio_prompt: Prompt used for audio demo.
+        vision_audio_prompt: Prompt used for vision-audio demo.
     """
     print("\n" + "=" * 64)
     print("🔍  --compare-hf: loading HuggingFace model for comparison ...")
@@ -781,6 +855,18 @@ def run_compare_hf(
         hf_vision = _hf_generate_vision(model_id, image_path, vision_prompt, max_new_tokens)
         _print_side_by_side("VISION", onnx_outputs["vision"], hf_vision)
 
+    if "audio" in onnx_outputs and has_audio:
+        print("Running HF audio generation ...")
+        hf_audio = _hf_generate_audio(model_id, audio_path, audio_prompt, max_new_tokens)
+        _print_side_by_side("AUDIO", onnx_outputs["audio"], hf_audio)
+
+    if "vision-audio" in onnx_outputs and has_image and has_audio:
+        print("Running HF vision + audio generation ...")
+        hf_va = _hf_generate_vision_audio(
+            model_id, image_path, audio_path, vision_audio_prompt, max_new_tokens
+        )
+        _print_side_by_side("VISION + AUDIO", onnx_outputs["vision-audio"], hf_va)
+
 
 # ---------------------------------------------------------------------------
 # CLI
@@ -845,10 +931,22 @@ def parse_args() -> argparse.Namespace:
     )
     parser.add_argument(
         "--dtype",
-        choices=["f32", "f16"],
+        choices=["f32", "f16", "bf16"],
         default="f32",
         help="Weight/activation dtype to use (default: %(default)s).",
     )
+    parser.add_argument(
+        "--device",
+        choices=["cpu", "cuda", "webgpu"],
+        default="cpu",
+        help="Device for inference (default: %(default)s).",
+    )
+    parser.add_argument(
+        "--ep",
+        choices=["default", "cpu", "cuda", "webgpu", "onnx-standard", "trt-rtx"],
+        default="default",
+        help="Execution provider for ONNX model build.",
+    )
     parser.add_argument(
         "--compare-hf",
         action="store_true",
@@ -871,8 +969,14 @@ def main() -> int:
     # and returns a ModelPackage containing all sub-models.
     # ------------------------------------------------------------------
     load_weights = not args.no_weights
-    print(f"Building ONNX models from {args.model_id!r} (dtype={args.dtype}) ...")
-    pkg = build(args.model_id, dtype=args.dtype, load_weights=load_weights)
+    ep = args.ep
+    print(f"Building ONNX models from {args.model_id!r} (dtype={args.dtype}, ep={ep}) ...")
+    pkg = build(
+        args.model_id,
+        dtype=args.dtype,
+        load_weights=load_weights,
+        execution_provider=ep,
+    )
     config = pkg.config
     print(f"Package components: {list(pkg.keys())}")
     print(
@@ -896,10 +1000,12 @@ def main() -> int:
     # are handled — audio_session is None when the model has no audio component.
     # ------------------------------------------------------------------
     print("\nCreating ONNX Runtime sessions ...")
-    vision_session = OnnxModelSession(pkg["vision"])
-    audio_session = OnnxModelSession(pkg["audio"]) if "audio" in pkg else None
-    embedding_session = OnnxModelSession(pkg["embedding"])
-    decoder_session = OnnxModelSession(pkg["decoder"])
+    vision_session = OnnxModelSession(pkg["vision"], device=args.device)
+    audio_session = (
+        OnnxModelSession(pkg["audio"], device=args.device) if "audio" in pkg else None
+    )
+    embedding_session = OnnxModelSession(pkg["embedding"], device=args.device)
+    decoder_session = OnnxModelSession(pkg["decoder"], device=args.device)
 
     # ------------------------------------------------------------------
     # Step 3: Load the HuggingFace processor.
@@ -943,6 +1049,8 @@ def main() -> int:
 
     text_prompt = args.prompt or "Explain the theory of general relativity in simple terms."
     vision_prompt = args.prompt or "Describe what you see in this image in detail."
+    audio_prompt = args.prompt or "Transcribe the following audio."
+    vision_audio_prompt = args.prompt or "Describe the image and transcribe the audio."
 
     # Collect ONNX outputs for optional --compare-hf side-by-side display
     onnx_outputs: dict[str, str] = {}
@@ -978,21 +1086,23 @@ def main() -> int:
         elif mode == "audio":
             if not has_audio:
                 continue
-            demo_audio(
+            result = demo_audio(
                 audio_session=audio_session,
                 embedding_session=embedding_session,
                 decoder_session=decoder_session,
                 tokenizer=tokenizer,
+                processor=processor,
                 config=config,
                 audio_path=args.audio,
-                prompt=args.prompt or "Transcribe the following audio.",
+                prompt=audio_prompt,
                 max_new_tokens=max_tokens,
             )
+            onnx_outputs["audio"] = result
 
         elif mode == "vision-audio":
             if not has_image or not has_audio:
                 continue
-            demo_vision_audio(
+            result = demo_vision_audio(
                 vision_session=vision_session,
                 audio_session=audio_session,
                 embedding_session=embedding_session,
@@ -1002,19 +1112,24 @@ def main() -> int:
                 config=config,
                 image_path=args.image,
                 audio_path=args.audio,
-                prompt=args.prompt or "Describe the image and transcribe the audio.",
+                prompt=vision_audio_prompt,
                 max_new_tokens=max_tokens,
             )
+            onnx_outputs["vision-audio"] = result
 
     if args.compare_hf and onnx_outputs:
         run_compare_hf(
             model_id=args.model_id,
             onnx_outputs=onnx_outputs,
             has_image=has_image,
+            has_audio=has_audio,
             image_path=args.image,
+            audio_path=args.audio,
             max_new_tokens=max_tokens,
             text_prompt=text_prompt,
             vision_prompt=vision_prompt,
+            audio_prompt=audio_prompt,
+            vision_audio_prompt=vision_audio_prompt,
         )
 
     return 0
diff --git a/scripts/generate_golden.py b/scripts/generate_golden.py
index b7a4c1a0..a0689804 100644
--- a/scripts/generate_golden.py
+++ b/scripts/generate_golden.py
@@ -465,6 +465,204 @@ def _generate_speech_to_text(case: TestCase, json_path: Path, device: str) -> No
         )
 
 
+def _try_register_qwen3_asr() -> None:
+    """Register Qwen3-ASR config with transformers if available.
+
+    The ``qwen_asr`` pip package provides the config and model classes
+    but does not auto-register with transformers' ``AutoConfig``.  We
+    do that here so ``AutoConfig.from_pretrained`` can load the config
+    from HuggingFace without ``auto_map`` in the repo.
+    """
+    try:
+        from qwen_asr.core.transformers_backend.configuration_qwen3_asr import (
+            Qwen3ASRConfig,
+        )
+        from transformers import AutoConfig
+
+        # register() is a no-op if already registered.
+        AutoConfig.register("qwen3_asr", Qwen3ASRConfig)
+    except ImportError:
+        # Optional dependency is not installed; skip registration and
+        # let speech-language generation proceed via other supported paths.
+        pass
+
+
+def _generate_speech_language(case: TestCase, json_path: Path, device: str) -> None:
+    """Generate golden data for a speech-language model.
+
+    Supports two architectures:
+
+    * **Gemma4-style**: Uses ``AutoModelForImageTextToText`` with a
+      multimodal processor that combines text prompts and audio.
+    * **Qwen3-ASR-style**: Uses the ``qwen_asr`` package with its own
+      processor and chat template (``trust_remote_code`` required).
+
+    The model type is auto-detected from the HuggingFace config.
+    """
+    import librosa
+    import torch
+
+    from mobius._testing.golden import save_generation_json, save_golden_ref
+
+    audio_path = Path("testdata") / case.audio[0]
+    audio_array, _sample_rate = librosa.load(str(audio_path), sr=16000)
+
+    model, processor, forward_model = _load_speech_language_model(case, device)
+
+    processed, prompt_for_golden = _prepare_speech_language_inputs(
+        case, model, processor, audio_array, audio_path, device
+    )
+
+    # L4: single forward pass
+    with torch.no_grad():
+        outputs = forward_model(**processed)
+
+    last_logits = outputs.logits[0, -1, :].cpu().numpy()
+    golden = _extract_logits_golden(last_logits)
+    input_ids_np = processed["input_ids"].cpu().numpy()
+
+    # L5: greedy generation
+    generated_ids = None
+    if "L5" in case.level:
+        with torch.no_grad():
+            gen = model.generate(
+                **processed,
+                max_new_tokens=case.generation_params.get("max_new_tokens", 50),
+                do_sample=False,
+            )
+        input_len = processed["input_ids"].shape[1]
+        gen_seq = gen.sequences if hasattr(gen, "sequences") else gen
+        generated_ids = gen_seq[0, input_len:].cpu().numpy()
+
+    save_golden_ref(
+        json_path,
+        top1_id=golden["top1_id"],
+        top2_id=golden["top2_id"],
+        top10_ids=golden["top10_ids"],
+        top10_logits=golden["top10_logits"],
+        logits_summary=golden["logits_summary"],
+        input_ids=input_ids_np,
+    )
+
+    if generated_ids is not None:
+        tokenizer = processor.tokenizer if hasattr(processor, "tokenizer") else processor
+        generated_text = tokenizer.decode(generated_ids.tolist(), skip_special_tokens=True)
+        gen_path = json_path.with_name(json_path.stem + "_generation.json")
+        save_generation_json(
+            gen_path,
+            model_id=case.model_id,
+            prompt=prompt_for_golden,
+            generated_tokens=generated_ids.tolist(),
+            generated_text=generated_text,
+        )
+
+
+def _load_speech_language_model(case: TestCase, device: str) -> tuple:
+    """Load a speech-language model and processor.
+
+    Returns ``(model, processor, forward_model)`` where
+    *forward_model* is the module whose ``forward()`` produces logits
+    (may differ from *model* for nested architectures like Qwen3-ASR).
+    """
+    import torch
+    import transformers
+
+    # Try to register qwen3_asr classes before loading config,
+    # since the HF repo lacks auto_map and trust_remote_code alone
+    # won't resolve it.
+    _try_register_qwen3_asr()
+
+    config = transformers.AutoConfig.from_pretrained(
+        case.model_id, trust_remote_code=case.trust_remote_code
+    )
+    model_type = getattr(config, "model_type", "")
+
+    if model_type == "qwen3_asr":
+        from qwen_asr.core.transformers_backend.modeling_qwen3_asr import (
+            Qwen3ASRForConditionalGeneration,
+        )
+
+        model = Qwen3ASRForConditionalGeneration.from_pretrained(
+            case.model_id, torch_dtype=torch.float32
+        )
+        model = model.to(device).eval()
+        processor = transformers.AutoProcessor.from_pretrained(
+            case.model_id, trust_remote_code=True
+        )
+        # Qwen3-ASR wraps a thinker; the thinker produces logits.
+        forward_model = model.thinker
+    else:
+        # Gemma4-style: AutoModelForImageTextToText
+        from mobius._testing.torch_reference import (
+            load_torch_multimodal_model,
+        )
+
+        model, _tokenizer, processor = load_torch_multimodal_model(
+            case.model_id, device=device
+        )
+        forward_model = model
+
+    return model, processor, forward_model
+
+
+def _prepare_speech_language_inputs(
+    case: TestCase,
+    model: object,
+    processor: object,
+    audio_array: np.ndarray,
+    audio_path: Path,
+    device: str,
+) -> tuple:
+    """Build model inputs and a prompt string for the golden file.
+
+    Returns ``(processed, prompt_for_golden)`` where *processed* is a
+    dict/BatchEncoding ready for ``model(**processed)`` and
+    *prompt_for_golden* is the string saved in the generation JSON.
+    """
+    # Detect Qwen3-ASR by processor class name (avoids redundant
+    # config download).
+    is_qwen3_asr = "Qwen3ASR" in type(processor).__name__
+
+    if is_qwen3_asr:
+        # Qwen3-ASR prompt: system + user with audio placeholder
+        messages = [
+            {"role": "system", "content": ""},
+            {
+                "role": "user",
+                "content": [{"type": "audio", "audio": ""}],
+            },
+        ]
+        text_prompt = processor.apply_chat_template(
+            messages, add_generation_prompt=True, tokenize=False
+        )
+        processed = processor(
+            text=text_prompt,
+            audio=[audio_array],
+            return_tensors="pt",
+        ).to(device)
+        prompt_for_golden = str(audio_path)
+    else:
+        # Gemma4-style: text prompt + audio
+        prompt_text = case.prompts[0]
+        if hasattr(processor, "apply_chat_template"):
+            content: list[dict[str, str]] = [
+                {"type": "audio", "audio": str(audio_path)},
+                {"type": "text", "text": prompt_text},
+            ]
+            messages = [{"role": "user", "content": content}]
+            prompt_text = processor.apply_chat_template(
+                messages, tokenize=False, add_generation_prompt=True
+            )
+        processed = processor(
+            text=prompt_text,
+            audio=[audio_array],
+            return_tensors="pt",
+        ).to(device)
+        prompt_for_golden = case.prompts[0]
+
+    return processed, prompt_for_golden
+
+
 def _generate_audio_feature_extraction(case: TestCase, json_path: Path, device: str) -> None:
     """Generate golden data for audio feature extraction (Wav2Vec2 etc.).
 
@@ -565,6 +763,7 @@ def _generate_image_classification(case: TestCase, json_path: Path, device: str)
     "image-text-to-text": _generate_vision_language,
     "image-classification": _generate_image_classification,
     "speech-to-text": _generate_speech_to_text,
+    "speech-language": _generate_speech_language,
     "audio-feature-extraction": _generate_audio_feature_extraction,
 }
 
diff --git a/src/mobius/_flags.py b/src/mobius/_flags.py
index d11b2938..44d6f930 100644
--- a/src/mobius/_flags.py
+++ b/src/mobius/_flags.py
@@ -79,6 +79,11 @@ class _Flags:
          - ``True``
          - Suppress "has no constant value" warnings from the initializer
            deduplication pass.
+       * - ``ort_lower_opset_for_ep``
+         - ``MOBIUS_ORT_LOWER_OPSET_FOR_EP``
+         - ``True``
+         - Lower the ONNX opset declaration to 23 for non-CPU EPs
+           (ORT ≤1.24.x workaround).
     """
 
     suppress_dedup_warning: bool = dataclasses.field(
@@ -98,6 +103,19 @@ class _Flags:
     Set ``MOBIUS_ORT_CUDA_GROUPED_RMSNORM_WORKAROUND=1`` when targeting CUDA.
     """
 
+    ort_lower_opset_for_ep: bool = dataclasses.field(
+        default_factory=lambda: _env_bool("MOBIUS_ORT_LOWER_OPSET_FOR_EP", True)
+    )
+    """Lower the ONNX default-domain opset declaration to 23 when creating
+    ORT sessions on non-CPU execution providers (CUDA, TRT, etc.).
+
+    ORT ≤1.24.x EPs don't register kernels for opset 24 standard ops
+    (Squeeze, Reshape, etc.) even though the semantics are unchanged.
+    Lowering the import declaration lets the EP find its existing kernels.
+    Set ``MOBIUS_ORT_LOWER_OPSET_FOR_EP=0`` to disable once ORT adds
+    opset 24 kernel support.
+    """
+
 
 # Global singleton — import and use this directly.
 flags = _Flags()
diff --git a/src/mobius/_registry.py b/src/mobius/_registry.py
index 8d311ea9..3c7014e7 100644
--- a/src/mobius/_registry.py
+++ b/src/mobius/_registry.py
@@ -805,7 +805,7 @@ def _create_default_registry() -> ModelRegistry:
 
     # --- Speech ---
     "whisper": "openai/whisper-tiny",
-    "qwen3_asr": "Qwen/Qwen3-ASR-2B-Instruct",
+    "qwen3_asr": "Qwen/Qwen3-ASR-0.6B",
     "speecht5": "microsoft/speecht5_asr",
     "sew": "asapp/sew-tiny-100k",
     "sew-d": "asapp/sew-d-tiny-100k",
diff --git a/src/mobius/_testing/ort_inference.py b/src/mobius/_testing/ort_inference.py
index 6e4b2b27..a78e8c7e 100644
--- a/src/mobius/_testing/ort_inference.py
+++ b/src/mobius/_testing/ort_inference.py
@@ -9,6 +9,7 @@
 
 from __future__ import annotations
 
+import logging
 import tempfile
 from pathlib import Path
 
@@ -16,8 +17,47 @@
 import onnx_ir as ir
 import onnxruntime_easy as ort_easy
 
+from mobius._flags import flags
 from mobius._model_package import ModelPackage
 
+logger = logging.getLogger(__name__)
+
+# Maximum default-domain ONNX opset that ORT ≤1.24.x CUDA/TRT EPs
+# register kernels for.  Models built with a higher opset can be
+# loaded after lowering the declared import — the op semantics have
+# not changed, only the version label.
+_MAX_EP_OPSET = 23
+
+
+def _should_lower_opset(model: ir.Model, device: str) -> bool:
+    """Return True when lowering the opset import is safe and needed.
+
+    Lowering is only attempted when
+    :attr:`~mobius._flags._Flags.ort_lower_opset_for_ep` is enabled and the
+    target device is non-CPU with a default-domain opset exceeding
+    ``_MAX_EP_OPSET``.
+
+    Lowering is *not* safe when the graph contains ops that were first
+    introduced in a post-23 opset (e.g. ``TensorScatter``).  In that
+    case the model requires genuine opset 24+ support and lowering
+    would produce an invalid model.
+    """
+    if not flags.ort_lower_opset_for_ep:
+        return False
+    if device == "cpu":
+        return False
+    current_opset = model.opset_imports.get("", 0)
+    if current_opset <= _MAX_EP_OPSET:
+        return False
+
+    # Ops that were *introduced* in opset 24 and have no opset 23
+    # equivalent.  If any appear in the graph, lowering is unsafe.
+    opset_24_only_ops = {"TensorScatter"}
+    for node in model.graph:
+        if node.domain == "" and node.op_type in opset_24_only_ops:
+            return False
+    return True
+
 
 class OnnxModelSession:
     """Wraps an ``onnxruntime_easy.EasySession`` for an ``ir.Model``.
@@ -47,9 +87,30 @@ def __init__(
                 )
             model = next(iter(model.values()))
 
+        # Workaround: ORT ≤1.24.x CUDA/TRT EPs don't register kernels
+        # for ONNX opset 24 standard ops (Squeeze, Reshape, etc.).  The
+        # op semantics are identical to opset 23, so lowering the import
+        # declaration lets the EP find its existing kernels.  The model
+        # object is restored to its original opset after saving.
+        device = load_kwargs.get("device", "cpu")
+        lower_opset = _should_lower_opset(model, device)
+        original_opset = model.opset_imports.get("", 0) if lower_opset else 0
+        if lower_opset:
+            logger.info(
+                "Lowering default-domain opset from %d to %d for %s",
+                original_opset,
+                _MAX_EP_OPSET,
+                device,
+            )
+            model.opset_imports[""] = _MAX_EP_OPSET
+
         self._tmpdir = tempfile.TemporaryDirectory()
         self._model_path = str(Path(self._tmpdir.name) / "model.onnx")
-        ir.save(model, self._model_path, external_data="model.onnx.data")
+        try:
+            ir.save(model, self._model_path, external_data="model.onnx.data")
+        finally:
+            if lower_opset:
+                model.opset_imports[""] = original_opset
 
         self._session = ort_easy.load(self._model_path, **load_kwargs)
         self._input_names = [inp.name for inp in self._session.get_inputs()]
diff --git a/src/mobius/components/__init__.py b/src/mobius/components/__init__.py
index 39f4e1be..36c82aef 100644
--- a/src/mobius/components/__init__.py
+++ b/src/mobius/components/__init__.py
@@ -36,6 +36,7 @@
     "GatedRMSNorm",
     "Gemma3MultiModalProjector",
     "Gemma4AudioEncoder",
+    "ClippableLinear",
     "GroupNorm",
     "GQAContext",
     "INT64_MAX",
@@ -175,7 +176,7 @@
     EncoderDecoderAttention,
 )
 from mobius.components._gated_deltanet import GatedDeltaNet
-from mobius.components._gemma4_audio import Gemma4AudioEncoder
+from mobius.components._gemma4_audio import ClippableLinear, Gemma4AudioEncoder
 from mobius.components._lightning_attention import LightningAttention
 from mobius.components._lora import LoRALinear
 from mobius.components._mamba_block import Mamba2Block, MambaBlock
diff --git a/src/mobius/components/_gemma4_audio.py b/src/mobius/components/_gemma4_audio.py
index 70971258..454dcb31 100644
--- a/src/mobius/components/_gemma4_audio.py
+++ b/src/mobius/components/_gemma4_audio.py
@@ -23,9 +23,10 @@
     attention_context_left=13, output_proj_dims=1536, rms_norm_eps=1e-6,
     residual_weight=0.5, gradient_clipping=1e10
 
-Notes on omissions:
-- ``use_clipped_linears=True``: ``Gemma4ClippableLinear`` buffers are ±inf by
-  default (no-op clamping at inference). Implemented as plain ``Linear``.
+Notes:
+- ``use_clipped_linears=True``: ``Gemma4ClippableLinear`` has learned
+  ``input_{min,max}`` and ``output_{min,max}`` buffers that clamp activations
+  before and after the linear projection. Implemented as ``ClippableLinear``.
 - ``attention_logit_cap=50.0``: Soft-capping ``tanh(logits/cap)*cap`` IS
   implemented using standard ONNX ops (MatMul/Tanh/Div/Mul). The ONNX
   Attention op's native ``softcap`` attribute cannot be used because the
@@ -80,6 +81,50 @@ def _swish(op: builder.OpBuilder, x: ir.Value) -> ir.Value:
 # ---------------------------------------------------------------------------
 
 
+class ClippableLinear(nn.Module):
+    """Linear layer with learned input/output activation clamping.
+
+    Matches ``Gemma4ClippableLinear`` in HuggingFace transformers.
+    The checkpoint stores learned ``input_{min,max}`` and ``output_{min,max}``
+    scalars that clamp activations before and after the linear projection::
+
+        x = clamp(x, input_min, input_max)
+        x = x @ weight.T [+ bias]
+        x = clamp(x, output_min, output_max)
+
+    Args:
+        in_features: Input feature dimension.
+        out_features: Output feature dimension.
+        bias: Whether to include a bias term (default: False).
+    """
+
+    def __init__(
+        self,
+        in_features: int,
+        out_features: int,
+        bias: bool = False,
+    ):
+        super().__init__()
+        self.weight = nn.Parameter([out_features, in_features])
+        self.bias = nn.Parameter([out_features]) if bias else None
+        # Learned activation clipping bounds (scalar)
+        self.input_min = nn.Parameter([])
+        self.input_max = nn.Parameter([])
+        self.output_min = nn.Parameter([])
+        self.output_max = nn.Parameter([])
+
+    def forward(self, op: builder.OpBuilder, x: ir.Value) -> ir.Value:
+        # Clamp input activations
+        x = op.Clip(x, self.input_min, self.input_max)
+        # Linear: x @ weight.T [+ bias]
+        w_t = op.Transpose(self.weight, perm=[1, 0])
+        result = op.MatMul(x, w_t)
+        if self.bias is not None:
+            result = op.Add(result, self.bias)
+        # Clamp output activations
+        return op.Clip(result, self.output_min, self.output_max)
+
+
 class Gemma4ConvSubsampling(nn.Module):
     """2-stage Conv2d subsampling for Gemma4 audio features.
 
@@ -195,8 +240,8 @@ def __init__(
         self._gradient_clipping = gradient_clipping
 
         self.pre_layer_norm = RMSNorm(hidden_size, eps=rms_norm_eps)
-        self.ffw_layer_1 = Linear(hidden_size, hidden_size * 4, bias=False)
-        self.ffw_layer_2 = Linear(hidden_size * 4, hidden_size, bias=False)
+        self.ffw_layer_1 = ClippableLinear(hidden_size, hidden_size * 4, bias=False)
+        self.ffw_layer_2 = ClippableLinear(hidden_size * 4, hidden_size, bias=False)
         self.post_layer_norm = RMSNorm(hidden_size, eps=rms_norm_eps)
 
     def forward(self, op: builder.OpBuilder, x: ir.Value):
@@ -246,10 +291,10 @@ def __init__(
         self._gradient_clipping = gradient_clipping
 
         self.pre_layer_norm = RMSNorm(hidden_size, eps=rms_norm_eps)
-        self.linear_start = Linear(hidden_size, hidden_size * 2, bias=False)
+        self.linear_start = ClippableLinear(hidden_size, hidden_size * 2, bias=False)
         self.depthwise_conv1d = CausalDepthwiseConv1d(hidden_size, conv_kernel_size)
         self.conv_norm = RMSNorm(hidden_size, eps=rms_norm_eps)
-        self.linear_end = Linear(hidden_size, hidden_size, bias=False)
+        self.linear_end = ClippableLinear(hidden_size, hidden_size, bias=False)
 
     def forward(self, op: builder.OpBuilder, x: ir.Value):
         residual = x
@@ -336,11 +381,11 @@ def __init__(
         self._k_scale = math.log(1 + math.e) / math.log(2)
 
         # Q/K/V: no bias (HF nn.Linear(..., bias=False))
-        self.q_proj = Linear(hidden_size, hidden_size, bias=False)
-        self.k_proj = Linear(hidden_size, hidden_size, bias=False)
-        self.v_proj = Linear(hidden_size, hidden_size, bias=False)
+        self.q_proj = ClippableLinear(hidden_size, hidden_size, bias=False)
+        self.k_proj = ClippableLinear(hidden_size, hidden_size, bias=False)
+        self.v_proj = ClippableLinear(hidden_size, hidden_size, bias=False)
         # post: no bias (HF has no .bias key for self_attn.post in checkpoint)
-        self.post = Linear(hidden_size, hidden_size, bias=False)
+        self.post = ClippableLinear(hidden_size, hidden_size, bias=False)
 
         # Learnable per-head-dim scale applied to Q after projection
         self.per_dim_scale = nn.Parameter([self._head_dim])
@@ -383,8 +428,10 @@ def _build_causal_window_mask(self, op: builder.OpBuilder, seq_len: ir.Value) ->
 
         # Causal: j ≤ i  ↔  diff ≥ 0
         causal = op.GreaterOrEqual(diff, zero)  # bool [T, T]
-        # In window: j ≥ i - (context_left - 1)  ↔  diff < context_left
-        ctx = op.Constant(value_int=self._attention_context_left)
+        # In window: j ≥ i - (context_left - 2)  ↔  diff < context_left - 1
+        # HF uses left_window_size = attention_context_left - 1 (e.g. 12 for
+        # config value 13) so the window covers positions [i-11, i] (12 frames).
+        ctx = op.Constant(value_int=self._attention_context_left - 1)
         in_window = op.Less(diff, ctx)  # bool [T, T]
 
         allowed = op.And(causal, in_window)
diff --git a/src/mobius/models/gemma4.py b/src/mobius/models/gemma4.py
index 8e31ff73..2478dc4e 100644
--- a/src/mobius/models/gemma4.py
+++ b/src/mobius/models/gemma4.py
@@ -37,6 +37,7 @@
 from mobius._weight_utils import vlm_decoder_weights, vlm_embedding_weights
 from mobius.components import (
     MLP,
+    ClippableLinear,
     Linear,
     RMSNorm,
     create_attention_bias,
@@ -113,10 +114,10 @@ def __init__(
         super().__init__()
         self.num_heads = num_heads
         self.head_dim = hidden_size // num_heads
-        self.q_proj = Linear(hidden_size, num_heads * self.head_dim, bias=False)
-        self.k_proj = Linear(hidden_size, num_heads * self.head_dim, bias=False)
-        self.v_proj = Linear(hidden_size, num_heads * self.head_dim, bias=False)
-        self.o_proj = Linear(num_heads * self.head_dim, hidden_size, bias=False)
+        self.q_proj = ClippableLinear(hidden_size, num_heads * self.head_dim, bias=False)
+        self.k_proj = ClippableLinear(hidden_size, num_heads * self.head_dim, bias=False)
+        self.v_proj = ClippableLinear(hidden_size, num_heads * self.head_dim, bias=False)
+        self.o_proj = ClippableLinear(num_heads * self.head_dim, hidden_size, bias=False)
         self.q_norm = RMSNorm(self.head_dim, eps=norm_eps)
         self.k_norm = RMSNorm(self.head_dim, eps=norm_eps)
         self.v_norm = _Gemma4ScaleFreeRMSNorm(self.head_dim, eps=norm_eps)
@@ -286,13 +287,15 @@ def __init__(
         self.post_feedforward_layernorm = RMSNorm(hidden_size, eps=norm_eps)
         # Gated MLP: activation(gate_proj) * up_proj -> down_proj (SwiGLU/GEGLU style)
         # HF uses gelu_pytorch_tanh (GELU with tanh approximation); read from config.
+        # Vision encoder uses ClippableLinear for all projections.
         self.mlp = MLP(
             ArchitectureConfig(
                 hidden_size=hidden_size,
                 intermediate_size=intermediate_size,
                 hidden_act=hidden_act,
                 rms_norm_eps=norm_eps,
-            )
+            ),
+            linear_class=ClippableLinear,
         )
 
     def forward(
@@ -1196,11 +1199,20 @@ def __init__(self, config: Gemma4Config):
         if self._per_layer_dim:
             self._num_layers = config.num_hidden_layers
             vocab_per_layer = getattr(config, "vocab_size_per_layer_input", 0)
-            self.embed_tokens_per_layer = Gemma3TextScaledWordEmbedding(
-                vocab_per_layer,
-                config.num_hidden_layers * self._per_layer_dim,
-                config.pad_token_id,
-                embed_scale=float(self._per_layer_dim**0.5),
+            # Use per-layer embedding tables instead of one giant [V, L*D] table.
+            # Each [V, D] table has only V*D elements (e.g. 262144*256 = 67M),
+            # well under the ORT CUDA Gather int32 limit (~2.1B).  This also
+            # avoids the post-Gather reshape and per-layer axis-2 slicing.
+            self.embed_tokens_per_layer = nn.ModuleList(
+                [
+                    Gemma3TextScaledWordEmbedding(
+                        vocab_per_layer,
+                        self._per_layer_dim,
+                        config.pad_token_id,
+                        embed_scale=float(self._per_layer_dim**0.5),
+                    )
+                    for _ in range(config.num_hidden_layers)
+                ]
             )
             self.per_layer_model_projection = Linear(
                 config.hidden_size,
@@ -1216,8 +1228,8 @@ def _compute_per_layer_inputs(
         op: builder.OpBuilder,
         input_ids: ir.Value | None,
         inputs_embeds: ir.Value,
-    ) -> ir.Value | None:
-        """Compute per-layer input embeddings ``[B, S, num_layers, per_layer_dim]``.
+    ) -> list[ir.Value] | None:
+        """Compute per-layer input embeddings, one ``[B, S, per_layer_dim]`` per layer.
 
         HF's ``Gemma4Model.forward`` replaces image/audio token positions with
         ``pad_token_id`` (0) *before* calling ``embed_tokens_per_layer``.  We
@@ -1237,10 +1249,9 @@ def _compute_per_layer_inputs(
         )
         proj = self.per_layer_projection_norm(op, proj)
 
+        # Mask multimodal token IDs to pad_token_id (0)
+        masked_ids: ir.Value | None = None
         if input_ids is not None:
-            # Mask multimodal positions to pad_token_id (0) so image/audio slots
-            # use PAD embeddings in the per-layer path — matching HF line 40:
-            #   llm_input_ids[multimodal_mask] = pad_token_id
             pad = op.Constant(value_int=0)
             masked_ids = input_ids
             if self._image_token_id:
@@ -1255,15 +1266,21 @@ def _compute_per_layer_inputs(
                     pad,
                     masked_ids,
                 )
-            token_emb = self.embed_tokens_per_layer(op, masked_ids)
-            token_emb = op.Reshape(
-                token_emb,
-                op.Constant(value_ints=[0, 0, self._num_layers, self._per_layer_dim]),
-            )
-            proj = op.Add(proj, token_emb)
 
-        # Scale combined result by 2**-0.5 (matches HF project_per_layer_inputs)
-        return op.Mul(proj, float(0.5**0.5))
+        # Per-layer embeddings: each table is [V, per_layer_dim] — small enough
+        # to avoid ORT CUDA Gather int32 overflow (onnxruntime#28107).
+        per_layer_results: list[ir.Value] = []
+        for i in range(self._num_layers):
+            # Slice proj along axis 2 for this layer: [B, S, L, D] → [B, S, D]
+            proj_i = op.Squeeze(op.Slice(proj, starts=[i], ends=[i + 1], axes=[2]), [2])
+
+            if masked_ids is not None:
+                token_emb_i = self.embed_tokens_per_layer[i](op, masked_ids)
+                proj_i = op.Add(proj_i, token_emb_i)
+
+            per_layer_results.append(op.Mul(proj_i, float(0.5**0.5)))
+
+        return per_layer_results
 
     def forward(
         self,
@@ -1324,12 +1341,7 @@ def forward(
         for i, (layer, layer_type, past_kv) in enumerate(
             zip(self.layers, self.layer_types, past_kvs)
         ):
-            if per_layer_inputs is not None:
-                idx = op.Constant(value_ints=[i])
-                pli = op.Gather(per_layer_inputs, idx, axis=2)
-                per_layer_input = op.Squeeze(pli, [2])
-            else:
-                per_layer_input = None
+            per_layer_input = per_layer_inputs[i] if per_layer_inputs is not None else None
             hidden_states, present_kv = layer(
                 op,
                 hidden_states=hidden_states,
@@ -1826,7 +1838,17 @@ def preprocess_weights(
                     renamed["decoder." + suffix] = value
                 else:
                     # All other text weights nest under decoder.model.*
-                    renamed["decoder.model." + suffix] = value
+                    onnx_key = "decoder.model." + suffix
+                    if suffix == "embed_tokens_per_layer.weight":
+                        # HF stores one [V, L*D] weight; split into L separate
+                        # [V, D] tables matching our nn.ModuleList layout.
+                        num_layers = self.config.num_hidden_layers
+                        per_layer_dim = self.decoder.model._per_layer_dim
+                        for i in range(num_layers):
+                            shard = value[:, i * per_layer_dim : (i + 1) * per_layer_dim]
+                            renamed[f"decoder.model.embed_tokens_per_layer.{i}.weight"] = shard
+                    else:
+                        renamed[onnx_key] = value
                     if suffix == "embed_tokens.weight":
                         # Token embedding is shared with the embedding sub-model
                         renamed["embedding.embed_tokens.weight"] = value
diff --git a/testdata/cases/audio/qwen3-asr-2b.yaml b/testdata/cases/audio/qwen3-asr-2b.yaml
deleted file mode 100644
index 9019788d..00000000
--- a/testdata/cases/audio/qwen3-asr-2b.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-model_id: "Qwen/Qwen3-ASR-2B-Instruct"
-revision: "main"
-task_type: "speech-to-text"
-dtype: "float32"
-
-inputs:
-  audio:
-    - "652-129742-0006.flac"
-
-level: "L4"
-
-skip_reason: "Qwen3-ASR-2B is too large for CI golden data generation."
-notes: "Qwen3-ASR-2B-Instruct. Large speech recognition model."
diff --git a/testdata/cases/audio/unispeech-sat-tiny.yaml b/testdata/cases/audio/unispeech-sat-tiny.yaml
index 1edad3ec..89e2463a 100644
--- a/testdata/cases/audio/unispeech-sat-tiny.yaml
+++ b/testdata/cases/audio/unispeech-sat-tiny.yaml
@@ -10,3 +10,5 @@ inputs:
 level: "L4"
 
 notes: "UniSpeechSat tiny random. Speaker-aware self-supervised audio encoder."
+
+skip_reason: "Model only has .bin weights (no safetensors)"
diff --git a/testdata/cases/audio/unispeech-tiny.yaml b/testdata/cases/audio/unispeech-tiny.yaml
index 4e91477b..94837f85 100644
--- a/testdata/cases/audio/unispeech-tiny.yaml
+++ b/testdata/cases/audio/unispeech-tiny.yaml
@@ -10,3 +10,5 @@ inputs:
 level: "L4"
 
 notes: "UniSpeech tiny random. Multi-task audio pre-training encoder."
+
+skip_reason: "Model only has .bin weights (no safetensors)"
diff --git a/testdata/cases/audio/whisper-tiny.yaml b/testdata/cases/audio/whisper-tiny.yaml
index f43f17f2..0eee3675 100644
--- a/testdata/cases/audio/whisper-tiny.yaml
+++ b/testdata/cases/audio/whisper-tiny.yaml
@@ -15,4 +15,4 @@ level: "L4+L5"
 
 notes: "Whisper tiny. Speech-to-text encoder-decoder."
 
-skip_reason: "Whisper encoder needs audio input_features (not text input_ids)"
+skip_reason: ""
diff --git a/testdata/cases/causal-lm/gemma-4-26b-a4b.yaml b/testdata/cases/causal-lm/gemma-4-26b-a4b.yaml
new file mode 100644
index 00000000..7b0501fc
--- /dev/null
+++ b/testdata/cases/causal-lm/gemma-4-26b-a4b.yaml
@@ -0,0 +1,17 @@
+model_id: "google/gemma-4-26B-A4B-it"
+revision: "main"
+task_type: "text-generation"
+dtype: "float32"
+
+inputs:
+  prompts:
+    - "Here is my poem:"
+
+level: "L4+L5"
+
+generation:
+  max_new_tokens: 20
+  do_sample: false
+
+skip_reason: "Gated repo (google/gemma-4-26B-A4B-it requires authentication). Model is 26B — too large for CI."
+notes: "Gemma 4 26B-A4B (text-only path). MoE (64 experts, top-2), KV sharing (num_kv_shared_layers=20). No audio encoder."
diff --git a/testdata/cases/causal-lm/gemma-4-31b.yaml b/testdata/cases/causal-lm/gemma-4-31b.yaml
new file mode 100644
index 00000000..c9acf8dc
--- /dev/null
+++ b/testdata/cases/causal-lm/gemma-4-31b.yaml
@@ -0,0 +1,17 @@
+model_id: "google/gemma-4-31B-it"
+revision: "main"
+task_type: "text-generation"
+dtype: "float32"
+
+inputs:
+  prompts:
+    - "Here is my poem:"
+
+level: "L4+L5"
+
+generation:
+  max_new_tokens: 20
+  do_sample: false
+
+skip_reason: "Gated repo (google/gemma-4-31B-it requires authentication). Model is 31B — too large for CI."
+notes: "Gemma 4 31B dense (text-only path). KV sharing, sliding+global attention, logit softcapping. No audio encoder, no MoE."
diff --git a/testdata/cases/causal-lm/gemma-4-e2b.yaml b/testdata/cases/causal-lm/gemma-4-e2b.yaml
index ce8d59f2..0efd812a 100644
--- a/testdata/cases/causal-lm/gemma-4-e2b.yaml
+++ b/testdata/cases/causal-lm/gemma-4-e2b.yaml
@@ -13,5 +13,4 @@ generation:
   max_new_tokens: 20
   do_sample: false
 
-skip_reason: "Gated repo (google/gemma-4-E2B-it requires authentication). Model is 2B but gated."
 notes: "Gemma 4 E2B Any-to-Any (text-only path). Dual RoPE, KV sharing (num_kv_shared_layers=20), double-wide MLP, sliding+global attention, logit softcapping."
diff --git a/testdata/cases/causal-lm/gemma-4-e4b.yaml b/testdata/cases/causal-lm/gemma-4-e4b.yaml
new file mode 100644
index 00000000..e97b8bb5
--- /dev/null
+++ b/testdata/cases/causal-lm/gemma-4-e4b.yaml
@@ -0,0 +1,17 @@
+model_id: "google/gemma-4-E4B-it"
+revision: "main"
+task_type: "text-generation"
+dtype: "float32"
+
+inputs:
+  prompts:
+    - "Here is my poem:"
+
+level: "L4+L5"
+
+generation:
+  max_new_tokens: 20
+  do_sample: false
+
+skip_reason: "Gated repo (google/gemma-4-E4B-it requires authentication)."
+notes: "Gemma 4 E4B Any-to-Any (text-only path). Dual RoPE, KV sharing, double-wide MLP, sliding+global attention, logit softcapping, audio encoder."
diff --git a/testdata/cases/speech/gemma-4-e2b-it-audio.yaml b/testdata/cases/speech/gemma-4-e2b-it-audio.yaml
new file mode 100644
index 00000000..270cf10f
--- /dev/null
+++ b/testdata/cases/speech/gemma-4-e2b-it-audio.yaml
@@ -0,0 +1,18 @@
+model_id: "google/gemma-4-E2B-it"
+revision: "main"
+task_type: "speech-language"
+dtype: "float32"
+
+inputs:
+  prompts:
+    - "Transcribe this audio."
+  audio:
+    - "652-129742-0006.flac"
+
+level: "L4+L5"
+
+generation:
+  max_new_tokens: 50
+  do_sample: false
+
+notes: "Gemma 4 E2B audio input. 4-model split: decoder + vision + audio + embedding. Gemma4AudioEncoder with ClippableLinear layers."
diff --git a/testdata/cases/speech/gemma-4-e4b-it-audio.yaml b/testdata/cases/speech/gemma-4-e4b-it-audio.yaml
new file mode 100644
index 00000000..f270332a
--- /dev/null
+++ b/testdata/cases/speech/gemma-4-e4b-it-audio.yaml
@@ -0,0 +1,19 @@
+model_id: "google/gemma-4-E4B-it"
+revision: "main"
+task_type: "speech-language"
+dtype: "float32"
+
+inputs:
+  prompts:
+    - "Transcribe this audio."
+  audio:
+    - "652-129742-0006.flac"
+
+level: "L4+L5"
+
+generation:
+  max_new_tokens: 50
+  do_sample: false
+
+skip_reason: "Large model, requires significant compute."
+notes: "Gemma 4 E4B audio input. 4-model split: decoder + vision + audio + embedding. Gemma4AudioEncoder with ClippableLinear layers."
diff --git a/testdata/cases/speech/qwen3-asr.yaml b/testdata/cases/speech/qwen3-asr.yaml
index 22c136b6..dcb1828b 100644
--- a/testdata/cases/speech/qwen3-asr.yaml
+++ b/testdata/cases/speech/qwen3-asr.yaml
@@ -1,9 +1,12 @@
-model_id: "Qwen/Qwen3-ASR"
+model_id: "Qwen/Qwen3-ASR-0.6B"
 revision: "main"
 task_type: "speech-language"
 dtype: "float32"
+trust_remote_code: true
 
 inputs:
+  prompts:
+    - "Transcribe this audio."
   audio:
     - "652-129742-0006.flac"
 
@@ -13,5 +16,5 @@ generation:
 
 level: "L4+L5"
 
-skip_reason: "Requires audio preprocessing pipeline not yet supported in test harness."
+skip_reason: ""
 notes: "Qwen3-ASR speech recognition. 3-model split: audio_encoder + embedding + decoder."
diff --git a/testdata/cases/vision-language/gemma-4-26b-a4b-it.yaml b/testdata/cases/vision-language/gemma-4-26b-a4b-it.yaml
index 5636987f..ac1dbeb1 100644
--- a/testdata/cases/vision-language/gemma-4-26b-a4b-it.yaml
+++ b/testdata/cases/vision-language/gemma-4-26b-a4b-it.yaml
@@ -15,5 +15,5 @@ generation:
   max_new_tokens: 30
   do_sample: false
 
-skip_reason: "Gated repo (google/gemma-4-26B-A4B-it requires authentication). Model is 26B — too large for CI golden data generation."
+skip_reason: "Gated repo (google/gemma-4-26B-A4B-it requires authentication). Model is 26B — too large for CI."
 notes: "Gemma 4 26B-A4B Image-Text-to-Text. 3-model split: decoder + vision + embedding. MoE layers, KV sharing (num_kv_shared_layers=20). No audio encoder."
diff --git a/testdata/cases/vision-language/gemma-4-31b-it.yaml b/testdata/cases/vision-language/gemma-4-31b-it.yaml
new file mode 100644
index 00000000..5336cbde
--- /dev/null
+++ b/testdata/cases/vision-language/gemma-4-31b-it.yaml
@@ -0,0 +1,19 @@
+model_id: "google/gemma-4-31B-it"
+revision: "main"
+task_type: "image-text-to-text"
+dtype: "float32"
+
+inputs:
+  prompts:
+    - "Describe this image in detail."
+  images:
+    - "pipeline-cat-chonk.jpeg"
+
+level: "L4+L5"
+
+generation:
+  max_new_tokens: 30
+  do_sample: false
+
+skip_reason: "Gated repo (google/gemma-4-31B-it requires authentication). Model is 31B — too large for CI."
+notes: "Gemma 4 31B dense Image-Text-to-Text. 3-model split: decoder + vision + embedding. No audio encoder, no MoE."
diff --git a/testdata/cases/vision-language/gemma-4-e2b-it.yaml b/testdata/cases/vision-language/gemma-4-e2b-it.yaml
index c1d6574d..142df219 100644
--- a/testdata/cases/vision-language/gemma-4-e2b-it.yaml
+++ b/testdata/cases/vision-language/gemma-4-e2b-it.yaml
@@ -15,5 +15,4 @@ generation:
   max_new_tokens: 30
   do_sample: false
 
-skip_reason: "Gated repo (google/gemma-4-E2B-it requires authentication)."
 notes: "Gemma 4 E2B Vision-Language. 3-model split: decoder + vision + embedding. Pre-patchified vision input (pixel_values [B,N,3P^2] + pixel_position_ids)."
diff --git a/testdata/cases/vision-language/gemma-4-e4b-it.yaml b/testdata/cases/vision-language/gemma-4-e4b-it.yaml
new file mode 100644
index 00000000..f8498615
--- /dev/null
+++ b/testdata/cases/vision-language/gemma-4-e4b-it.yaml
@@ -0,0 +1,19 @@
+model_id: "google/gemma-4-E4B-it"
+revision: "main"
+task_type: "image-text-to-text"
+dtype: "float32"
+
+inputs:
+  prompts:
+    - "Describe this image in detail."
+  images:
+    - "pipeline-cat-chonk.jpeg"
+
+level: "L4+L5"
+
+generation:
+  max_new_tokens: 30
+  do_sample: false
+
+skip_reason: "Gated repo (google/gemma-4-E4B-it requires authentication)."
+notes: "Gemma 4 E4B Vision-Language. 3-model split: decoder + vision + embedding. Pre-patchified vision input (pixel_values [B,N,3P^2] + pixel_position_ids). Audio encoder also present."
diff --git a/testdata/golden/audio/unispeech-sat-tiny.json b/testdata/golden/audio/unispeech-sat-tiny.json
new file mode 100644
index 00000000..951eb0b7
--- /dev/null
+++ b/testdata/golden/audio/unispeech-sat-tiny.json
@@ -0,0 +1,37 @@
+{
+  "top1_id": 1,
+  "top2_id": 8,
+  "top10_ids": [
+    1,
+    8,
+    2,
+    6,
+    14,
+    5,
+    9,
+    10,
+    0,
+    12
+  ],
+  "top10_logits": [
+    "0x1.16e0580000000p+0",
+    "0x1.05e64a0000000p+0",
+    "0x1.fc27b60000000p-1",
+    "0x1.facd2a0000000p-1",
+    "0x1.82c97e0000000p-1",
+    "0x1.5264860000000p-1",
+    "0x1.fb07cc0000000p-2",
+    "0x1.8171b80000000p-2",
+    "0x1.21baf00000000p-2",
+    "0x1.198ec40000000p-2"
+  ],
+  "logits_summary": [
+    "0x1.16e0580000000p+0",
+    "-0x1.0f72300000000p+1",
+    "0x1.4000000000000p-26",
+    "0x1.ffff578727788p-1"
+  ],
+  "input_ids": [
+    0
+  ]
+}
diff --git a/testdata/golden/audio/unispeech-tiny.json b/testdata/golden/audio/unispeech-tiny.json
new file mode 100644
index 00000000..e137afbc
--- /dev/null
+++ b/testdata/golden/audio/unispeech-tiny.json
@@ -0,0 +1,37 @@
+{
+  "top1_id": 3,
+  "top2_id": 1,
+  "top10_ids": [
+    3,
+    1,
+    9,
+    12,
+    5,
+    7,
+    15,
+    6,
+    11,
+    13
+  ],
+  "top10_logits": [
+    "0x1.3660120000000p+1",
+    "0x1.c42bd20000000p-1",
+    "0x1.bd64680000000p-1",
+    "0x1.70382a0000000p-1",
+    "0x1.07aebe0000000p-1",
+    "0x1.6054380000000p-2",
+    "0x1.2c247c0000000p-3",
+    "0x1.33e2da0000000p-5",
+    "-0x1.18fce40000000p-7",
+    "-0x1.945f5a0000000p-6"
+  ],
+  "logits_summary": [
+    "0x1.3660120000000p+1",
+    "-0x1.c8bc300000000p+0",
+    "0x1.a400000000000p-25",
+    "0x1.fffb29aa8d27fp-1"
+  ],
+  "input_ids": [
+    0
+  ]
+}
diff --git a/testdata/golden/audio/whisper-tiny.json b/testdata/golden/audio/whisper-tiny.json
index ae4d89c7..b983a4fa 100644
--- a/testdata/golden/audio/whisper-tiny.json
+++ b/testdata/golden/audio/whisper-tiny.json
@@ -14,22 +14,22 @@
     50260
   ],
   "top10_logits": [
-    "0x1.7c7fe80000000p+4",
-    "0x1.3fe61a0000000p+4",
-    "0x1.f79f7e0000000p+3",
-    "0x1.f30d780000000p+3",
-    "0x1.ed4c2a0000000p+3",
-    "0x1.eb5cc00000000p+3",
-    "0x1.ce3a540000000p+3",
-    "0x1.ce205c0000000p+3",
-    "0x1.c46df80000000p+3",
-    "0x1.be00bc0000000p+3"
+    "0x1.7c7fee0000000p+4",
+    "0x1.3fe6280000000p+4",
+    "0x1.f79f900000000p+3",
+    "0x1.f30d700000000p+3",
+    "0x1.ed4c340000000p+3",
+    "0x1.eb5cca0000000p+3",
+    "0x1.ce3a460000000p+3",
+    "0x1.ce20440000000p+3",
+    "0x1.c46df00000000p+3",
+    "0x1.be00b00000000p+3"
   ],
   "logits_summary": [
-    "0x1.7c7fe80000000p+4",
-    "-0x1.37f2000000000p+3",
-    "0x1.794745645896ep+1",
-    "0x1.8fe40dbf892e5p+0"
+    "0x1.7c7fee0000000p+4",
+    "-0x1.37f1f40000000p+3",
+    "0x1.794743ddc4b5cp+1",
+    "0x1.8fe40a48acec0p+0"
   ],
   "input_ids": [
     50258
diff --git a/testdata/golden/audio/whisper-tiny_generation.json b/testdata/golden/audio/whisper-tiny_generation.json
new file mode 100644
index 00000000..8bb08cf3
--- /dev/null
+++ b/testdata/golden/audio/whisper-tiny_generation.json
@@ -0,0 +1,30 @@
+{
+  "model_id": "openai/whisper-tiny",
+  "prompt": "652-129742-0006.flac",
+  "generated_tokens": [
+    7807,
+    295,
+    34993,
+    34406,
+    13,
+    3664,
+    3554,
+    21058,
+    43125,
+    11,
+    1821,
+    666,
+    14770,
+    11,
+    5127,
+    5139,
+    11,
+    8532,
+    293,
+    18030,
+    281,
+    3196,
+    13
+  ],
+  "generated_text": " Call of Flower mayonnaise. Take cold boiled cauliflower, break into branches, adding salt, pepper and vinegar to season."
+}
diff --git a/testdata/golden/causal-lm/gemma-4-26b-a4b.json b/testdata/golden/causal-lm/gemma-4-26b-a4b.json
new file mode 100644
index 00000000..e31d9eb7
--- /dev/null
+++ b/testdata/golden/causal-lm/gemma-4-26b-a4b.json
@@ -0,0 +1,41 @@
+{
+  "top1_id": 563,
+  "top2_id": 642,
+  "top10_ids": [
+    563,
+    642,
+    108,
+    768,
+    564,
+    107,
+    914,
+    1017,
+    623,
+    236743
+  ],
+  "top10_logits": [
+    "0x1.4521ba0000000p+3",
+    "0x1.7273aa0000000p+2",
+    "0x1.4ee5380000000p+2",
+    "0x1.1b08f20000000p+2",
+    "0x1.1aa4d60000000p+2",
+    "0x1.11c3d40000000p+2",
+    "0x1.082f140000000p+2",
+    "0x1.e178920000000p+1",
+    "0x1.b4c8540000000p+1",
+    "0x1.a5f3bc0000000p+1"
+  ],
+  "logits_summary": [
+    "0x1.4521ba0000000p+3",
+    "-0x1.7d35760000000p+4",
+    "-0x1.dd30dee90da40p+3",
+    "0x1.84cf73a4cb9e5p+1"
+  ],
+  "input_ids": [
+    8291,
+    563,
+    1041,
+    27355,
+    236787
+  ]
+}
diff --git a/testdata/golden/causal-lm/gemma-4-26b-a4b_generation.json b/testdata/golden/causal-lm/gemma-4-26b-a4b_generation.json
new file mode 100644
index 00000000..a213bed9
--- /dev/null
+++ b/testdata/golden/causal-lm/gemma-4-26b-a4b_generation.json
@@ -0,0 +1,27 @@
+{
+  "model_id": "google/gemma-4-26B-A4B-it",
+  "prompt": "Here is my poem:",
+  "generated_tokens": [
+    563,
+    1041,
+    27355,
+    236787,
+    563,
+    1041,
+    27355,
+    236787,
+    563,
+    1041,
+    27355,
+    236787,
+    563,
+    1041,
+    27355,
+    236787,
+    563,
+    1041,
+    27355,
+    236787
+  ],
+  "generated_text": " is my poem: is my poem: is my poem: is my poem: is my poem:"
+}
diff --git a/testdata/golden/causal-lm/gemma-4-31b.json b/testdata/golden/causal-lm/gemma-4-31b.json
new file mode 100644
index 00000000..1c990e95
--- /dev/null
+++ b/testdata/golden/causal-lm/gemma-4-31b.json
@@ -0,0 +1,41 @@
+{
+  "top1_id": 1041,
+  "top2_id": 3307,
+  "top10_ids": [
+    1041,
+    3307,
+    240017,
+    759,
+    236762,
+    506,
+    238378,
+    496,
+    571,
+    537
+  ],
+  "top10_logits": [
+    "0x1.ff8efe0000000p+3",
+    "0x1.bb271c0000000p+3",
+    "0x1.699eee0000000p+3",
+    "0x1.64ef940000000p+3",
+    "0x1.40d6a40000000p+3",
+    "0x1.3a49ce0000000p+3",
+    "0x1.3007d80000000p+3",
+    "0x1.27f5920000000p+3",
+    "0x1.1c9d740000000p+3",
+    "0x1.1af9c00000000p+3"
+  ],
+  "logits_summary": [
+    "0x1.ff8efe0000000p+3",
+    "-0x1.3cceec0000000p+4",
+    "-0x1.d9e1359f1bd20p+2",
+    "0x1.89543352f8eb0p+1"
+  ],
+  "input_ids": [
+    8291,
+    563,
+    1041,
+    27355,
+    236787
+  ]
+}
diff --git a/testdata/golden/causal-lm/gemma-4-31b_generation.json b/testdata/golden/causal-lm/gemma-4-31b_generation.json
new file mode 100644
index 00000000..a6bec907
--- /dev/null
+++ b/testdata/golden/causal-lm/gemma-4-31b_generation.json
@@ -0,0 +1,27 @@
+{
+  "model_id": "google/gemma-4-31B-it",
+  "prompt": "Here is my poem:",
+  "generated_tokens": [
+    1041,
+    27355,
+    236787,
+    1041,
+    27355,
+    236787,
+    1041,
+    27355,
+    236787,
+    1041,
+    27355,
+    236787,
+    1041,
+    27355,
+    236787,
+    1041,
+    27355,
+    236787,
+    1041,
+    27355
+  ],
+  "generated_text": " my poem: my poem: my poem: my poem: my poem: my poem: my poem"
+}
diff --git a/testdata/golden/causal-lm/gemma-4-e2b.json b/testdata/golden/causal-lm/gemma-4-e2b.json
new file mode 100644
index 00000000..f8e50178
--- /dev/null
+++ b/testdata/golden/causal-lm/gemma-4-e2b.json
@@ -0,0 +1,41 @@
+{
+  "top1_id": 5715,
+  "top2_id": 1041,
+  "top10_ids": [
+    5715,
+    1041,
+    1590,
+    8291,
+    563,
+    106,
+    1174,
+    108,
+    532,
+    107
+  ],
+  "top10_logits": [
+    "-0x1.a8075e0000000p+2",
+    "-0x1.8bf65e0000000p+3",
+    "-0x1.9abfcc0000000p+3",
+    "-0x1.b3b2880000000p+3",
+    "-0x1.d7dfb80000000p+3",
+    "-0x1.efa9840000000p+3",
+    "-0x1.09d00a0000000p+4",
+    "-0x1.0da9860000000p+4",
+    "-0x1.10a1720000000p+4",
+    "-0x1.1287920000000p+4"
+  ],
+  "logits_summary": [
+    "-0x1.a8075e0000000p+2",
+    "-0x1.d80c380000000p+4",
+    "-0x1.ba689efc82a00p+4",
+    "0x1.bb8275bb18aacp-1"
+  ],
+  "input_ids": [
+    8291,
+    563,
+    1041,
+    27355,
+    236787
+  ]
+}
diff --git a/testdata/golden/causal-lm/gemma-4-e2b_generation.json b/testdata/golden/causal-lm/gemma-4-e2b_generation.json
new file mode 100644
index 00000000..f3bfd5a3
--- /dev/null
+++ b/testdata/golden/causal-lm/gemma-4-e2b_generation.json
@@ -0,0 +1,27 @@
+{
+  "model_id": "google/gemma-4-E2B-it",
+  "prompt": "Here is my poem:",
+  "generated_tokens": [
+    5715,
+    563,
+    1041,
+    27355,
+    236787,
+    5715,
+    563,
+    1041,
+    27355,
+    236787,
+    5715,
+    563,
+    1041,
+    27355,
+    236787,
+    5715,
+    563,
+    1041,
+    27355,
+    236787
+  ],
+  "generated_text": " Here is my poem: Here is my poem: Here is my poem: Here is my poem:"
+}
diff --git a/testdata/golden/causal-lm/gemma-4-e4b.json b/testdata/golden/causal-lm/gemma-4-e4b.json
new file mode 100644
index 00000000..8a50bb6b
--- /dev/null
+++ b/testdata/golden/causal-lm/gemma-4-e4b.json
@@ -0,0 +1,41 @@
+{
+  "top1_id": 5715,
+  "top2_id": 1590,
+  "top10_ids": [
+    5715,
+    1590,
+    107,
+    108,
+    8291,
+    27355,
+    563,
+    1174,
+    29195,
+    568
+  ],
+  "top10_logits": [
+    "0x1.1effe80000000p+4",
+    "0x1.81ee9c0000000p+3",
+    "0x1.72e22a0000000p+3",
+    "0x1.6d1c380000000p+3",
+    "0x1.6aba360000000p+3",
+    "0x1.4159400000000p+3",
+    "0x1.3cb4c80000000p+3",
+    "0x1.2a817c0000000p+3",
+    "0x1.2332e40000000p+3",
+    "0x1.22f51c0000000p+3"
+  ],
+  "logits_summary": [
+    "0x1.1effe80000000p+4",
+    "-0x1.5d25220000000p+4",
+    "-0x1.6c60be83b7e6ep+3",
+    "0x1.83370bea84159p+1"
+  ],
+  "input_ids": [
+    8291,
+    563,
+    1041,
+    27355,
+    236787
+  ]
+}
diff --git a/testdata/golden/causal-lm/gemma-4-e4b_generation.json b/testdata/golden/causal-lm/gemma-4-e4b_generation.json
new file mode 100644
index 00000000..2cb047a1
--- /dev/null
+++ b/testdata/golden/causal-lm/gemma-4-e4b_generation.json
@@ -0,0 +1,27 @@
+{
+  "model_id": "google/gemma-4-E4B-it",
+  "prompt": "Here is my poem:",
+  "generated_tokens": [
+    5715,
+    8291,
+    563,
+    1041,
+    27355,
+    236787,
+    5715,
+    8291,
+    563,
+    1041,
+    27355,
+    236787,
+    5715,
+    8291,
+    563,
+    1041,
+    27355,
+    236787,
+    5715,
+    8291
+  ],
+  "generated_text": " HereHere is my poem: HereHere is my poem: HereHere is my poem: HereHere"
+}
diff --git a/testdata/golden/speech/gemma-4-e2b-it-audio.json b/testdata/golden/speech/gemma-4-e2b-it-audio.json
new file mode 100644
index 00000000..a3cabe77
--- /dev/null
+++ b/testdata/golden/speech/gemma-4-e2b-it-audio.json
@@ -0,0 +1,280 @@
+{
+  "top1_id": 1980,
+  "top2_id": 17832,
+  "top10_ids": [
+    1980,
+    17832,
+    236780,
+    17095,
+    236755,
+    3284,
+    63893,
+    1408,
+    138,
+    1018
+  ],
+  "top10_logits": [
+    "-0x1.c61ac20000000p+1",
+    "-0x1.d4dfe40000000p+1",
+    "-0x1.0e4a300000000p+2",
+    "-0x1.4b56480000000p+3",
+    "-0x1.d0422a0000000p+3",
+    "-0x1.e300f00000000p+3",
+    "-0x1.0a7cbc0000000p+4",
+    "-0x1.0ba0660000000p+4",
+    "-0x1.1238840000000p+4",
+    "-0x1.18bf9c0000000p+4"
+  ],
+  "logits_summary": [
+    "-0x1.c61ac20000000p+1",
+    "-0x1.dfd4400000000p+4",
+    "-0x1.d6c2d961af700p+4",
+    "0x1.45b30cce8f27ep-1"
+  ],
+  "input_ids": [
+    2,
+    105,
+    2364,
+    107,
+    256000,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258883,
+    5183,
+    17038,
+    672,
+    9855,
+    236761,
+    106,
+    107,
+    105,
+    4368,
+    107
+  ]
+}
diff --git a/testdata/golden/speech/gemma-4-e2b-it-audio_generation.json b/testdata/golden/speech/gemma-4-e2b-it-audio_generation.json
new file mode 100644
index 00000000..59a608df
--- /dev/null
+++ b/testdata/golden/speech/gemma-4-e2b-it-audio_generation.json
@@ -0,0 +1,30 @@
+{
+  "model_id": "google/gemma-4-E2B-it",
+  "prompt": "Transcribe this audio.",
+  "generated_tokens": [
+    1980,
+    86742,
+    123051,
+    236761,
+    12774,
+    7445,
+    50890,
+    106377,
+    236764,
+    2541,
+    1131,
+    15092,
+    236764,
+    8009,
+    9551,
+    236764,
+    22300,
+    532,
+    48399,
+    531,
+    3409,
+    236761,
+    106
+  ],
+  "generated_text": "Coliflower mayonnaise. Take cold boiled cauliflower, break into branches, adding salt, pepper and vinegar to season."
+}
diff --git a/testdata/golden/speech/gemma-4-e4b-it-audio.json b/testdata/golden/speech/gemma-4-e4b-it-audio.json
new file mode 100644
index 00000000..e67ac953
--- /dev/null
+++ b/testdata/golden/speech/gemma-4-e4b-it-audio.json
@@ -0,0 +1,280 @@
+{
+  "top1_id": 1980,
+  "top2_id": 17832,
+  "top10_ids": [
+    1980,
+    17832,
+    7029,
+    82747,
+    17095,
+    236775,
+    3284,
+    2542,
+    37967,
+    35838
+  ],
+  "top10_logits": [
+    "0x1.7a96440000000p+4",
+    "0x1.7853300000000p+4",
+    "0x1.4dfc160000000p+4",
+    "0x1.4970620000000p+4",
+    "0x1.3bf0b40000000p+4",
+    "0x1.2b2e840000000p+4",
+    "0x1.252e500000000p+4",
+    "0x1.0ff69e0000000p+4",
+    "0x1.0ecdd80000000p+4",
+    "0x1.e4ea280000000p+3"
+  ],
+  "logits_summary": [
+    "0x1.7a96440000000p+4",
+    "-0x1.de5fd80000000p+4",
+    "-0x1.951c05e817e67p+4",
+    "0x1.028d05a0556f1p+2"
+  ],
+  "input_ids": [
+    2,
+    105,
+    2364,
+    107,
+    256000,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258881,
+    258883,
+    5183,
+    17038,
+    672,
+    9855,
+    236761,
+    106,
+    107,
+    105,
+    4368,
+    107
+  ]
+}
diff --git a/testdata/golden/speech/gemma-4-e4b-it-audio_generation.json b/testdata/golden/speech/gemma-4-e4b-it-audio_generation.json
new file mode 100644
index 00000000..30043657
--- /dev/null
+++ b/testdata/golden/speech/gemma-4-e4b-it-audio_generation.json
@@ -0,0 +1,30 @@
+{
+  "model_id": "google/gemma-4-E4B-it",
+  "prompt": "Transcribe this audio.",
+  "generated_tokens": [
+    1980,
+    86742,
+    123051,
+    236761,
+    12774,
+    7445,
+    50890,
+    106377,
+    236764,
+    2541,
+    1131,
+    15092,
+    236764,
+    8009,
+    9551,
+    236764,
+    22300,
+    532,
+    48399,
+    531,
+    3409,
+    236761,
+    106
+  ],
+  "generated_text": "Coliflower mayonnaise. Take cold boiled cauliflower, break into branches, adding salt, pepper and vinegar to season."
+}
diff --git a/testdata/golden/speech/qwen3-asr.json b/testdata/golden/speech/qwen3-asr.json
new file mode 100644
index 00000000..ee7d51ab
--- /dev/null
+++ b/testdata/golden/speech/qwen3-asr.json
@@ -0,0 +1,170 @@
+{
+  "top1_id": 11528,
+  "top2_id": 4128,
+  "top10_ids": [
+    11528,
+    4128,
+    102064,
+    6207,
+    13806,
+    151645,
+    73353,
+    29021,
+    31633,
+    60740
+  ],
+  "top10_logits": [
+    "0x1.bdd5f80000000p+4",
+    "0x1.3ade700000000p+4",
+    "0x1.25e2640000000p+4",
+    "0x1.f50dda0000000p+3",
+    "0x1.e830be0000000p+3",
+    "0x1.e3a7700000000p+3",
+    "0x1.e15bca0000000p+3",
+    "0x1.d6fb280000000p+3",
+    "0x1.cc90c20000000p+3",
+    "0x1.bea7380000000p+3"
+  ],
+  "logits_summary": [
+    "-0x1.2516140000000p+1",
+    "0x1.d376820000000p+1",
+    "-0x1.1ff44e0000000p+4",
+    "0x1.bdd5f80000000p+4"
+  ],
+  "input_ids": [
+    151644,
+    8948,
+    198,
+    151645,
+    198,
+    151644,
+    872,
+    198,
+    151669,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151676,
+    151670,
+    151645,
+    198,
+    151644,
+    77091,
+    198
+  ]
+}
diff --git a/testdata/golden/speech/qwen3-asr_generation.json b/testdata/golden/speech/qwen3-asr_generation.json
new file mode 100644
index 00000000..a0e5e607
--- /dev/null
+++ b/testdata/golden/speech/qwen3-asr_generation.json
@@ -0,0 +1,37 @@
+{
+  "model_id": "Qwen/Qwen3-ASR-0.6B",
+  "prompt": "testdata/652-129742-0006.flac",
+  "generated_tokens": [
+    11528,
+    6364,
+    151704,
+    34,
+    4943,
+    76773,
+    1231,
+    13459,
+    1064,
+    25,
+    11778,
+    9255,
+    65085,
+    95870,
+    11,
+    1438,
+    1119,
+    23091,
+    11,
+    7842,
+    12021,
+    11,
+    24353,
+    11,
+    323,
+    46105,
+    311,
+    3200,
+    13,
+    151645
+  ],
+  "generated_text": "language English<asr_text>Cauliflower mayonnaise: Take cold boiled cauliflower, break into branches, adding salt, pepper, and vinegar to season."
+}
\ No newline at end of file
diff --git a/testdata/golden/vision-language/gemma-4-26b-a4b-it.json b/testdata/golden/vision-language/gemma-4-26b-a4b-it.json
new file mode 100644
index 00000000..adc7c7da
--- /dev/null
+++ b/testdata/golden/vision-language/gemma-4-26b-a4b-it.json
@@ -0,0 +1,323 @@
+{
+  "top1_id": 236776,
+  "top2_id": 902,
+  "top10_ids": [
+    236776,
+    902,
+    2267,
+    2094,
+    818,
+    31534,
+    13238,
+    13973,
+    195463,
+    91561
+  ],
+  "top10_logits": [
+    "0x1.bb88b80000000p+4",
+    "0x1.6fbcda0000000p+4",
+    "0x1.6ef8dc0000000p+4",
+    "0x1.6097e40000000p+4",
+    "0x1.4fcc7c0000000p+4",
+    "0x1.2ae5d40000000p+4",
+    "0x1.2195140000000p+4",
+    "0x1.20134c0000000p+4",
+    "0x1.1f27260000000p+4",
+    "0x1.1c632a0000000p+4"
+  ],
+  "logits_summary": [
+    "0x1.bb88b80000000p+4",
+    "-0x1.a8824a0000000p+4",
+    "-0x1.004256afc37dfp+3",
+    "0x1.f17a4df0e3b27p+1"
+  ],
+  "input_ids": [
+    2,
+    105,
+    2364,
+    107,
+    255999,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258882,
+    82858,
+    672,
+    2471,
+    528,
+    8052,
+    236761,
+    106,
+    107,
+    105,
+    4368,
+    107,
+    100,
+    45518,
+    107,
+    101
+  ]
+}
diff --git a/testdata/golden/vision-language/gemma-4-26b-a4b-it_generation.json b/testdata/golden/vision-language/gemma-4-26b-a4b-it_generation.json
new file mode 100644
index 00000000..c13b1d7a
--- /dev/null
+++ b/testdata/golden/vision-language/gemma-4-26b-a4b-it_generation.json
@@ -0,0 +1,37 @@
+{
+  "model_id": "google/gemma-4-26B-A4B-it",
+  "prompt": "Describe this image in detail.",
+  "generated_tokens": [
+    236776,
+    7860,
+    5719,
+    3831,
+    496,
+    7497,
+    236772,
+    236760,
+    39046,
+    236764,
+    8614,
+    532,
+    12819,
+    593,
+    18230,
+    236789,
+    236751,
+    5866,
+    9378,
+    699,
+    2378,
+    531,
+    1447,
+    580,
+    496,
+    7613,
+    236772,
+    26987,
+    3761,
+    236761
+  ],
+  "generated_text": "A medium shot shows a thick-furred, tan and gray Pallas's cat walking from left to right on a snow-covered surface."
+}
diff --git a/testdata/golden/vision-language/gemma-4-31b-it.json b/testdata/golden/vision-language/gemma-4-31b-it.json
new file mode 100644
index 00000000..1c6ef452
--- /dev/null
+++ b/testdata/golden/vision-language/gemma-4-31b-it.json
@@ -0,0 +1,323 @@
+{
+  "top1_id": 236776,
+  "top2_id": 2267,
+  "top10_ids": [
+    236776,
+    2267,
+    902,
+    92736,
+    2094,
+    818,
+    562,
+    125617,
+    195463,
+    13973
+  ],
+  "top10_logits": [
+    "0x1.c8c0a40000000p+4",
+    "0x1.6f78de0000000p+4",
+    "0x1.2e78320000000p+4",
+    "0x1.299e400000000p+4",
+    "0x1.2352dc0000000p+4",
+    "0x1.217a280000000p+4",
+    "0x1.12cb1a0000000p+4",
+    "0x1.119bac0000000p+4",
+    "0x1.0c4c540000000p+4",
+    "0x1.0a902e0000000p+4"
+  ],
+  "logits_summary": [
+    "0x1.c8c0a40000000p+4",
+    "-0x1.582e000000000p+4",
+    "-0x1.5634f38f0645ep+1",
+    "0x1.fcbe18bf553b7p+1"
+  ],
+  "input_ids": [
+    2,
+    105,
+    2364,
+    107,
+    255999,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258882,
+    82858,
+    672,
+    2471,
+    528,
+    8052,
+    236761,
+    106,
+    107,
+    105,
+    4368,
+    107,
+    100,
+    45518,
+    107,
+    101
+  ]
+}
diff --git a/testdata/golden/vision-language/gemma-4-31b-it_generation.json b/testdata/golden/vision-language/gemma-4-31b-it_generation.json
new file mode 100644
index 00000000..3070b670
--- /dev/null
+++ b/testdata/golden/vision-language/gemma-4-31b-it_generation.json
@@ -0,0 +1,37 @@
+{
+  "model_id": "google/gemma-4-31B-it",
+  "prompt": "Describe this image in detail.",
+  "generated_tokens": [
+    236776,
+    1494,
+    236772,
+    2925,
+    236764,
+    7860,
+    5719,
+    3831,
+    496,
+    593,
+    18230,
+    236789,
+    236751,
+    5866,
+    9378,
+    699,
+    2378,
+    531,
+    1447,
+    3418,
+    496,
+    7613,
+    236772,
+    26987,
+    3761,
+    236761,
+    669,
+    5866,
+    563,
+    496
+  ],
+  "generated_text": "A high-angle, medium shot shows a Pallas's cat walking from left to right across a snow-covered surface. The cat is a"
+}
diff --git a/testdata/golden/vision-language/gemma-4-e2b-it.json b/testdata/golden/vision-language/gemma-4-e2b-it.json
new file mode 100644
index 00000000..cbbc3ff4
--- /dev/null
+++ b/testdata/golden/vision-language/gemma-4-e2b-it.json
@@ -0,0 +1,319 @@
+{
+  "top1_id": 2094,
+  "top2_id": 8291,
+  "top10_ids": [
+    2094,
+    8291,
+    236776,
+    902,
+    1408,
+    2267,
+    1018,
+    1174,
+    818,
+    92736
+  ],
+  "top10_logits": [
+    "0x1.22aaf40000000p+4",
+    "0x1.d43c300000000p+2",
+    "0x1.9a38da0000000p+2",
+    "0x1.638fea0000000p+2",
+    "0x1.1e06120000000p+2",
+    "0x1.018ac60000000p+2",
+    "0x1.92d1040000000p+1",
+    "0x1.c6c4060000000p+0",
+    "0x1.9aec800000000p+0",
+    "0x1.10e6ae0000000p+0"
+  ],
+  "logits_summary": [
+    "0x1.22aaf40000000p+4",
+    "-0x1.aed2a80000000p+4",
+    "-0x1.289e9657c46e0p+4",
+    "0x1.82753e85ba58ap+1"
+  ],
+  "input_ids": [
+    2,
+    105,
+    2364,
+    107,
+    255999,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258882,
+    82858,
+    672,
+    2471,
+    528,
+    8052,
+    236761,
+    106,
+    107,
+    105,
+    4368,
+    107
+  ]
+}
diff --git a/testdata/golden/vision-language/gemma-4-e2b-it_generation.json b/testdata/golden/vision-language/gemma-4-e2b-it_generation.json
new file mode 100644
index 00000000..14d1186c
--- /dev/null
+++ b/testdata/golden/vision-language/gemma-4-e2b-it_generation.json
@@ -0,0 +1,37 @@
+{
+  "model_id": "google/gemma-4-E2B-it",
+  "prompt": "Describe this image in detail.",
+  "generated_tokens": [
+    2094,
+    563,
+    496,
+    10807,
+    529,
+    496,
+    5213,
+    115042,
+    1018,
+    528,
+    496,
+    54530,
+    11690,
+    6514,
+    236761,
+    108,
+    8291,
+    236789,
+    236751,
+    496,
+    9813,
+    6492,
+    236787,
+    108,
+    1018,
+    21288,
+    53121,
+    107,
+    236829,
+    139
+  ],
+  "generated_text": "This is a photograph of a **raccoon** in a snowy outdoor setting.\n\nHere's a detailed description:\n\n**Subject:**\n*   "
+}
diff --git a/testdata/golden/vision-language/gemma-4-e4b-it.json b/testdata/golden/vision-language/gemma-4-e4b-it.json
new file mode 100644
index 00000000..46f6ae26
--- /dev/null
+++ b/testdata/golden/vision-language/gemma-4-e4b-it.json
@@ -0,0 +1,319 @@
+{
+  "top1_id": 2094,
+  "top2_id": 236776,
+  "top10_ids": [
+    2094,
+    236776,
+    902,
+    8291,
+    1408,
+    2267,
+    236777,
+    1018,
+    818,
+    100
+  ],
+  "top10_logits": [
+    "0x1.ac9f4c0000000p+4",
+    "0x1.5cf86a0000000p+4",
+    "0x1.580a4e0000000p+4",
+    "0x1.4ffddc0000000p+4",
+    "0x1.44c5000000000p+4",
+    "0x1.34a4b00000000p+4",
+    "0x1.0cde3e0000000p+4",
+    "0x1.0801de0000000p+4",
+    "0x1.edfa680000000p+3",
+    "0x1.e94f9c0000000p+3"
+  ],
+  "logits_summary": [
+    "0x1.ac9f4c0000000p+4",
+    "-0x1.a288e80000000p+4",
+    "-0x1.c10b3e7a0c0a3p+3",
+    "0x1.16beba1f49e04p+2"
+  ],
+  "input_ids": [
+    2,
+    105,
+    2364,
+    107,
+    255999,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258880,
+    258882,
+    82858,
+    672,
+    2471,
+    528,
+    8052,
+    236761,
+    106,
+    107,
+    105,
+    4368,
+    107
+  ]
+}
diff --git a/testdata/golden/vision-language/gemma-4-e4b-it_generation.json b/testdata/golden/vision-language/gemma-4-e4b-it_generation.json
new file mode 100644
index 00000000..91e3847d
--- /dev/null
+++ b/testdata/golden/vision-language/gemma-4-e4b-it_generation.json
@@ -0,0 +1,37 @@
+{
+  "model_id": "google/gemma-4-E4B-it",
+  "prompt": "Describe this image in detail.",
+  "generated_tokens": [
+    2094,
+    563,
+    496,
+    10807,
+    529,
+    496,
+    5213,
+    236765,
+    201968,
+    1018,
+    528,
+    496,
+    54530,
+    236764,
+    11690,
+    3453,
+    236761,
+    108,
+    8291,
+    563,
+    496,
+    9813,
+    6492,
+    236787,
+    108,
+    1018,
+    21288,
+    53121,
+    107,
+    818
+  ],
+  "generated_text": "This is a photograph of a **wolverine** in a snowy, outdoor environment.\n\nHere is a detailed description:\n\n**Subject:**\nThe"
+}
diff --git a/tests/e2e_golden_test.py b/tests/e2e_golden_test.py
index 3fe94323..b0617e8d 100644
--- a/tests/e2e_golden_test.py
+++ b/tests/e2e_golden_test.py
@@ -12,6 +12,9 @@
     pytest tests/e2e_golden_test.py -k "qwen2_5-0_5b"    # by model
     pytest tests/e2e_golden_test.py -m golden              # L4 only
     pytest tests/e2e_golden_test.py -m generation          # L5 only
+
+    # Run on CUDA GPU:
+    MOBIUS_TEST_DEVICE=cuda pytest tests/e2e_golden_test.py -v
 """
 
 from __future__ import annotations
@@ -45,6 +48,75 @@
 _TESTDATA_DIR = Path(__file__).resolve().parent.parent / "testdata"
 
 
+def _get_test_device_kwargs() -> dict[str, str]:
+    """Return OnnxModelSession kwargs from environment variables.
+
+    Set ``MOBIUS_TEST_DEVICE`` to ``cuda`` to run on GPU.
+    Set ``MOBIUS_TEST_EP`` to override the execution provider
+    (e.g. ``CUDAExecutionProvider``).
+    """
+    kwargs: dict[str, str] = {}
+    device = os.environ.get("MOBIUS_TEST_DEVICE", "").lower()
+    if device:
+        kwargs["device"] = device
+    ep = os.environ.get("MOBIUS_TEST_EP", "")
+    if ep:
+        kwargs["providers"] = [ep]
+    return kwargs
+
+
+def _make_empty_kv_cache(
+    session: OnnxModelSession,
+    config: object,
+) -> dict[str, np.ndarray]:
+    """Create empty KV cache feeds using the ORT session's declared shapes.
+
+    Uses the model's own shape declarations so that per-layer dimension
+    variations (e.g. KV sharing in Gemma4) are handled correctly.
+    The sequence/time dimension is set to 0.
+    """
+    feeds: dict[str, np.ndarray] = {}
+    # Fallback values from config
+    default_kv_heads = getattr(config, "num_key_value_heads", 1)
+    default_head_dim = getattr(config, "head_dim", 64)
+    layer_types = getattr(config, "layer_types", None) or []
+
+    for name in session.input_names:
+        if not name.startswith("past_key_values."):
+            continue
+        shape = session.get_input_shape(name)
+        if shape is not None and len(shape) >= 2:
+            # Use declared shape; set dynamic/zero dims appropriately
+            parts = name.split(".")
+            layer_idx = int(parts[1]) if len(parts) >= 3 and parts[1].isdigit() else 0
+            ltype = (
+                layer_types[layer_idx] if layer_idx < len(layer_types) else "full_attention"
+            )
+            if ltype in ("linear_attention", "mamba", "mamba2"):
+                # Fixed-size recurrent state: replace symbolic dims with 1
+                static = [d if isinstance(d, int) and d > 0 else 1 for d in shape]
+            else:
+                # KV cache: use declared dims but seq=0
+                static = []
+                for i, d in enumerate(shape):
+                    if isinstance(d, int) and d > 0:
+                        static.append(d)
+                    elif i == 0:
+                        static.append(1)  # batch
+                    elif i == 2:
+                        static.append(0)  # seq dim
+                    else:
+                        static.append(default_kv_heads if i == 1 else default_head_dim)
+            feeds[name] = np.zeros(static, dtype=np.float32)
+        else:
+            # Fallback: standard shape
+            feeds[name] = np.zeros(
+                (1, default_kv_heads, 0, default_head_dim),
+                dtype=np.float32,
+            )
+    return feeds
+
+
 @pytest.fixture(autouse=True)
 def _use_temp_hf_cache(tmp_path):
     """Redirect HuggingFace downloads to a per-test temp dir.
@@ -146,12 +218,13 @@ def _open_decoder_session(pkg: ModelPackage) -> OnnxModelSession:
     which is the decoder component that produces logits.
     Seq2seq packages: uses the ``"decoder"`` key.
     """
+    device_kwargs = _get_test_device_kwargs()
     if len(pkg) == 1:
-        return OnnxModelSession(pkg)
+        return OnnxModelSession(pkg, **device_kwargs)
     if "model" in pkg:
-        return OnnxModelSession(pkg["model"])
+        return OnnxModelSession(pkg["model"], **device_kwargs)
     if "decoder" in pkg:
-        return OnnxModelSession(pkg["decoder"])
+        return OnnxModelSession(pkg["decoder"], **device_kwargs)
     raise KeyError(f"Cannot find decoder model in package. Keys: {sorted(pkg.keys())}")
 
 
@@ -170,7 +243,7 @@ def _run_seq2seq_prefill(
     seq_len = input_ids.shape[1]
 
     # Step 1: Run encoder
-    enc_session = OnnxModelSession(pkg["encoder"])
+    enc_session = OnnxModelSession(pkg["encoder"], **_get_test_device_kwargs())
     try:
         enc_feeds = {
             "input_ids": input_ids,
@@ -192,7 +265,7 @@ def _run_seq2seq_prefill(
         )
 
     # Step 2: Run decoder with encoder output + decoder start token
-    dec_session = OnnxModelSession(pkg["decoder"])
+    dec_session = OnnxModelSession(pkg["decoder"], **_get_test_device_kwargs())
     try:
         decoder_start_id = getattr(config, "decoder_start_token_id", 0) or 0
         dec_input_ids = np.array([[decoder_start_id]], dtype=np.int64)
@@ -511,13 +584,20 @@ def _run_vision_language_prefill(
     }
 
     # --- Step 1: Run vision encoder ---
-    vis_session = OnnxModelSession(pkg["vision"])
+    vis_session = OnnxModelSession(pkg["vision"], **_get_test_device_kwargs())
     try:
         vis_feeds: dict[str, np.ndarray] = {}
         for name in vis_session.input_names:
             if name in processed:
                 val = processed[name]
                 vis_feeds[name] = val if isinstance(val, np.ndarray) else np.array(val)
+            else:
+                # Handle HF↔ONNX name mismatches (e.g. HF "image_position_ids"
+                # vs ONNX "pixel_position_ids").
+                for hf_key, val in processed.items():
+                    if hf_key.replace("image_", "pixel_") == name:
+                        vis_feeds[name] = val if isinstance(val, np.ndarray) else np.array(val)
+                        break
         vis_out = vis_session.run(vis_feeds)
     finally:
         vis_session.close()
@@ -527,7 +607,7 @@ def _run_vision_language_prefill(
     vis_hidden = vis_out[vis_hidden_key]
 
     # --- Step 2: Run embedding model ---
-    emb_session = OnnxModelSession(pkg["embedding"])
+    emb_session = OnnxModelSession(pkg["embedding"], **_get_test_device_kwargs())
     try:
         emb_feeds: dict[str, np.ndarray] = {
             "input_ids": processed["input_ids"].astype(np.int64),
@@ -538,6 +618,11 @@ def _run_vision_language_prefill(
                 emb_feeds[name] = vis_out[name]
             elif name == "image_features":
                 emb_feeds[name] = vis_hidden
+            elif name not in emb_feeds:
+                # Provide empty tensor for unused modalities (e.g. audio_features)
+                shape = emb_session.get_input_shape(name) or []
+                static_shape = [d if isinstance(d, int) and d > 0 else 0 for d in shape]
+                emb_feeds[name] = np.zeros(static_shape, dtype=np.float32)
         emb_out = emb_session.run(emb_feeds)
     finally:
         emb_session.close()
@@ -549,11 +634,13 @@ def _run_vision_language_prefill(
     # --- Step 3: Run decoder ---
     # VL packages may use "model" or "decoder" for the text decoder.
     dec_key = "model" if "model" in pkg else "decoder"
-    dec_session = OnnxModelSession(pkg[dec_key])
+    dec_session = OnnxModelSession(pkg[dec_key], **_get_test_device_kwargs())
     try:
         seq_len = inputs_embeds.shape[1]
+        kv_cache = _make_empty_kv_cache(dec_session, config)
         dec_feeds: dict[str, np.ndarray] = {
             "inputs_embeds": inputs_embeds,
+            **kv_cache,
         }
         # Pass through processor outputs that match decoder inputs
         # (e.g., attention_mask, position_ids with model-specific shapes)
@@ -577,13 +664,6 @@ def _run_vision_language_prefill(
                     )
                 else:
                     dec_feeds[name] = np.arange(seq_len, dtype=np.int64).reshape(1, -1)
-            elif name.startswith("past_key_values."):
-                num_kv_heads = getattr(config, "num_key_value_heads", 1)
-                head_dim = getattr(config, "head_dim", 64)
-                dec_feeds[name] = np.zeros(
-                    (1, num_kv_heads, 0, head_dim),
-                    dtype=np.float32,
-                )
         outputs = dec_session.run(dec_feeds)
     finally:
         dec_session.close()
@@ -689,11 +769,18 @@ def _run_vl_generation(
     }
 
     # --- Step 1: vision encoder ---
-    vis_session = OnnxModelSession(pkg["vision"])
+    vis_session = OnnxModelSession(pkg["vision"], **_get_test_device_kwargs())
     try:
-        vis_feeds: dict[str, np.ndarray] = {
-            name: processed[name] for name in vis_session.input_names if name in processed
-        }
+        vis_feeds: dict[str, np.ndarray] = {}
+        for name in vis_session.input_names:
+            if name in processed:
+                vis_feeds[name] = processed[name]
+            else:
+                # HF↔ONNX name mismatch (e.g. image_position_ids → pixel_position_ids)
+                for hf_key, val in processed.items():
+                    if hf_key.replace("image_", "pixel_") == name:
+                        vis_feeds[name] = val if isinstance(val, np.ndarray) else np.array(val)
+                        break
         vis_out = vis_session.run(vis_feeds)
     finally:
         vis_session.close()
@@ -703,12 +790,12 @@ def _run_vl_generation(
     # --- Step 2: embedding (prefill) ---
     # VL packages use "decoder" as the decoder key
     dec_key = "decoder" if "decoder" in pkg else "model"
-    dec_session = OnnxModelSession(pkg[dec_key])
-    emb_session = OnnxModelSession(pkg["embedding"])
+    dec_session = OnnxModelSession(pkg[dec_key], **_get_test_device_kwargs())
+    emb_session = OnnxModelSession(pkg["embedding"], **_get_test_device_kwargs())
 
     # Find the image features input name on the embedding model
     image_feat_input = next(
-        (n for n in emb_session.input_names if n != "input_ids"),
+        (n for n in emb_session.input_names if "image" in n),
         None,
     )
 
@@ -718,6 +805,12 @@ def _run_vl_generation(
         }
         if image_feat_input is not None:
             emb_feeds[image_feat_input] = vis_hidden
+        # Provide empty tensors for unused modalities (e.g. audio_features)
+        for name in emb_session.input_names:
+            if name not in emb_feeds:
+                shape = emb_session.get_input_shape(name) or []
+                static_shape = [d if isinstance(d, int) and d > 0 else 0 for d in shape]
+                emb_feeds[name] = np.zeros(static_shape, dtype=np.float32)
         emb_out = emb_session.run(emb_feeds)
         inputs_embeds = emb_out[next(iter(emb_out))]  # [1, seq_len, hidden_size]
 
@@ -731,12 +824,15 @@ def _run_vl_generation(
         spatial_merge = getattr(config, "spatial_merge_size", 2)
 
         # --- Step 4: prefill decoder ---
-        past_cache = _make_vl_decoder_cache_feeds(dec_session, config)
+        past_cache = _make_empty_kv_cache(dec_session, config)
         dec_feeds: dict[str, np.ndarray] = {
             "inputs_embeds": inputs_embeds,
             "attention_mask": np.ones((batch_size, prompt_seq_len), dtype=np.int64),
             **past_cache,
         }
+        # Gemma4 decoder requires input_ids alongside inputs_embeds
+        if "input_ids" in dec_session.input_names:
+            dec_feeds["input_ids"] = processed["input_ids"].astype(np.int64)
         # Track the next decode position (may differ from token count for MRoPE
         # because image tokens consume fewer positions than tokens: image group
         # advances current_pos by max(H, W), not by num_image_tokens).
@@ -782,6 +878,12 @@ def _run_vl_generation(
             step_emb_feeds: dict[str, np.ndarray] = {"input_ids": next_token}
             if image_feat_input is not None:
                 step_emb_feeds[image_feat_input] = empty_image
+            # Provide empty tensors for other modalities
+            for name in emb_session.input_names:
+                if name not in step_emb_feeds:
+                    shape = emb_session.get_input_shape(name) or []
+                    static_shape = [d if isinstance(d, int) and d > 0 else 0 for d in shape]
+                    step_emb_feeds[name] = np.zeros(static_shape, dtype=np.float32)
             step_emb_out = emb_session.run(step_emb_feeds)
             step_embeds = step_emb_out[next(iter(step_emb_out))]  # [1, 1, hidden_size]
 
@@ -791,6 +893,9 @@ def _run_vl_generation(
                 "attention_mask": np.ones((batch_size, total_len), dtype=np.int64),
                 **past_cache,
             }
+            # Gemma4 decoder requires input_ids alongside inputs_embeds
+            if "input_ids" in dec_session.input_names:
+                step_feeds["input_ids"] = next_token
             if "position_ids" in dec_session.input_names:
                 if uses_mrope:
                     # Use the true MRoPE position (not the token count), since
@@ -816,6 +921,292 @@ def _run_vl_generation(
     return np.concatenate(generated, axis=1)[0]  # [generated_len]
 
 
+# ---------------------------------------------------------------------------
+# Multimodal prefill helpers (speech-to-text, speech-language, text-only VL)
+# ---------------------------------------------------------------------------
+
+
+def _run_speech_to_text_prefill(
+    pkg: ModelPackage,
+    case: GoldenTestCase,
+    golden: GoldenRef,
+    config: object,
+) -> dict[str, np.ndarray]:
+    """Run encoder → decoder for speech-to-text models (e.g. Whisper).
+
+    Unlike seq2seq text models, the encoder takes ``input_features``
+    (mel spectrogram) rather than ``input_ids``.
+    """
+    import librosa
+    import transformers
+
+    # Load audio and extract features
+    processor = transformers.AutoProcessor.from_pretrained(
+        case.model_id, trust_remote_code=case.trust_remote_code
+    )
+    audio_path = _TESTDATA_DIR / case.audio[0]
+    audio_array, _sr = librosa.load(str(audio_path), sr=16000)
+    processed = processor(audio_array, sampling_rate=16000, return_tensors="np")
+
+    # Step 1: Run encoder
+    enc_session = OnnxModelSession(pkg["encoder"], **_get_test_device_kwargs())
+    try:
+        enc_feeds: dict[str, np.ndarray] = {}
+        for name in enc_session.input_names:
+            if name in processed:
+                enc_feeds[name] = processed[name].astype(np.float32)
+        enc_outputs = enc_session.run(enc_feeds)
+    finally:
+        enc_session.close()
+
+    enc_hidden = None
+    for key in ("encoder_hidden_states", "last_hidden_state"):
+        if key in enc_outputs:
+            enc_hidden = enc_outputs[key]
+            break
+    if enc_hidden is None:
+        raise KeyError(
+            f"Encoder output missing hidden states. Keys: {sorted(enc_outputs.keys())}"
+        )
+
+    # Step 2: Run decoder with encoder output + decoder start token
+    dec_session = OnnxModelSession(pkg["decoder"], **_get_test_device_kwargs())
+    try:
+        decoder_start_id = getattr(config, "decoder_start_token_id", 0) or 0
+        dec_input_ids = np.array([[decoder_start_id]], dtype=np.int64)
+
+        dec_feeds: dict[str, np.ndarray] = {
+            "encoder_hidden_states": enc_hidden,
+        }
+
+        # Map decoder inputs by name — whisper uses "decoder_input_ids"
+        for name in dec_session.input_names:
+            if name in dec_feeds:
+                continue
+            if name in ("input_ids", "decoder_input_ids"):
+                dec_feeds[name] = dec_input_ids
+            elif name == "encoder_attention_mask":
+                enc_seq_len = enc_hidden.shape[1]
+                dec_feeds[name] = np.ones((1, enc_seq_len), dtype=np.int64)
+            elif name == "position_ids":
+                dec_feeds[name] = np.zeros((1, 1), dtype=np.int64)
+            elif name.startswith("past_key_values."):
+                num_kv_heads = getattr(config, "num_key_value_heads", None) or getattr(
+                    config, "num_attention_heads", 1
+                )
+                head_dim = getattr(config, "head_dim", None) or (
+                    getattr(config, "d_model", 256)
+                    // getattr(config, "decoder_attention_heads", 1)
+                )
+                dec_feeds[name] = np.zeros(
+                    (1, num_kv_heads, 0, head_dim),
+                    dtype=np.float32,
+                )
+        outputs = dec_session.run(dec_feeds)
+    finally:
+        dec_session.close()
+
+    return outputs
+
+
+def _run_text_only_multimodel_prefill(
+    pkg: ModelPackage,
+    golden: GoldenRef,
+    config: object,
+) -> dict[str, np.ndarray]:
+    """Run embedding → decoder for text-only input on a multi-model package.
+
+    Multi-model packages (e.g. Gemma4 VL) require text to go through the
+    embedding model first since the decoder only accepts ``inputs_embeds``.
+    """
+    device_kwargs = _get_test_device_kwargs()
+    input_ids = np.array(golden.input_ids, dtype=np.int64).reshape(1, -1)
+
+    # Step 1: Run embedding with text-only input (no image/audio features)
+    emb_session = OnnxModelSession(pkg["embedding"], **device_kwargs)
+    try:
+        emb_feeds: dict[str, np.ndarray] = {"input_ids": input_ids}
+        # Provide empty features for any non-text inputs
+        for name in emb_session.input_names:
+            if name not in emb_feeds:
+                shape = emb_session.get_input_shape(name) or []
+                static_shape = [d if isinstance(d, int) and d > 0 else 0 for d in shape]
+                emb_feeds[name] = np.zeros(static_shape, dtype=np.float32)
+        emb_out = emb_session.run(emb_feeds)
+    finally:
+        emb_session.close()
+
+    inputs_embeds = emb_out[next(iter(emb_out))]
+
+    # Step 2: Run decoder
+    dec_key = "model" if "model" in pkg else "decoder"
+    dec_session = OnnxModelSession(pkg[dec_key], **device_kwargs)
+    try:
+        seq_len = inputs_embeds.shape[1]
+        kv_cache = _make_empty_kv_cache(dec_session, config)
+        dec_feeds: dict[str, np.ndarray] = {
+            "inputs_embeds": inputs_embeds,
+            **kv_cache,
+        }
+        for name in dec_session.input_names:
+            if name in dec_feeds:
+                continue
+            if name == "input_ids":
+                dec_feeds[name] = input_ids
+            elif name == "attention_mask":
+                dec_feeds[name] = np.ones((1, seq_len), dtype=np.int64)
+            elif name == "position_ids":
+                dec_feeds[name] = np.arange(seq_len, dtype=np.int64).reshape(1, -1)
+        outputs = dec_session.run(dec_feeds)
+    finally:
+        dec_session.close()
+
+    return outputs
+
+
+def _run_speech_language_prefill(
+    pkg: ModelPackage,
+    case: GoldenTestCase,
+    golden: GoldenRef,
+    config: object,
+) -> dict[str, np.ndarray]:
+    """Run audio encoder → embedding → decoder for speech-language models.
+
+    The audio encoder produces audio features which are fed to the
+    embedding model along with input_ids, then the decoder runs on
+    inputs_embeds.
+    """
+    import librosa
+    import transformers
+
+    device_kwargs = _get_test_device_kwargs()
+
+    # Load audio and extract features
+    processor = transformers.AutoProcessor.from_pretrained(
+        case.model_id, trust_remote_code=case.trust_remote_code
+    )
+    audio_path = _TESTDATA_DIR / case.audio[0]
+    audio_array, _sr = librosa.load(str(audio_path), sr=16000)
+
+    # Use the feature extractor component for audio.
+    # For Qwen3-ASR, AutoProcessor returns a tokenizer (not the
+    # full Qwen3ASRProcessor) because the HF repo lacks auto_map.
+    # Fall back to WhisperFeatureExtractor which is what Qwen3-ASR
+    # actually uses under the hood.
+    fe = getattr(processor, "feature_extractor", None)
+    if fe is None or not hasattr(fe, "sampling_rate"):
+        fe = transformers.WhisperFeatureExtractor.from_pretrained(case.model_id)
+    audio_processed = fe(
+        [audio_array],
+        sampling_rate=16000,
+        return_tensors="np",
+        padding=False,
+    )
+
+    # Step 1: Run audio encoder
+    audio_key = "audio" if "audio" in pkg else "audio_encoder"
+    audio_session = OnnxModelSession(pkg[audio_key], **device_kwargs)
+    try:
+        audio_feeds: dict[str, np.ndarray] = {}
+        for name in audio_session.input_names:
+            if name in audio_processed:
+                audio_feeds[name] = audio_processed[name].astype(np.float32)
+            elif name == "input_features" and "input_features" in audio_processed:
+                audio_feeds[name] = audio_processed["input_features"].astype(np.float32)
+        audio_out = audio_session.run(audio_feeds)
+    finally:
+        audio_session.close()
+
+    audio_hidden = audio_out[next(iter(audio_out))]
+    # Audio encoder output is [batch, seq, hidden]; embedding expects
+    # [num_tokens, hidden] (no batch dim).
+    if audio_hidden.ndim == 3:
+        audio_hidden = audio_hidden[0]  # squeeze batch
+
+    # Build input_ids from golden reference
+    input_ids = np.array(golden.input_ids, dtype=np.int64).reshape(1, -1)
+
+    # Adjust audio placeholder count to match encoder output.
+    # The HF processor may generate a different number of audio tokens
+    # than the ONNX encoder actually produces.  Re-build input_ids so
+    # the placeholder count matches the encoder output exactly.
+    num_encoder_tokens = audio_hidden.shape[0]
+    audio_token_id = getattr(config, "audio_token_id", None)
+    if audio_token_id is None:
+        thinker_cfg = getattr(config, "thinker_config", None)
+        if thinker_cfg is not None:
+            audio_token_id = getattr(thinker_cfg, "audio_token_id", None)
+    if audio_token_id is not None:
+        flat = input_ids[0].tolist()
+        num_placeholders = flat.count(audio_token_id)
+        if num_placeholders != num_encoder_tokens:
+            # Replace existing placeholders with correct count
+            new_ids: list[int] = []
+            replaced = False
+            for tok in flat:
+                if tok == audio_token_id:
+                    if not replaced:
+                        new_ids.extend([audio_token_id] * num_encoder_tokens)
+                        replaced = True
+                    # Skip remaining old placeholders
+                else:
+                    new_ids.append(tok)
+            input_ids = np.array(new_ids, dtype=np.int64).reshape(1, -1)
+
+    # Step 2: Run embedding with input_ids + audio features
+    emb_session = OnnxModelSession(pkg["embedding"], **device_kwargs)
+    try:
+        emb_feeds: dict[str, np.ndarray] = {"input_ids": input_ids}
+        for name in emb_session.input_names:
+            if name in emb_feeds:
+                continue
+            if name == "audio_features":
+                emb_feeds[name] = audio_hidden
+            elif name in audio_out:
+                emb_feeds[name] = audio_out[name]
+            else:
+                # Empty features for unused modalities (e.g. image)
+                shape = emb_session.get_input_shape(name) or []
+                static_shape = [d if isinstance(d, int) and d > 0 else 0 for d in shape]
+                emb_feeds[name] = np.zeros(static_shape, dtype=np.float32)
+        emb_out = emb_session.run(emb_feeds)
+    finally:
+        emb_session.close()
+
+    inputs_embeds = emb_out[next(iter(emb_out))]
+
+    # Step 3: Run decoder
+    dec_key = "model" if "model" in pkg else "decoder"
+    dec_session = OnnxModelSession(pkg[dec_key], **device_kwargs)
+    try:
+        seq_len = inputs_embeds.shape[1]
+        kv_cache = _make_empty_kv_cache(dec_session, config)
+        dec_feeds: dict[str, np.ndarray] = {
+            "inputs_embeds": inputs_embeds,
+            **kv_cache,
+        }
+        for name in dec_session.input_names:
+            if name in dec_feeds:
+                continue
+            if name == "input_ids":
+                dec_feeds[name] = input_ids
+            elif name == "attention_mask":
+                dec_feeds[name] = np.ones((1, seq_len), dtype=np.int64)
+            elif name == "position_ids":
+                pos = np.arange(seq_len, dtype=np.int64).reshape(1, -1)
+                # MRoPE models expect 3D position_ids: (dims, batch, seq)
+                pos_shape = dec_session.get_input_shape(name)
+                if pos_shape and len(pos_shape) == 3:
+                    ndims = pos_shape[0] if isinstance(pos_shape[0], int) else 3
+                    pos = np.tile(pos, (ndims, 1, 1))
+                dec_feeds[name] = pos
+        outputs = dec_session.run(dec_feeds)
+    finally:
+        dec_session.close()
+
+    return outputs
+
+
 # ---------------------------------------------------------------------------
 # L4 Tests: Checkpoint Verified
 # ---------------------------------------------------------------------------
@@ -848,6 +1239,10 @@ def test_prefill_argmax_matches_golden(self, case: GoldenTestCase) -> None:
         # Seq2seq models require running encoder → decoder
         if case.task_type == "seq2seq":
             outputs = _run_seq2seq_prefill(pkg, golden, config)
+        elif case.task_type == "speech-to-text":
+            outputs = _run_speech_to_text_prefill(pkg, case, golden, config)
+        elif case.task_type == "speech-language":
+            outputs = _run_speech_language_prefill(pkg, case, golden, config)
         elif case.task_type == "image-text-to-text":
             outputs = _run_vision_language_prefill(pkg, case, config)
         elif case.task_type == "image-classification":
@@ -864,6 +1259,9 @@ def test_prefill_argmax_matches_golden(self, case: GoldenTestCase) -> None:
                 outputs = session.run(feeds)
             finally:
                 session.close()
+        elif len(pkg) > 1 and "embedding" in pkg:
+            # Multi-model text-generation (e.g. Gemma4 VL text-only)
+            outputs = _run_text_only_multimodel_prefill(pkg, golden, config)
         else:
             session = _open_decoder_session(pkg)
             try:
@@ -1020,17 +1418,23 @@ def test_generation_matches_golden(self, case: GoldenTestCase) -> None:
             "cannot determine KV cache dimensions for generation"
         )
 
-        new_tokens = (
-            _run_vl_generation(
+        if case.task_type == "image-text-to-text":
+            new_tokens = _run_vl_generation(
                 pkg,
                 case,
                 config,
                 max_new_tokens=case.generation_params.get("max_new_tokens", 30),
                 eos_token_id=case.generation_params.get("eos_token_id"),
             )
-            if case.task_type == "image-text-to-text"
-            else _run_causal_lm_generation(pkg, case, golden)
-        )
+        elif len(pkg) > 1 and "embedding" in pkg:
+            # Multi-model text-generation (e.g. Gemma4) — L5 generation
+            # requires embedding → decoder loop, not yet implemented.
+            pytest.skip(
+                f"L5 generation for multi-model text-generation "
+                f"not yet implemented ({case.case_id})"
+            )
+        else:
+            new_tokens = _run_causal_lm_generation(pkg, case, golden)
 
         # --- Diagnostics ---
         expected_tokens = np.array(expected_token_ids, dtype=np.int64)