Skip to content

Commit 6025fbb

Browse files
justinchubyCopilot
andauthored
Document f32 input casting for vision/audio encoders in multimodal skill (#271)
Update the multimodal-models skill to document why vision/audio encoder graphs accept f32 inputs and Cast to model dtype at graph entry. ## What this adds New section in `.agents/skills/multimodal-models/SKILL.md`: - **Why:** ORT GenAI image/audio processors always output f32, regardless of model dtype - **How:** Encoder graph adds `Cast(f32 → model_dtype)` at entry; weights remain in f16/bf16 - **When:** Handled automatically by mobius with `--runtime ort-genai` - **Error without it:** `Type Error: Type parameter (T) bound to different types` This documents the pattern introduced in PR #265. --------- Signed-off-by: Justin Chu <justinchu@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent f027a8e commit 6025fbb

1 file changed

Lines changed: 68 additions & 0 deletions

File tree

  • .agents/skills/multimodal-models

.agents/skills/multimodal-models/SKILL.md

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -214,9 +214,77 @@ models this overflows ORT's CUDA Gather kernel. **Workaround:** Split into
214214
L separate `Embedding([V, D])` tables via `nn.ModuleList`, and use `Slice`
215215
instead of `Gather` for per-layer projection indexing.
216216

217+
## Vision/audio encoder f32 input casting
218+
219+
> **This applies to ALL multimodal models and ALL inference paths**
220+
> not just ORT GenAI, and not architecture-specific.
221+
222+
Image and audio preprocessing universally produces **float32** output.
223+
This is true across all frameworks and runtimes:
224+
225+
- **PIL / torchvision:** Pixel normalization outputs f32
226+
- **torchaudio / librosa:** Mel spectrograms are f32
227+
- **ORT GenAI image_processor:** Resize, normalize, tile → f32
228+
- **ORT GenAI audio_processor:** Feature extraction → f32
229+
- **ORT Python API:** Custom preprocessing pipelines → typically f32
230+
- **Foundry Local:** Uses GenAI processors → f32
231+
232+
This means vision and audio encoder ONNX graphs must accept f32 inputs
233+
even when the model is built in f16 or bf16. The encoder adds a
234+
`Cast(f32 → model_dtype)` at its graph entry point so that any runtime
235+
can feed it preprocessed data without worrying about the model's
236+
internal precision.
237+
238+
### How it works
239+
240+
```
241+
Input (f32 from ANY preprocessor — PIL, torchaudio, GenAI, etc.)
242+
243+
Cast(to=FLOAT16) ← inserted automatically by mobius
244+
245+
Vision/Audio encoder (weights in f16/bf16)
246+
247+
Output (model_dtype)
248+
```
249+
250+
Encoder weights still use the requested dtype (f16/bf16) for memory
251+
efficiency — only the graph inputs are f32. The Cast is a lightweight
252+
op with negligible overhead.
253+
254+
### Why f32 is the universal preprocessing dtype
255+
256+
Preprocessing involves floating-point arithmetic (mean subtraction,
257+
std division, resampling interpolation) where f32 is the natural
258+
precision. Converting to f16/bf16 before these operations would lose
259+
precision in the preprocessing itself. The model's internal precision
260+
only matters after the preprocessed data enters the encoder.
261+
262+
### What mobius does
263+
264+
Mobius always builds encoder graphs with f32 inputs — this is the
265+
default behavior, not gated behind any flag. It works correctly
266+
regardless of the inference runtime:
267+
268+
- `--runtime ort-genai` → f32 inputs (GenAI processors output f32)
269+
- No `--runtime` flag → f32 inputs (ORT Python API, custom runtimes)
270+
- Foundry Local → f32 inputs (uses GenAI internally)
271+
272+
Without the Cast-at-input, ORT throws a type mismatch error:
273+
```
274+
Type Error: Type parameter (T) bound to different types
275+
(tensor(float) and tensor(float16))
276+
```
277+
278+
### For model authors
279+
280+
If you're adding a new multimodal model, you don't need to handle this
281+
manually — mobius inserts the Cast automatically for all encoder graphs.
282+
If the model dtype is already f32, no Cast is needed.
283+
217284
## Cross-references
218285

219286
- **Multimodal debugging:** `.agents/skills/debugging-multimodal/SKILL.md`
220287
- **ORT GenAI config:** `.agents/skills/ort-genai-config/SKILL.md`
221288
- **Weight name alignment:** `.agents/skills/weight-name-alignment/SKILL.md`
222289
- **Reusable components (ClippableLinear):** `.agents/skills/reusable-components/SKILL.md`
290+
- **Profiling:** `.agents/skills/profiling-onnx-models/SKILL.md`

0 commit comments

Comments
 (0)