Document f32 input casting for vision/audio encoders in multimodal skill - #271
Merged
Conversation
Add section to multimodal-models skill explaining why vision and audio encoder graphs accept f32 inputs and Cast to model dtype: - ORT GenAI processors always output f32 - Encoder adds Cast(f32 → f16/bf16) at graph entry - Weights remain in requested dtype for memory efficiency - Handled automatically by mobius with --runtime ort-genai - Explains the type mismatch error users see without the Cast Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Performance Comparison
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Add callout clarifying the Cast-at-input pattern is universal for all vision/audio models (Gemma3, Gemma4, LLaVA, Phi-3-Vision, Qwen-VL, Whisper, etc.), not architecture-specific. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Rewrite the f32 input casting section to explain this is a universal pattern across ALL inference paths (ORT Python API, Foundry Local, custom runtimes), not GenAI-specific. f32 is the natural preprocessing dtype because PIL, torchaudio, librosa, and all preprocessing libraries output f32. The Cast-at-input is always inserted by mobius regardless of --runtime flag. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Document which tasks have the f32+Cast pattern (5 tasks), which still use config.dtype as input (VisionLanguageTask), and which single-model tasks correctly use hardcoded FLOAT (7 tasks). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Add MultimodalTask (pixel_values + audio_features) and AudioCTCTask (input_features) to the needs-migration list. Move AudioCTCTask out of the single-model correct list. 3 tasks need migration total. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Remove per-task migration status table. The section now explains the general principle concisely without listing internal task status. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
37 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update the multimodal-models skill to document why vision/audio encoder graphs accept f32 inputs and Cast to model dtype at graph entry.
What this adds
New section in
.agents/skills/multimodal-models/SKILL.md:Cast(f32 → model_dtype)at entry; weights remain in f16/bf16--runtime ort-genaiType Error: Type parameter (T) bound to different typesThis documents the pattern introduced in PR #265.