Add OpenBMB MiniCPM-V 4.6 support - #464
Open
justinchuby wants to merge 4 commits into
Open
Conversation
Implement the packed variable-resolution SigLIP2 vision tower, ragged crop batching, 16x/4x mergers, Qwen3.5 hybrid decoder integration, image/video token mixing, config extraction, standardized three-model export, and ORT GenAI metadata. Add synthetic and real-weight parity tests plus committed image prefill and deterministic generation goldens. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds first-class support for OpenBMB MiniCPM‑V‑4.6 to mobius’ multimodal pipeline by introducing a native model implementation, a dedicated 3-model Vision-Language task layout (decoder/vision_encoder/embedding), registry/config wiring, ORT GenAI export integration, and new integration + golden coverage.
Changes:
- Introduce
MiniCPMV46ForConditionalGeneration(packed NaViT/SigLIP2 vision + Qwen3.5 hybrid decoder) plus unit tests and weight-routing. - Add
MiniCPMVLTaskand register the newminicpmv4_6model_type across configs, registry, and ORT GenAI export behavior. - Add new L4/L5 test case + golden artifacts, plus ORT GenAI package-load and real-weight vision parity integration tests.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/ort_genai_test.py | Adds ORT GenAI integration test to verify MiniCPM-V-4.6 package loads. |
| tests/integration_test.py | Adds real-weight SigLIP2/merger vision parity test for MiniCPM‑V‑4.6. |
| tests/_test_configs.py | Adds tiny MiniCPM‑V‑4.6 VL config for graph-build coverage. |
| testdata/golden/vision-language/minicpm-v-4_6.json | Adds L4 golden logits/tokens snapshot for MiniCPM‑V‑4.6. |
| testdata/golden/vision-language/minicpm-v-4_6_generation.json | Adds L5 deterministic generation golden for MiniCPM‑V‑4.6. |
| testdata/cases/vision-language/minicpm-v-4_6.yaml | Adds L4/L5 golden case definition for MiniCPM‑V‑4.6. |
| src/mobius/tasks/_vision_language_3model.py | Adds MiniCPMVLTask and packed target_sizes vision I/O contract. |
| src/mobius/tasks/init.py | Exposes MiniCPMVLTask and registers it in task registry. |
| src/mobius/models/minicpmv4_6.py | Implements MiniCPM‑V‑4.6 model (vision tower + mergers + hybrid decoder + embedding fusion). |
| src/mobius/models/minicpmv4_6_test.py | Adds synthetic vision parity + embedding placeholder-mixing tests. |
| src/mobius/models/init.py | Exports MiniCPMV46ForConditionalGeneration. |
| src/mobius/integrations/ort_genai/auto_export.py | Adds MiniCPM-specific ORT GenAI handling (no image_processor.json; preserve HF preprocessor config; adjust vision input mapping). |
| src/mobius/_registry.py | Registers minicpmv4_6 model_type and default model id. |
| src/mobius/_configs/per_model/_minicpmv4_6_vision.py | Adds per-model vision-config extraction hook for the composite MiniCPM config. |
| src/mobius/_configs/per_model/init.py | Imports the new MiniCPM vision extractor to register hooks. |
| src/mobius/_configs/_sub_configs.py | Extends VisionConfig with MiniCPM merger parameters. |
| src/mobius/_configs/_base.py | Adds video_token_id and downsample_mode to the shared architecture config. |
| README.md | Updates supported Multimodal model list to include MiniCPM‑V 4.6. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Keep packed patch coordinate mapping on CUDA by replacing ArgMax with a segment-count reduction. Cast through float for ReduceMean and Compress so BF16 vision graphs load and run while restoring the model dtype at each boundary. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Emit float compatibility casts around ReduceMean and Compress only for BF16, where ORT requires them. Keep native FP16 kernels on the fast path and remove four unnecessary casts without changing outputs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Fall back to the architecture's 2x2 merger kernels when Hugging Face explicitly supplies null values, and add a regression test for both window and final merger fields. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Member
Author
|
Looks like we need compress and argmax in ORT cuda |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
minicpmv4_6support foropenbmb/MiniCPM-V-4.6/ Transformers 5.7+decoder,vision_encoder, andembeddingmodelsReduceMean/CompressValidation
Core confidence levels
rtol=atol=1e-4)rtol=atol=1e-3lintrunner -aclean; three independent reviews found no significant issuesDRYness and optimized graph audit
The implementation reuses existing public building blocks wherever semantics match:
Qwen35TextModelplus the shared hybrid cache/task machineryVisionEncoderLayer,VisionAttention,LayerNorm,Linear, and activation helpersVisionLanguageTask,build_decoder_from_embeds, andbuild_embedding_from_featuresThe existing
InputMixerwas intentionally not reused: it accepts one token ID and rank-3 per-batch vision features, while MiniCPM has two placeholder IDs sharing one rank-2 packed feature stream. Adapting it would add shape/expand/gather nodes and obscure the exact HF ordering; the MiniCPM embedding graph is already only 12 nodes.Weighted, CUDA-optimized FP16 graph evidence:
Initializer folding reduces unweighted vision transposes from 175 to 5; the remaining five are architectural raster/window permutations. Decoder transposes/reshapes are Qwen3.5 attention/DeltaNet layouts. The 128 FP16 decoder casts come from the shared numerically-stable Qwen3.5 DeltaNet implementation. BF16 vision retains six casts because ORT requires FLOAT boundaries for
ReduceMeanandCompress; FP16 now emits only two metadata casts after gating those compatibility casts to BF16.CUDA (RTX A1000, driver 573.44, CUDA 12.8, ORT 1.28.0)
Runtime setup was verified with
torch 2.11.0+cu128,torch.cuda.is_available() == True, andCUDAExecutionProviderpresent. Profiles reject silent CPU fallback.FP32 CPU vs CUDA, nonzero 1024x1024 image, ragged grids
[[32,32], [40,24] x6]:0.00485945, mean abs0.000139655, cosine0.999999404, no NaNsConcatx36,Subx1), 0 memcpy events; logits max abs0.471408, mean abs0.00219277, cosine0.999997973, identical argmax760, no NaNs0.9999992, no NaNsMemcpy audit:
int64scalars/vectors only),944 ustotal; no CPU compute nodes remain after replacingArgMaxwithGreaterOrEqual + ReduceSumReduced precision on CUDA:
0.0691971, mean abs0.000724234, cosine0.999961913, no NaNs0.0691971, mean abs0.000666107, cosine0.999961913, no NaNs0.0500653, mean abs0.00798651, cosine0.999991179, identical argmax760, no NaNsReduceMean/Compressfixes, with no NaNs. BF16 ONNX vision vs per-unit HF BF16 CUDA: max abs0.265625, mean abs0.00541074, cosine0.99873763.Exact commands:
Stage parity/profiling used
OnnxModelSession(..., device="cuda", enable_profiling=True)forvision_encoder,embedding, anddecoder, with a hard assertion that each stage had CUDA node placement.CI status
All completed lint, analysis, platform, L1, L3, architecture, benchmark, and fast-integration jobs pass. The initial L4 runner was cancelled and the initial L5 runner disappeared without a failed step or downloadable test log. Failed-job reruns were requested; the self-hosted A10 jobs remained queued without assigned runners. The exact L5 CI command passes locally on CUDA in 201.34 seconds.
Runtime limitations / waivers
target_sizes. The export preservespreprocessor_config.json; the full image pipeline and generation are validated directly through ONNX Runtime CUDA. Runtime callers must inject HF-preprocessed tensors until native processor support lands.onnxruntime-genai-cuda 0.15.2currently crashes with Windows access violation-1073741819while loading the FP16 hybrid package. Raw ORT CUDA executes all three stages and exact L5 generation successfully, so this is isolated to the GenAI hybrid runtime path.CausalConvWithStatenode in BF16 (Provider type ... is not set). BF16 vision/embedding are validated; FP16 is the supported end-to-end reduced-precision CUDA path.