Add Qwen VL/3.5 support and refactor model builder infrastructure by Copilot · Pull Request #2386 · microsoft/onnxruntime-genai

Copilot · 2026-08-08T00:08:22Z

Summary

Adds Qwen VL and Qwen 3.5 family support while restructuring the Python model builder around shared cache, RoPE/M-RoPE, MTP, quantization, and checkpoint-loader abstractions. Quantized checkpoint formats are normalized by loaders so builders can emit common ONNX graph structures with less architecture-specific code.

Key changes

Adds and validates Qwen2.5-VL, Qwen3-VL, and Qwen3.5 dense/MoE text builders, including 3D position IDs, native com.microsoft::MRotaryEmbedding, gated attention, linear attention, and shared experts.
Centralizes RoPE configuration and graph emission, including sectioned/interleaved M-RoPE, rotary casting, YaRN/LongRoPE handling, and architecture-specific parameter normalization.
Moves KV, convolution, and recurrent cache topology and naming into the base builder, including mixed local/full windowed caches, paged-cache behavior, and generated genai_config.json templates.
Extends the C++ configuration and recurrent-state runtime to consume explicit convolution and recurrent cache name templates.
Adds reusable base-builder emission for fused operations and quantized linear/MoE paths, including CausalConvWithState, GatedRMSNorm, GatedAdd, LinearAttentionGate, and native QMoE.
Introduces a shared MTP builder and automatic Qwen3.5 MTP export, with checkpoint adaptation handled by loaders and updated examples/documentation.
Reorganizes quantized checkpoint handling into dedicated AWQ, GPTQ, ModelOpt, Olive, Quark, Qwen, and GPT-OSS loaders with declarative naming, dtype, exclusion, and layout metadata.
Adds native packed-expert support for GPT-OSS MXFP4 and Quark QMoE checkpoints, including repacking/parity coverage and signed-scale handling.
Moves quantization utilities into models.quantization, clarifies the KV-cache quantization scheme/scale contract, and updates builder options, imports, documentation, and tests.
Simplifies LFM2 and other architecture builders by routing shared cache, RoPE, and operator behavior through the base implementation.

Validation

Full Python builder suite: 466 passed, 3 skipped.
Focused loader and CodeQL regression suite: 97 passed, 3 skipped.

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>

azure-pipelines · 2026-08-08T00:08:30Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

This PR extends the Python model-builder stack to cover Qwen VL / Qwen 3.5 model families, while centralizing hybrid-cache naming, RoPE/M-RoPE handling, and quantization utilities into shared builder/runtime infrastructure (with corresponding C++ config/runtime support for recurrent-cache templates).

Changes:

Refactors builder configuration initialization and RoPE/M-RoPE emission paths (including fp32 RoPE compute/casting support and canonicalization of RoPE settings).
Introduces shared cache-name templating for KV/conv/recurrent caches (and emits corresponding genai_config.json templates), plus C++ runtime support for recurrent template discovery.
Moves quantization-related helpers into a dedicated onnxruntime_genai.models.quantization package and updates docs/tests accordingly; adds Qwen-focused builder unit coverage.

Reviewed changes

Copilot reviewed 36 out of 40 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/python/models/test_yarn_rope_parity.py	Updates RoPE parity tests to use `Model.make_config_init` canonicalization.
test/python/models/test_qwen_3_5_text_only.py	Updates Qwen3.5 text-only test description to reflect 3D MRoPE position_ids.
test/python/create/create_dummy_qwen_3.5_models.py	Updates dummy Qwen3.5 configs to include conv/recurrent cache name templates.
test/python/builder/test_windowed_kv_cache.py	Adjusts sys.path injection to new models package layout.
test/python/builder/test_tied_embeddings.py	Adjusts sys.path injection to new models package layout.
test/python/builder/test_qwen_vl.py	Adds Qwen VL / Qwen 3.5 builder unit tests (MRoPE, gating, cache naming, contrib ops).
test/python/builder/test_quantized_kv_cache.py	Updates quantization import expectations to the new `quantization` package.
test/python/builder/test_quant_config.py	Adds tests for the new structured quantization config module location.
test/python/builder/test_qmoe_weights.py	Expands MoE/QMoE builder tests and updates quantization module import paths.
test/python/builder/test_precision.py	Updates quantization module import paths.
test/python/builder/test_mixed_precision_config.py	Adjusts sys.path injection to new models package layout.
test/python/builder/test_kv_cache_calibration.py	Updates kv-cache calibration module path to `models.quantization`.
test/python/builder/test_hunyuan.py	Updates Hunyuan RoPE tests to use `make_config_init` + `make_rope_init`.
test/python/builder/test_gqa_attributes.py	Adjusts sys.path injection to new models package layout.
test/models/qwen3-5/genai_config.json	Adds conv/recurrent cache name templates for the test model config.
src/python/setup.py.in	Adds `onnxruntime_genai.models.quantization` to packaged modules.
src/python/py/models/README.md	Updates docs to reference the new kv-cache calibration module path.
src/python/py/models/quantization/quantization.md	Adds documentation for unified quantization options and mixed precision.
src/python/py/models/quantization/quant_config.py	Introduces structured `QuantConfig` + validation + back-compat adapters.
src/python/py/models/quantization/kv_cache_calibration.py	Updates module docs/examples to new `models.quantization` path.
src/python/py/models/quantization/cuda_quantizer.py	Adds CUDA quantization/prepacking helper utilities (torch-backed).
src/python/py/models/quantization/init.py	Exposes the quantization public API surface.
src/python/py/models/builders/phi.py	Updates LongRoPE handling and MoE routing hooks to base-class MoE orchestration.
src/python/py/models/builders/nemotron.py	Aligns MLP output publication with new base `mlp_attrs["output_0"]` flow.
src/python/py/models/builders/llama.py	Updates LLaMA3 RoPE handling to use canonical rope_parameters.
src/python/py/models/builders/lfm2.py	Simplifies LFM2 conv-cache wiring to shared cache infrastructure + CausalConvWithState helper.
src/python/py/models/builders/internlm.py	Removes hard-coded model_type override.
src/python/py/models/builders/hunyuan.py	Moves Hunyuan dynamic-theta adjustment into `make_rope_init`.
src/python/py/models/builders/granite.py	Renames hybrid MoE class to `GraniteMoEHybridModel`.
src/python/py/models/builders/gptoss.py	Refactors GPT-OSS MoE flow to base MoE hooks and updates weight naming paths.
src/python/py/models/builders/gemma.py	Marks Gemma2 TODO and keeps builder alignment with base changes.
src/python/py/models/builders/chatglm.py	Minor cleanup of partial rotary factor assignment.
src/python/py/models/builders/base.py	Central refactor: config canonicalization, cache-name templating, RoPE/MRoPE emission, MoE orchestration, quant config integration.
src/python/py/models/builders/init.py	Updates exported builder class names and Qwen MoE class rename.
src/python/py/models/builder.py	Updates model routing, model_type assignment logic, and kv-cache quantization imports.
src/models/recurrent_state.cpp	Uses explicit conv/recurrent cache name templates and discovers recurrent layers by input-name scanning.
src/models/model_type.h	Expands model-type categorization (LLM/VLM/QwenVL family) and adds MMM/Pipe helpers.
src/config.h	Adds default templates for conv/recurrent cache names and output_cross_qk naming.
src/config.cpp	Parses new `past_recurrent_names` / `present_recurrent_names` decoder template fields.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Tianlei Wu (tianleiwu)

Requesting changes for one packaging regression.

This PR moves builders/quant_config.py to models/quantization/quant_config.py, but builders/qwen.py still imports QuantConfig from .quant_config and builders/qwen_mtp.py still imports resolve_dtype from .quant_config. Running python3 -m pytest test/python/builder/test_qwen_vl.py -q fails during collection with ModuleNotFoundError: No module named 'models.builders.quant_config', so none of the new Qwen tests run.

Please update both imports to use ..quantization (or preserve a compatibility module if the old path is part of the supported API), then rerun the focused test.

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>

kunal-vaishnavi · 2026-08-26T02:46:05Z

/azp run macOS arm64 Build

azure-pipelines · 2026-08-26T02:46:16Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>

kunal-vaishnavi · 2026-08-26T05:39:10Z

/azp run macOS arm64 Build

azure-pipelines · 2026-08-26T05:39:22Z

Azure Pipelines: Successfully started running 1 pipeline(s).

PR microsoft#2386 refactored make_moe_init to write the resolved MoE op into moe_attrs["moe_op_type"], but the shared emitters (make_moe_expert_initializers and make_moe_op) read moe_attrs["op_type"], which stays at its "MoE" default. Quantized experts therefore emitted the float MoE op, which rejects uint8 weights. Set both keys so quantized MoE models correctly emit QMoE. Co-Authored-By: Claude <noreply@anthropic.com>

## Summary - restore the legacy Qwen-style convolution and recurrent state tensor names as the decoder defaults - preserve explicit state bindings emitted by current model builders - add direct config coverage and a separate legacy CUDA fixture that executes a first decode ## Background PR #2386 introduced configurable fixed-state bindings with generic `past.%d.*` defaults. Published Qwen 3.5 packages created before that change omit these fields because the earlier runtime derived `conv_state` and `recurrent_state` names from the KV-cache templates. Those packages therefore stopped binding their recurrent state inputs. The current builders already emit explicit fixed-state mappings for models that use different names, so restoring the legacy Qwen-compatible defaults fixes old packages without changing modern generated configs. ## Validation - CUDA 12.8 Release build succeeded - `DecoderStateGroupsConfigTest.UsesLegacyRecurrentStateBindingDefaults` passed - `ModelTests.LegacyRecurrentStateBindingDefaultsRunFirstDecode` passed - `ModelTests.RecurrentStateCudaGraphCaptureMatchesEager` passed with the restored explicit-binding fixture - modern explicit-binding Qwen 3.5 CUDA tests passed (2/2) - real legacy Qwen 3.5 package smoke test passed model creation, token append, and first generation

Fix INT4 QDQ export and TRT-RTX GPU inference for **Qwen/Qwen3.6-35B-A3B**. A model exported from this checkpoint with the current main-branch builder and `-e NvTensorRtRtx` fails to load when CPU fallback is disabled. Restoring compatible operators alone allows GPU execution but produces whitespace because a separate residual-wiring regression leaves the MoE results unused. This PR fixes both parts of that export path. The tested checkpoint is `Qwen/Qwen3.6-35B-A3B` at revision `995ad96eacd98c81ed38be0c5b274b04031597b0`. It uses the Qwen3.5-family MoE text builder, so the provider-specific expansions are implemented in the shared builder infrastructure rather than special-cased by checkpoint name. Two regressions were introduced by [24ad4e7](24ad4e7) (#2386): 1. **Unsupported fused operators are emitted for TRT-RTX.** The Qwen3.6-35B-A3B decoder contains 40 `GatedAdd`, 30 `LinearAttentionGate`, 30 `GatedRMSNorm`, and 20 `MRotaryEmbedding` nodes. The tested TRT-RTX parser rejects these operators, and strict session creation fails with `This session contains graph nodes that are assigned to the default CPU EP, but fallback to CPU EP has been explicitly disabled by the user.` 2. **The combined MoE output is disconnected from the residual stream.** `make_moe_subgraph` returns the combined expert output, but `make_moe` ignores that return value. The next layer's residual add therefore reuses the attention output instead of consuming the routed experts plus gated shared expert. Restore `layernorm_attrs["skip_input"]` to the combined MoE output, including the final decoder layer. The operator expansions are registered only when `self.ep == "trt-rtx"`: | Fused operator | TRT-RTX-compatible graph | | --- | --- | | `GatedAdd` | Multiply the shared expert output by its gate, then add the routed expert output. | | `LinearAttentionGate` | FP32 bias/softplus/decay calculation, cast back to the model dtype, and a separate sigmoid beta gate. | | `GatedRMSNorm` | Per-head RMS normalization followed by FP32 SiLU gating and a cast back to the model dtype. | | `MRotaryEmbedding` | Gather T/H/W positions and rotary caches, select columns for sectioned or interleaved layouts, rotate pairs, preserve the unrotated tail, and restore the packed head shape. | Small shape/index constants stay embedded as ONNX `Constant` nodes so shape inference can read them when weights are saved as external data. The expansions reuse existing `make_*` node helpers. MRoPE ownership mapping, position gathering, cache selection, rotation, and output assembly are separate public builder methods. Other providers retain their existing fused/unfused operator selection; the missing MoE residual assignment is corrected for all providers using this Qwen builder. The full model was exported from the source builder with: ```powershell python src/python/py/models/builder.py ` -i path/to/Qwen3.6-35B-A3B -o path/to/qwen36-int4-trt-rtx ` -p int4 -e NvTensorRtRtx ` --extra_options hf_token=false quant_mode=int4 int4_block_size=32 ` use_qdq=true exclude_embeds=false int4_accuracy_level=4 ` enable_cuda_graph=true qmoe_block_size=128 ``` Local validation used Windows, an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU, normal `onnxruntime==1.30.0`, and a dynamically registered TRT-RTX ABI execution provider: - **474 Qwen and related builder tests pass.** Coverage includes provider dispatch, FP16/FP32 gate comparisons against fused reference operators, sectioned/interleaved rotary layouts with distinct T/H/W positions and uneven or zero-sized sections, full/partial rotation, external-data serialization, invalid rotary attributes, empty batch/sequence dimensions, and MoE residual/shape handling. - **38 GPU numerical comparisons pass** against fused CPU reference operators. CPU fallback is disabled for every TRT-RTX session, and the execution profiles contain only TRT-RTX events. - **Fresh Qwen3.6-35B-A3B INT4 QDQ export passes.** The decoder and MTP graphs contain none of the four unsupported fused operators; all 40 decoder MoE outputs and the MTP MoE output have downstream consumers. - **Full-model text generation passes** with both released GenAI `0.17.1` and source-built GenAI `0.18.0-dev` built with CUDA 13.4. Greedy generation answers `4` to `What is 2 + 2? Answer briefly.` CPU fallback is explicitly disabled; each profile contains two TRT-RTX execution events and no CPU execution events. - **Follow-up hardening at `41f2543f` passes the expanded 474-test suite and 38 GPU comparisons.** RMSNorm uses an explicit group count so empty batch/sequence dimensions do not make reshape inference ambiguous. MRoPE rejects invalid rotary dimensions and section values, while preserving the fused operator's fixed-slot interleaving semantics. The full-model export/generation results above were collected before this follow-up; a new full-model run was not performed for these edge-case changes. - Ruff checks and `git diff --check` pass. For the generation checks, CUDA graph capture and shared KV buffers were disabled at runtime. These are text-decoder smoke tests, not a broad accuracy benchmark or validation of MTP decoding, paged attention, or full multimodal generation. Distinct multimodal rotary positions are covered separately by the numerical tests.

Use MRotaryEmbedding contrib op for Qwen VL model builders

53c6c92

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>

Copilot AI assigned Copilot and kunal-vaishnavi Aug 8, 2026

Copilot created this pull request from a session on behalf of kunal-vaishnavi August 8, 2026 00:08

View session

github-advanced-security Bot found potential problems Aug 8, 2026

View reviewed changes

Comment thread src/python/py/models/builders/qwen.py Fixed

kunal-vaishnavi added 2 commits August 8, 2026 08:46

Add Qwen-2.5 VL and Qwen-3 VL text-only model types

3f3f0e5

Validate Qwen-2.5 and Qwen-3 series of models

d507b95

kunal-vaishnavi mentioned this pull request Aug 15, 2026

Qwen3.6: export ModelOpt NVFP4/FP8 checkpoints natively #2351

Merged

kunal-vaishnavi added 2 commits August 15, 2026 10:21

Validate Qwen-3.5 dense model

2e814bf

Simplify LFM models and integrate new caches into base class

6ce342b

Copilot started work on behalf of kunal-vaishnavi August 19, 2026 21:33

View session

kunal-vaishnavi changed the title ~~Use MRotaryEmbedding contrib op for Qwen2.5-VL/Qwen3-VL M-RoPE instead of manual graph construction~~ Add Qwen VL/Qwen 3.5 builders and shared hybrid cache support Aug 19, 2026

Copilot finished work on behalf of kunal-vaishnavi August 19, 2026 21:38

Copilot AI requested a review from kunal-vaishnavi August 19, 2026 21:38

kunal-vaishnavi added 2 commits August 20, 2026 07:37

Refactor RoPE parameter handling

923854b

Validate Qwen-3.5 MoE model

727cf4c

kunal-vaishnavi marked this pull request as ready for review August 20, 2026 23:01

kunal-vaishnavi requested a review from a team as a code owner August 20, 2026 23:01

Copilot AI lite review requested due to automatic review settings August 20, 2026 23:01

Copilot AI reviewed Aug 20, 2026

View reviewed changes

Comment thread src/python/py/models/builders/base.py

Comment thread src/python/py/models/builders/base.py

Comment thread src/python/py/models/builders/base.py

Comment thread src/python/py/models/builder.py

kunal-vaishnavi added 2 commits August 20, 2026 23:13

Update todo comments

6c1a4ac

Merge branch 'main' into copilot/copilotmrope-variants-support

011a747

github-advanced-security Bot found potential problems Aug 20, 2026

View reviewed changes

Comment thread src/python/py/models/builders/base.py

Comment thread src/python/py/models/builders/base.py Fixed

Comment thread src/python/py/models/builders/phi.py Fixed

kunal-vaishnavi added 2 commits August 20, 2026 23:39

Merge branch 'main' into copilot/copilotmrope-variants-support

6143f7e

Restore MTP changes for now

fb4852e

Copilot started reviewing on behalf of kunal-vaishnavi August 21, 2026 00:18

View session

Tianlei Wu (tianleiwu) requested changes Aug 21, 2026

View reviewed changes

kunal-vaishnavi added 2 commits August 21, 2026 00:19

Update Copilot reviewer instructions

b3f5387

Fix model builder after MTP restore

7282e79

Copilot started work on behalf of kunal-vaishnavi August 25, 2026 22:12

View session

Move DML GatedAdd fallback to EP expansion

62cb323

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>

Copilot finished work on behalf of kunal-vaishnavi August 25, 2026 22:19

Remove fallback from base method now that expansion has it

f9a4e01

Copilot started work on behalf of kunal-vaishnavi August 26, 2026 02:40

View session

Copilot finished work on behalf of kunal-vaishnavi August 26, 2026 02:43

Copilot started work on behalf of kunal-vaishnavi August 26, 2026 03:31

View session

Fix recurrent state test model config

cbf840a

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>

Copilot finished work on behalf of kunal-vaishnavi August 26, 2026 03:34

Copilot started work on behalf of kunal-vaishnavi August 26, 2026 04:19

View session

Fix checked-in recurrent test model config

6e4eaa2

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>

Copilot finished work on behalf of kunal-vaishnavi August 26, 2026 04:21

Tianlei Wu (tianleiwu) approved these changes Aug 26, 2026

View reviewed changes

kunal-vaishnavi approved these changes Aug 26, 2026

View reviewed changes

kunal-vaishnavi enabled auto-merge (squash) August 26, 2026 14:28

Tianlei Wu (tianleiwu) merged commit 24ad4e7 into main Aug 26, 2026
16 of 17 checks passed

Tianlei Wu (tianleiwu) deleted the copilot/copilotmrope-variants-support branch August 26, 2026 17:19

David Fan (jiafatom) mentioned this pull request Sep 2, 2026

Restore legacy recurrent state binding fallback #2515

Merged

Tianlei Wu (tianleiwu) mentioned this pull request Sep 16, 2026

feat: add LFM2-MoE support #2575

Merged

Yuri Khrustalev (ykhrustalev) mentioned this pull request Sep 16, 2026

fix: restore the moe_attrs keys the MoE emitters read #2581

Closed

Yuri Khrustalev (ykhrustalev) mentioned this pull request Sep 26, 2026

[Bug] Old int4_ extra option names are silently ignored #2629

Closed

xieofxie mentioned this pull request Sep 28, 2026

Regression: GenAI 0.17.0 rejects older Qwen3.5 text exports with rank-2 position_ids #2639

Closed

Hrishith Thadicherla (hthadicherla) mentioned this pull request Sep 30, 2026

Fix TRT-RTX export and inference for Qwen3.6-35B-A3B #2654

Merged

Navigation Menu

Repository navigation

Add Qwen VL/3.5 support and refactor model builder infrastructure - #2386

Add Qwen VL/3.5 support and refactor model builder infrastructure#2386
Tianlei Wu (tianleiwu) merged 65 commits into
mainfrom
copilot/copilotmrope-variants-support

Copilot AI commented Aug 8, 2026 •

edited by kunal-vaishnavi

Loading

Uh oh!

azure-pipelines Bot commented Aug 8, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tianlei Wu (tianleiwu) left a comment

Uh oh!

kunal-vaishnavi commented Aug 26, 2026

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

kunal-vaishnavi commented Aug 26, 2026

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Copilot AI commented Aug 8, 2026 • edited by kunal-vaishnavi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key changes

Validation

Uh oh!

azure-pipelines Bot commented Aug 8, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tianlei Wu (tianleiwu) left a comment

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi commented Aug 26, 2026

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

kunal-vaishnavi commented Aug 26, 2026

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Copilot AI commented Aug 8, 2026 •

edited by kunal-vaishnavi

Loading