Add Qwen VL/3.5 support and refactor model builder infrastructure

.github/instructions/python-model-builder.instructions.md

@@ -4,17 +4,30 @@ applyTo: "src/python/py/models/**"
 
 # Python Model Builder - Copilot Instructions
 
-When generating or reviewing code in the Python Model Builder (`src/python/py/models/`), evaluate all changes against the criteria defined in the following files:
+When generating or reviewing code in the Model Builder (`src/python/py/models/`), evaluate all changes against the criteria defined in the following files:
 
 - [README.md](../../src/python/py/models/README.md)
 - [DESIGN.md](../../src/python/py/models/DESIGN.md)
+- [LOADERS.md](../../src/python/py/models/loaders/LOADERS.md)
+- [QUANTIZATION.md](../../src/python/py/models/quantization/QUANTIZATION.md)
 
-Read both documents to understand the intended usage, supported models, design principles, and architectural constraints before suggesting or reviewing any code changes in this area.
+Read all documents to understand the intended usage, supported models, design principles, and architectural constraints before suggesting or reviewing any code changes in this area.
 
 ## Code Style Guidelines
 
-1. When a node is inserted into the model, prefer using `self.make_op_name` as the wrapper method for `self.make_node` + `self.make_value` calls.
-2. Ignore any CodeQL warnings about how an __init__ method calls an overridden method. These warnings are false positives and can be safely ignored. The warning message is: "this call to ABC in an initialization method is overwritten by XYZ".
-3. Find ways to reduce code duplication by reusing existing functionality and implementing common patterns.
-4. Discover ways to leverage the use of shared code in the base classes to avoid code duplication and improve maintainability.
-5. For any new `extra_options` that are added, make sure that they are documented in both `README.md` and `builder.py`. In `README.md`, there should be a description of the option and its purpose. There should be a usage example thereafter showing how to use the option when calling the model builder from wheel or from source. In `builder.py`, there should be a description of the option and its purpose, its default value, and any possible values. Any constraints or limitations should also be documented. Make sure the documentation is consistent across both files.
+- Lots of AI-generated code will be proposed in pull requests. Scrutinize every single line of code related to the model builder in every PR very heavily. Most of the generated code is not actually necessary if designed correctly. Most PRs do not get this right and most proposers do not understand this.
+- When a node is inserted into the model, prefer using `self.make_op_name` as the wrapper method for `self.make_node` + `self.make_value` calls.
+- Do NOT use global variables in any logic. Any static variables should be declared inside a standard Python method (e.g. `init` methods, a regular Python function, etc).
+- Ignore any CodeQL warnings about how an __init__ method calls an overridden method. These warnings are false positives and can be safely ignored. The warning message is: "this call to ABC in an initialization method is overwritten by XYZ".
+- Find ways to reduce code duplication by reusing existing functionality and implementing common patterns.
+- Discover ways to leverage the use of shared code in the base classes to avoid code duplication and improve maintainability.
+- For any new `extra_options` that are added, make sure that they are documented in both `README.md` and `builder.py`. In `README.md`, there should be a description of the option and its purpose. There should be a usage example thereafter showing how to use the option when calling the model builder from wheel or from source. In `builder.py`, there should be a description of the option and its purpose, its default value, and any possible values. Any constraints or limitations should also be documented. Make sure the documentation is consistent across both files.
+- Always raise an issue if `staticmethod` is used in any method added EXCEPT if the method it is attached to is a `from_pretrained` method. The use of `staticmethod` should otherwise be avoided. If you find a case where it is used, raise an issue to find a better way to implement the functionality without using `staticmethod`.
+- Always raise an issue if a variable or method is named with a leading underscore. The use of leading underscores should be avoided. If you find a case where it is used, raise an issue to find a better way to implement the functionality without using leading underscores.
+- Always raise an issue if there is a lot of complicated logic in a method. If you find a case where there is a lot of complicated logic, raise an issue to find a better way to implement the functionality without using complicated logic.
+- Avoid putting customized logic inside the base class constructor (i.e. avoid putting inside `Model.__init__`). Prefer using `make_description_init` methods that wrap that customized logic in a standalone `init` method.
+- When many new variables are created inside the model builder (e.g. many variables to control how an MoE subgraph looks, how quantized KV caches work, etc.), you should move those variables into settings stored within `_attrs` dictionaries. Each setting should be documented, explain its purpose, and mention when it's used.
+- Ensure that existing dispatcher methods only continue to dispatch across branches. For example, `make_matmul_op`, `make_attention_op`, and `make_mlp_op` should only dispatch across branches. If you find a case where a dispatcher method is doing more than dispatching, raise an issue to find a better way to implement the functionality without using dispatcher methods for more than dispatching.
+- As new data types and new quantization schemas are introduced, make sure that the core feature of the model builder still remains. The logic in `base.py` + each model class should be generic enough. It is the job of the parser class from the new data type or new quantization schema to convert the model into a form that the model builder can understand. The core model builder logic should not be modified directly to support new data types or new quantization schemas. If you find a case where the model builder is being modified to support new data types or new quantization schemas, raise an issue to find a better way to implement the functionality without modifying the model builder.
+- Make sure that any bit unpacking and repacking logic is done when the quantized model is parsed through the parser class. The model builder should not need to be modified to support new quantization schemas. If you find a case where the model builder is being modified to support new quantization schemas, raise an issue to find a better way to implement the functionality without modifying the model builder.
+- Make sure all new methods are added inside the `Model` class. If you find a case where a new method is being added outside of the `Model` class, raise an issue to find a better way to implement the functionality without adding new methods outside of the `Model` class.

examples/python/qwen-3.6-mtp.md

@@ -58,16 +58,15 @@ offset RMSNorm convention.
 
 ## Exporting the model
 
-The model builder emits the MTP head as a separate `mtp.onnx` when `enable_mtp=true`. The main
-model must also expose its hidden state (`include_hidden_states=true`) so the head has an input
-to consume:
+When the model configuration declares MTP layers with `mtp_num_hidden_layers`, the model builder
+emits the MTP head as a separate `mtp.onnx`. The main model also exposes its hidden state
+automatically so the head has an input to consume. Models without MTP layers omit this file:
 
 ```bash
 python -m onnxruntime_genai.models.builder \
     -i <path-to-qwen3.6-hf-checkpoint> \
     -o <output-dir> \
-    -p fp16 -e cuda \
-    --extra_options enable_mtp=true include_hidden_states=true exclude_embeds=false
+    -p fp16 -e cuda
 ```
 
 This produces, in `<output-dir>`:
@@ -77,7 +76,10 @@ This produces, in `<output-dir>`:
   RMSNorms, and a copy of the embedding + `lm_head`);
 * `genai_config.json` — carrying an `mtp` section and the decoder's `hidden_states` output.
 
-`Qwen35MtpHead` (in `src/python/py/models/builders/qwen_mtp.py`) builds the head by reusing the
+To run without MTP, remove the `model.mtp` section from `genai_config.json`. The exported ONNX
+files do not need to be rebuilt.
+
+`Qwen35MTPHead` (in `src/python/py/models/builders/qwen.py`) builds the head by reusing the
 parent `Qwen35MoeTextModel` machinery (`_make_full_attention`, `make_moe`, mRoPE, the residual
 chain) for the single layer, and loads the `mtp.*` + shared embedding/`lm_head` weights directly
 from the source safetensors — HF `transformers` discards the `mtp.*` weights on load, so they

examples/python/qwen-3.6-mtp.py

@@ -22,7 +22,7 @@
 Both require:
   * the main Qwen3.6 decoder exported with ``include_hidden_states=true`` (so it exposes
     a ``hidden_states`` output), and
-  * the MTP head (``mtp.onnx``, exported with ``enable_mtp=true``), loaded as a standalone
+    * the MTP head (``mtp.onnx``), exported when the model configuration declares MTP layers
     model whose ``hidden_states`` input is fed the main model's last hidden state.
 
 See ``qwen-3.6-mtp.md`` for the design and export instructions.

src/config.h

@@ -23,9 +23,13 @@ struct Config {
     static constexpr std::string_view PositionIdsName = "position_ids";
     static constexpr std::string_view PastKeyName = "past_key_values.%d.key";
     static constexpr std::string_view PastValueName = "past_key_values.%d.value";
+    static constexpr std::string_view PastConvName = "past.%d.conv";
+    static constexpr std::string_view PastRecurrentName = "past.%d.recurrent";
     static constexpr std::string_view LogitsName = "logits";
     static constexpr std::string_view PresentKeyName = "present.%d.key";
     static constexpr std::string_view PresentValueName = "present.%d.value";
+    static constexpr std::string_view PresentConvName = "present.%d.conv";
+    static constexpr std::string_view PresentRecurrentName = "present.%d.recurrent";
     static constexpr std::string_view HiddenStatesName = "hidden_states";
     static constexpr std::string_view RnnStatesName = "rnn_states";
     static constexpr std::string_view RnnStatesPrevName = "rnn_states_prev";
@@ -42,6 +46,7 @@ struct Config {
     static constexpr std::string_view AudioProjectionModeName = "audio_projection_mode";
     static constexpr std::string_view AudioFeaturesName = "audio_features";
     static constexpr std::string_view NumAudioTokens = "num_audio_tokens";
+    static constexpr std::string_view OutputCrossQKName = "output_cross_qk_%d";
 
     // Vision encoder names
     static constexpr std::string_view PixelValuesName = "pixel_values";
@@ -416,8 +421,8 @@ struct Config {
         // ones. Empty when the model has no windowed paged layers.
         std::string block_table_windowed{Defaults::BlockTableWindowedName};
         std::string attention_metadata{Defaults::AttentionMetadataName};
-        std::string past_conv_names{"past_conv.%d"};  // Conv cache input name template (LFM2)
-        std::string past_recurrent_names;
+        std::string past_conv_names{Defaults::PastConvName};  // Conv cache input name template (LFM2)
+        std::string past_recurrent_names{Defaults::PastRecurrentName};
 
         // Last hidden-state input (e.g. the MTP head consumes the main model's hidden state).
         // Empty unless the model graph takes a hidden_states input.
@@ -440,10 +445,10 @@ struct Config {
         std::string present_key_names{Defaults::PresentKeyName};
         std::string present_value_names{Defaults::PresentValueName};
         std::string present_names;  // When key/value pairs are combined
-        std::string output_cross_qk_names{"output_cross_qk_%d"};
+        std::string output_cross_qk_names{Defaults::OutputCrossQKName};
         std::string rnn_states{Defaults::RnnStatesName};
-        std::string present_conv_names{"present_conv.%d"};  // Conv cache output name template (LFM2)
-        std::string present_recurrent_names;
+        std::string present_conv_names{Defaults::PresentConvName};  // Conv cache output name template (LFM2)
+        std::string present_recurrent_names{Defaults::PresentRecurrentName};
         std::string hidden_states;  // Last hidden state output (when exported with include_hidden_states; e.g. fed to the MTP head)
 
         // RNNT decoder outputs

src/models/model_type.h

@@ -17,7 +17,7 @@ namespace Generators {
 struct ModelType {
   inline static bool IsLLM(const std::string& model_type) {
     // Large-language model (LLM)
-    static constexpr std::array<std::string_view, 27> LLM = {"chatglm", "decoder", "ernie4_5", "gemma", "gemma2", "gemma3_text", "gemma4_text", "gpt2", "gptoss", "granite", "granitemoehybrid", "hunyuandensev1", "internlm2", "lfm2", "llama", "mistral", "nemotron", "olmo", "phi", "phimoe", "phi3", "phi3small", "qwen2", "qwen3", "qwen3_5_moe_text", "qwen3_5_text", "smollm3"};
+    static constexpr std::array<std::string_view, 31> LLM = {"chatglm", "decoder", "ernie4_5", "gemma", "gemma2", "gemma3_text", "gemma3_vl_text", "gemma4_text", "gpt2", "gptoss", "granite", "granitemoehybrid", "hunyuandensev1", "internlm2", "lfm2", "llama", "mistral", "mistral3_text", "nemotron", "olmo", "phi", "phimoe", "phi3", "phi3small", "qwen2", "qwen2_5_vl_text", "qwen3", "qwen3_vl_text", "qwen3_5_moe_text", "qwen3_5_text", "smollm3"};
     return std::find(LLM.begin(), LLM.end(), model_type) != LLM.end();
   }
 
@@ -27,23 +27,24 @@ struct ModelType {
     return std::find(VLM.begin(), VLM.end(), model_type) != VLM.end();
   }
 
-  inline static bool IsQwenVLFamily(const std::string& model_type) {
-    // Qwen-VL family: models requiring 3D mRoPE position IDs
-    static constexpr std::array<std::string_view, 5> QwenVL = {"fara", "qwen2_5_vl", "qwen3_vl", "qwen3_5", "qwen3_5_moe"};
-    return std::find(QwenVL.begin(), QwenVL.end(), model_type) != QwenVL.end();
-  }
-
-  inline static bool IsPixtralFamily(const std::string& model_type) {
-    // Pixtral family: per-image vision loop with variable resolution
-    return model_type == "mistral3";
-  }
-
   inline static bool IsALM(const std::string& model_type) {
     // Audio-language model (ALM)
     static constexpr std::array<std::string_view, 1> ALM = {"whisper"};
     return std::find(ALM.begin(), ALM.end(), model_type) != ALM.end();
   }
 
+  inline static bool IsMMM(const std::string& model_type) {
+    // Multi-modal model (MMM)
+    static constexpr std::array<std::string_view, 2> MMM = {"gemma4", "phi4mm"};
+    return std::find(MMM.begin(), MMM.end(), model_type) != MMM.end();
+  }
+
+  inline static bool IsPipe(const std::string& model_type) {
+    // Pipeline (Pipe)
+    static constexpr std::array<std::string_view, 1> Pipe = {"decoder-pipeline"};
+    return std::find(Pipe.begin(), Pipe.end(), model_type) != Pipe.end();
+  }
+
   inline static bool IsRNNT(const std::string& model_type) {
     // RNNT models bypass the search/logits pipeline entirely.
     static constexpr std::array<std::string_view, 1> rnnt_types = {"nemotron_speech"};
@@ -61,16 +62,15 @@ struct ModelType {
     return IsRNNT(model_type) || IsTDT(model_type);
   }
 
-  inline static bool IsMMM(const std::string& model_type) {
-    // Multi-modal model (MMM)
-    static constexpr std::array<std::string_view, 2> MMM = {"gemma4", "phi4mm"};
-    return std::find(MMM.begin(), MMM.end(), model_type) != MMM.end();
+  inline static bool IsQwenVLFamily(const std::string& model_type) {
+    // Qwen-VL family: models requiring 3D mRoPE position IDs
+    static constexpr std::array<std::string_view, 10> QwenVL = {"fara", "qwen2_5_vl_text", "qwen2_5_vl", "qwen3_vl", "qwen3_vl_text", "qwen3_5", "qwen3_5_moe", "qwen3_5_moe_text", "qwen3_5_text", "videochat_flash_qwen"};
+    return std::find(QwenVL.begin(), QwenVL.end(), model_type) != QwenVL.end();
   }
 
-  inline static bool IsPipe(const std::string& model_type) {
-    // Pipeline (Pipe)
-    static constexpr std::array<std::string_view, 1> Pipe = {"decoder-pipeline"};
-    return std::find(Pipe.begin(), Pipe.end(), model_type) != Pipe.end();
+  inline static bool IsPixtralFamily(const std::string& model_type) {
+    // Pixtral family: per-image vision loop with variable resolution
+    return model_type == "mistral3";
   }
 
   inline static bool IsLFM2(const std::string& model_type) {