Skip to content

Fix Quark/AWQ weight loading for Qwen3-VL-4B text model - #2143

Merged
kunal-vaishnavi merged 3 commits into
microsoft:mainfrom
anilmartha:fix_quark_awq_qwen3_vl_text_weight_load2
May 18, 2026
Merged

Fix Quark/AWQ weight loading for Qwen3-VL-4B text model#2143
kunal-vaishnavi merged 3 commits into
microsoft:mainfrom
anilmartha:fix_quark_awq_qwen3_vl_text_weight_load2

Conversation

@anilmartha

Copy link
Copy Markdown
Contributor

Summary

Fixes Quark/AWQ weight loading for the Qwen3-VL-4B text model when building the ONNX model from a quantized Hugging Face checkpoint.
Previously:

  • Qwen3VLTextModel.load_weights always went through Qwen3VLForConditionalGeneration.from_pretrained, which doesn't understand quantized checkpoints (Quark / AWQ / GPTQ) or .gguf files.
  • The shared QuantizedModel loader didn't handle VLM checkpoint conventions (vision-tower weights, model.language_model.* prefix) or Quark's weight_quantizer.{scale,zero_point} naming.

Changes

src/python/py/models/builders/qwen.py

  • In Qwen3VLTextModel.load_weights, route quantized models (self.quant_type is not None) and .gguf inputs to the base class implementation, which loads weights via QuantizedModel. Plain HF FP checkpoints continue to use Qwen3VLForConditionalGeneration.from_pretrained as before.
    src/python/py/models/quantized_model.py
  • Skip vision-tower tensors (model.visual.*, model.vision.*, visual.*) so the LLM weight loader only consumes language-model weights.
  • Normalize the VLM prefix model.language_model.* to model.* so the existing LLM regex/tensor-parsing logic keeps working unchanged.
  • Normalize Quark naming:
    • *.weight_quantizer.scale*.weight_scale
    • *.weight_quantizer.zero_point*.weight_zero_point

Copilot AI review requested due to automatic review settings May 8, 2026 06:44
@anilmartha
anilmartha requested a review from a team as a code owner May 8, 2026 06:44
@anilmartha

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree company="AMD"

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes Qwen3-VL-4B text model weight loading when building ONNX from quantized Hugging Face checkpoints (Quark/AWQ/GPTQ) or GGUF inputs by routing those cases through the shared quantized/GGUF loader and expanding the quantized tensor name-normalization logic to handle VLM checkpoint conventions.

Changes:

  • Route Qwen3VLTextModel.load_weights to the base-class loader for quantized checkpoints and .gguf inputs (keeping the existing HF FP loading path unchanged).
  • Extend QuantizedModel weight-key handling to (a) skip vision-tower tensors in VLM checkpoints, (b) normalize model.language_model.* to model.*, and (c) normalize Quark weight_quantizer.{scale,zero_point} naming to the existing weight_{scale,zero_point} convention.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
src/python/py/models/quantized_model.py Adds VLM/Quark-specific tensor-key filtering and normalization so existing LLM parsing logic can consume Qwen3-VL language weights.
src/python/py/models/builders/qwen.py Ensures Qwen3-VL text builder uses the shared quantized/GGUF loading path for non-FP checkpoints.

Comment thread src/python/py/models/quantized_model.py Outdated
…ression tests

Extract the inline VLM/Quark key-normalisation logic from QuantizedModel.__init__
into a module-level normalize_vlm_weight_name() function so it can be unit-tested
independently without writing safetensors files or constructing a full model.

Add six regression tests covering:
- Vision-tower key filtering (model.visual.*, model.vision.*, visual.*)
- Pass-through for non-vision/non-VLM keys
- model.language_model.* -> model.* prefix rewriting
- Quark weight_quantizer.scale -> weight_scale renaming
- Quark weight_quantizer.zero_point -> weight_zero_point renaming
- Combined VLM prefix + Quark renaming in a single key

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants