[Model]Add Qwen3-Omni quantization Ascend NPU adaptation and optimization

[tanhaoan333]

What this PR does / why we need it?

This pull request is for quantization adaptation of Qwen3Omni, and it achieves operator-level optimization and AUT (Auto-Quantization Tuning) component optimization through patch-based modifications.

Does this PR introduce any user-facing change?

How was this patch tested?

• vLLM version: v0.15.0
• vLLM main: vllm-project/vllm@83b47f6

[github-actions]

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

• A PR should do only one thing, smaller PRs enable faster reviews.
• Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
• Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

[gemini-code-assist]

Summary of Changes

Hello @tanhaoan333, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the support for the Qwen3Omni model on NPU (Neural Processing Unit) platforms. It introduces specialized NPU-optimized implementations for the model's audio attention and encoder layers, alongside crucial adjustments to the dynamic quantization pipeline. These changes aim to improve both the performance and accuracy of Qwen3Omni when running with quantization on NPU hardware, and ensure robust handling of quantization configurations.

Highlights

• NPU Optimization for Qwen3Omni: Implemented NPU-specific optimizations for Qwen3OmniMoE audio attention and encoder components, leveraging torch_npu for improved performance and efficiency on NPU hardware.
• Dynamic Quantization Adaptation: Adapted the dynamic quantization method (W8A8_DYNAMIC) to correctly handle tensor dimensions for pertoken_scale during NPU matrix multiplication, preventing shape mismatches.
• Quantization Configuration Refinement: Refined the quantization configuration logic to accurately determine the quantization type and identify layers that should be skipped for models with mixed precision, improving the flexibility and correctness of auto-quantization tuning.
• Type Consistency in MoE MLP: Ensured type consistency in Mixture-of-Experts (MoE) MLP operations by explicitly casting tensor types to torch.int32 for cumsum operations, which can be crucial for NPU compatibility and performance.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• vllm_ascend/ops/fused_moe/moe_mlp.py
  • Casted group_list and group_list[:, 1] to torch.int32 before cumsum operations.
  • Added a blank line at the end of the file.
• vllm_ascend/patch/worker/init.py
  • Added an import for patch_qwen3_omni_moe_thinker.
• vllm_ascend/patch/worker/patch_qwen3_omni_moe_thinker.py
  • Added a new file containing NPU-optimized Qwen3OmniMoeAudioAttention, Qwen3OmniMoeAudioEncoder, and Qwen3OmniMoeAudioEncoderLayer classes.
  • Implemented patching logic to replace original transformers and vllm components with their NPU-optimized counterparts when NPU is available.
  • Defined WeightsMapper and packed_modules_mapping for Qwen3OmniMoeThinkerForConditionalGeneration.
• vllm_ascend/quantization/methods/w8a8_dynamic.py
  • Added logic to squeeze quantized_x and pertoken_scale if pertoken_scale has 2 dimensions before npu_quant_matmul.
  • Added logic to unsqueeze the output after npu_quant_matmul if the input was squeezed.
• vllm_ascend/quantization/modelslim_config.py
  • Modified get_linear_quant_type to check if any weight within a prefix is 'FLOAT' to determine the quantization type, defaulting to 'W8A8_DYNAMIC'.
  • Updated is_layer_skipped_ascend to check if any weight within a prefix is 'FLOAT' to determine if a layer should be skipped.

Activity

• No activity (comments, reviews, etc.) was provided in the context.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces quantization support for the Qwen3Omni model on Ascend NPUs. This is achieved by adding NPU-specific patches for the model's audio encoder and attention mechanisms, leveraging fused kernels for performance optimization. The changes also include fixes and improvements for quantization logic, particularly for handling fused layers and varied tensor shapes from quantization operations. My review found a critical issue related to a missing import in the new patch file, which would cause a runtime error.

[gemini-code-assist]

math.sqrt is used here, but the math module is not imported in this file. This will cause a NameError at runtime. Please add import math at the top of the file.