[Bugfix]Fix deepseek 3.2 C8 precision by rotary tensor

[Yaphets24]

What this PR does / why we need it?

During the attention quantization process of DeepSeek V3.2, it is necessary to retrieve the Hadamard matrix from the weights to facilitate the computation.

Does this PR introduce any user-facing change?

No. But there will be two new tensor in quant weight.

How was this patch tested?

• vLLM version: v0.18.0
• vLLM main: vllm-project/vllm@8b63257

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the vllm_ascend framework to support DeepseekV2 3.2 models, specifically by integrating C8 rotation tensors into the attention mechanism. It refactors the handling of query and key rotation components, introduces a dedicated quantization scheme for these new rotation tensors, and updates the model loading process to correctly map and initialize them. These changes are crucial for optimizing the performance and accuracy of DeepseekV2 models on Ascend hardware.

Highlights

• DeepseekV2 C8 Rotation Tensor Support: Introduced support for DeepseekV2 3.2 C8 rotation tensors by splitting the previously unified qk_hadamard tensor into separate q_hadamard and k_hadamard tensors within the AscendSFAImpl attention mechanism.
• Dynamic Quantization for Indexer Rotation: Implemented a new AscendSFAQuantAttentionMethod for INT8_DYNAMIC quantization specifically for attention layers, which creates indexer.q_rot and indexer.k_rot parameters to handle the rotation tensors.
• DeepseekV2 Weight Loading Patch: Added a new patch file (patch_deepseekv2.py) that overrides the DeepseekV2ForCausalLM.load_weights method to correctly remap and load indexer.q_rot and indexer.k_rot weights during model initialization.
• Quantization Configuration Updates: Modified the modelslim_config.py to recognize and enable indexer_quant_type for attention layers, allowing the system to apply specific quantization methods to the indexer rotation components.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for a new attention mechanism feature (C8 rot tensor) for Deepseek V2 models on Ascend hardware. The changes involve splitting the Hadamard tensor for Q and K in sparse attention, adding a patch to load new indexer rotation tensor weights, and updating quantization configurations to support this. My review has identified a critical issue in the new weight loading patch that would prevent most of the model's weights from being loaded due to a hardcoded layer limit. I've also pointed out a performance improvement opportunity in the attention implementation to avoid redundant tensor creation. Addressing these points will ensure the feature works correctly and efficiently.

[gemini-code-assist]

This block of code contains two issues:

1. Critical Bug: It uses a hardcoded NUM_HIDDEN_LAYERS = 4 (defined on line 16) to skip loading weights for layers with an index of 4 or greater. This will prevent loading most of the model's layers, as models like DeepseekV2 have far more than 4 layers. This logic should be removed to ensure the entire model is loaded.
2. Bad Practice: The import re statement is inside a loop. Imports should be at the top of the file for performance and style. Since the logic using re should be removed, this import becomes unnecessary.

[gemini-code-assist]

The Hadamard matrix is being created twice, which is inefficient. Since q_hadamard and k_hadamard are initialized with the same value, you can create the tensor once and assign it to both, avoiding redundant computation and memory allocation.

        if self.use_sparse_c8_indexer and (AscendSFAImpl.q_hadamard is None or AscendSFAImpl.k_hadamard is None):
            hadamard_matrix = torch.tensor(scipy.linalg.hadamard(128), dtype=torch.bfloat16, device="npu") / (128**0.5)
            if AscendSFAImpl.q_hadamard is None:
                AscendSFAImpl.q_hadamard = hadamard_matrix
            if AscendSFAImpl.k_hadamard is None:
                AscendSFAImpl.k_hadamard = hadamard_matrix

[github-actions]

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

• A PR should do only one thing, smaller PRs enable faster reviews.
• Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
• Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

[rjg-lyh]

This PR replaces the default matrix values with more precise weights loaded during model initialization, enhancing overall network performance in the C8 scenario. Thanks for this contribution.

[MengqingCao]

LGTM