[310p] Qwen3.5 moe support - #7346
Tflowers-0129 wants to merge 4 commits into
Conversation
Signed-off-by: Tflowers-0129 <2906339855@qq.com>
Signed-off-by: Tflowers-0129 <2906339855@qq.com>
Signed-off-by: Tflowers-0129 <2906339855@qq.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly expands the capabilities of the vLLM-Ascend project by integrating Qwen3.5 models, particularly their Gated Delta Network attention, onto the Ascend 310P architecture. The changes focus on providing robust PyTorch-based fallback implementations for key operations, optimizing KV cache handling, and enhancing the stability of paged attention through improved diagnostics. These updates ensure that Qwen3.5 models can leverage the 310P platform effectively, despite the absence of Triton support. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces 310P-specific implementations and fallbacks for various operations, including attention, MoE, and layernorm, to address the lack of Triton support on 310P. The changes are well-structured, with clear comments explaining the rationale behind the adaptations. New test cases for get_310p_attention_kernel_block_sizes enhance coverage, and improved logging and preflight checks contribute to robustness. However, a critical issue was identified in the causal_conv1d_fn_pytorch function regarding inconsistent tensor dimension handling, which could lead to incorrect calculations or runtime errors.
| if x.dim() == 3: | ||
| if x.shape[0] == 1: | ||
| x = x.squeeze(0) | ||
| elif x.shape[1] == 1: | ||
| x = x.squeeze(1).transpose(0, 1) | ||
| else: | ||
| raise RuntimeError(f"Unsupported x shape for causal_conv1d_fn_pytorch: {tuple(x.shape)}") | ||
| if x.dim() != 2: | ||
| raise RuntimeError(f"Unsupported x ndim for causal_conv1d_fn_pytorch: {x.dim()}") | ||
|
|
||
| feature_dim = x.shape[0] |
There was a problem hiding this comment.
The causal_conv1d_fn_pytorch function's docstring and internal checks indicate that the input x should always be 2D, specifically (dim, cu_seq_len). However, the code block from lines 119-125 attempts to handle a 3D input, which is inconsistent with the function's expected behavior and the query_start_loc check at line 105. Furthermore, the feature_dim calculation at line 129 (x.shape[1]) is incorrect if x is (dim, cu_seq_len), as it would yield cu_seq_len instead of dim. This discrepancy can lead to incorrect dimension matching with weight.shape[0] and runtime errors.
# Normalize x to [dim, total_tokens]
# The docstring and query_start_loc check imply x is always 2D (dim, cu_seq_len).
if x.dim() != 2:
raise RuntimeError(f"Unsupported x ndim for causal_conv1d_fn_pytorch: {x.dim()}, expected 2D (dim, total_tokens).")
feature_dim = x.shape[0]
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
1 participant
What this PR does / why we need it?
Does this PR introduce any user-facing change?
How was this patch tested?