[BugFix] fix fiaV2 contiguous err in GQA - #13456
Conversation
Signed-off-by: zhaoyhh <zhaoyuhui14@h-partners.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a bug in the GQA (Grouped Query Attention) implementation within the vLLM Ascend backend. By enforcing contiguous memory layouts for key and value tensors, it resolves potential compatibility issues when invoking the NPU-specific fused attention kernel. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request ensures that the key and value tensors are contiguous before being passed to the npu_fused_infer_attention_score_v2 function in vllm_ascend/attention/attention_v1.py. The review feedback correctly points out that the PR title and description violate the repository's style guide and provides a compliant suggestion.
| key.contiguous(), | ||
| value.contiguous(), |
There was a problem hiding this comment.
The Pull Request title and description do not adhere to the repository style guide. The PR description is currently empty, and the PR title is missing the module prefix.\n\nPlease update the PR title and description to match the following suggested formats:\n\nSuggested PR Title:\n\nmarkdown\n[Attention][BugFix] fix fiaV2 contiguous err in GQA\n\n\nSuggested PR Summary:\n\nmarkdown\n### What this PR does / why we need it?\n\nThis PR fixes a contiguous tensor error in Grouped Query Attention (GQA) when using `npu_fused_infer_attention_score_v2`. In GQA, the key and value tensors can be non-contiguous (e.g., due to slicing or striding). Passing non-contiguous tensors to `npu_fused_infer_attention_score_v2` causes runtime errors. This is resolved by calling `.contiguous()` on the `key` and `value` tensors before passing them to the attention score function.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nTested with existing attention tests and GQA workloads on Ascend NPU.\n
References
- The Pull Request title and summary must follow the repository style guide format. The title should be in the format
[Branch][Module][Action] Pull Request Title, and the summary should contain sections for 'What this PR does / why we need it?', 'Does this PR introduce any user-facing change?', and 'How was this patch tested?'. (link)
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. Tip 💡 Consider Linking a Related Issue or RFCYour PR title contains the [BugFix] tag, indicating a bug fix or new feature. Linking a related issue or RFC in the PR description is strongly encouraged — it gives reviewers helpful context and speeds up the review. You can use any of these keywords:
🙏 Thanks for helping us keep the project well-organized! |
|
/rerun Failed:
|
|
/rerun Rerun:
|
…into vllm-new # By shenhui-cli (8) and others # Via GitHub (1) and shenhui-cli (1) * 'vllm-new' of https://github.com/shenhui-cli/vllm-ascend: (34 commits) When the PR modifies any files in the CSRC folder, skip the test case filtering logic and execute all test cases by default. [MRV2][Bugfix] Fix two triton ops errors in model_runner_v2 num_nans_kernel and apply_panalties (vllm-project#13159) [Cherry-pick][main][Doc][BugFix] Update proxy script name in DeepSeek-V3.2 tutorial (from vllm-project#13537) (vllm-project#13575) [Refactor][quantization]remove mxfp_compat compatibility shim and inline dtype references (vllm-project#13447) [MTP][BugFix] Preserve MoE weight loaders during online updates (vllm-project#13337) [Feature][xlite] Support MLA and DSA (`DeepseekV3ForCausalLM`, `DeepseekV32ForCausalLM` and `GlmMoeDsaForCausalLM`) in the xlite adapter (vllm-project#13378) [BugFix][xlite] Fix DP metadata handling in XliteWrapper and pass down the max number of tokens across all DP ranks for collective communications. (vllm-project#11213) [Refactor][Ops] Move expert routing into router classes (vllm-project#13417) [Platform][Refactor] Refactor NPUPlatform for better organization and clarity (vllm-project#13484) [BugFix] fix fiaV2 contiguous err in GQA (vllm-project#13456) [feature][KV Offload] Support Sparse KV Cache Offload (vllm-project#13026) [Bugfix][MRV2]Skip D2H copy and synchronize when spec decode is not active (vllm-project#13382) [Bugfix] Change AscendSFAIndexerCacheSpec to inherit MLAAttentionSpec (vllm-project#12849) [BugFix][FusedMoE] Restore BF16 quant method initialization (vllm-project#13412) [Doc] Fix link errors and add section anchors (vllm-project#13485) [TEST]Revise the A3 case (vllm-project#13495) [CI] modify default cann_version and add build_type support in nightly (vllm-project#13480) [CI]Improve logging, optimize function-level recommendation algorithm, and add early exit for no product code changes (vllm-project#13472) This PR restricts the test case discovery scope to the following directories: tests/e2e/pull_request/ tests/ut/ This PR restricts the test case discovery scope to the following directories: tests/e2e/pull_request/ tests/ut/ ... # Conflicts: # .github/workflows/scripts/test_selector.py
### What this PR does / why we need it? When CANN is updated to 9.1.0, the fused_infer_attention operator added a new constraint that key and value should be contiguous when input into this FIAV2 operator. This PR makes the implementation. ### Does this PR introduce _any_ user-facing change? NO. ### How was this patch tested? CI passed. - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@0351e9a Signed-off-by: zhaoyhh <zhaoyuhui14@h-partners.com> Co-authored-by: zhaoyhh <zhaoyuhui14@h-partners.com>
### What this PR does / why we need it? When CANN is updated to 9.1.0, the fused_infer_attention operator added a new constraint that key and value should be contiguous when input into this FIAV2 operator. This PR makes the implementation. ### Does this PR introduce _any_ user-facing change? NO. ### How was this patch tested? CI passed. - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@0351e9a Signed-off-by: zhaoyhh <zhaoyuhui14@h-partners.com> Co-authored-by: zhaoyhh <zhaoyuhui14@h-partners.com>
### What this PR does / why we need it? When CANN is updated to 9.1.0, the fused_infer_attention operator added a new constraint that key and value should be contiguous when input into this FIAV2 operator. This PR makes the implementation. ### Does this PR introduce _any_ user-facing change? NO. ### How was this patch tested? CI passed. - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@0351e9a Signed-off-by: zhaoyhh <zhaoyuhui14@h-partners.com> Co-authored-by: zhaoyhh <zhaoyuhui14@h-partners.com>
4 participants
What this PR does / why we need it?
When CANN is updated to 9.1.0, the fused_infer_attention operator added a new constraint that key and value should be contiguous when input into this FIAV2 operator. This PR makes the implementation.
Does this PR introduce any user-facing change?
NO.
How was this patch tested?
CI passed.