[310P]: add torch chunk gated delta rule and 910b parity ut - #7594
wangxiyuan merged 5 commits into
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a PyTorch-based implementation of the chunked gated delta rule, specifically designed for the Ascend 310P environment. This provides a robust fallback mechanism for this critical operation within the Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces a PyTorch fallback implementation for chunk_gated_delta_rule, intended for the Ascend 310P platform. The implementation supports various input formats and is accompanied by unit tests. My review has identified one high-severity issue regarding the use of DeprecationWarning for an unsupported parameter, which should be changed to an error to prevent silent failures.
As per the repository's style guide, here are suggestions for the pull request title and description:
Suggested PR Title:
[Feat/310p][Ops][Feature] Add PyTorch fallback for chunk_gated_delta_ruleSuggested PR Summary:
### What this PR does / why we need it?
This pull request introduces a PyTorch-based fallback implementation for the `chunk_gated_delta_rule` operation, specifically for the Ascend 310P platform. This is necessary to provide a functional equivalent for environments where a specialized hardware kernel is not available, ensuring broader compatibility and enabling testing/development on different platforms.
The implementation is aligned with the `torch_chunk_gated_delta_rule` from the `transformers` library (for Qwen3-Next models) and supports both standard (BTHD) and variable-length (TND with `cu_seqlens`) inputs.
### Does this PR introduce _any_ user-facing change?
Yes, this PR adds a new operator `chunk_gated_delta_rule_pytorch` under `vllm_ascend._310p.ops.fla`. This is an internal-facing change for developers working on model support, but does not affect end-users of the vLLM API directly.
### How was this patch tested?
The patch was tested by adding a new unit test file (`tests/ut/_310p/ops/test_chunk_gated_delta_rule_310.py`). The tests cover:
- Correctness of output shapes and dtypes.
- The variable-length input path using `cu_seqlens`.
- Equivalence between TND (token-major) and BTHD (batch-major) input formats for variable-length sequences.
All tests pass, ensuring the implementation is correct for the tested scenarios.|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
- add _310p chunk_gated_delta_rule_pytorch with vLLM-compatible interface - align internal chunk recurrence math with transformers torch reference flow - add CPU UT for shape/dtype and varlen interface path Signed-off-by: Tflowers-0129 <2906339855@qq.com>
Signed-off-by: Tflowers-0129 <2906339855@qq.com>
- normalize 3D TND inputs to 4D internal layout when cu_seqlens is provided - keep existing BTHD interface unchanged - return TND output when TND input is used - add UT to verify TND and BTHD varlen parity Signed-off-by: Tflowers-0129 <2906339855@qq.com>
f44c5e6 to
a892e12
Compare
…ject#7594) ### What this PR does / why we need it? RFC vllm-project#7394 Add a PyTorch implementation of the chunk gated delta rule on 310P. ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? UT --------- Signed-off-by: Tflowers-0129 <2906339855@qq.com>
…ject#7594) ### What this PR does / why we need it? RFC vllm-project#7394 Add a PyTorch implementation of the chunk gated delta rule on 310P. ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? UT --------- Signed-off-by: Tflowers-0129 <2906339855@qq.com> Signed-off-by: 01267596 <xiongkai123@cmbchina.com>
…ject#7594) ### What this PR does / why we need it? RFC vllm-project#7394 Add a PyTorch implementation of the chunk gated delta rule on 310P. ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? UT --------- Signed-off-by: Tflowers-0129 <2906339855@qq.com>
…ject#7594) ### What this PR does / why we need it? RFC vllm-project#7394 Add a PyTorch implementation of the chunk gated delta rule on 310P. ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? UT --------- Signed-off-by: Tflowers-0129 <2906339855@qq.com> Signed-off-by: zouyida2052 <zouyida2002@gmail.com>
…ject#7594) ### What this PR does / why we need it? RFC vllm-project#7394 Add a PyTorch implementation of the chunk gated delta rule on 310P. ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? UT --------- Signed-off-by: Tflowers-0129 <2906339855@qq.com>
…ject#7594) ### What this PR does / why we need it? RFC vllm-project#7394 Add a PyTorch implementation of the chunk gated delta rule on 310P. ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? UT --------- Signed-off-by: Tflowers-0129 <2906339855@qq.com> Signed-off-by: nanxing <1014662416@qq.com>
…ject#7594) ### What this PR does / why we need it? RFC vllm-project#7394 Add a PyTorch implementation of the chunk gated delta rule on 310P. ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? UT --------- Signed-off-by: Tflowers-0129 <2906339855@qq.com>
…ject#7594) ### What this PR does / why we need it? RFC vllm-project#7394 Add a PyTorch implementation of the chunk gated delta rule on 310P. ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? UT --------- Signed-off-by: Tflowers-0129 <2906339855@qq.com>
What this PR does / why we need it?
RFC #7394
Add a PyTorch implementation of the chunk gated delta rule on 310P.
Does this PR introduce any user-facing change?
NO
How was this patch tested?
UT