Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .agents/skills/vllm-ascend-release/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -672,15 +672,15 @@ vLLM Ascend v0.18.0rc1版本发布🎉 此版本是针对vLLM v0.18.0系列版

🥳 本版本新增9位新开发者,感谢社区开发者的持续贡献!
💥 C8(INT8 KV cache)支持GQA attention模型,同时适配DeepSeek-V3.1 PD分离场景
💥 DeepSeek模型通过新MLA算子支持A5硬件
💥 DeepSeek模型通过新MLA算子支持Ascend 950系列产品
🆕 Flash Comm V1支持VL模型的MLA,解除多模态服务限制
🆕 支持speculative decoding中target和draft模型使用不同attention backend
🆕 VL MoE模型支持SP,`sp_threshold`替换为vLLM原生`sp_min_token_num`
🆕 Qwen VL模型支持`w8a8_mxfp8`量化
🚀 Triton算子重编译优化,提升算子性能
🚀 Qwen3.5/Qwen3-Next GDN prefill路径优化,预构建chunk metadata减少h2d同步开销
🚀 FIA prefill context merge路径简化,提升运行时效率
🐞 torch-npu 和 triton-ascend 依赖版本更新,请参考官方release note
🐞 TorchNPU 和 triton-ascend 依赖版本更新,请参考官方release note
🐞 修复PD分离场景decode节点因DP节点shape不对齐导致卡住的问题
🐞 修复单卡部署多实例显存 OOM 问题
🐞 修复 DeepSeek v3.1 C8在MTP + full decode + full graph模式下的问题
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ This is the first release candidate of v0.14.0 for vLLM Ascend. Please follow th

### Dependencies

- torch-npu is upgraded to 2.9.0 [#6112](https://github.com/vllm-project/vllm-ascend/pull/6112)
- TorchNPU is upgraded to 2.9.0 [#6112](https://github.com/vllm-project/vllm-ascend/pull/6112)

### Deprecation & Breaking Changes

Expand Down Expand Up @@ -92,7 +92,7 @@ We mainly focus on quality and performance improvement in this release. The spec
### Dependencies

- CANN is upgraded to 8.5.0
- torch-npu is upgraded to 2.8.0.post1. Please note that the post version will not be installed by default. Please install it by hand from [pypi mirror](https://mirrors.huaweicloud.com/ascend/repos/pypi/torch-npu/).
- TorchNPU is upgraded to 2.8.0.post1. Please note that the post version will not be installed by default. Please install it by hand from [pypi mirror](https://mirrors.huaweicloud.com/ascend/repos/pypi/torch-npu/).
- triton-ascend is upgraded to 3.2.0

### Deprecation & Breaking Changes
Expand Down Expand Up @@ -187,7 +187,7 @@ We're excited to announce the release of v0.11.0 for vLLM Ascend. This is the of

### Upgrade notice

- torch-npu is upgraded to 2.7.1.post1. Please note that the package is pushed to [pypi mirror](https://mirrors.huaweicloud.com/ascend/repos/pypi/torch-npu/). So it's hard to add it to auto dependence. Please install it by yourself.
- TorchNPU is upgraded to 2.7.1.post1. Please note that the package is pushed to [pypi mirror](https://mirrors.huaweicloud.com/ascend/repos/pypi/torch-npu/). So it's hard to add it to auto dependence. Please install it by yourself.
- CANN is upgraded to 8.3.rc2.

### Known Issues
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ For detailed information on supported models, please refer to [supported models]
- Software:
- Python >= 3.10, < 3.13
- CANN == 9.0.1 (For Ascend HDK version, please refer to the [Release Notes](https://www.hiascend.com/document/detail/zh/canncommercial/900/releasenote/releasenote_0000.html))
- PyTorch == 2.10.0, torch-npu == 2.10.0.post2
- PyTorch == 2.10.0, TorchNPU == 2.10.0.post2
- vLLM (the same version as vllm-ascend)

## Getting Started
Expand Down
2 changes: 1 addition & 1 deletion README.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ vLLM 昇腾插件 (`vllm-ascend`) 是一个由社区维护的让vLLM在Ascend NP
- 软件:
- Python >= 3.10, < 3.13
- CANN == 9.0.1 (Ascend HDK 版本详见 [版本说明](https://www.hiascend.com/document/detail/zh/canncommercial/900/releasenote/releasenote_0000.html))
- PyTorch == 2.10.0, torch-npu == 2.10.0.post2
- PyTorch == 2.10.0, TorchNPU == 2.10.0.post2
- vLLM (与vllm-ascend版本一致)

## 开始使用
Expand Down
2 changes: 1 addition & 1 deletion docs/source/community/versioning_policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ Notes:

## Software dependency management

- `torch-npu`: Ascend Extension for PyTorch (torch-npu) releases a stable version to [PyPI](https://pypi.org/project/torch-npu)
- `torch-npu`: TorchNPU releases a stable version to [PyPI](https://pypi.org/project/torch-npu)
every 3 months, a development version (aka the POC version) every month, and a nightly version every day.
The PyPI stable version **CAN** be used in vLLM Ascend final version, the monthly dev version **ONLY CAN** be used in
vLLM Ascend RC version for rapid iteration, and the nightly version **CANNOT** be used in any vLLM Ascend version or branch.
2 changes: 1 addition & 1 deletion docs/source/developer_guide/contribution/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ docker run --rm --name vllm-ascend-ut \
sed -i 's|ports.ubuntu.com|mirrors.huaweicloud.com|g' /etc/apt/sources.list
pip config set global.index-url https://mirrors.huaweicloud.com/repository/pypi/simple/

# For torch-npu dev version or x86 machine
# For TorchNPU dev version or x86 machine
export PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu/ https://mirrors.huaweicloud.com/ascend/repos/pypi"

# src path
Expand Down
4 changes: 2 additions & 2 deletions docs/source/faqs.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Below series are NOT supported yet:
- Atlas 200I A2 (Ascend-cann-kernels-310b) unplanned yet
- Ascend 910, Ascend 910 Pro B (Ascend-cann-kernels-910) unplanned yet

From a technical view, vllm-ascend supports devices if torch-npu is supported. Otherwise, we have to implement it by using custom ops. We also welcome you to join us to improve together.
From a technical view, vllm-ascend supports devices if TorchNPU is supported. Otherwise, we have to implement it by using custom ops. We also welcome you to join us to improve together.

### 2. How to get our docker containers?

Expand Down Expand Up @@ -229,7 +229,7 @@ ACL graph capture can still fail when the runtime resources required by the sele

### 18. How to install custom version of torch_npu?

torch-npu will be overridden when installing vllm-ascend. If you need to install a specific version of torch-npu, you can manually install the specified version of torch-npu after vllm-ascend is installed.
TorchNPU will be overridden when installing vllm-ascend. If you need to install a specific version of TorchNPU, you can manually install the specified version of TorchNPU after vllm-ascend is installed.

### 19. On certain systems (e.g., Kylin OS), `docker pull` may fail with an `invalid tar header` error

Expand Down
18 changes: 9 additions & 9 deletions docs/source/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ This document describes how to install vllm-ascend manually.
| Software | Supported version | Note |
|---------------|----------------------------------|-------------------------------------------|
| Ascend HDK | Refer to the documentation [CANN 9.0.1](https://www.hiascend.com/document/detail/zh/canncommercial/900/releasenote/releasenote_0000.html) | Required for CANN |
| CANN | == 9.0.1 | Required for vllm-ascend and torch-npu |
| torch-npu | == 2.10.0 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
| torch | == 2.10.0 | Required for torch-npu and vllm, No need to install manually, it will be auto installed in below steps |
| CANN | == 9.0.1 | Required for vllm-ascend and TorchNPU |
| TorchNPU | == 2.10.0 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
| torch | == 2.10.0 | Required for TorchNPU and vllm, No need to install manually, it will be auto installed in below steps |
| NNAL | == 9.0.1 | Required for libatb.so, enables advanced tensor operations |

```{note}
Expand All @@ -31,9 +31,9 @@ Atlas inference products require CANN 9.1.0 . The Atlas A2/A3 requirements in th
| Software | Supported version | Note |
|---------------|----------------------------------|-------------------------------------------|
| Ascend HDK | Refer to the documentation [CANN 9.1.0](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910beta1/releasenote/9.1.0-beta.1/release-note.md) | Required for CANN |
| CANN | == 9.1.0 | Required for vllm-ascend and torch-npu |
| torch-npu | == 2.10.0 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
| torch | == 2.10.0 | Required for torch-npu and vllm, No need to install manually, it will be auto installed in below steps |
| CANN | == 9.1.0 | Required for vllm-ascend and TorchNPU |
| TorchNPU | == 2.10.0 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
| torch | == 2.10.0 | Required for TorchNPU and vllm, No need to install manually, it will be auto installed in below steps |
| NNAL | == 9.1.0 | Required for libatb.so, enables advanced tensor operations |
| triton / triton-ascend | Not supported | Uninstalled in `Dockerfile.310p` |

Expand Down Expand Up @@ -153,10 +153,10 @@ apt-get update -y && apt-get install -y gcc g++ cmake libnuma-dev wget git curl
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
```

**[Optional]** Then configure the extra-index of `pip` if you are working on an x86 machine or using torch-npu dev version:
**[Optional]** Then configure the extra-index of `pip` if you are working on an x86 machine or using TorchNPU dev version:

```bash
# For torch-npu dev version or x86 machine
# For TorchNPU dev version or x86 machine
pip config set global.extra-index-url "https://download.pytorch.org/whl/cpu/"
```

Expand Down Expand Up @@ -259,7 +259,7 @@ pip uninstall -y triton-ascend triton
:::

```{note}
To build custom operators, gcc/g++ higher than 8 and C++17 or higher are required. If you are using `pip install -e .` and encounter a torch-npu version conflict, please install with `pip install --no-build-isolation -e .` to build on system env.
To build custom operators, gcc/g++ higher than 8 and C++17 or higher are required. If you are using `pip install -e .` and encounter a TorchNPU version conflict, please install with `pip install --no-build-isolation -e .` to build on system env.
If you encounter other problems during compiling, it is probably because an unexpected compiler is being used, you may export `CXX_COMPILER` and `C_COMPILER` in the environment to specify your g++ and gcc locations before compiling.

If you are building in a CPU-only environment where `npu-smi` is unavailable, you need to set `SOC_VERSION` before `pip install -e .` so the build can target the correct chip. You can refer to `Dockerfile*` defaults, for example:
Expand Down
12 changes: 6 additions & 6 deletions docs/source/quick_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@ This section guides you through container-based environment setup and large mode
| Software | Supported version | Note |
|---------------|----------------------------------|-------------------------------------------|
| Ascend HDK | Refer to the documentation [CANN 9.0.1](https://www.hiascend.com/document/detail/zh/canncommercial/900/releasenote/releasenote_0000.html) | Required for CANN |
| CANN | == 9.0.1 | Required for vllm-ascend and torch-npu |
| torch-npu | == 2.10.0 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
| torch | == 2.10.0 | Required for torch-npu and vllm, No need to install manually, it will be auto installed in below steps |
| CANN | == 9.0.1 | Required for vllm-ascend and TorchNPU |
| TorchNPU | == 2.10.0 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
| torch | == 2.10.0 | Required for TorchNPU and vllm, No need to install manually, it will be auto installed in below steps |
| NNAL | == 9.0.1 | Required for libatb.so, enables advanced tensor operations |

```{note}
Expand All @@ -51,9 +51,9 @@ Atlas 200I Pro requires additional device nodes and driver mounts. See [Set up u
| Software | Supported version | Note |
|---------------|----------------------------------|-------------------------------------------|
| Ascend HDK | Refer to the documentation [CANN 9.1.0](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910beta1/releasenote/9.1.0-beta.1/release-note.md) | Required for CANN |
| CANN | == 9.1.0 | Required for vllm-ascend and torch-npu |
| torch-npu | == 2.10.0 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
| torch | == 2.10.0 | Required for torch-npu and vllm, No need to install manually, it will be auto installed in below steps |
| CANN | == 9.1.0 | Required for vllm-ascend and TorchNPU |
| TorchNPU | == 2.10.0 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
| torch | == 2.10.0 | Required for TorchNPU and vllm, No need to install manually, it will be auto installed in below steps |
| NNAL | == 9.1.0 | Required for libatb.so, enables advanced tensor operations |
| triton / triton-ascend | Not supported | Uninstalled in `Dockerfile.310p` |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ All MoE models supported by vLLM-Ascend.
But we have only verified the performance on deepseek-v3.1/r1 models.

> [!IMPORTANT]
> Ascend A5 does not support using EPLB with quant type "W4A8MXFP4", "W4A16", "W4A16MXFP4".
> Ascend 950 Products does not support using EPLB with quant type "W4A8MXFP4", "W4A16", "W4A16MXFP4".

### MOE QuantType

Expand Down
16 changes: 8 additions & 8 deletions docs/source/user_guide/feature_guide/kv_pool.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ export PYTHONHASHSEED=0
cat /etc/hccn.conf
```

For A5 series, additionally mount:
For Ascend 950 Products, additionally mount:
* devices: `/dev/ummu`, `/dev/uburma`
* commands: `/usr/bin/urma_admin`
* configurations: `/lib/route.conf`, `/etc/hccl_rootinfo.json`
Expand All @@ -85,7 +85,7 @@ export PYTHONHASHSEED=0

| Hardware | Dependencies | Export Command | Description |
| :--- | :--- | :--- | :--- |
| 800 I/T A5 series | HDK >=25.6 with mooncake >= v0.3.11 <br>CANN >= 9.1.0 | # UBOE<br> `export ASCEND_GLOBAL_RESOURCE_CONFIG='{"comm_resource_config.protocol_desc":["uboe:device"]}'` <br> # UB<br>`export ASCEND_LOCAL_COMM_RES='{"version":"1.3"}'` | Configure the required environment variables based on the communication protocol to use. |
| Ascend 950 Products | HDK >=25.6 with mooncake >= v0.3.11 <br>CANN >= 9.1.0 | # UBOE<br> `export ASCEND_GLOBAL_RESOURCE_CONFIG='{"comm_resource_config.protocol_desc":["uboe:device"]}'` <br> # UB<br>`export ASCEND_LOCAL_COMM_RES='{"version":"1.3"}'` | Configure the required environment variables based on the communication protocol to use. |
| 800 I/T A3 series | HDK >= 26.0<br>or HDK >= 25.5 with mooncake >= v0.3.11<br>CANN >= 9.0.0<br>LingQu Computing Network >= 1.5 | `export ASCEND_ENABLE_USE_FABRIC_MEM=1` | **Recommended**. Enables unified memory address direct transmission scheme. With SSD offload, see [Fabric memory size alignment](#122-fabric-memory-size-alignment-a3--ascend_enable_use_fabric_mem1) — memory sizes must be aligned to 1GB. |
| 800 I/T A3 series | If any dependency above is not met | `export ASCEND_BUFFER_POOL=4:8` | Configures the number and size of buffers on the NPU Device for aggregation and KV transfer (e.g., `4:8` means 4 buffers of 8MB). |
| 800 I/T A2 series | HDK >= 25.5 is recommended | `export HCCL_INTRA_ROCE_ENABLE=1` | Required by direct transmission scheme on 800 I/T A2 series|
Expand Down Expand Up @@ -157,9 +157,9 @@ export ACL_OP_INIT_MODE=1
export ASCEND_ENABLE_USE_FABRIC_MEM=1
#A2
#export HCCL_INTRA_ROCE_ENABLE=1
#A5 UBOE
#Ascend 950 Products UBOE
#export ASCEND_GLOBAL_RESOURCE_CONFIG='{"comm_resource_config.protocol_desc":["uboe:device"]}'
#A5 UB
#Ascend 950 Products UB
#export ASCEND_LOCAL_COMM_RES='{"version":"1.3"}'

#Minimum retransmission timeout of the RDMA, equals 4.096 μs * 2 ^ timeout.
Expand Down Expand Up @@ -240,9 +240,9 @@ export ACL_OP_INIT_MODE=1
export ASCEND_ENABLE_USE_FABRIC_MEM=1
#A2
#export HCCL_INTRA_ROCE_ENABLE=1
#A5 UBOE
#Ascend 950 Products UBOE
#export ASCEND_GLOBAL_RESOURCE_CONFIG='{"comm_resource_config.protocol_desc":["uboe:device"]}'
#A5 UB
#Ascend 950 Products UB
#export ASCEND_LOCAL_COMM_RES='{"version":"1.3"}'
export HCCL_RDMA_TIMEOUT=17
export ASCEND_CONNECT_TIMEOUT=10000
Expand Down Expand Up @@ -361,9 +361,9 @@ export ACL_OP_INIT_MODE=1
export ASCEND_ENABLE_USE_FABRIC_MEM=1
#A2
#export HCCL_INTRA_ROCE_ENABLE=1
#A5 UBOE
#Ascend 950 Products UBOE
#export ASCEND_GLOBAL_RESOURCE_CONFIG='{"comm_resource_config.protocol_desc":["uboe:device"]}'
#A5 UB
#Ascend 950 Products UB
#export ASCEND_LOCAL_COMM_RES='{"version":"1.3"}'
export HCCL_RDMA_TIMEOUT=17
export ASCEND_CONNECT_TIMEOUT=10000
Expand Down
Loading