[doc] Upload doc for qwen3.5-27B and qwen3.5-397B-A17B on Ascend by ppppeng · Pull Request #7313 · vllm-project/vllm-ascend

ppppeng · 2026-03-16T09:27:52Z

What this PR does / why we need it?

Upload doc for qwen3.5-27B and qwen3.5-397B-A17B on Ascend
Base on vllm-ascend:v0.17.0rc1

Does this PR introduce any user-facing change?

How was this patch tested?

vLLM version: v0.17.0
vLLM main: vllm-project/vllm@4034c3d

gemini-code-assist · 2026-03-16T09:28:07Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the project's support for Qwen3.5 models on Ascend hardware by providing detailed user documentation and corresponding end-to-end tests. The changes enable users to effectively deploy, verify, and evaluate both the Qwen3.5-27B and Qwen3.5-397B-A17B models, enhancing the platform's utility for AI inference on specialized hardware.

Highlights

New Model Documentation: Added comprehensive documentation for deploying and evaluating Qwen3.5-27B and Qwen3.5-397B-A17B models on Ascend platforms.
Ascend Support: The documentation details environment setup, single-node and multi-node deployment, functional verification, and performance/accuracy evaluation specific to Ascend hardware.
End-to-End Testing: Introduced new end-to-end test cases for Qwen3.5 models, including distributed execution with tensor parallelism and configurations for full decode only and speculative decoding.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

docs/source/tutorials/models/Qwen3.5-27B.md
- Added new documentation for Qwen3.5-27B model deployment and evaluation on Ascend.
docs/source/tutorials/models/Qwen3.5-397B-A17B.md
- Added new documentation for Qwen3.5-397B-A17B model deployment and evaluation on Ascend, including multi-node setup.
tests/e2e/multicard/4-cards/test_qwen3_5.py
- Added end-to-end test cases for Qwen3.5-27B and Qwen3.5-35B-A3B models, covering distributed execution, tensor parallelism, and specific compilation/speculative decoding configurations.

Ignored Files

Ignored by pattern: .github/workflows/** (1)
- .github/workflows/scripts/config.yaml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

Signed-off-by: pppeng <zepengliu912@qq.com>

gemini-code-assist

Code Review

This pull request adds documentation for the Qwen3.5-27B and Qwen3.5-397B-A17B models, along with new end-to-end tests. The documentation is a valuable addition, but I've found a few issues in the provided shell commands and some inconsistencies that could lead to user confusion or errors. I've left specific comments to address these points. The new tests seem correct and follow existing patterns.

gemini-code-assist · 2026-03-16T09:30:57Z

+--no-enable-prefix-caching \
+--speculative_config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' \
+--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
+--additional-config '{"enable_cpu_binding":true}'


The vllm serve command is syntactically incorrect. This line is missing a trailing \ to continue the command on the next line. This will cause the --async-scheduling argument on the next line to be executed as a separate command, which is not the intention.

Suggested change

--additional-config '{"enable_cpu_binding":true}'

--additional-config '{"enable_cpu_binding":true}' \

gemini-code-assist · 2026-03-16T09:30:57Z

+
+The parameters are explained as follows:
+
+- `--data-parallel-size` 1 and `--tensor-parallel-size` 16 are common settings for data parallelism (DP) and tensor parallelism (TP) sizes.


The explanation for --tensor-parallel-size states that 16 is a common setting, but the example script on line 109 uses a value of 2. This discrepancy can be confusing for users. Please either update the example to use 16 or adjust the explanation to reflect the example's value and provide context for when to use different values.

gemini-code-assist · 2026-03-16T09:30:57Z

+--no-enable-prefix-caching \
+--speculative_config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' \
+--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
+--additional-config '{"enable_cpu_binding":true, "multistream_overlap_shared_expert": true}'


The vllm serve command is syntactically incorrect. This line is missing a trailing \ to continue the command on the next line. This will cause the --async-scheduling argument on the next line to be executed as a separate command, which is not the intention.

Suggested change

--additional-config '{"enable_cpu_binding":true, "multistream_overlap_shared_expert": true}'

--additional-config '{"enable_cpu_binding":true, "multistream_overlap_shared_expert": true}' \

gemini-code-assist · 2026-03-16T09:30:57Z

+--data-parallel-rpc-port 13389 \
+--seed 1024 \
+--tensor-parallel-size 8 \
+--served-model-name qwen3 \


The served-model-name is qwen3 for Node 1, but it is qwen3.5 for Node 0 (line 183). This should be consistent across all nodes in a multi-node deployment. Please change it to qwen3.5.

Suggested change

--served-model-name qwen3 \

--served-model-name qwen3.5 \

Signed-off-by: pppeng <zepengliu912@qq.com>

Signed-off-by: pppeng <60355449+ppppeng@users.noreply.github.com>

github-actions · 2026-03-16T10:04:44Z

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

A PR should do only one thing, smaller PRs enable faster reviews.
Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

Signed-off-by: pppeng <60355449+ppppeng@users.noreply.github.com>

shaopeng-666 · 2026-03-17T10:59:25Z

/approve

MengqingCao · 2026-03-17T12:39:03Z

+    - (2) Decode requests are prioritized for scheduling, and prefill requests are scheduled only if there is available capacity.
+    - Generally, if `--max-num-batched-tokens` is set to a larger value, the overall latency will be lower, but the pressure on GPU memory (activation value usage) will be greater.
+- `--gpu-memory-utilization` represents the proportion of HBM that vLLM will use for actual inference. Its essential function is to calculate the available kv_cache size. During the warm-up phase (referred to as profile run in vLLM), vLLM records the peak GPU memory usage during an inference process with an input size of `--max-num-batched-tokens`. The available kv_cache size is then calculated as: `--gpu-memory-utilization` * HBM size - peak GPU memory usage. Therefore, the larger the value of `--gpu-memory-utilization`, the more kv_cache can be used. However, since the GPU memory usage during the warm-up phase may differ from that during actual inference (e.g., due to uneven EP load), setting `--gpu-memory-utilization` too high may lead to OOM (Out of Memory) issues during actual inference. The default value is `0.9`.
+- `--no-enable-prefix-caching` indicates that prefix caching is disabled. To enable it, remove this option.


Actually we also need to set --mamba-cache-mode align to enable prefix caching now, this comment will confuse users

BTW, I think we can add a script to illustrate how to run qwen3.5 with prefix caching

I have add comments to prefixcache

Signed-off-by: pppeng <zepengliu912@qq.com>

…scend into qwen3next_graph * 'qwen3next_graph' of https://github.com/845473182/vllm-ascend: (62 commits) [doc] Refresh the documentation for DeepSeek-V3.2 (vllm-project#7403) [bugfix][accuracy] Fix ds indexer accuracy problem caused by k rope (vllm-project#7341) [P/D] LayerwiseConnector supports the virtual push functionality on node D. (vllm-project#7361) [CI] Add PAT_TOKEN when checkout (vllm-project#7400) [main2main] upgrade vllm to 0308 (vllm-project#7213) [CI] add scheduled stale issue management (vllm-project#7354) [CI] expand issue labeler rules for feature/model triage (vllm-project#7356) [Bugfix] Assertion error when decode prefix cache fully hits (vllm-project#7236) [doc] Refresh the documentation for GLM-4.7 (vllm-project#7292) [BugFix]A2 MOE method&& layerwise MTP bugfix && Mamba gdn_metadata bugfix (vllm-project#7364) [doc] Upload doc for qwen3.5-27B and qwen3.5-397B-A17B on Ascend (vllm-project#7313) [bugfix]Enable dispatch_ffn_combine feature for qwen3.5 (vllm-project#7066) [bugfix] fix unzip file path for fia operator (vllm-project#7367) [Perf] Optimize bias handling in AscendRMSNorm (vllm-project#7226) [eagle3][pcp] fix bug for eagle3 and cp enable (vllm-project#7309) [Bugfix] fix TransposeKvCacheByBlock op error report in plog (vllm-project#7235) [Feature]Supports DSv3.1 PD separation and C8 quantization (vllm-project#7222) [main][bugfix] Fixed the problem that eagle3 will crash in FULL_DECODE_ONLY (vllm-project#7290) [xlite][Bugfix] Support mrope and deepstack features in xlite backend (vllm-project#7295) [model_runner_v2]optimize the performance of the _topk_log_softmax_kernel (vllm-project#7221) ...

…m-project#7313) ### What this PR does / why we need it? Upload doc for qwen3.5-27B and qwen3.5-397B-A17B on Ascend Base on vllm-ascend:v0.17.0rc1 - vLLM version: v0.17.0 - vLLM main: vllm-project/vllm@4034c3d --------- Signed-off-by: pppeng <zepengliu912@qq.com> Signed-off-by: pppeng <60355449+ppppeng@users.noreply.github.com>

ppppeng requested review from LCAIZJ, Yikun and wangxiyuan as code owners March 16, 2026 09:27

upload markdown for qwen3.5-27B and qwen3.5-397B-A17B

13d688c

Signed-off-by: pppeng <zepengliu912@qq.com>

gemini-code-assist bot reviewed Mar 16, 2026

View reviewed changes

ppppeng force-pushed the qwen3.5_doc branch from abcc9d8 to 13d688c Compare March 16, 2026 09:31

ppppeng and others added 4 commits March 16, 2026 17:34

update doc

d77e2d2

Signed-off-by: pppeng <zepengliu912@qq.com>

update index.md for qwen3.5

ebe55a8

Signed-off-by: pppeng <60355449+ppppeng@users.noreply.github.com>

fix lint

000681a

Signed-off-by: pppeng <60355449+ppppeng@users.noreply.github.com>

fix lint

4ef89fa

Signed-off-by: pppeng <60355449+ppppeng@users.noreply.github.com>

github-actions bot added the documentation Improvements or additions to documentation label Mar 16, 2026

fix lint

a4a27d0

Signed-off-by: pppeng <60355449+ppppeng@users.noreply.github.com>

zzzzwwjj approved these changes Mar 17, 2026

View reviewed changes

shaopeng-666 approved these changes Mar 17, 2026

View reviewed changes

MengqingCao reviewed Mar 17, 2026

View reviewed changes

update docs about prefixcache on qwen3.5

3851f47

Signed-off-by: pppeng <zepengliu912@qq.com>

MengqingCao approved these changes Mar 17, 2026

View reviewed changes

MengqingCao merged commit a457d0f into vllm-project:main Mar 17, 2026
22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[doc] Upload doc for qwen3.5-27B and qwen3.5-397B-A17B on Ascend#7313

[doc] Upload doc for qwen3.5-27B and qwen3.5-397B-A17B on Ascend#7313
MengqingCao merged 7 commits intovllm-project:mainfrom
ppppeng:qwen3.5_doc

ppppeng commented Mar 16, 2026 •

edited by github-actions bot

Loading

Uh oh!

gemini-code-assist bot commented Mar 16, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Mar 16, 2026

Uh oh!

gemini-code-assist bot Mar 16, 2026

Uh oh!

gemini-code-assist bot Mar 16, 2026

Uh oh!

gemini-code-assist bot Mar 16, 2026

Uh oh!

github-actions bot commented Mar 16, 2026

Uh oh!

shaopeng-666 commented Mar 17, 2026

Uh oh!

MengqingCao Mar 17, 2026

Uh oh!

MengqingCao Mar 17, 2026

Uh oh!

ppppeng Mar 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	--additional-config '{"enable_cpu_binding":true}'
	--additional-config '{"enable_cpu_binding":true}' \


		The parameters are explained as follows:

		- `--data-parallel-size` 1 and `--tensor-parallel-size` 16 are common settings for data parallelism (DP) and tensor parallelism (TP) sizes.

Conversation

ppppeng commented Mar 16, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

gemini-code-assist bot commented Mar 16, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 16, 2026

Uh oh!

shaopeng-666 commented Mar 17, 2026

Uh oh!

MengqingCao Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

MengqingCao Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

ppppeng Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ppppeng commented Mar 16, 2026 •

edited by github-actions bot

Loading