[TEST]Add initial prefix cache case for nightly test#3709
[TEST]Add initial prefix cache case for nightly test#3709wangxiyuan merged 1 commit intovllm-project:mainfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new end-to-end test for prefix caching and modifies the aisbench tool to support it. The changes are generally good, but I have a couple of suggestions to improve the test's efficiency and the robustness of the result parsing logic.
In the new test file, the vLLM server is started twice to run two separate test cases. This is inefficient and can be done within a single server instance, which will significantly speed up the test.
In tools/aisbench.py, the new get_TTFT function uses string slicing to extract a numeric value, which is brittle. I've suggested a more robust implementation using regular expressions.
| with RemoteOpenAIServer(model, | ||
| server_args, | ||
| server_port=port, | ||
| env_dict=env_dict, | ||
| auto_port=False): | ||
| run_aisbench_cases(model, port, aisbench_warm_up) | ||
| result = run_aisbench_cases(model, port, aisbench_cases0) | ||
| TTFT0 = get_TTFT(result) | ||
| with RemoteOpenAIServer(model, | ||
| server_args, | ||
| server_port=port, | ||
| env_dict=env_dict, | ||
| auto_port=False): | ||
| run_aisbench_cases(model, port, aisbench_warm_up) | ||
| result = run_aisbench_cases(model, port, aisbench_cases75) | ||
| TTFT75 = get_TTFT(result) |
There was a problem hiding this comment.
Starting the RemoteOpenAIServer twice is inefficient and significantly slows down the test. The server can be started once to run both test cases, and the warm-up also only needs to be run once. This will make the test execute much faster.
with RemoteOpenAIServer(model,
server_args,
server_port=port,
env_dict=env_dict,
auto_port=False):
run_aisbench_cases(model, port, aisbench_warm_up)
result = run_aisbench_cases(model, port, aisbench_cases0)
TTFT0 = get_TTFT(result)
result = run_aisbench_cases(model, port, aisbench_cases75)
TTFT75 = get_TTFT(result)| def get_TTFT(result): | ||
| TTFT = result[0][0].loc["TTFT", "Average"][:-3] | ||
| return float(TTFT) |
There was a problem hiding this comment.
Slicing with [:-3] is brittle because it assumes the units will always be 3 characters long (e.g., ' ms'). If the units change (e.g., to 's') or are formatted differently, this will break or produce incorrect results. It's more robust to parse the numeric value from the string using a regular expression.
| def get_TTFT(result): | |
| TTFT = result[0][0].loc["TTFT", "Average"][:-3] | |
| return float(TTFT) | |
| def get_TTFT(result): | |
| ttft_str = result[0][0].loc["TTFT", "Average"] | |
| match = re.match(r"^\d+(\.\d*)?", ttft_str) | |
| if not match: | |
| raise ValueError(f"Could not parse TTFT value from '{ttft_str}'") | |
| return float(match.group(0)) |
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
dc128ee to
12613d7
Compare
Signed-off-by: jiangyunfan1 <jiangyunfan1@h-partners.com>
12613d7 to
28a9e32
Compare
### What this PR does / why we need it? This PR adds the initial prefix cache case for nightly test for Qwen3-32b-int8 on A3, we need test them daily. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By running the test - vLLM version: v0.11.0rc3 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.0 Signed-off-by: jiangyunfan1 <jiangyunfan1@h-partners.com> Signed-off-by: luolun <luolun1995@cmbchina.com>
### What this PR does / why we need it? This PR adds the initial prefix cache case for nightly test for Qwen3-32b-int8 on A3, we need test them daily. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By running the test - vLLM version: v0.11.0rc3 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.0 Signed-off-by: jiangyunfan1 <jiangyunfan1@h-partners.com> Signed-off-by: hwhaokun <haokun0405@163.com>
### What this PR does / why we need it? This PR adds the initial prefix cache case for nightly test for Qwen3-32b-int8 on A3, we need test them daily. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By running the test - vLLM version: v0.11.0rc3 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.0 Signed-off-by: jiangyunfan1 <jiangyunfan1@h-partners.com> Signed-off-by: nsdie <yeyifan@huawei.com>
### What this PR does / why we need it? This PR adds the initial prefix cache case for nightly test for Qwen3-32b-int8 on A3, we need test them daily. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By running the test - vLLM version: v0.11.0rc3 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.0 Signed-off-by: jiangyunfan1 <jiangyunfan1@h-partners.com>
… to `.yaml` (#6503) ### What this PR does / why we need it? This PR refactors the nightly single-node model test by migrating test configurations from Python scripts to a more maintainable `YAML-based` format. | Original PR | Python (`.py`) | YAML (`.yaml`) | | :--- | :--- | :--- | | [#3568](#3568) | `test_deepseek_r1_0528_w8a8_eplb.py` | `DeepSeek-R1-0528-W8A8.yaml` | | [#3631](#3631) | `test_deepseek_r1_0528_w8a8.py` | `DeepSeek-R1-0528-W8A8.yaml` | | [#5874](#5874) | `test_deepseek_r1_w8a8_hbm.py` | `DeepSeek-R1-W8A8-HBM.yaml` | | [#3908](#3908) | `test_deepseek_v3_2_w8a8.py` | `DeepSeek-V3.2-W8A8.yaml` | | [#5682](#5682) | `test_kimi_k2_thinking.py` | `Kimi-K2-Thinking.yaml` | | [#4111](#4111) | `test_mtpx_deepseek_r1_0528_w8a8.py` | `MTPX-DeepSeek-R1-0528-W8A8.yaml` | | [#3733](#3733) | `test_prefix_cache_deepseek_r1_0528_w8a8.py` | `Prefix-Cache-DeepSeek-R1-0528-W8A8.yaml` | | [#6543](#6543) | `test_qwen3_235b_w8a8.py` | `Qwen3-235B-A22B-W8A8.yaml` | | [#6543](#6543) | `test_qwen3_235b_a22b_w8a8_eplb.py` | `Qwen3-235B-A22B-W8A8.yaml` | | [#3973](#3973) | `test_qwen3_30b_w8a8.py` | `Qwen3-30B-A3B-W8A8.yaml` | | [#3541](#3541) | `test_qwen3_32b_int8.py` | `Qwen3-32B-Int8.yaml` | | [#3757](#3757) | `test_qwq_32b.py` | `QwQ-32B.yaml` | | [#5616](#5616) | `test_qwen3_next_w8a8.py` | `Qwen3-Next-80B-A3B-Instruct-W8A8.yaml` | | [#3541](#3541) | `test_qwen2_5_vl_7b.py` | `Qwen2.5-VL-7B-Instruct.yaml` | | [#5301](#5301) | `test_qwen2_5_vl_7b_epd.py` | `Qwen2.5-VL-7B-Instruct-EPD.yaml` | | [#3707](#3707) | `test_qwen2_5_vl_32b.py` | `Qwen2.5-VL-32B-Instruct.yaml` | | [#3676](#3676) | `test_qwen3_32b_int8_a3_feature_stack3.py` | `Qwen3-32B-Int8-A3-Feature-Stack3.yaml` | | [#3709](#3709) | `test_prefix_cache_qwen3_32b_int8.py` | `Prefix-Cache-Qwen3-32B-Int8.yaml` | | [#5395](#5395) | `test_qwen3_next.py` | `Qwen3-Next-80B-A3B-Instruct-A2.yaml` | | [#3474](#3474) | `test_qwen3_32b.py` | `Qwen3-32B.yaml` | | [#3541](#3541) | `test_qwen3_32b_int8.py` | `Qwen3-32B-Int8-A2.yaml` | ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - vLLM version: v0.15.0 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.15.0 --------- Signed-off-by: MrZ20 <2609716663@qq.com>
… to `.yaml` (vllm-project#6503) ### What this PR does / why we need it? This PR refactors the nightly single-node model test by migrating test configurations from Python scripts to a more maintainable `YAML-based` format. | Original PR | Python (`.py`) | YAML (`.yaml`) | | :--- | :--- | :--- | | [vllm-project#3568](vllm-project#3568) | `test_deepseek_r1_0528_w8a8_eplb.py` | `DeepSeek-R1-0528-W8A8.yaml` | | [vllm-project#3631](vllm-project#3631) | `test_deepseek_r1_0528_w8a8.py` | `DeepSeek-R1-0528-W8A8.yaml` | | [vllm-project#5874](vllm-project#5874) | `test_deepseek_r1_w8a8_hbm.py` | `DeepSeek-R1-W8A8-HBM.yaml` | | [vllm-project#3908](vllm-project#3908) | `test_deepseek_v3_2_w8a8.py` | `DeepSeek-V3.2-W8A8.yaml` | | [vllm-project#5682](vllm-project#5682) | `test_kimi_k2_thinking.py` | `Kimi-K2-Thinking.yaml` | | [vllm-project#4111](vllm-project#4111) | `test_mtpx_deepseek_r1_0528_w8a8.py` | `MTPX-DeepSeek-R1-0528-W8A8.yaml` | | [vllm-project#3733](vllm-project#3733) | `test_prefix_cache_deepseek_r1_0528_w8a8.py` | `Prefix-Cache-DeepSeek-R1-0528-W8A8.yaml` | | [vllm-project#6543](vllm-project#6543) | `test_qwen3_235b_w8a8.py` | `Qwen3-235B-A22B-W8A8.yaml` | | [vllm-project#6543](vllm-project#6543) | `test_qwen3_235b_a22b_w8a8_eplb.py` | `Qwen3-235B-A22B-W8A8.yaml` | | [vllm-project#3973](vllm-project#3973) | `test_qwen3_30b_w8a8.py` | `Qwen3-30B-A3B-W8A8.yaml` | | [vllm-project#3541](vllm-project#3541) | `test_qwen3_32b_int8.py` | `Qwen3-32B-Int8.yaml` | | [vllm-project#3757](vllm-project#3757) | `test_qwq_32b.py` | `QwQ-32B.yaml` | | [vllm-project#5616](vllm-project#5616) | `test_qwen3_next_w8a8.py` | `Qwen3-Next-80B-A3B-Instruct-W8A8.yaml` | | [vllm-project#3541](vllm-project#3541) | `test_qwen2_5_vl_7b.py` | `Qwen2.5-VL-7B-Instruct.yaml` | | [vllm-project#5301](vllm-project#5301) | `test_qwen2_5_vl_7b_epd.py` | `Qwen2.5-VL-7B-Instruct-EPD.yaml` | | [vllm-project#3707](vllm-project#3707) | `test_qwen2_5_vl_32b.py` | `Qwen2.5-VL-32B-Instruct.yaml` | | [vllm-project#3676](vllm-project#3676) | `test_qwen3_32b_int8_a3_feature_stack3.py` | `Qwen3-32B-Int8-A3-Feature-Stack3.yaml` | | [vllm-project#3709](vllm-project#3709) | `test_prefix_cache_qwen3_32b_int8.py` | `Prefix-Cache-Qwen3-32B-Int8.yaml` | | [vllm-project#5395](vllm-project#5395) | `test_qwen3_next.py` | `Qwen3-Next-80B-A3B-Instruct-A2.yaml` | | [vllm-project#3474](vllm-project#3474) | `test_qwen3_32b.py` | `Qwen3-32B.yaml` | | [vllm-project#3541](vllm-project#3541) | `test_qwen3_32b_int8.py` | `Qwen3-32B-Int8-A2.yaml` | ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - vLLM version: v0.15.0 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.15.0 --------- Signed-off-by: MrZ20 <2609716663@qq.com>
What this PR does / why we need it?
This PR adds the initial prefix cache case for nightly test for Qwen3-32b-int8 on A3, we need test them daily.
Does this PR introduce any user-facing change?
No
How was this patch tested?
By running the test