Skip to content

[CI] Add wait logic for each individual case#6036

Merged
wangxiyuan merged 6 commits intovllm-project:mainfrom
Potabk:0112_fix
Jan 20, 2026
Merged

[CI] Add wait logic for each individual case#6036
wangxiyuan merged 6 commits intovllm-project:mainfrom
Potabk:0112_fix

Conversation

@Potabk
Copy link
Copy Markdown
Collaborator

@Potabk Potabk commented Jan 20, 2026 •
edited by github-actions bot
Loading

What this PR does / why we need it?

Wait until the NPU memory is clean

Does this PR introduce any user-facing change?

How was this patch tested?

@Potabk
fix
ff1f957 
Signed-off-by: wangli <wangli858794774@gmail.com>
@Potabk Potabk requested a review from wangxiyuan as a code owner January 20, 2026 06:45
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 20, 2026

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

  • A PR should do only one thing, smaller PRs enable faster reviews.
  • Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
  • Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jan 20, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a utility decorator wait_until_npu_memory_free to ensure sufficient NPU memory is available before running tests, which helps prevent flaky tests. The changes are generally good, but I have identified a couple of high-severity issues in the implementation of the decorator. Firstly, the docstring for one of the parameters is incorrect, which could lead to misuse. Secondly, the waiting loop is a busy-wait loop that will cause high CPU usage. I've provided suggestions to fix these issues.

Comment thread tests/e2e/conftest.py Outdated
Comment thread tests/e2e/conftest.py Outdated
@Potabk Potabk added ready read for review ready-for-test start test by label for PR labels Jan 20, 2026
Potabk and others added 5 commits January 20, 2026 14:50
@Potabk
fix
5c28398 
Signed-off-by: wangli <wangli858794774@gmail.com>
@Potabk
also add wait for pcp
acc250a 
Signed-off-by: wangli <wangli858794774@gmail.com>
@Potabk
fix
6031f54 
Signed-off-by: wangli <wangli858794774@gmail.com>
@Potabk
fix log
2d67b7f 
Signed-off-by: wangli <wangli858794774@gmail.com>
@leo-pony
add successfully print
3c67a06 
Signed-off-by: leo-pony <nengjunma@outlook.com>
@wangxiyuan wangxiyuan merged commit 8cf1e8d into vllm-project:main Jan 20, 2026
20 checks passed
@Potabk Potabk deleted the 0112_fix branch January 21, 2026 01:17
845473182 pushed a commit to 845473182/vllm-ascend that referenced this pull request Jan 21, 2026
Merge branch 'main' of https://github.com/vllm-project/vllm-ascend in…
188edfa 
…to FIA_rebase

* 'main' of https://github.com/vllm-project/vllm-ascend: (24 commits)
  add dispath_ffn_combine_bf16 (vllm-project#5866)
  [BugFix] Fix input parameter bug of dispatch_gmm_combine_decode[RFC: issue 5476] (vllm-project#5932)
  [1/N][Feat] Xlite Qwen3 MoE Support (vllm-project#5951)
  [Bugfix] Fix setting of `speculative_config.enforce_eager` for dsv32 (vllm-project#5945)
  [bugfix][mm] change get_num_encoder_tokens to get_num_encoder_embeds in recompute_schedule.py (vllm-project#5132)
  [Bugfix] fix pcp qwen full graph FIA bug (vllm-project#6037)
  [Bugfix]Fixed precision issues caused by pooled request pooling (vllm-project#6049)
  【main】【bugfix】Resolved memory deallocation failure in the pooling layer under re-computation workloads. (vllm-project#6045)
  [main][Bugfix] Fixed an problem related to embeddings sharing (vllm-project#5967)
  [Feature]refactor the npugraph_ex config, support online-infer with static kernel (vllm-project#5775)
  [CI][Lint] Show lint diff on failure (vllm-project#5956)
  [CI] Add wait logic for each individual case (vllm-project#6036)
  [CI] Add DeepSeek-V3.2-W8A8 nightly ci test (vllm-project#4633)
  model runner v2 support triton of penalty (vllm-project#5854)
  [Docs][Model] Support Qwen3-VL-Embedding & Qwen3-VL-Reranker (vllm-project#6034)
  [Tests] move qwen3 performance test from nightly to e2e (vllm-project#5980)
  [Bugfix] fix bug of pcp+mtp+async scheduler (vllm-project#5994)
  [Main2Main] Upgrade vllm commit to releases/v0.14.0 (vllm-project#5988)
  [Ops] Add layernorm for qwen3Next (vllm-project#5765)
  [Doc] Add layer_sharding additional config for DeepSeek-V3.2-W8A8 (vllm-project#5921)
  ...
starmountain1997 pushed a commit to starmountain1997/vllm-ascend that referenced this pull request Jan 31, 2026
@Potabk @leo-pony @asinkLuno
[CI] Add wait logic for each individual case (vllm-project#6036)
b3be4f9 
### What this PR does / why we need it?
Wait until the NPU memory is clean
### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@2c24bc6

---------

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: leo-pony <nengjunma@outlook.com>
Co-authored-by: leo-pony <nengjunma@outlook.com>
ZRJ026 pushed a commit to ZRJ026/vllm-ascend that referenced this pull request Feb 28, 2026
@Potabk @leo-pony
[CI] Add wait logic for each individual case (vllm-project#6036)
a002574 
### What this PR does / why we need it?
Wait until the NPU memory is clean
### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@2c24bc6

---------

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: leo-pony <nengjunma@outlook.com>
Co-authored-by: leo-pony <nengjunma@outlook.com>
Signed-off-by: zrj026 <zhangrunjiang026@gmail.com>
maoxx241 pushed a commit to maoxx241/vllm-ascend that referenced this pull request Mar 2, 2026
@Potabk @leo-pony @maoxx241
[CI] Add wait logic for each individual case (vllm-project#6036)
27fb72f 
### What this PR does / why we need it?
Wait until the NPU memory is clean
### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@2c24bc6

---------

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: leo-pony <nengjunma@outlook.com>
Co-authored-by: leo-pony <nengjunma@outlook.com>
ZRJ026 pushed a commit to ZRJ026/vllm-ascend that referenced this pull request Mar 4, 2026
@Potabk @leo-pony
[CI] Add wait logic for each individual case (vllm-project#6036)
b06867a 
### What this PR does / why we need it?
Wait until the NPU memory is clean
### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@2c24bc6

---------

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: leo-pony <nengjunma@outlook.com>
Co-authored-by: leo-pony <nengjunma@outlook.com>
Signed-off-by: zrj026 <zhangrunjiang026@gmail.com>
LCAIZJ pushed a commit to LCAIZJ/vllm-ascend that referenced this pull request Mar 7, 2026
@Potabk @leo-pony @LCAIZJ
[CI] Add wait logic for each individual case (vllm-project#6036)
a7c66e5 
### What this PR does / why we need it?
Wait until the NPU memory is clean
### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@2c24bc6

---------

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: leo-pony <nengjunma@outlook.com>
Co-authored-by: leo-pony <nengjunma@outlook.com>
wangyibo1005 added a commit to wangyibo1005/vllm-ascend that referenced this pull request Apr 1, 2026
@wangyibo1005
[BugFix]Fix compilation errors for operators dispatch_gmm_combine_dec…
5c34070 
…ode/moe_combine_normal/moe_dispatch_normal

Signed-off-by: Wangyibo1005 <2633333316@qq.com>

[Misc] Refactor aclgraph accuracy test to use logprob-based comparison (vllm-project#7455)

Replace text-match assertions with a two-tier logprob accuracy check:

- Prefill (token 0): assert token ID is identical between eager baseline
and compiled mode, then verify logprob matches within `atol`.
- Decode (tokens 1-2): if chosen tokens match, compare logprobs
directly; if they differ, cross-lookup the baseline token in the
compiled model's top-20 distribution and assert the assigned logprob is
within `decode_atol` (defaults to 2x atol). This tolerates minor argmax
drift caused by floating-point differences while still catching
distribution divergence.

- vLLM version: v0.17.0
- vLLM main:
vllm-project/vllm@8a68046

---------

Signed-off-by: wangli <wangli858794774@gmail.com>

fix

123

delete

[CI] Add wait logic for each individual case (vllm-project#6036)

Wait until the NPU memory is clean

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@2c24bc6

---------

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: leo-pony <nengjunma@outlook.com>
Co-authored-by: leo-pony <nengjunma@outlook.com>

123123
wangyibo1005 added a commit to wangyibo1005/vllm-ascend that referenced this pull request Apr 1, 2026
@wangyibo1005
[BugFix]Fix compilation errors for operators dispatch_gmm_combine_dec…
1bdbccf 
…ode/moe_combine_normal/moe_dispatch_normal

Signed-off-by: Wangyibo1005 <2633333316@qq.com>

[Misc] Refactor aclgraph accuracy test to use logprob-based comparison (vllm-project#7455)

Replace text-match assertions with a two-tier logprob accuracy check:

- Prefill (token 0): assert token ID is identical between eager baseline
and compiled mode, then verify logprob matches within `atol`.
- Decode (tokens 1-2): if chosen tokens match, compare logprobs
directly; if they differ, cross-lookup the baseline token in the
compiled model's top-20 distribution and assert the assigned logprob is
within `decode_atol` (defaults to 2x atol). This tolerates minor argmax
drift caused by floating-point differences while still catching
distribution divergence.

- vLLM version: v0.17.0
- vLLM main:
vllm-project/vllm@8a68046

---------

Signed-off-by: wangli <wangli858794774@gmail.com>

fix

123

delete

[CI] Add wait logic for each individual case (vllm-project#6036)

Wait until the NPU memory is clean

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@2c24bc6

---------

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: leo-pony <nengjunma@outlook.com>
Co-authored-by: leo-pony <nengjunma@outlook.com>

123123

123
wangyibo1005 added a commit to wangyibo1005/vllm-ascend that referenced this pull request Apr 1, 2026
@wangyibo1005
[BugFix]Fix compilation errors for operators dispatch_gmm_combine_dec…
41ff207 
…ode/moe_combine_normal/moe_dispatch_normal

Signed-off-by: Wangyibo1005 <2633333316@qq.com>

[Misc] Refactor aclgraph accuracy test to use logprob-based comparison (vllm-project#7455)

Replace text-match assertions with a two-tier logprob accuracy check:

- Prefill (token 0): assert token ID is identical between eager baseline
and compiled mode, then verify logprob matches within `atol`.
- Decode (tokens 1-2): if chosen tokens match, compare logprobs
directly; if they differ, cross-lookup the baseline token in the
compiled model's top-20 distribution and assert the assigned logprob is
within `decode_atol` (defaults to 2x atol). This tolerates minor argmax
drift caused by floating-point differences while still catching
distribution divergence.

- vLLM version: v0.17.0
- vLLM main:
vllm-project/vllm@8a68046

---------

Signed-off-by: wangli <wangli858794774@gmail.com>

fix

123

delete

[CI] Add wait logic for each individual case (vllm-project#6036)

Wait until the NPU memory is clean

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@2c24bc6

---------

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: leo-pony <nengjunma@outlook.com>
Co-authored-by: leo-pony <nengjunma@outlook.com>

123123

123

pre-commit
wangyibo1005 added a commit to wangyibo1005/vllm-ascend that referenced this pull request Apr 7, 2026
@wangyibo1005
[BugFix]Fix compilation errors for operators dispatch_gmm_combine_dec…
9ef853d 
…ode/moe_combine_normal/moe_dispatch_normal

Signed-off-by: Wangyibo1005 <2633333316@qq.com>

[Misc] Refactor aclgraph accuracy test to use logprob-based comparison (vllm-project#7455)

Replace text-match assertions with a two-tier logprob accuracy check:

- Prefill (token 0): assert token ID is identical between eager baseline
and compiled mode, then verify logprob matches within `atol`.
- Decode (tokens 1-2): if chosen tokens match, compare logprobs
directly; if they differ, cross-lookup the baseline token in the
compiled model's top-20 distribution and assert the assigned logprob is
within `decode_atol` (defaults to 2x atol). This tolerates minor argmax
drift caused by floating-point differences while still catching
distribution divergence.

- vLLM version: v0.17.0
- vLLM main:
vllm-project/vllm@8a68046

---------

Signed-off-by: wangli <wangli858794774@gmail.com>

fix

123

delete

[CI] Add wait logic for each individual case (vllm-project#6036)

Wait until the NPU memory is clean

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@2c24bc6

---------

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: leo-pony <nengjunma@outlook.com>
Co-authored-by: leo-pony <nengjunma@outlook.com>

123123

123

pre-commit

pick use
wangyibo1005 added a commit to wangyibo1005/vllm-ascend that referenced this pull request Apr 8, 2026
@wangyibo1005
[BugFix]Fix compilation errors for operators dispatch_gmm_combine_dec…
ed63eb3 
…ode/moe_combine_normal/moe_dispatch_normal

Signed-off-by: Wangyibo1005 <2633333316@qq.com>

[Misc] Refactor aclgraph accuracy test to use logprob-based comparison (vllm-project#7455)

Replace text-match assertions with a two-tier logprob accuracy check:

- Prefill (token 0): assert token ID is identical between eager baseline
and compiled mode, then verify logprob matches within `atol`.
- Decode (tokens 1-2): if chosen tokens match, compare logprobs
directly; if they differ, cross-lookup the baseline token in the
compiled model's top-20 distribution and assert the assigned logprob is
within `decode_atol` (defaults to 2x atol). This tolerates minor argmax
drift caused by floating-point differences while still catching
distribution divergence.

- vLLM version: v0.17.0
- vLLM main:
vllm-project/vllm@8a68046

---------

Signed-off-by: wangli <wangli858794774@gmail.com>

fix

123

delete

[CI] Add wait logic for each individual case (vllm-project#6036)

Wait until the NPU memory is clean

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@2c24bc6

---------

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: leo-pony <nengjunma@outlook.com>
Co-authored-by: leo-pony <nengjunma@outlook.com>

123123

123

pre-commit

pick use

123
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wangxiyuan wangxiyuan Awaiting requested review from wangxiyuan wangxiyuan is a code owner

1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

module:tests ready read for review ready-for-test start test by label for PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Potabk @wangxiyuan @leo-pony