Skip to content

[bugfix] fix test_camem failed with triton-ascend#5492

Merged
wangxiyuan merged 2 commits intovllm-project:mainfrom
Meihan-chen:triton_ci
Jan 5, 2026
Merged

[bugfix] fix test_camem failed with triton-ascend#5492
wangxiyuan merged 2 commits intovllm-project:mainfrom
Meihan-chen:triton_ci

Conversation

@Meihan-chen
Copy link
Copy Markdown
Collaborator

@Meihan-chen Meihan-chen commented Dec 29, 2025 •
edited
Loading

What this PR does / why we need it?

This fixes a bug that occurred when running test_camem.py in the triton-ascend environment NPU function error: aclrtGetMemInfo(ACL_HBM_MEM, &device_free, &device_total)

Does this PR introduce any user-facing change?

How was this patch tested?

@wxsIcey wxsIcey added ready read for review ready-for-test start test by label for PR labels Dec 29, 2025
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Dec 29, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the codebase to address an e2e test failure involving Triton. The core change involves centralizing the import of torch_npu._inductor from multiple Triton kernel files into a single location within the NPUWorker's initialization. This side-effect-only import is now performed once when a worker starts, ensuring proper and timely initialization before any Triton operations are executed. This change improves code structure and resolves potential issues arising from multiple or improperly timed imports. The implementation appears correct and aligns with the project's established coding practices.

@Meihan-chen Meihan-chen force-pushed the triton_ci branch 2 times, most recently from d1bffdc to 41cd56e Compare December 29, 2025 14:32
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Dec 29, 2025

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

  • A PR should do only one thing, smaller PRs enable faster reviews.
  • Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
  • Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

Yikun
Yikun reviewed Dec 30, 2025
View reviewed changes
Comment thread vllm_ascend/worker/worker.py
adapt_patch()
from vllm.triton_utils import HAS_TRITON
if HAS_TRITON:
import torch_npu._inductor # noqa: F401
Copy link
Copy Markdown
Member

@Yikun Yikun Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would mind adding a note to show why need import this?

Copy link
Copy Markdown
Collaborator Author

@Meihan-chen Meihan-chen Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! I've added a comment explaining.

Comment thread vllm_ascend/ops/triton/fla/fused_qkvzba_split_reshape.py

if HAS_TRITON:
import torch_npu._inductor # noqa: F401
from vllm.triton_utils import tl, triton
Copy link
Copy Markdown
Member

@Yikun Yikun Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems we need to add a check Forbid import torch_npu._inductor in vllm_ascend/ops/triton/
https://github.com/vllm-project/vllm/blob/main/.pre-commit-config.yaml#L132

Copy link
Copy Markdown
Collaborator Author

@Meihan-chen Meihan-chen Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This import of torch_npu._inductor fixes graph mode running errors with triton-ascend. Once added, the issue no longer occurs, so future Triton ops won't need similar imports in ops/triton. Therefore, we don't need a dedicated pre-commit specifically for this.

@Yikun
Copy link
Copy Markdown
Member

Yikun commented Dec 30, 2025 •
edited
Loading

Please fullfill commit msg and change commit title to a meaningful title.

@Meihan-chen Meihan-chen changed the title [bugfix] fix e2e test failed with triton [bugfix] fix test_camem.py aclrtGetMemInfo failed with triton-ascend Dec 30, 2025
@Meihan-chen Meihan-chen changed the title [bugfix] fix test_camem.py aclrtGetMemInfo failed with triton-ascend [bugfix] fix aclrtGetMemInfo faild when running test_camem with triton-ascend Dec 30, 2025
@Meihan-chen Meihan-chen changed the title [bugfix] fix aclrtGetMemInfo faild when running test_camem with triton-ascend [bugfix] fix aclrtGetMemInfo faild issue when running test_camem with triton-ascend Dec 30, 2025
@Meihan-chen Meihan-chen changed the title [bugfix] fix aclrtGetMemInfo faild issue when running test_camem with triton-ascend [bugfix] fix test_camem failed with triton-ascend Dec 30, 2025
@Meihan-chen
Copy link
Copy Markdown
Collaborator Author

Meihan-chen commented Dec 30, 2025

Please fullfill commit msg and change commit title to a meaningful title.

Thanks, I have modified it

@Meihan-chen
[bugfix] fix test file failed with triton
3554796 
Signed-off-by: Meihan-chen <jcccx.cmh@gmail.com>
@Meihan-chen
add note
c111b5d 
Signed-off-by: Meihan-chen <jcccx.cmh@gmail.com>
@wangxiyuan wangxiyuan merged commit 16b1bee into vllm-project:main Jan 5, 2026
19 checks passed
zyz111222 pushed a commit to zyz111222/vllm-ascend that referenced this pull request Jan 5, 2026
@Meihan-chen @zyz111222
[bugfix] fix test_camem failed with triton-ascend (vllm-project#5492)
5e98abf 
### What this PR does / why we need it?
This fixes a bug that occurred when running `test_camem.py` in the
triton-ascend environment `NPU function error:
aclrtGetMemInfo(ACL_HBM_MEM, &device_free, &device_total)`

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@5326c89

---------

Signed-off-by: Meihan-chen <jcccx.cmh@gmail.com>
845473182 pushed a commit to 845473182/vllm-ascend that referenced this pull request Jan 6, 2026
Merge branch 'main' of https://github.com/vllm-project/vllm-ascend in…
3650848 
…to FIA_rebase

* 'main' of https://github.com/vllm-project/vllm-ascend: (58 commits)
  [Main2Main] Upgrade vllm commit to 0106 (vllm-project#5617)
  [CI]update bisheng version (vllm-project#5621)
  [UT][PCP&DCP] UT for block_table.py (vllm-project#5032)
  [Main2Main] Upgrade vllm commit to 0105 (vllm-project#5595)
  [CI] mv ops to correct path (vllm-project#5615)
  [BugFix] Fix Smoke Testing Bug for DSR1 longseq (vllm-project#5613)
  Revert "[Feat] enable hierarchical mc2 ops on A2 by default (vllm-project#5545)" (vllm-project#5611)
  [TRITON][TEST]Add nightly test for triton split_qkv_rmsnorm_rope (vllm-project#5267)
  [perf] Fix MLAPO weight disposal for KV-consumer MLA in PD-mix deploy... (vllm-project#5192)
  [docs] Correct image about prefill phase of PCP (vllm-project#5598)
  [CI] update triton-ascend version (vllm-project#5584)
  [P/D]Remove mooncake kvpool unused parameter `local_hostname` (vllm-project#5574)
  [Bugfix] record cos and sin cache in AscendRotaryEmbedding (vllm-project#5516)
  [bugfix] fix test_camem failed with triton-ascend (vllm-project#5492)
  [UT]add triton ops ut :  test_fused_qkvzba_split_reshape_cat (vllm-project#5474)
  [CI] Download models from ms (vllm-project#5405)
  Docs: Add A3 Docker image guidance for Atlas A3 machines (vllm-project#5256)
  [Doc] Add NNAL installation guide and requirements (vllm-project#5235)
  Add the requirement of arctic-inference which  speculative decoding with suffix_decode  (vllm-project#5045)
  [BugFix][Fusion] Fix graph fusion failure problem (vllm-project#5253)
  ...
Rozwel-dx pushed a commit to Rozwel-dx/vllm-ascend that referenced this pull request Jan 8, 2026
@Meihan-chen @Rozwel-dx
[bugfix] fix test_camem failed with triton-ascend (vllm-project#5492)
986b072 
### What this PR does / why we need it?
This fixes a bug that occurred when running `test_camem.py` in the
triton-ascend environment `NPU function error:
aclrtGetMemInfo(ACL_HBM_MEM, &device_free, &device_total)`

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@5326c89

---------

Signed-off-by: Meihan-chen <jcccx.cmh@gmail.com>
aipaes pushed a commit to aipaes/vllm-ascend that referenced this pull request Jan 15, 2026
@Meihan-chen
[bugfix] fix test_camem failed with triton-ascend (vllm-project#5492)
11cf571 
### What this PR does / why we need it?
This fixes a bug that occurred when running `test_camem.py` in the
triton-ascend environment `NPU function error:
aclrtGetMemInfo(ACL_HBM_MEM, &device_free, &device_total)`

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@5326c89

---------

Signed-off-by: Meihan-chen <jcccx.cmh@gmail.com>
ZRJ026 pushed a commit to ZRJ026/vllm-ascend that referenced this pull request Feb 28, 2026
@Meihan-chen
[bugfix] fix test_camem failed with triton-ascend (vllm-project#5492)
8eeabb3 
### What this PR does / why we need it?
This fixes a bug that occurred when running `test_camem.py` in the
triton-ascend environment `NPU function error:
aclrtGetMemInfo(ACL_HBM_MEM, &device_free, &device_total)`

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@5326c89

---------

Signed-off-by: Meihan-chen <jcccx.cmh@gmail.com>
Signed-off-by: zrj026 <zhangrunjiang026@gmail.com>
maoxx241 pushed a commit to maoxx241/vllm-ascend that referenced this pull request Mar 2, 2026
@Meihan-chen @maoxx241
[bugfix] fix test_camem failed with triton-ascend (vllm-project#5492)
cf1657d 
### What this PR does / why we need it?
This fixes a bug that occurred when running `test_camem.py` in the
triton-ascend environment `NPU function error:
aclrtGetMemInfo(ACL_HBM_MEM, &device_free, &device_total)`

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@5326c89

---------

Signed-off-by: Meihan-chen <jcccx.cmh@gmail.com>
ZRJ026 pushed a commit to ZRJ026/vllm-ascend that referenced this pull request Mar 4, 2026
@Meihan-chen
[bugfix] fix test_camem failed with triton-ascend (vllm-project#5492)
4a04370 
### What this PR does / why we need it?
This fixes a bug that occurred when running `test_camem.py` in the
triton-ascend environment `NPU function error:
aclrtGetMemInfo(ACL_HBM_MEM, &device_free, &device_total)`

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@5326c89

---------

Signed-off-by: Meihan-chen <jcccx.cmh@gmail.com>
Signed-off-by: zrj026 <zhangrunjiang026@gmail.com>
LCAIZJ pushed a commit to LCAIZJ/vllm-ascend that referenced this pull request Mar 7, 2026
@Meihan-chen @LCAIZJ
[bugfix] fix test_camem failed with triton-ascend (vllm-project#5492)
015e97d 
### What this PR does / why we need it?
This fixes a bug that occurred when running `test_camem.py` in the
triton-ascend environment `NPU function error:
aclrtGetMemInfo(ACL_HBM_MEM, &device_free, &device_total)`

- vLLM version: v0.13.0
- vLLM main:
vllm-project/vllm@5326c89

---------

Signed-off-by: Meihan-chen <jcccx.cmh@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Yikun Yikun Yikun left review comments

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build module:ops ready read for review ready-for-test start test by label for PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Meihan-chen @Yikun @wangxiyuan @wxsIcey