Repository navigation
[Bugfix] Synchronize NPU worker device - #17666
Merged
ZT-AIA merged 1 commit intoOct 15, 2026
Merged
Conversation
Contributor
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. Tip 💡 Consider Linking a Related Issue or RFCYour PR title contains the [BugFix] tag, indicating a bug fix or new feature. Linking a related issue or RFC in the PR description is strongly encouraged — it gives reviewers helpful context and speeds up the review. You can use any of these keywords:
🙏 Thanks for helping us keep the project well-organized! |
Contributor
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request ports the Ascend backend requirement from upstream vLLM #52914, ensuring that device-side operations are properly synchronized. By adding an explicit synchronization barrier to the NPU worker, the changes guarantee that device streams are idle before returning from pause or wake-up transitions, preventing race conditions when subsequent tasks like weight synchronization are initiated. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
Suggested PR Title:
[Worker][Feature] Implement synchronize_device on NPUWorker and add NPU state transition testsSuggested PR Summary:
### What this PR does / why we need it?
This PR implements `synchronize_device()` on `NPUWorker` to block until all in-flight NPU work has completed. This ensures compatibility with older vLLM releases (such as v0.28.0) that do not define `WorkerBase.synchronize_device` in upstream vLLM. It also adds a call to `synchronize_device()` at the end of `wake_up()` to guarantee that the device is idle before a caller starts same-device work (e.g., RLHF weight synchronization).
Additionally, this PR introduces helper utilities and tests to verify these state transitions:
- `npu_stream_busy.py`: Helper to queue real matmuls to simulate a busy NPU stream (substituting `torch.cuda._sleep`).
- `test_npu_device_synchronize.py`: Asserts that `synchronize_device()` and `wake_up()` wait for in-flight NPU work.
- `test_pause_device_idle.py`: Verifies end-to-end that a completed pause in data parallel mode leaves the NPU stream idle.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
This patch was tested with newly added unit tests in `tests/ut/worker/test_worker_v1.py` and new E2E state transition tests for both single-card and two-card RLHF setups.
SeaAsters
force-pushed
the
codex/npu-pause-synchronize-v2
branch
3 times, most recently
from
September 29, 2026 02:42
7062f21 to
b72c61d
Compare
SeaAsters
force-pushed
the
codex/npu-pause-synchronize-v2
branch
2 times, most recently
from
October 12, 2026 07:34
285afb3 to
013f68d
Compare
SeaAsters
marked this pull request as ready for review
October 12, 2026 07:38
Ronald1995
reviewed
Oct 12, 2026
Contributor
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
1 similar comment
Contributor
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
SeaAsters
force-pushed
the
codex/npu-pause-synchronize-v2
branch
from
October 12, 2026 09:10
013f68d to
9ce265d
Compare
wangxiyuan
pushed a commit
that referenced
this pull request
Oct 13, 2026
…and wakeup (#17766) ### What this PR does / why we need it? Ports the Ascend backend requirement from vLLM vllm-project/vllm#52914: a resolved pause promises an idle device, and backends outside `torch.accelerator` must provide their own wait. For main branch, there is also a PR #17666 working on this issue. Without this synchronize, in RL scenario, if we sleep exactly when dummy_batch executes, will cause the following issues: ``` [36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] Traceback (most recent call last): ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] File "/autotest/pytorch/torch_autotest/model/run/Qwen35_VL_397B_GRPO/acc_/vllm/vllm/v1/executor/multiproc_executor.py", line 982, in worker_busy_loop ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] output = func(*args, **kwargs) ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] ^^^^^^^^^^^^^^^^^^^^^ ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] File "/autotest/pytorch/torch_autotest/model/run/Qwen35_VL_397B_GRPO/acc_/vllm-ascend/vllm_ascend/worker/worker.py", line 228, in sleep ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] allocator.sleep(offload_tags=("weights",) if level == 1 else tuple()) ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] File "/autotest/pytorch/torch_autotest/model/run/Qwen35_VL_397B_GRPO/acc_/vllm-ascend/vllm_ascend/device_allocator/camem.py", line 220, in sleep ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] torch.npu.empty_cache() ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] File "/usr/local/lib/python3.12/site-packages/torch_npu/npu/memory.py", line 180, in empty_cache ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] torch_npu._C._npu_emptyCache() ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] RuntimeError: npuSynchronizeDevice:../torch_npu/csrc/core/npu/NPUStream.cpp:714 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015 ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tested on A3 platform, using following Qwen3.5-397B script: ```sh # master node: export VLLM_SERVER_DEV_MODE=1 vllm serve /data/LLM/mindSpeed_mm/mm_qwen3_5/grpo/ckpt_hf/Qwen3.5-397B-A17B --tensor-parallel-size 8 --data-parallel-size 4 --data-parallel-size-local 2 --data-parallel-start-rank 0 --data-parallel-address 80.5.21.123 --data-parallel-rpc-port 12345 --max-num-seqs 96 --max-model-len 71680 --max-num-batched-tokens 16384 --enable_expert_parallel --port 20002 --gpu-memory-utilization 0.9 --compilation_config '{"cudagraph_capture_sizes":[4,8,16,32,64,128,192,256,384],"cudagraph_mode": "FULL_DECODE_ONLY"}' --enable-sleep-mode --additional-config '{"weight_nz_mode": 0}' # slave node: export VLLM_SERVER_DEV_MODE=1 vllm serve /data/LLM/mindSpeed_mm/mm_qwen3_5/grpo/ckpt_hf/Qwen3.5-397B-A17B --tensor-parallel-size 8 --data-parallel-size 4 --data-parallel-size-local 2 --data-parallel-start-rank 2 --headless --data-parallel-address 80.5.21.123 --data-parallel-rpc-port 12345 --max-num-seqs 96 --max-model-len 71680 --max-num-batched-tokens 16384 --enable_expert_parallel --port 20002 --gpu-memory-utilization 0.9 --compilation_config '{"cudagraph_capture_sizes":[4,8,16,32,64,128,192,256,384],"cudagraph_mode": "FULL_DECODE_ONLY"}' --enable-sleep-mode --additional-config '{"weight_nz_mode": 0}' ``` and curl this sleep command: ```sh curl -X POST "http://127.0.0.1:20002/sleep?level=1" ``` #### Before this fix The testing case will reply these following error messages without this patch: ``` (ApiServer_1 pid=779773) INFO: Started server process [779773] (ApiServer_1 pid=779773) INFO: Waiting for application startup. (ApiServer_0 pid=779772) INFO: Application startup complete. (ApiServer_1 pid=779773) INFO: Application startup complete. (EngineCore_DP0 pid=779770) INFO 09-29 19:59:07 [core.py:761] Waiting for in-flight requests to complete before sleeping... (EngineCore_DP1 pid=779771) INFO 09-29 19:59:07 [core.py:761] Waiting for in-flight requests to complete before sleeping... (Worker_DP1_TP0_EP8 pid=780969) INFO 09-29 19:59:07 [acl_graph.py:243] Replaying aclgraph (Worker_DP0_TP0_EP0 pid=780976) INFO 09-29 19:59:07 [acl_graph.py:243] Replaying aclgraph (EngineCore_DP0 pid=779770) INFO 09-29 19:59:08 [block_pool.py:490] Successfully reset prefix cache (EngineCore_DP1 pid=779771) INFO 09-29 19:59:08 [block_pool.py:490] Successfully reset prefix cache (Worker_DP0_TP7_EP7 pid=780984) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP1_EP1 pid=780977) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP5_EP5 pid=780981) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP4_EP4 pid=780980) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP3_EP3 pid=780979) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP0_EP0 pid=780976) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP2_EP2 pid=780978) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP4_EP12 pid=780975) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP7_EP15 pid=780986) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP3_EP11 pid=780974) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP0_EP8 pid=780969) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP1_EP9 pid=780970) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP5_EP13 pid=780982) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP6_EP6 pid=780983) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP6_EP14 pid=780985) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP2_EP10 pid=780972) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] WorkerProc hit an exception. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] Traceback (most recent call last): (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] File "/autotest/pytorch/torch_autotest/model/run/Qwen35_VL_397B_GRPO/acc_/vllm/vllm/v1/executor/multiproc_executor.py", line 982, in worker_busy_loop (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] output = func(*args, **kwargs) (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] ^^^^^^^^^^^^^^^^^^^^^ (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] File "/autotest/pytorch/torch_autotest/model/run/Qwen35_VL_397B_GRPO/acc_/vllm-ascend/vllm_ascend/worker/worker.py", line 229, in sleep (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] allocator.sleep(offload_tags=("weights",) if level == 1 else tuple()) (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] File "/autotest/pytorch/torch_autotest/model/run/Qwen35_VL_397B_GRPO/acc_/vllm-ascend/vllm_ascend/device_allocator/camem.py", line 220, in sleep (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] torch.npu.empty_cache() (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] File "/usr/local/lib/python3.12/site-packages/torch_npu/npu/memory.py", line 178, in empty_cache (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] torch_npu._C._npu_emptyCache() (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] RuntimeError: npuSynchronizeDevice:../torch_npu/csrc/core/npu/NPUStream.cpp:714 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507011 (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] [ERROR] 2026-09-29-19:59:10 (PID:780977, Device:1, RankID:-1) ERR00100 PTA call acl api failed (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] [Error]: Model execution failed. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] Rectify the fault based on the error information in the ascend log. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] [PID: 780977] 2026-09-29-19:59:10.232.584 Invalid_Argument(EL0003): The argument is invalid. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] Solution: Try again with a valid argument. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] TraceBack (most recent call last): (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] An error occurs on the device(chipId:0, dieId:1), the serial number is 8, the error is aivec error, core id is 5, error code = 0x800000, dump info: pc start: 0x1242001f8074, current: 0x1242001f8294, vec error info: 0x601d9073df, mte error info: 0xe003000077, ifu error info: 0x22da8102081c0, ccu error info: 0xde118eb01d540ea3, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c103cf5000, aic cond: 0. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] The extend info: errcode:(0x800000, 0, 0) errorStr: MTE accesses an invalid GM address or the cross-device memory access times out. fixp_error0 info: 0x3000077, fixp_error1 info: 0xe0, fsmId:0, tslot:7, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] For details, see the troubleshooting document on the Ascend official website. Search for the keyword "AI Core Error".[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:1227] (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] The argument is invalid. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] rtMemGetInfoByType execution failed, reason=driver error:invalid handle[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:72] (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] rtDeviceSynchronizeWithTimeout execution failed, reason=the model stream execute failed[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:72] (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] ``` #### After this fix After adding this patch, server can sleep correctly: ```sh (ApiServer_1 pid=987951) INFO: Started server process [987951] (ApiServer_1 pid=987951) INFO: Waiting for application startup. (ApiServer_1 pid=987951) INFO: Application startup complete. (EngineCore_DP0 pid=987948) INFO 09-29 20:14:51 [core.py:761] Waiting for in-flight requests to complete before sleeping... (EngineCore_DP1 pid=987949) INFO 09-29 20:14:51 [core.py:761] Waiting for in-flight requests to complete before sleeping... (Worker_DP1_TP0_EP8 pid=989150) INFO 09-29 20:14:51 [acl_graph.py:243] Replaying aclgraph (Worker_DP0_TP0_EP0 pid=989149) INFO 09-29 20:14:51 [acl_graph.py:243] Replaying aclgraph (EngineCore_DP0 pid=987948) INFO 09-29 20:14:52 [block_pool.py:490] Successfully reset prefix cache (EngineCore_DP1 pid=987949) INFO 09-29 20:14:52 [block_pool.py:490] Successfully reset prefix cache (Worker_DP1_TP1_EP9 pid=989151) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP0_EP8 pid=989150) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP7_EP15 pid=989164) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP3_EP3 pid=989157) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP2_EP2 pid=989156) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP2_EP10 pid=989153) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP7_EP7 pid=989161) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP1_EP1 pid=989152) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP3_EP11 pid=989154) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP6_EP14 pid=989163) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP6_EP6 pid=989160) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP4_EP12 pid=989155) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP5_EP5 pid=989159) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP0_EP0 pid=989149) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP5_EP13 pid=989162) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP4_EP4 pid=989158) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP0_EP8 pid=989150) INFO 09-29 20:14:57 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.20 GiB memory is still in use. (Worker_DP0_TP5_EP5 pid=989159) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.96 GiB memory is still in use. (Worker_DP1_TP4_EP12 pid=989155) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.20 GiB memory is still in use. (Worker_DP1_TP1_EP9 pid=989151) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.97 GiB memory is still in use. (Worker_DP0_TP1_EP1 pid=989152) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.97 GiB memory is still in use. (Worker_DP0_TP3_EP3 pid=989157) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.97 GiB memory is still in use. (Worker_DP1_TP6_EP14 pid=989163) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.19 GiB memory is still in use. (Worker_DP0_TP6_EP6 pid=989160) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.20 GiB memory is still in use. (Worker_DP0_TP7_EP7 pid=989161) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.95 GiB memory is still in use. (Worker_DP1_TP5_EP13 pid=989162) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.95 GiB memory is still in use. (Worker_DP0_TP2_EP2 pid=989156) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.20 GiB memory is still in use. (Worker_DP0_TP4_EP4 pid=989158) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.19 GiB memory is still in use. (Worker_DP1_TP7_EP15 pid=989164) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.96 GiB memory is still in use. (Worker_DP0_TP0_EP0 pid=989149) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.20 GiB memory is still in use. (EngineCore_DP0 pid=987948) INFO 09-29 20:14:59 [abstract.py:327] It took 6.552437 seconds to fall asleep. (Worker_DP1_TP2_EP10 pid=989153) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.19 GiB memory is still in use. (Worker_DP1_TP3_EP11 pid=989154) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.96 GiB memory is still in use. (EngineCore_DP1 pid=987949) INFO 09-29 20:14:59 [abstract.py:327] It took 7.087198 seconds to fall asleep. (ApiServer_1 pid=987951) INFO: 127.0.0.1:47916 - "POST /sleep?level=1 HTTP/1.1" 200 OK ``` - vLLM main: vllm-project/vllm@ee0da84 Signed-off-by: YuhanBai <yuhan.bai0830@gmail.com>
lijiahang226
approved these changes
Oct 13, 2026
Contributor
Author
|
/rerun Rerun (failed jobs only):
|
SeaAsters
force-pushed
the
codex/npu-pause-synchronize-v2
branch
from
October 13, 2026 11:17
d9b5ecd to
4bd6b2e
Compare
vLLM #52914 makes a resolved pause promise an idle device and mirrors the same contract on the wake transition, which means backends outside `torch.accelerator` must supply their own wait. Ascend supplied none: `pause_generation()` could return while the last dummy batch a DP rank launched was still executing, and `wake_up()` left its tail work (the level-2 buffer restores and the sleep-mode cleanup) unwaited, so a caller starting a weight sync right after wake-up raced it. Add `NPUWorker.synchronize_device()`, which blocks on `torch.npu.synchronize()`, and end `wake_up()` with it. The method stays on the worker rather than relying on inheritance because `wake_up` calls it unconditionally, while vLLM releases predating #52914 define no `WorkerBase.synchronize_device` (verified: the method is absent in v0.28.0 and present from v0.29.0 onward). `sleep()` gets the same treatment, bracketing the memory transition with a barrier on each side: one before the baseline `mem_get_info()` snapshot that the freed-bytes assertion depends on, and one after the optional RL extra cleanup -- whose `destroy_hccl()`/aclgraph work is the last thing that can touch memory before the allocator unmaps it. Tests: - UT: every `wake_up` test now patches the barrier and asserts it is reached, `synchronize_device()` delegates to `torch.npu.synchronize()`, and the method is asserted to live on `NPUWorker` itself. The sleep tests assert both sleep barriers fire, on the cleanup-enabled and cleanup-disabled paths alike. - E2E, 1 card: the barrier drains in-flight NPU work, and `wake_up()` does not return while the stream still owns queued work. - E2E, 2 cards: the Ascend port of upstream's DP pause check -- inflate the dummy batches the pause consensus manufactures, then require an idle device the moment `pause_generation()` returns. `torch.cuda._sleep`, which upstream uses to keep a CUDA stream busy, does not exist on NPU, so the shared helper queues real matmuls without waiting and probes with `torch.npu.current_stream().query()` instead. The DP pause test uses `Qwen/Qwen2.5-0.5B-Instruct`, a small text-only checkpoint the a3 runners already pre-download. CI runs with `VLLM_USE_MODELSCOPE=1` and `HF_HUB_OFFLINE=1`, so an unlisted checkpoint cannot be fetched on the runners. Signed-off-by: SeaAsters <keine214@qq.com> Co-authored-by: OpenAI Codex <codex@openai.com>
SeaAsters
force-pushed
the
codex/npu-pause-synchronize-v2
branch
from
October 13, 2026 12:54
4bd6b2e to
a4743df
Compare
zzzzwwjj
pushed a commit
that referenced
this pull request
Oct 15, 2026
…18338) ### What this PR does / why we need it? Ports the Ascend backend requirement from vLLM vllm-project/vllm#52914: a resolved pause promises an idle device, and backends outside `torch.accelerator` must provide their own wait. For main branch, there is also a PR #17666 working on this issue. Without this synchronize, in RL scenario, if we sleep exactly when dummy_batch executes, will cause the following issues: ``` [36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] Traceback (most recent call last): ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] File "/autotest/pytorch/torch_autotest/model/run/Qwen35_VL_397B_GRPO/acc_/vllm/vllm/v1/executor/multiproc_executor.py", line 982, in worker_busy_loop ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] output = func(*args, **kwargs) ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] ^^^^^^^^^^^^^^^^^^^^^ ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] File "/autotest/pytorch/torch_autotest/model/run/Qwen35_VL_397B_GRPO/acc_/vllm-ascend/vllm_ascend/worker/worker.py", line 228, in sleep ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] allocator.sleep(offload_tags=("weights",) if level == 1 else tuple()) ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] File "/autotest/pytorch/torch_autotest/model/run/Qwen35_VL_397B_GRPO/acc_/vllm-ascend/vllm_ascend/device_allocator/camem.py", line 220, in sleep ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] torch.npu.empty_cache() ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] File "/usr/local/lib/python3.12/site-packages/torch_npu/npu/memory.py", line 180, in empty_cache ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] torch_npu._C._npu_emptyCache() ^[[36m(vLLMHttpServer pid=3455188, ip=80.5.21.123)^[[0m (Worker_DP6_TP11_EP107 pid=3455539) ERROR 09-29 15:05:09 [multiproc_executor.py:987] RuntimeError: npuSynchronizeDevice:../torch_npu/csrc/core/npu/NPUStream.cpp:714 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015 ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tested on A3 platform, using following Qwen3.5-397B script: ```sh # master node: export VLLM_SERVER_DEV_MODE=1 vllm serve /data/LLM/mindSpeed_mm/mm_qwen3_5/grpo/ckpt_hf/Qwen3.5-397B-A17B --tensor-parallel-size 8 --data-parallel-size 4 --data-parallel-size-local 2 --data-parallel-start-rank 0 --data-parallel-address 80.5.21.123 --data-parallel-rpc-port 12345 --max-num-seqs 96 --max-model-len 71680 --max-num-batched-tokens 16384 --enable_expert_parallel --port 20002 --gpu-memory-utilization 0.9 --compilation_config '{"cudagraph_capture_sizes":[4,8,16,32,64,128,192,256,384],"cudagraph_mode": "FULL_DECODE_ONLY"}' --enable-sleep-mode --additional-config '{"weight_nz_mode": 0}' # slave node: export VLLM_SERVER_DEV_MODE=1 vllm serve /data/LLM/mindSpeed_mm/mm_qwen3_5/grpo/ckpt_hf/Qwen3.5-397B-A17B --tensor-parallel-size 8 --data-parallel-size 4 --data-parallel-size-local 2 --data-parallel-start-rank 2 --headless --data-parallel-address 80.5.21.123 --data-parallel-rpc-port 12345 --max-num-seqs 96 --max-model-len 71680 --max-num-batched-tokens 16384 --enable_expert_parallel --port 20002 --gpu-memory-utilization 0.9 --compilation_config '{"cudagraph_capture_sizes":[4,8,16,32,64,128,192,256,384],"cudagraph_mode": "FULL_DECODE_ONLY"}' --enable-sleep-mode --additional-config '{"weight_nz_mode": 0}' ``` and curl this sleep command: ```sh curl -X POST "http://127.0.0.1:20002/sleep?level=1" ``` #### Before this fix The testing case will reply these following error messages without this patch: ``` (ApiServer_1 pid=779773) INFO: Started server process [779773] (ApiServer_1 pid=779773) INFO: Waiting for application startup. (ApiServer_0 pid=779772) INFO: Application startup complete. (ApiServer_1 pid=779773) INFO: Application startup complete. (EngineCore_DP0 pid=779770) INFO 09-29 19:59:07 [core.py:761] Waiting for in-flight requests to complete before sleeping... (EngineCore_DP1 pid=779771) INFO 09-29 19:59:07 [core.py:761] Waiting for in-flight requests to complete before sleeping... (Worker_DP1_TP0_EP8 pid=780969) INFO 09-29 19:59:07 [acl_graph.py:243] Replaying aclgraph (Worker_DP0_TP0_EP0 pid=780976) INFO 09-29 19:59:07 [acl_graph.py:243] Replaying aclgraph (EngineCore_DP0 pid=779770) INFO 09-29 19:59:08 [block_pool.py:490] Successfully reset prefix cache (EngineCore_DP1 pid=779771) INFO 09-29 19:59:08 [block_pool.py:490] Successfully reset prefix cache (Worker_DP0_TP7_EP7 pid=780984) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP1_EP1 pid=780977) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP5_EP5 pid=780981) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP4_EP4 pid=780980) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP3_EP3 pid=780979) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP0_EP0 pid=780976) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP2_EP2 pid=780978) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP4_EP12 pid=780975) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP7_EP15 pid=780986) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP3_EP11 pid=780974) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP0_EP8 pid=780969) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP1_EP9 pid=780970) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP5_EP13 pid=780982) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP6_EP6 pid=780983) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP6_EP14 pid=780985) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP2_EP10 pid=780972) INFO 09-29 19:59:08 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] WorkerProc hit an exception. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] Traceback (most recent call last): (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] File "/autotest/pytorch/torch_autotest/model/run/Qwen35_VL_397B_GRPO/acc_/vllm/vllm/v1/executor/multiproc_executor.py", line 982, in worker_busy_loop (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] output = func(*args, **kwargs) (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] ^^^^^^^^^^^^^^^^^^^^^ (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] File "/autotest/pytorch/torch_autotest/model/run/Qwen35_VL_397B_GRPO/acc_/vllm-ascend/vllm_ascend/worker/worker.py", line 229, in sleep (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] allocator.sleep(offload_tags=("weights",) if level == 1 else tuple()) (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] File "/autotest/pytorch/torch_autotest/model/run/Qwen35_VL_397B_GRPO/acc_/vllm-ascend/vllm_ascend/device_allocator/camem.py", line 220, in sleep (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] torch.npu.empty_cache() (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] File "/usr/local/lib/python3.12/site-packages/torch_npu/npu/memory.py", line 178, in empty_cache (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] torch_npu._C._npu_emptyCache() (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] RuntimeError: npuSynchronizeDevice:../torch_npu/csrc/core/npu/NPUStream.cpp:714 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507011 (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] [ERROR] 2026-09-29-19:59:10 (PID:780977, Device:1, RankID:-1) ERR00100 PTA call acl api failed (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] [Error]: Model execution failed. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] Rectify the fault based on the error information in the ascend log. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] [PID: 780977] 2026-09-29-19:59:10.232.584 Invalid_Argument(EL0003): The argument is invalid. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] Solution: Try again with a valid argument. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] TraceBack (most recent call last): (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] An error occurs on the device(chipId:0, dieId:1), the serial number is 8, the error is aivec error, core id is 5, error code = 0x800000, dump info: pc start: 0x1242001f8074, current: 0x1242001f8294, vec error info: 0x601d9073df, mte error info: 0xe003000077, ifu error info: 0x22da8102081c0, ccu error info: 0xde118eb01d540ea3, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c103cf5000, aic cond: 0. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] The extend info: errcode:(0x800000, 0, 0) errorStr: MTE accesses an invalid GM address or the cross-device memory access times out. fixp_error0 info: 0x3000077, fixp_error1 info: 0xe0, fsmId:0, tslot:7, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] For details, see the troubleshooting document on the Ascend official website. Search for the keyword "AI Core Error".[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:1227] (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] The argument is invalid. (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] rtMemGetInfoByType execution failed, reason=driver error:invalid handle[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:72] (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] rtDeviceSynchronizeWithTimeout execution failed, reason=the model stream execute failed[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:72] (Worker_DP0_TP1_EP1 pid=780977) ERROR 09-29 19:59:10 [multiproc_executor.py:987] ``` #### After this fix After adding this patch, server can sleep correctly: ```sh (ApiServer_1 pid=987951) INFO: Started server process [987951] (ApiServer_1 pid=987951) INFO: Waiting for application startup. (ApiServer_1 pid=987951) INFO: Application startup complete. (EngineCore_DP0 pid=987948) INFO 09-29 20:14:51 [core.py:761] Waiting for in-flight requests to complete before sleeping... (EngineCore_DP1 pid=987949) INFO 09-29 20:14:51 [core.py:761] Waiting for in-flight requests to complete before sleeping... (Worker_DP1_TP0_EP8 pid=989150) INFO 09-29 20:14:51 [acl_graph.py:243] Replaying aclgraph (Worker_DP0_TP0_EP0 pid=989149) INFO 09-29 20:14:51 [acl_graph.py:243] Replaying aclgraph (EngineCore_DP0 pid=987948) INFO 09-29 20:14:52 [block_pool.py:490] Successfully reset prefix cache (EngineCore_DP1 pid=987949) INFO 09-29 20:14:52 [block_pool.py:490] Successfully reset prefix cache (Worker_DP1_TP1_EP9 pid=989151) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP0_EP8 pid=989150) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP7_EP15 pid=989164) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP3_EP3 pid=989157) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP2_EP2 pid=989156) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP2_EP10 pid=989153) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP7_EP7 pid=989161) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP1_EP1 pid=989152) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP3_EP11 pid=989154) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP6_EP14 pid=989163) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP6_EP6 pid=989160) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP4_EP12 pid=989155) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP5_EP5 pid=989159) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP0_EP0 pid=989149) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP5_EP13 pid=989162) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP0_TP4_EP4 pid=989158) INFO 09-29 20:14:52 [camem.py:198] CaMem sleep: offloading 138/154 allocations (tags=('weights',)) (Worker_DP1_TP0_EP8 pid=989150) INFO 09-29 20:14:57 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.20 GiB memory is still in use. (Worker_DP0_TP5_EP5 pid=989159) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.96 GiB memory is still in use. (Worker_DP1_TP4_EP12 pid=989155) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.20 GiB memory is still in use. (Worker_DP1_TP1_EP9 pid=989151) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.97 GiB memory is still in use. (Worker_DP0_TP1_EP1 pid=989152) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.97 GiB memory is still in use. (Worker_DP0_TP3_EP3 pid=989157) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.97 GiB memory is still in use. (Worker_DP1_TP6_EP14 pid=989163) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.19 GiB memory is still in use. (Worker_DP0_TP6_EP6 pid=989160) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.20 GiB memory is still in use. (Worker_DP0_TP7_EP7 pid=989161) INFO 09-29 20:14:58 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.95 GiB memory is still in use. (Worker_DP1_TP5_EP13 pid=989162) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.95 GiB memory is still in use. (Worker_DP0_TP2_EP2 pid=989156) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.20 GiB memory is still in use. (Worker_DP0_TP4_EP4 pid=989158) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.19 GiB memory is still in use. (Worker_DP1_TP7_EP15 pid=989164) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.96 GiB memory is still in use. (Worker_DP0_TP0_EP0 pid=989149) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.20 GiB memory is still in use. (EngineCore_DP0 pid=987948) INFO 09-29 20:14:59 [abstract.py:327] It took 6.552437 seconds to fall asleep. (Worker_DP1_TP2_EP10 pid=989153) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 4.19 GiB memory is still in use. (Worker_DP1_TP3_EP11 pid=989154) INFO 09-29 20:14:59 [worker.py:235] Sleep mode (level=1) freed 51.30 GiB memory, 3.96 GiB memory is still in use. (EngineCore_DP1 pid=987949) INFO 09-29 20:14:59 [abstract.py:327] It took 7.087198 seconds to fall asleep. (ApiServer_1 pid=987951) INFO: 127.0.0.1:47916 - "POST /sleep?level=1 HTTP/1.1" 200 OK ``` - vLLM main: vllm-project/vllm@ced6857 --------- Signed-off-by: YuhanBai <yuhan.bai0830@gmail.com>
tangdafu
pushed a commit
to tangdafu/vllm-ascend
that referenced
this pull request
Oct 16, 2026
### What this PR does / why we need it? Ports the Ascend backend requirement from vLLM [#52914](vllm-project/vllm#52914): a resolved pause promises an idle device, and backends outside `torch.accelerator` must provide their own wait. Ascend provided none. `pause_generation()` could return while the last dummy batch a DP rank launched was still executing, and `wake_up()` left its tail work (the level-2 buffer restores and the sleep-mode cleanup) unwaited -- so a caller starting a weight sync right after wake-up raced it. This adds `NPUWorker.synchronize_device()`, which blocks on `torch.npu.synchronize()`, and ends `wake_up()` with it. The method stays on the worker rather than relying on inheritance because `wake_up` calls it unconditionally, while vLLM releases predating #52914 define no `WorkerBase.synchronize_device`. Verified against the vLLM tree (`vllm/v1/worker/worker_base.py`): the method is **absent in v0.28.0** and **present from v0.29.0 onward**. v0.28.0 is in the support matrix, so on that line the override is the only barrier that exists -- there is nothing to inherit, and `wake_up()` would otherwise raise `AttributeError`. `sleep()` gets the same treatment, bracketing the memory transition with a barrier on each side: one before the baseline `mem_get_info()` snapshot that the freed-bytes assertion depends on, and one after the optional RL extra cleanup -- whose `destroy_hccl()`/aclgraph work is the last thing that can touch memory before the allocator unmaps it. Without them, `sleep()` racing an in-flight dummy batch fails inside `CaMemAllocator.sleep()`'s `empty_cache()` (`AclrtSynchronizeDeviceWithTimeout`, `507011`/`507015`) -- the same failure the `releases/v0.30.0` fix in vllm-project#18338 addresses. ### Does this PR introduce _any_ user-facing change? No. Control-plane only; no effect on model output. ### How was this patch tested? Unit tests (`tests/ut/worker/test_worker_v1.py`): every `wake_up` test now patches the barrier and asserts it is reached, `synchronize_device()` delegates to `torch.npu.synchronize()`, and the method is asserted to live on `NPUWorker` itself. Unit tests (`tests/ut/worker/test_kv_cache_allocation_scope.py`) and `test_sleep_uses_rl_extra_cleanup`: the sleep tests now patch the barrier and assert both sleep barriers fire, covering the cleanup-enabled and cleanup-disabled paths. They also stop executing a real `torch.npu.synchronize()` on NPU hosts, which previously made these specs behave differently on hardware versus CPU runners. NPU system tests: - `tests/e2e/pull_request/one_card/rlhf/state_transitions/test_npu_device_synchronize.py` -- on real hardware, `synchronize_device()` and `wake_up()` must leave the NPU stream idle. - `tests/e2e/pull_request/two_card/rlhf/state_transitions/test_pause_device_idle.py` -- the Ascend port of upstream's `test_dp_pause_completion_implies_device_idle`. In DPLB the pause path itself launches the dummy batch whose completion the pause promise covers, so the test inflates the dummy batches the pause consensus manufactures and requires an idle device the moment `pause_generation()` returns. It skips (rather than passing vacuously) on vLLM without the #52914 barrier. - `tests/e2e/pull_request/one_card/rlhf/state_transitions/npu_stream_busy.py` -- the device-busy helper both files share. `torch.cuda._sleep(200_000_000)`, which the upstream test uses to keep a CUDA stream busy, cannot run on NPU: `torch.npu` exposes no `_sleep`, and `torch.cuda._sleep` raises `AttributeError: module 'torch._C' has no attribute '_cuda_sleep'`. `enqueue_npu_busy_work()` queues real matmuls without waiting instead, and `torch.npu.current_stream().query()` reports the same "is the stream quiet" fact as `torch.cuda.current_stream().query()`. Re-verified on Ascend 910B3 (2 chips) against the vLLM main lane (`0.30.0`, the pinned `.github/vllm-main-verified.commit`), i.e. after the sleep barriers were added -- 84 passed: | suite | result | |---|---| | `tests/ut/worker/test_worker_v1.py` + `tests/ut/worker/test_kv_cache_allocation_scope.py` | 81 passed | | `tests/e2e/pull_request/one_card/rlhf/state_transitions/test_npu_device_synchronize.py` | 2 passed | | `tests/e2e/pull_request/two_card/rlhf/state_transitions/test_pause_device_idle.py` | 1 passed (DP=2 engine, not skipped) | The DP pause test uses `Qwen/Qwen2.5-0.5B-Instruct`, a small text-only checkpoint the a3 runners already pre-download. CI runs with `VLLM_USE_MODELSCOPE=1` and `HF_HUB_OFFLINE=1`, so an unlisted checkpoint cannot be fetched on the runners. An earlier A/B check of the pause barrier itself, run on the vLLM 0.29.0 line before the sleep barriers were added, kept a DP=2 engine busy for ~200 ms and then called `pause_generation()`: | | fixed | barrier neutralised | |---|---|---| | `pause_generation()` wall time | 0.229 s | 0.036 s | | device idle when it returned | True | **False** | ### Upstream context Ports the Ascend backend requirement from vLLM [#52914](vllm-project/vllm#52914). - vLLM main: vllm-project/vllm@8cc9aa5 Signed-off-by: SeaAsters <keine214@qq.com> Co-authored-by: OpenAI Codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it?
Ports the Ascend backend requirement from vLLM
#52914: a resolved pause
promises an idle device, and backends outside
torch.acceleratormust providetheir own wait.
Ascend provided none.
pause_generation()could return while the last dummybatch a DP rank launched was still executing, and
wake_up()left its tail work(the level-2 buffer restores and the sleep-mode cleanup) unwaited -- so a caller
starting a weight sync right after wake-up raced it.
This adds
NPUWorker.synchronize_device(), which blocks ontorch.npu.synchronize(), and endswake_up()with it. The method stays on theworker rather than relying on inheritance because
wake_upcalls itunconditionally, while vLLM releases predating #52914 define
no
WorkerBase.synchronize_device.Verified against the vLLM tree (
vllm/v1/worker/worker_base.py): the method isabsent in v0.28.0 and present from v0.29.0 onward. v0.28.0 is in the
support matrix, so on that line the override is the only barrier that exists --
there is nothing to inherit, and
wake_up()would otherwise raiseAttributeError.sleep()gets the same treatment, bracketing the memory transition with abarrier on each side: one before the baseline
mem_get_info()snapshot that thefreed-bytes assertion depends on, and one after the optional RL extra cleanup --
whose
destroy_hccl()/aclgraph work is the last thing that can touch memorybefore the allocator unmaps it. Without them,
sleep()racing an in-flightdummy batch fails inside
CaMemAllocator.sleep()'sempty_cache()(
AclrtSynchronizeDeviceWithTimeout,507011/507015) -- the same failure thereleases/v0.30.0fix in #18338 addresses.Does this PR introduce any user-facing change?
No. Control-plane only; no effect on model output.
How was this patch tested?
Unit tests (
tests/ut/worker/test_worker_v1.py): everywake_uptest now patchesthe barrier and asserts it is reached,
synchronize_device()delegates totorch.npu.synchronize(), and the method is asserted to live onNPUWorkeritself.
Unit tests (
tests/ut/worker/test_kv_cache_allocation_scope.py) andtest_sleep_uses_rl_extra_cleanup: the sleep tests now patch the barrier andassert both sleep barriers fire, covering the cleanup-enabled and
cleanup-disabled paths. They also stop executing a real
torch.npu.synchronize()on NPU hosts, which previously made these specs behave differently on hardware
versus CPU runners.
NPU system tests:
tests/e2e/pull_request/one_card/rlhf/state_transitions/test_npu_device_synchronize.py-- on real hardware,
synchronize_device()andwake_up()must leave the NPUstream idle.
tests/e2e/pull_request/two_card/rlhf/state_transitions/test_pause_device_idle.py-- the Ascend port of upstream's
test_dp_pause_completion_implies_device_idle.In DPLB the pause path itself launches the dummy batch whose completion the pause
promise covers, so the test inflates the dummy batches the pause consensus
manufactures and requires an idle device the moment
pause_generation()returns.It skips (rather than passing vacuously) on vLLM without the #52914 barrier.
tests/e2e/pull_request/one_card/rlhf/state_transitions/npu_stream_busy.py-- the device-busy helper both files share.
torch.cuda._sleep(200_000_000), which the upstream test uses to keep a CUDAstream busy, cannot run on NPU:
torch.npuexposes no_sleep, andtorch.cuda._sleepraisesAttributeError: module 'torch._C' has no attribute '_cuda_sleep'.enqueue_npu_busy_work()queues real matmuls without waitinginstead, and
torch.npu.current_stream().query()reports the same "is the streamquiet" fact as
torch.cuda.current_stream().query().Re-verified on Ascend 910B3 (2 chips) against the vLLM main lane
(
0.30.0, the pinned.github/vllm-main-verified.commit), i.e. after the sleepbarriers were added -- 84 passed:
tests/ut/worker/test_worker_v1.py+tests/ut/worker/test_kv_cache_allocation_scope.pytests/e2e/pull_request/one_card/rlhf/state_transitions/test_npu_device_synchronize.pytests/e2e/pull_request/two_card/rlhf/state_transitions/test_pause_device_idle.pyThe DP pause test uses
Qwen/Qwen2.5-0.5B-Instruct, a small text-only checkpointthe a3 runners already pre-download. CI runs with
VLLM_USE_MODELSCOPE=1andHF_HUB_OFFLINE=1, so an unlisted checkpoint cannot be fetched on the runners.An earlier A/B check of the pause barrier itself, run on the vLLM 0.29.0 line
before the sleep barriers were added, kept a DP=2 engine busy for ~200 ms and then
called
pause_generation():pause_generation()wall timeUpstream context
Ports the Ascend backend requirement from vLLM
#52914.