[Feature][PCP] Support decode-only FULL CUDA graphs - #53867
Merged
Merged
Conversation
pisceskkk
force-pushed
the
codex/pcp-full-main
branch
3 times, most recently
from
August 27, 2026 02:41
622dd24 to
88fad4c
Compare
LucasWilkinson
previously requested changes
Aug 27, 2026
LucasWilkinson
left a comment
LucasWilkinson
left a comment
Collaborator
There was a problem hiding this comment.
Im not sure we want this?! I dont it makes sense to have a PCP deployment doing decodes (i.e. PCP is primarily for P/D). Is there a use case im missing?
Contributor
Author
imo PCP isn’t only used in P/D deployments. In our experience on Ascend, for models like GLM-5.2 and DeepSeek-V4, PCP provides significant prefill performance improvements, resulting in noticeable throughput gains even in mixed deployments. The average TPOT is also better than without PCP. |
Contributor
Author
And for example, based on the data shown in PR #52162 — "8× H20, 1/4-depth DeepSeek-V3.2 FP8 checkpoint at 95/97/99% hit rates" — PCP performs slightly better than TP under mixed deployment as well. (That said, I’m still not sure how useful the PR #52162 is or whether it should be merged, so it’s just pending for now.) 
|
Collaborator
|
oh sorry i missed #52162 (basically makes it a PCP x DP agg deployment which makes sense to me 👍), with that context this makes more sense, thanks! will try to review that soon |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
pisceskkk
force-pushed
the
codex/pcp-full-main
branch
from
August 27, 2026 06:49
88fad4c to
c8d4bd4
Compare
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
pisceskkk
marked this pull request as ready for review
August 27, 2026 12:05
pisceskkk
requested review from
AndreasKaratzas,
WoosukKwon,
mgoin,
njhill,
vadiklyutiy and
yewentao256
as code owners
August 27, 2026 12:05
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f832b6a7a7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -0,0 +1,154 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
There was a problem hiding this comment.
This file contains focused PCPManager and set_dummy_context unit tests, while tests/v1/worker/test_gpu_pcp_manager.py is the existing nearby suite and is already extended by this commit. Move these cases into that file instead of creating a parallel suite; repository guidance explicitly requires extending a nearby test file whenever one fits.
AGENTS.md reference: AGENTS.md:L88-L89
Useful? React with 👍 / 👎.
sunny-rain-63
pushed a commit
to sunny-rain-63/vllm-ascend
that referenced
this pull request
Sep 12, 2026
### What this PR does / why we need it? Add Ascend MRV2 handling for PCP combined with DP, reusing the upstream PCP dispatch sizing and input partitioning paths. When a DP replica becomes idle, its dummy batch bypasses PCP partitioning. Saved PCP attention state may be absent or belong to the preceding real batch. Runtime dummy inputs use runner buffers, while decode graphs capture persistent PCP buffers. - Build dummy PCP attention contexts from the current batch, block tables and rank-local slot mappings, including the DSA metadata path. - Refresh persistent PCP buffers before dummy execution. Keep the implementation in `AscendPCPManager`; `NPUModelRunner.prepare_dummy_attn` delegates or takes the existing non-PCP path. - Synchronize replicated speculative drafts independently of the target's PCP-local DP state, with every DP replica participating, including idle and decode replicas. - Restrict PCP+DP graph configuration to eager or `FULL_DECODE_ONLY`. Upstream validation dependency: vllm-project/vllm#54523, merged as [7c2f1ff4958eaf0818405e9192c71608fe4a16b1](vllm-project/vllm@7c2f1ff), moves the PCP+DP restriction from common `ParallelConfig` validation to CUDA/ROCm platform checks. This PR relies on that change and contains no Ascend validator bypass or Pydantic schema rebuilding. Related upstream PR: vllm-project/vllm#53867 ([Feature][PCP] Support decode-only FULL CUDA graphs). Once the paired vLLM includes #53867, adapt its PCP `prepare_inputs_to_capture` entry point to create `AscendInputBatch` directly in persistent PCP buffers. After capture and real-to-idle replay validation, remove `AscendPCPManager.prepare_dummy_attn` and the runner override, as tracked by `TODO(wzx0726)`. The Ascend dummy attention-context handling and replicated-draft DP synchronization remain necessary. ### Does this PR introduce _any_ user-facing change? Adds Ascend-side MRV2 handling for PCP combined with DP. No new CLI options or environment variables are introduced. Rebased onto Ascend main `ad86348b0cb2324d643df6a496f2d9c3879e4481`. The draft conflict resolution preserves vLLM 0.28.0 `num_tokens_across_dp` forwarding and applies fresh synchronization for replicated drafts on the main2main `dp_sync` interface. **Compatibility and remaining validation:** - The previously tested baseline was Ascend `b1c91857c16bde10c2fa7f5d7548b7666a86d4bb` with vLLM `e6bfe03ad73a3330cb427885aa90d97a12e1c704`. - Removing the validator bypass requires vLLM #54523. Ascend main currently pins `b2f685834a6456197e7033966fdef52a23f1abcd`, which predates that change and still rejects PCP+DP. The local vLLM checkout remains at the user-selected `7c2f1ff4958eaf0818405e9192c71608fe4a16b1`. This PR does not change the main2main pin or reintroduce the bypass. - Compatibility work against that newer revision is still pending, including the added `prepare_dummy_attn` argument and removal of `InputBatch.max_seq_len_np`. The current branch is not yet validated to run against that revision. - Multi-card Attention/MLA/DSA inference, real-to-idle DP transitions during `FULL_DECODE_ONLY` replay, speculative-decoding numerical correctness, and EP/TP combinations require model-level validation on the final paired sources. ### How was this patch tested? The latest rebase onto `ad86348b0` resolves an append-location conflict in `test_pcp_manager_v2.py`, retaining both the main DCP padding regression and this PR's tests. AST comparison confirms that all main tests and the final PR tests are preserved, and the other changed files match the automatic merge. Ruff lint/format, syntax parsing, and `git diff --check` passed. Runtime tests were not rerun for this conflict-only update. For the preceding rebase onto `f8481287a`, **6 targeted candidate-method cases passed** in an isolated process in the Ascend container: five main2main draft-sync cases and one mocked v0.28.0 argument-routing case. The exact candidate `propose` method was loaded into the installed runtime classes; native dependencies and the rest of the runtime remained at their installed versions. This is method-level regression evidence, not full rebased-source or dual-version runtime validation. All eight changed Python files passed Ruff lint/format checks, syntax parsing, and `git diff --check`. Full unit-suite and model validation on the final paired sources remain pending. The runtime adaptation suite previously passed on the earlier baseline in an isolated Ascend-container test directory, using CPU tensors and mocks for device kernels: ```bash python3 -m pytest -q tests/ut/worker/test_pcp_manager_v2.py tests/ut/worker/test_model_runner_v2.py tests/ut/worker/test_attn_utils_v2.py tests/ut/worker/test_mtp_pcp_speculator_v2.py ``` - **75 passed** (14 existing `torch.jit` deprecation warnings). This is prior-baseline evidence, not a test result for vLLM `7c2f1ff`. - Covers missing/stale dummy PCP state, rank-local mappings, persistent buffer contents and storage, DSA metadata forwarding, non-PCP fallback, graph restrictions, and replicated draft DP synchronization. - The removed validator bypass's dedicated tests are also removed; the existing unrelated platform test is retained. - For the removal, targeted Ruff lint/format checks, Python syntax checks and `git diff --check` passed. No runtime tests were rerun against the upgraded vLLM. The local `format.sh ci` entry point remains unavailable because its shell lacks `pre-commit`. - vLLM main: vllm-project/vllm@b2f6858 --------- Signed-off-by: wzx0726 <zhexuanwu12@gmail.com>
Collaborator
|
/ci run |
|
✅ Triggered Buildkite CI #88559 for commit |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Contributor
Author
|
/ci run |
|
✅ Triggered Buildkite CI #88585 for commit |
Contributor
|
Hi @pisceskkk, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
Contributor
Author
|
/ci run |
|
✅ Triggered Buildkite CI #88659 for commit |
johnnysluckydays
pushed a commit
to johnnysluckydays/vllm-ascend
that referenced
this pull request
Sep 14, 2026
### What this PR does / why we need it? Add Ascend MRV2 handling for PCP combined with DP, reusing the upstream PCP dispatch sizing and input partitioning paths. When a DP replica becomes idle, its dummy batch bypasses PCP partitioning. Saved PCP attention state may be absent or belong to the preceding real batch. Runtime dummy inputs use runner buffers, while decode graphs capture persistent PCP buffers. - Build dummy PCP attention contexts from the current batch, block tables and rank-local slot mappings, including the DSA metadata path. - Refresh persistent PCP buffers before dummy execution. Keep the implementation in `AscendPCPManager`; `NPUModelRunner.prepare_dummy_attn` delegates or takes the existing non-PCP path. - Synchronize replicated speculative drafts independently of the target's PCP-local DP state, with every DP replica participating, including idle and decode replicas. - Restrict PCP+DP graph configuration to eager or `FULL_DECODE_ONLY`. Upstream validation dependency: vllm-project/vllm#54523, merged as [7c2f1ff4958eaf0818405e9192c71608fe4a16b1](vllm-project/vllm@7c2f1ff), moves the PCP+DP restriction from common `ParallelConfig` validation to CUDA/ROCm platform checks. This PR relies on that change and contains no Ascend validator bypass or Pydantic schema rebuilding. Related upstream PR: vllm-project/vllm#53867 ([Feature][PCP] Support decode-only FULL CUDA graphs). Once the paired vLLM includes #53867, adapt its PCP `prepare_inputs_to_capture` entry point to create `AscendInputBatch` directly in persistent PCP buffers. After capture and real-to-idle replay validation, remove `AscendPCPManager.prepare_dummy_attn` and the runner override, as tracked by `TODO(wzx0726)`. The Ascend dummy attention-context handling and replicated-draft DP synchronization remain necessary. ### Does this PR introduce _any_ user-facing change? Adds Ascend-side MRV2 handling for PCP combined with DP. No new CLI options or environment variables are introduced. Rebased onto Ascend main `ad86348b0cb2324d643df6a496f2d9c3879e4481`. The draft conflict resolution preserves vLLM 0.28.0 `num_tokens_across_dp` forwarding and applies fresh synchronization for replicated drafts on the main2main `dp_sync` interface. **Compatibility and remaining validation:** - The previously tested baseline was Ascend `b1c91857c16bde10c2fa7f5d7548b7666a86d4bb` with vLLM `e6bfe03ad73a3330cb427885aa90d97a12e1c704`. - Removing the validator bypass requires vLLM #54523. Ascend main currently pins `b2f685834a6456197e7033966fdef52a23f1abcd`, which predates that change and still rejects PCP+DP. The local vLLM checkout remains at the user-selected `7c2f1ff4958eaf0818405e9192c71608fe4a16b1`. This PR does not change the main2main pin or reintroduce the bypass. - Compatibility work against that newer revision is still pending, including the added `prepare_dummy_attn` argument and removal of `InputBatch.max_seq_len_np`. The current branch is not yet validated to run against that revision. - Multi-card Attention/MLA/DSA inference, real-to-idle DP transitions during `FULL_DECODE_ONLY` replay, speculative-decoding numerical correctness, and EP/TP combinations require model-level validation on the final paired sources. ### How was this patch tested? The latest rebase onto `ad86348b0` resolves an append-location conflict in `test_pcp_manager_v2.py`, retaining both the main DCP padding regression and this PR's tests. AST comparison confirms that all main tests and the final PR tests are preserved, and the other changed files match the automatic merge. Ruff lint/format, syntax parsing, and `git diff --check` passed. Runtime tests were not rerun for this conflict-only update. For the preceding rebase onto `f8481287a`, **6 targeted candidate-method cases passed** in an isolated process in the Ascend container: five main2main draft-sync cases and one mocked v0.28.0 argument-routing case. The exact candidate `propose` method was loaded into the installed runtime classes; native dependencies and the rest of the runtime remained at their installed versions. This is method-level regression evidence, not full rebased-source or dual-version runtime validation. All eight changed Python files passed Ruff lint/format checks, syntax parsing, and `git diff --check`. Full unit-suite and model validation on the final paired sources remain pending. The runtime adaptation suite previously passed on the earlier baseline in an isolated Ascend-container test directory, using CPU tensors and mocks for device kernels: ```bash python3 -m pytest -q tests/ut/worker/test_pcp_manager_v2.py tests/ut/worker/test_model_runner_v2.py tests/ut/worker/test_attn_utils_v2.py tests/ut/worker/test_mtp_pcp_speculator_v2.py ``` - **75 passed** (14 existing `torch.jit` deprecation warnings). This is prior-baseline evidence, not a test result for vLLM `7c2f1ff`. - Covers missing/stale dummy PCP state, rank-local mappings, persistent buffer contents and storage, DSA metadata forwarding, non-PCP fallback, graph restrictions, and replicated draft DP synchronization. - The removed validator bypass's dedicated tests are also removed; the existing unrelated platform test is retained. - For the removal, targeted Ruff lint/format checks, Python syntax checks and `git diff --check` passed. No runtime tests were rerun against the upgraded vLLM. The local `format.sh ci` entry point remains unavailable because its shell lacks `pre-commit`. - vLLM main: vllm-project/vllm@b2f6858 --------- Signed-off-by: wzx0726 <zhexuanwu12@gmail.com> Signed-off-by: tianming2009 <13246728590@163.com>
Contributor
Author
|
/ci run |
|
❌ This PR is 24 commits behind upstream |
Contributor
Author
|
/ci run |
|
❌ This PR is 1 commit behind upstream |
Contributor
Author
|
/ci run |
|
✅ Triggered Buildkite CI #88985 for commit |
like-0517
pushed a commit
to like-0517/vllm-ascend
that referenced
this pull request
Sep 15, 2026
### What this PR does / why we need it? Add Ascend MRV2 handling for PCP combined with DP, reusing the upstream PCP dispatch sizing and input partitioning paths. When a DP replica becomes idle, its dummy batch bypasses PCP partitioning. Saved PCP attention state may be absent or belong to the preceding real batch. Runtime dummy inputs use runner buffers, while decode graphs capture persistent PCP buffers. - Build dummy PCP attention contexts from the current batch, block tables and rank-local slot mappings, including the DSA metadata path. - Refresh persistent PCP buffers before dummy execution. Keep the implementation in `AscendPCPManager`; `NPUModelRunner.prepare_dummy_attn` delegates or takes the existing non-PCP path. - Synchronize replicated speculative drafts independently of the target's PCP-local DP state, with every DP replica participating, including idle and decode replicas. - Restrict PCP+DP graph configuration to eager or `FULL_DECODE_ONLY`. Upstream validation dependency: vllm-project/vllm#54523, merged as [7c2f1ff4958eaf0818405e9192c71608fe4a16b1](vllm-project/vllm@7c2f1ff), moves the PCP+DP restriction from common `ParallelConfig` validation to CUDA/ROCm platform checks. This PR relies on that change and contains no Ascend validator bypass or Pydantic schema rebuilding. Related upstream PR: vllm-project/vllm#53867 ([Feature][PCP] Support decode-only FULL CUDA graphs). Once the paired vLLM includes #53867, adapt its PCP `prepare_inputs_to_capture` entry point to create `AscendInputBatch` directly in persistent PCP buffers. After capture and real-to-idle replay validation, remove `AscendPCPManager.prepare_dummy_attn` and the runner override, as tracked by `TODO(wzx0726)`. The Ascend dummy attention-context handling and replicated-draft DP synchronization remain necessary. ### Does this PR introduce _any_ user-facing change? Adds Ascend-side MRV2 handling for PCP combined with DP. No new CLI options or environment variables are introduced. Rebased onto Ascend main `ad86348b0cb2324d643df6a496f2d9c3879e4481`. The draft conflict resolution preserves vLLM 0.28.0 `num_tokens_across_dp` forwarding and applies fresh synchronization for replicated drafts on the main2main `dp_sync` interface. **Compatibility and remaining validation:** - The previously tested baseline was Ascend `b1c91857c16bde10c2fa7f5d7548b7666a86d4bb` with vLLM `e6bfe03ad73a3330cb427885aa90d97a12e1c704`. - Removing the validator bypass requires vLLM #54523. Ascend main currently pins `b2f685834a6456197e7033966fdef52a23f1abcd`, which predates that change and still rejects PCP+DP. The local vLLM checkout remains at the user-selected `7c2f1ff4958eaf0818405e9192c71608fe4a16b1`. This PR does not change the main2main pin or reintroduce the bypass. - Compatibility work against that newer revision is still pending, including the added `prepare_dummy_attn` argument and removal of `InputBatch.max_seq_len_np`. The current branch is not yet validated to run against that revision. - Multi-card Attention/MLA/DSA inference, real-to-idle DP transitions during `FULL_DECODE_ONLY` replay, speculative-decoding numerical correctness, and EP/TP combinations require model-level validation on the final paired sources. ### How was this patch tested? The latest rebase onto `ad86348b0` resolves an append-location conflict in `test_pcp_manager_v2.py`, retaining both the main DCP padding regression and this PR's tests. AST comparison confirms that all main tests and the final PR tests are preserved, and the other changed files match the automatic merge. Ruff lint/format, syntax parsing, and `git diff --check` passed. Runtime tests were not rerun for this conflict-only update. For the preceding rebase onto `f8481287a`, **6 targeted candidate-method cases passed** in an isolated process in the Ascend container: five main2main draft-sync cases and one mocked v0.28.0 argument-routing case. The exact candidate `propose` method was loaded into the installed runtime classes; native dependencies and the rest of the runtime remained at their installed versions. This is method-level regression evidence, not full rebased-source or dual-version runtime validation. All eight changed Python files passed Ruff lint/format checks, syntax parsing, and `git diff --check`. Full unit-suite and model validation on the final paired sources remain pending. The runtime adaptation suite previously passed on the earlier baseline in an isolated Ascend-container test directory, using CPU tensors and mocks for device kernels: ```bash python3 -m pytest -q tests/ut/worker/test_pcp_manager_v2.py tests/ut/worker/test_model_runner_v2.py tests/ut/worker/test_attn_utils_v2.py tests/ut/worker/test_mtp_pcp_speculator_v2.py ``` - **75 passed** (14 existing `torch.jit` deprecation warnings). This is prior-baseline evidence, not a test result for vLLM `7c2f1ff`. - Covers missing/stale dummy PCP state, rank-local mappings, persistent buffer contents and storage, DSA metadata forwarding, non-PCP fallback, graph restrictions, and replicated draft DP synchronization. - The removed validator bypass's dedicated tests are also removed; the existing unrelated platform test is retained. - For the removal, targeted Ruff lint/format checks, Python syntax checks and `git diff --check` passed. No runtime tests were rerun against the upgraded vLLM. The local `format.sh ci` entry point remains unavailable because its shell lacks `pre-commit`. - vLLM main: vllm-project/vllm@b2f6858 --------- Signed-off-by: wzx0726 <zhexuanwu12@gmail.com> Signed-off-by: like-0517 <ithwlike@126.com>
Contributor
Author
|
/ci retry |
|
✅ Queued 1 failed job(s) for retry in Buildkite CI #88985. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Enable prefill context parallelism (PCP) to use decode-only FULL CUDA graphs through
FULL_DECODE_ONLYandFULL_AND_PIECEWISE.This change:
PCPManagerin one linear branch;FULLmode restricted because PCP prefill uses a separately dispatched execution path;Test Plan
Full GPU accuracy validation:
PIECEWISE,FULL_DECODE_ONLY, andFULL_AND_PIECEWISETest Result
All four services completed the full 1319-question dataset with zero invalid completions and zero request errors.
PIECEWISEFULL_DECODE_ONLYFULL_AND_PIECEWISEThe graph-mode deltas remain within 0.61 percentage points of eager on this evaluation. Service logs were also checked for graph replay, CUDA, NCCL, and EngineCore failures.
Current branch validation at
f832b6a7a75745bbf80dbb9763cea1d4ed587356:git diff --checkpre-commitworkflow passed in 5m56s