[CP]: Support CP V2 Strategy for dsv4 - #33532
Merged
Merged
Conversation
hzh0425
requested review from
1am9trash,
Fridge003,
Ying1123,
hnyls2002,
hubertlu-tw,
ispobock and
merrymercy
as code owners
August 4, 2026 09:56
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
hzh0425
requested review from
BBuf,
DarkSharpness,
Edwardf0t1,
HaiShaw,
HydraQYH,
Qiaolin-Yu,
YAMY1234,
celve,
ch-wan,
hebiao064,
kkHuang-amd,
rainj-me and
yuan-luo
as code owners
August 4, 2026 09:56
Collaborator
Author
|
/rerun-test test/registered/cp/test_dsv4_prefill_cp.py |
Contributor
|
Results for 🚀 |
hzh0425
force-pushed
the
cp/deepseek-v4-cpv2
branch
from
August 4, 2026 11:21
fc2dbe9 to
40dcabd
Compare
Collaborator
Author
|
/rerun-test test/registered/cp/test_dsv4_prefill_cp.py |
Contributor
|
Results for 🚀 |
41 tasks
Fridge003
reviewed
Aug 7, 2026
# Conflicts: # python/sglang/srt/layers/attention/dsv4/compressor.py
Collaborator
Author
|
/rerun-test test/registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py |
Contributor
|
Results for 🚀 |
Collaborator
Author
|
/rerun-test test/registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py |
Contributor
|
Results for 🚀 |
Fridge003
approved these changes
Aug 7, 2026
19 tasks
kfhfar
pushed a commit
to kfhfar/sglang
that referenced
this pull request
Aug 7, 2026
cursor Bot
pushed a commit
to learning-sketch/sglang
that referenced
this pull request
Aug 8, 2026
Clarify enable switches for shared-KV / layer-split / direct-KV and add an upstream-merged callout for sgl-project#33532. Co-authored-by: learning <learning-sketch@users.noreply.github.com>
Xia-Weiwen
pushed a commit
to Xia-Weiwen/sglang
that referenced
this pull request
Aug 10, 2026
5 tasks
saturn-acc
pushed a commit
to saturn-acc/sglang
that referenced
this pull request
Aug 16, 2026
AMD-yanfeiwang
added a commit
to AMD-yanfeiwang/sglang
that referenced
this pull request
Aug 20, 2026
backend, so on ROCm the model/runner would split tokens for CP-v2 while the HIP
attention backend built metadata for the un-split layout. Nothing gates CP-v2 by
platform (SGLANG_ENABLE_CP_V2, environ.py:319, and is_cp_v2_active have no
is_hip branch), so enabling it on ROCm produced wrong metadata rather than an
error. This ports the contract instead of gating it off.
The contract, mirroring deepseek_v4_backend.py: CP-v2 pads the causal metadata
rows for per-rank partitioning (P rows) while cache-write locations stay
one-per-logical-token (N rows). So the reindex must anchor its global-field
post-condition to N, not to the padded metadata length.
* apply_cp_reindex gains num_tokens (default None -> pre_global_len), keeping
the CP-v1 model-hook calls at deepseek_v4.py:2698 and
deepseek_v4_nextn.py:272 byte-compatible.
* init_compression_metadata gains num_tokens; its raw_out_loc/seq_lens_casual
shape-equality assert would otherwise fail on the first CP-v2 prefill, since
those two now legitimately differ.
* make_core_attn_metadata and init_forward_metadata_prefill thread num_tokens
and forward_batch so cp_v2_active and padded_num_tokens can be computed.
* HIP-only: unified_kv mirrors c4/c128_out_loc with a page offset
(UnifiedKvMetadata). Those are write addresses too, so they are checked to
stay global. They are nested under .unified and so cannot join
_CP_GLOBAL_FIELDS, which getattrs off self.
No kernel work was needed: the shared triton metadata kernel already implements
the num_write_tokens < bs masking CP-v2 relies on, and HIP's flashmla scheduler
metadata is a no-op (_create_flashmla_metadata returns None under is_hip).
dspark_block_size is deliberately not added to make_core_attn_metadata -- HIP
has no get_dspark_swa_page_indices, and it is unrelated to CP-v2.
NOT YET VALIDATED ON HARDWARE. Signatures and imports check out on ROCm; the
8xMI355X run is pending.
中文：将 CP V2(sgl-project#33532)移植到 DeepSeek-V4 HIP backend。sgl-project#33532 只接入了 CUDA backend，
因此在 ROCm 上 model/runner 会按 CP-v2 切分 token，而 HIP attention backend 仍按未切分
布局构造 metadata；且 CP-v2 没有任何平台门控，在 ROCm 上打开只会得到错误的 metadata 而
不是报错。本提交选择移植该能力而非关闭它。契约(对齐 deepseek_v4_backend.py)：CP-v2 会
把因果 metadata 行数补齐以便按 rank 切分(P 行)，而 cache 写入地址仍是每个逻辑 token 一行
(N 行)，因此 reindex 的全局字段后置条件必须以 N 为基准而非补齐后的长度。改动包括：
apply_cp_reindex 增加 num_tokens(默认 None 回退，保持 CP-v1 调用点二进制兼容)；
init_compression_metadata 增加 num_tokens(否则其 raw_out_loc/seq_lens_casual 形状相等
断言会在首次 CP-v2 预填充时失败)；make_core_attn_metadata 与
init_forward_metadata_prefill 透传 num_tokens 与 forward_batch；HIP 特有的
unified_kv 的 c4/c128_out_loc 同为写地址，一并校验其保持全局。无需改内核。尚未在硬件上
验证。
jakki-amd
pushed a commit
to jakki-amd/sglang
that referenced
this pull request
Sep 9, 2026
Atituiset
pushed a commit
to Atituiset/sglang
that referenced
this pull request
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Modifications
Accuracy Tests
Speed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #31148530718
Latest PR Test (Extra): ✅ Run #31148530638