[Perf] [NVIDIA] deepseek-v4.1: Reuse dequantized compressed KV in PD CP prefill (7/N) by BJWang-ant · Pull Request #40421 · sgl-project/sglang

BJWang-ant · 2026-09-20T06:40:38Z

Summary

DeepSeek V4.1 consumer layers can share the same compressed KV storage, but sparse prefill currently dequantizes that storage into a BF16 workspace on every layer.

Reuse the dequantized compressed KV within a forward to avoid repeated reads, dequantization, and workspace writes.

Changes

Limit reuse to CP sparse BF16 prefill with compression ratios 1 and 2.
Keep separate workspaces per compression ratio.
Track validity by source layer and workspace address within the per-forward gather cache.
Refresh on producer execution, source changes, workspace reallocation, or a new forward.
Continue updating SWA and top-k inputs for every layer.

This retains additional scratch memory for the separate workspaces.

Validation

GPU regression tests cover shared sources, source updates, ratio interleaving, consecutive forwards, and workspace resizing.
All 16 short/long-context smoke-test outputs match the baseline.
In a six-step trace, compressed KV dequantization calls decrease from 228 to 24 (89.5% fewer).

Performance

DeepSeek V4.1 Flash on 8×B200 with PD disaggregation; prefill uses TP4/EP4, CP, BCG, MegaMoE, and DSpark. Results are medians of three unprofiled runs using the unchanged benchmark script, with 512 requests and an output length of 500, which is 60k unprefix cache of 64k all input length.

precision

gsm8k test result

trace comparison

before

after

CI States

Latest PR Test (Base): 🚫 Run #41813675179
Latest PR Test (Extra): ❌ Run #41813675111
Latest PR Test (AMD ROCm 10): ❌ Run #41813675138

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

BJWang-ant · 2026-10-09T09:42:23Z

/tag-run-ci-label

BJWang-ant · 2026-10-09T09:44:03Z

/rerun-failed-ci

BJWang-ant · 2026-10-11T13:37:34Z

/rerun-failed-ci

BJWang-ant · 2026-10-12T02:22:43Z

/rerun-test test_deepseek_v41_flash_gb300.py

github-actions · 2026-10-12T02:23:06Z

🚀 4-gpu-gb300 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300.py

BJWang-ant · 2026-10-12T02:26:19Z

base pr pass：https://github.com/sgl-project/sglang/actions/runs/38134315111

BJWang-ant · 2026-10-20T07:40:48Z

/rerun-test test_dsv41_shared_prefill_kv.py test_deepseek_v41_pd_cp_dspark.py test_deepseek_v4_flash_fp4_h200.py test_deepseek_v4_flash_fp4_b200.py

github-actions · 2026-10-20T07:41:13Z

🚀 8-gpu-b300 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_pd_cp_dspark.py

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_h200.py

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_b200.py

⛔ test_dsv41_shared_prefill_kv.py: No test file found matching test_dsv41_shared_prefill_kv.py under test/registered/ or python/sglang/multimodal_gen/test/.

BJWang-ant · 2026-10-20T08:09:06Z

/rerun-test test_deepseek_v4_flash_fp4_megamoe_b200.py test_deepseek_v41_flash_gb300_dp4ep4.py test_deepseek_v41_flash_gb300_encoder_bcg.py test_deepseek_v41_flash_gb300_tp4ep4.py test_deepseek_v41_flash_gb300_trtllm_bounded_bcg.py

github-actions · 2026-10-20T08:09:28Z

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_megamoe_b200.py

🚀 4-gpu-gb300 (4 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300_dp4ep4.py
cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300_encoder_bcg.py
cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300_tp4ep4.py
cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300_trtllm_bounded_bcg.py

BJWang-ant · 2026-10-20T11:44:48Z

/rerun-test test_deepseek_v41_pd_cp_dspark.py test_deepseek_v4_flash_fp4_h200.py test_deepseek_v4_flash_fp4_b200.py test_deepseek_v4_flash_fp4_megamoe_b200.py test_deepseek_v41_flash_gb300_dp4ep4.py test_deepseek_v41_flash_gb300_encoder_bcg.py test_deepseek_v41_flash_gb300_tp4ep4.py test_deepseek_v41_flash_gb300_trtllm_bounded_bcg.py

github-actions · 2026-10-20T11:45:12Z

🚀 8-gpu-b300 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_pd_cp_dspark.py

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_h200.py

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_b200.py
cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_megamoe_b200.py

🚀 4-gpu-gb300 (4 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300_dp4ep4.py
cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300_encoder_bcg.py
cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300_tp4ep4.py
cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300_trtllm_bounded_bcg.py

Oasis-Git · 2026-10-22T07:10:05Z

PR ready. Will merge later

BJWang-ant · 2026-11-03T06:49:33Z

BJWang-ant · 2026-11-03T06:57:00Z

Required checks failed on 2c6ee0f; this PR cannot be merged until they pass.

Lint (run) Run pre-commit run --all-files locally and push the result. The run log shows any other failing step.

BJWang-ant requested review from Fridge003, HaiShaw, Qiaolin-Yu, hebiao064, ispobock and merrymercy as code owners September 20, 2026 06:40

github-actions Bot added the deepseek label Sep 20, 2026

BJWang-ant changed the title ~~[Perf] Reuse shared compressed KV dequantization in DeepSeek V4.1 CP prefill~~ [DSV4.1][PD][7/N][Perf] Reuse shared compressed KV dequantization in DeepSeek V4.1 CP prefill Sep 20, 2026

This was referenced Sep 20, 2026

DSV4.1 PD Road Map #39700

Open

[DSV4.1][PD][8/N] Fuse DeepSeek V4.1 prefill candidate selection and filtering #40536

Closed

[Perf] [NVIDIA] deepseek-v4.1: Fuse candidate selection and masking in PD prefill (8/N) #40436

Open

Oasis-Git mentioned this pull request Oct 2, 2026

[Roadmap] DeepSeek V4.1 Optimization #42170

Open

17 of 56 tasks

github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 9, 2026

BJWang-ant force-pushed the main-reuse-kvcache branch 4 times, most recently from c6d77af to b14c6e3 Compare October 11, 2026 11:05

BJWang-ant changed the title ~~[DSV4.1][PD][7/N][Perf] Reuse shared compressed KV dequantization in DeepSeek V4.1 CP prefill~~ [Perf] deepseek-v4.1: Reuse dequantized compressed KV in PD CP prefill (7/N) Oct 19, 2026

BJWang-ant changed the title ~~[Perf] deepseek-v4.1: Reuse dequantized compressed KV in PD CP prefill (7/N)~~ [Perf] [Nvidia] deepseek-v4.1: Reuse dequantized compressed KV in PD CP prefill (7/N) Oct 19, 2026

This comment has been minimized.

Sign in to view

BJWang-ant changed the title ~~[Perf] [Nvidia] deepseek-v4.1: Reuse dequantized compressed KV in PD CP prefill (7/N)~~ [Perf] [NVIDIA] deepseek-v4.1: Reuse dequantized compressed KV in PD CP prefill (7/N) Oct 19, 2026

BJWang-ant force-pushed the main-reuse-kvcache branch from b14c6e3 to 4e0159d Compare October 20, 2026 07:39

BJWang-ant force-pushed the main-reuse-kvcache branch 2 times, most recently from 86349c2 to e58302f Compare October 20, 2026 11:37

BJWang-ant requested review from BBuf, DarkSharpness, HydraQYH, celve and yuan-luo as code owners October 20, 2026 11:37

github-actions Bot added quant LLM Quantization jit-kernel labels Oct 20, 2026

BJWang-ant force-pushed the main-reuse-kvcache branch from e58302f to be92599 Compare October 21, 2026 02:37

Oasis-Git approved these changes Oct 22, 2026

View reviewed changes

BJWang-ant force-pushed the main-reuse-kvcache branch from be92599 to 2c6ee0f Compare November 3, 2026 06:33

This comment has been minimized.

Sign in to view

BJWang-ant force-pushed the main-reuse-kvcache branch from 2c6ee0f to 4d23ceb Compare November 4, 2026 03:07

BJWang-ant and others added 7 commits November 4, 2026 18:17

Reuse shared compressed KV dequantization in DeepSeek V4.1 CP prefill

c18922c

format

ee45a42

unit test update

01e2cab

support h200

9f256fe

remove log code

7a80d7a

update format

a769a00

fix bug

ac14286

BJWang-ant force-pushed the main-reuse-kvcache branch from 6405bb6 to ac14286 Compare November 4, 2026 10:17

Oasis-Git merged commit 06c60f4 into sgl-project:main Nov 5, 2026
146 of 165 checks passed

Conversation

BJWang-ant commented Sep 20, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Validation

Performance

precision

trace comparison

CI States

Uh oh!

BJWang-ant commented Oct 9, 2026

Uh oh!

BJWang-ant commented Oct 9, 2026

Uh oh!

BJWang-ant commented Oct 11, 2026

Uh oh!

BJWang-ant commented Oct 12, 2026

Uh oh!

github-actions Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BJWang-ant commented Oct 12, 2026

Uh oh!

This comment has been minimized.

BJWang-ant commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BJWang-ant commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BJWang-ant commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Oasis-Git commented Oct 22, 2026

Uh oh!

This comment has been minimized.

BJWang-ant commented Nov 3, 2026

Uh oh!

BJWang-ant commented Nov 3, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

BJWang-ant commented Sep 20, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Oct 12, 2026 •

edited

Loading

github-actions Bot commented Oct 20, 2026 •

edited

Loading

github-actions Bot commented Oct 20, 2026 •

edited

Loading

github-actions Bot commented Oct 20, 2026 •

edited

Loading