[DeepSeek-V4] Enable non-paged indexer by default for large prefill chunks - #30140
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
/rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py test/registered/models_e2e/test_deepseek_v4_flash_fp8_h200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py test/registered/disaggregation/test_disaggregation_dsv4.py |
|
Results for 🚀 🚀 |
Fridge003
added
release-highlight
…er by default for large prefill chunks (sgl-project#30140) (sgl-project#30436) Co-authored-by: YAMY <74099316+YAMY1234@users.noreply.github.com>
2 participants
Motivation
The non-paged DeepSeek-V4 indexer improves large local prefill queries, while its fixed gather cost can regress small queries. Enable it by default only in the measured safe region.
Modifications
Accuracy Tests
This follow-up only changes the default path selection; it does not change kernel math. The non-paged path was accuracy-tested as part of #29619.
Paged/non-paged logits and top-k parity remained bit-exact on H100 and GB300 (
rtol=0,atol=0). Current-head B200 unit tests passed:5 passed,12 subtests passed.Speed Tests and Profiling
End-to-end serving
Real-serving tests used OSL=1 and an effective local prefill chunk of 8,192 tokens/rank. Global concurrency was 64 for 8K–64K and 16 for 128K. Each row is a four-arm
OFF → default → default → OFFcomparison with a 600-second measurement window per arm. Pair gain compares adjacent default/control runs; ABBA gain is their position-balanced geometric mean. The server windows were fully saturated with no cache reuse or fatal errors.Both serial pairs were positive at every ISL. The effect is small at 8K–16K and increases at longer contexts as the gather cost is amortized over more prefix work.
Benchmark
The table below is a kernel-level benchmark used to choose the default-path boundary. It compares paged MQA with gather + non-paged MQA on B200.
Local Qis the number of query tokens handled by one rank in the current prefill forward, not the request ISL.C4 prefix rowsis the compressed K length already present before that query block; one C4 row represents four raw tokens. Run 1 and Run 2 repeat the same comparison in different execution orders. Gain is(paged - gather_and_nonpaged) / paged, so a positive value means non-paged is faster.Non-paged generally becomes more favorable as
Local Qand the C4 prefix grow because the gather is amortized over more Q-by-K work. The trend is not strictly monotonic at every shape, but the safe boundary is clear: every testedQ >= 8192case improved in both runs, while smaller Q can regress, especially with short prefixes.The same boundary holds across architectures: at
Q=6144, H100 and GB300 regressed by8.78%and49.33%; atQ=8192, they improved by8.82%and6.26%. Therefore, this change conservatively selects non-paged atLocal Q >= 8192and keeps paged below it. A real-serving GB300 profile atQ=8192also reduced the replaced p50 GPU span from1.524 msto1.068 ms, including gather.Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #28722513145
Latest PR Test (Extra): ✅ Run #28901105817