Skip to content

[Perf] dsv4: Compute request-window history rows once per batch - #45579

Merged
hnyls2002 merged 2 commits into
mainfrom
lsyin/rw-history-rows
Oct 21, 2026
Merged

hnyls2002 merged 2 commits into
mainfrom
lsyin/rw-history-rows

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator
  • window_layout now computes each history slot's source row (history_src: row_loc of the slot's request and position, or the zero row when invalid) once per batch and stores it in WindowLayout, instead of RequestWindow.buffer recomputing it in every layer through _history_src. It takes the window's zero_row to do so.
  • WindowLayout.copy_ copies the new field, so a captured layout refreshed in place for graph replay carries the current batch's rows.
  • The two window_layout callers (the CUDA and HIP DeepSeek V4 backends) and the tests that build a layout pass zero_row.
  • DeepSeek V4.1 Flash on GB300, DEP4: row_loc and _history_src take 4.1-4.8% and up to 2.9% of the scheduler thread at 4K tokens per rank, and 3.2% and 1.7% in a multi-turn prefill workload at 640 concurrent sessions, all per layer under get_swa_raw_buffer. With this change and the V4.1 index-K slot snapshot ([Perf] dsv4: Gather V4.1 index-K slots once per forward and cache the prefill index plan #45577) together, the 4K step goes from 106-110 to 98.7 ms (one run per arm); this change alone was not measured.

No behavior change; only applies with encoder SWA bounded replay.


CI States

Latest PR Test (Base): 🚫 Run #39675061764
Latest PR Test (Extra): ❌ Run #39675061708
Latest PR Test (AMD ROCm 10): 🚫 Run #39675061600

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test test_dsv41_request_window_copy.py test_q8kv8_sparse_prefill_backend.py test_deepseek_v41_flash_gb300_encoder_bcg.py test_unified_cache_linker_kl_dsv41.py test_deepseek_v4.py test_encoder_swa_replay.py test_dsv4_compressed_pools.py test_dsv41_late_layer_tail_readers.py

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

🚀 1-gpu-h100 (3 tests): ✅ View workflow run

cd test/ && python3 registered/kernels/ops/kvcache/test_dsv41_request_window_copy.py
cd test/ && python3 registered/kernels/ops/attention/test_q8kv8_sparse_prefill_backend.py
cd test/ && python3 registered/attention/unittests/dsv4/test_deepseek_v4.py

🚀 4-gpu-gb300 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300_encoder_bcg.py

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/linker/test_unified_cache_linker_kl_dsv41.py

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/attention/unittests/dsv4/test_deepseek_v4.py

🚀 ubuntu-latest (3 tests): ✅ View workflow run

cd test/ && python3 registered/unit/model_executor/test_encoder_swa_replay.py
cd test/ && python3 registered/unit/mem_cache/test_dsv4_compressed_pools.py
cd test/ && python3 registered/unit/layers/test_dsv41_late_layer_tail_readers.py

@hnyls2002 hnyls2002 added the run-ci CI: run the baseline test suite on this PR label Oct 21, 2026
@hnyls2002
hnyls2002 merged commit f7d4f06 into main Oct 21, 2026
78 of 120 checks passed
@hnyls2002
hnyls2002 deleted the lsyin/rw-history-rows branch October 21, 2026 21:01
kkHuang-amd pushed a commit to HaiShaw/sglang that referenced this pull request Oct 22, 2026
Resolve conflicts with sgl-project#45579 (history_src precomputed in the layout:
drop the HIP history-row cache, pass zero_row, add history_src to the
direct layout) and sgl-project#45586 (idle request-window layout). Gate the direct
window branches on _is_hip, move the direct test to AMD CI, and give the
HIP floor test mock a zero_row.
chaohu2000 added a commit to flytiger-eco/sglang-for-sail that referenced this pull request Oct 27, 2026
…project#45579)

(cherry picked from commit 7be625e1a97b421b119b49c26596a3145ab31a77)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant