Repository navigation
[Perf] dsv4: Compute request-window history rows once per batch - #45579
Merged
Merged
Conversation
hnyls2002
requested review from
Fridge003,
HaiShaw,
Qiaolin-Yu,
Ying1123,
alphabetc1,
hanming-lu,
hebiao064,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
October 21, 2026 09:46
Collaborator
Author
|
/rerun-test test_dsv41_request_window_copy.py test_q8kv8_sparse_prefill_backend.py test_deepseek_v41_flash_gb300_encoder_bcg.py test_unified_cache_linker_kl_dsv41.py test_deepseek_v4.py test_encoder_swa_replay.py test_dsv4_compressed_pools.py test_dsv41_late_layer_tail_readers.py |
Contributor
|
🚀 🚀 🚀 🚀 🚀 |
kkHuang-amd
pushed a commit
to HaiShaw/sglang
that referenced
this pull request
Oct 22, 2026
Resolve conflicts with sgl-project#45579 (history_src precomputed in the layout: drop the HIP history-row cache, pass zero_row, add history_src to the direct layout) and sgl-project#45586 (idle request-window layout). Gate the direct window branches on _is_hip, move the direct test to AMD CI, and give the HIP floor test mock a zero_row.
1 task done
This was referenced Oct 22, 2026
Open
5 tasks
chaohu2000
added a commit
to flytiger-eco/sglang-for-sail
that referenced
this pull request
Oct 27, 2026
…project#45579) (cherry picked from commit 7be625e1a97b421b119b49c26596a3145ab31a77)
4 of 5 tasks
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
window_layoutnow computes each history slot's source row (history_src:row_locof the slot's request and position, or the zero row when invalid) once per batch and stores it inWindowLayout, instead ofRequestWindow.bufferrecomputing it in every layer through_history_src. It takes the window'szero_rowto do so.WindowLayout.copy_copies the new field, so a captured layout refreshed in place for graph replay carries the current batch's rows.window_layoutcallers (the CUDA and HIP DeepSeek V4 backends) and the tests that build a layout passzero_row.row_locand_history_srctake 4.1-4.8% and up to 2.9% of the scheduler thread at 4K tokens per rank, and 3.2% and 1.7% in a multi-turn prefill workload at 640 concurrent sessions, all per layer underget_swa_raw_buffer. With this change and the V4.1 index-K slot snapshot ([Perf] dsv4: Gather V4.1 index-K slots once per forward and cache the prefill index plan #45577) together, the 4K step goes from 106-110 to 98.7 ms (one run per arm); this change alone was not measured.No behavior change; only applies with encoder SWA bounded replay.
CI States
Latest PR Test (Base): 🚫 Run #39675061764
Latest PR Test (Extra): ❌ Run #39675061708
Latest PR Test (AMD ROCm 10): 🚫 Run #39675061600