[DeepSeek-V4.1] Commit the engram decode history inside the hash kernel - #39138
Merged
Merged
Conversation
hnyls2002
requested review from
BBuf,
DarkSharpness,
Edwardf0t1,
Fridge003,
HaiShaw,
HydraQYH,
Ying1123,
celve,
ch-wan,
ispobock,
merrymercy and
yuan-luo
as code owners
September 11, 2026 22:36
kevin-mii
added a commit
to kevin-mii/sglang
that referenced
this pull request
Sep 12, 2026
…dense route, Engram if _is_hip dispatches from the model into deepseek_common/amd (fused mHC boundary with deferred coefficients, gfx950 dense helpers, index-Q one-launch, fused decode glue, SWA tail slicing), the Engram image select, the new SGLANG_OPT_HIP_* switches, and the ROCm defaults for the aiter batched GEMM and the FlashMLA backend. The Engram hash kernel runs on HIP too, so upstream's in-kernel decode commit (sgl-project#39138) serves both platforms and the history is written once per step. deepseek_v4_low_ratio_sources and _every_row_routed follow the ForwardBatch rename num_token_non_padded_cpu -> global_num_token_non_padded_cpu (the old name is no longer a field), which the HIP low-ratio path reads under the breakable prefill graph. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tZvxNDYPWtWc7ZtfXuybA
kevin-mii
added a commit
to kevin-mii/sglang
that referenced
this pull request
Sep 15, 2026
…dense route, Engram if _is_hip dispatches from the model into deepseek_common/amd (fused mHC boundary with deferred coefficients, gfx950 dense helpers, index-Q one-launch, fused decode glue, SWA tail slicing), the Engram image select, the new SGLANG_OPT_HIP_* switches, and the ROCm defaults for the aiter batched GEMM and the FlashMLA backend. The Engram hash kernel runs on HIP too, so upstream's in-kernel decode commit (sgl-project#39138) serves both platforms and the history is written once per step. deepseek_v4_low_ratio_sources and _every_row_routed follow the ForwardBatch rename num_token_non_padded_cpu -> global_num_token_non_padded_cpu (the old name is no longer a field), which the HIP low-ratio path reads under the breakable prefill graph. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tZvxNDYPWtWc7ZtfXuybA
kevin-mii
added a commit
to kevin-mii/sglang
that referenced
this pull request
Sep 16, 2026
…dense route, Engram if _is_hip dispatches from the model into deepseek_common/amd (fused mHC boundary with deferred coefficients, gfx950 dense helpers, index-Q one-launch, fused decode glue, SWA tail slicing), the Engram image select, the new SGLANG_OPT_HIP_* switches, and the ROCm defaults for the aiter batched GEMM and the FlashMLA backend. The Engram hash kernel runs on HIP too, so upstream's in-kernel decode commit (sgl-project#39138) serves both platforms and the history is written once per step. deepseek_v4_low_ratio_sources and _every_row_routed follow the ForwardBatch rename num_token_non_padded_cpu -> global_num_token_non_padded_cpu (the old name is no longer a field), which the HIP low-ratio path reads under the breakable prefill graph. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tZvxNDYPWtWc7ZtfXuybA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In decode every token is its request's newest token, so the engram hash kernel already holds the row that becomes the request's new n-gram history. This writes it from the kernel instead of the five torch ops that followed the kernel (
req_pool_indicescast,out_cache_loc == 0,where,flip,index_put). The kernel keeps one pure entry point,engram_hash_ids, and gains a decode one,engram_hash_ids_and_commit, which advanceshistoryin place (rows whose out_cache_loc is 0 are graph padding and write nothing) and skips the predecessor table that only the extend commit reads. The hasher is one node in the decode CUDA graph; extend keeps the torch commit, and verify is unchanged.Hash ids are integers, so the change is bitwise.
test_engram_hash.pyadds a 5-row decode case (a padded row aliasing a live slot) and a 100-row case over four programs, both checked against the naive oracle and the expected history table.A/B against the
dsv4.1base on DeepSeek-V4.1-Flash, TP4/EP4 on 4x GB300, one server at a time:CI States
Latest PR Test (Base): ❌ Run #34656639330
Latest PR Test (Extra): ❌ Run #34656639236
Latest PR Test (AMD ROCm 10): ❌ Run #34656639290