Repository navigation
[Fix] deepseek_v4: Enable monokernels for MoRI PD decode - #67
Merged
Merged
Conversation
Duyi-Wang
marked this pull request as ready for review
October 28, 2026 02:42
Duyi-Wang
requested review from
HaiShaw,
alexnails and
kevin-mii
as code owners
October 28, 2026 02:42
HaiShaw
approved these changes
Oct 28, 2026
AMD-yanfeiwang
added a commit
to AMD-yanfeiwang/sgl-amd-mono
that referenced
this pull request
Oct 28, 2026
Allow the DeepSeek-V4.1-Flash target and DSpark draft monokernels on a MoRI PD decode server, as sgl-project#67 does for DeepSeek-V4-Pro; SGLang serves V4.1 DSpark PD over MoRI since sgl-project/sglang#42457. The kernels cover decode and DSpark verify steps only, so the KV, compressor-state and index-key buffers they read are the ones MoRI fills. Prefill servers and other transfer backends stay refused. The README names the index-key layout switch that both peers must share, since the PD handshake does not compare it.
HaiShaw
pushed a commit
that referenced
this pull request
Oct 29, 2026
## Motivation The DeepSeek-V4.1-Flash monokernels refuse every PD server today: the target and DSpark draft specs do not declare `PD_DISAGGREGATION`, so `--enable-amd-monokernel` on a decode server stops with "PD disaggregation is in use and the spec does not accept it". #67 enabled MoRI PD decode for DeepSeek-V4-Pro; this does the same for V4.1. SGLang serves V4.1 DSpark PD over MoRI since sgl-project/sglang#42457. ## Modifications - `deepseek_v41/spec.py` and `deepseek_v41/draft.py` declare `PD_DISAGGREGATION`. - `_config_disable_reasons` refuses a prefill server and transfer backends other than MoRI, with the same reasons as V4-Pro. - `SGLANG_READS` pins `disaggregation_mode` and `disaggregation_transfer_backend`. - A unit test covers the gate; the README's MoRI PD note names both models. The kernels and their layouts are unchanged. They run decode and DSpark verify steps only, so all the state they read (paged SWA KV, compressor states, index keys, the draft's KV) lives in SGLang's own pools, which MoRI fills on the decode server. With a per-node shared Engram host table, decode also needs #65. Both peers must leave `SGLANG_HIP_DSV41_INDEX_K_PAGE8` unset: it changes the index keys' bytes but not their size, so SGLang's PD handshake does not catch a mismatch (the README says so). ## Accuracy Tests DeepSeek-V4.1-Flash + DSpark (block size 5), 1P1D on one MI355X node over MoRI with `SGLANG_RAGGED_VERIFY_MODE=static`: prefill without monokernels at the decode's TP. TP2 keeps the Engram tables in per-rank host memory on both peers (`--engram-config '{"placement":"cpu"}'`), as the AgentX TP2 recipe does; on GPU they leave TP2 no room for the KV cache. `lmsysorg/sglang-rocm:v0.5.22-rocm10-mi35x-20261028` running SGLang main. GSM8K is `sglang.test.few_shot_gsm8k` with 1319 questions, sent through the PD router. | decode | TP | `--enable-amd-monokernel` | startup | parallel 64 | parallel 8 | DSpark accept length | |---|---|---|---|---:|---:|---:| | `main` | any | yes | refused: PD not accepted | - | - | - | | this PR | 4 | no | serves | 0.889 | 0.908 | 3.39 | | this PR | 4 | yes | serves; target and draft kernels built on all ranks | 0.895 | 0.915 | 3.39 | | this PR | 2 | no | serves | 0.908 | 0.904 | 3.37 | | this PR | 2 | yes | serves; target and draft kernels built on all ranks | 0.904 | 0.906 | 3.38 | The monokernels run verify steps of up to 8 requests, so parallel 8 runs every verify step on them; at parallel 64 most steps are wider and take SGLang's own path. The spread matches run-to-run nondeterminism. Without PD, on the same image, the monokernels score 0.904 to 0.909 at parallel 64 and 0.903 to 0.914 at parallel 8 across Engram table placements. CPU, same image: `test_deepseek_v41_spec.py` passes, including the new PD test, and so does the `deepseek_v41` contract. The two `minimax` / `minimax_m31` contract failures are the same on `main`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allow DeepSeek-V4-Pro monokernels on MoRI PD decode servers. MoRI transfers into the same KV and compressor-state buffers the kernels use. Prefill and other transfer backends remain rejected; kernel math and KV layouts are unchanged.
Validation: CPU startup/contract checks passed. The same PD change was tested on MI355X / ROCm 10, P4/D8, DSpark block 6, with SGLang
a4a19332aband mono basedb7adb7:AgentX c1/c4 runs also completed, without a clear overall performance gain. This is a subset accuracy check, not a full evaluation.