[AMD] Support DeepSeek V4 DSpark on AMD HIP platform

[At1a8]

Motivation

Follow #30261, support dspark for deepseek v4 on AMD platform

Modifications

python/sglang/srt/layers/attention/deepseek_v4_backend_hip_radix.py
Enables HIP ragged verify CUDA graph support for DSpark. Adds ragged layout metadata handling, GPU prefill expansion/compression planning, token-tier graph keys, and a draft-worker token-count fix. Removes previous HIP NotImplementedError guards for ragged verify.

python/sglang/srt/layers/attention/dsv4/unified_kv_kernels/runtime.py
Adds a Triton scatter kernel and wrapper to write already normed/roped BF16 KV rows into unified KV by precomputed ring locations, skipping rows marked with negative locations.

python/sglang/srt/mem_cache/deepseek_v4_memory_pool.py
Adds a unified-KV store method for DSpark KV writes. It applies fused norm/RoPE to draft KV, then scatters BF16 keys into the unified SWA ring instead of the non-unified SWA KV pool.

python/sglang/srt/models/deepseek_v4_dspark.py
Routes DSpark draft KV storage and target-hidden KV injection through unified-KV writes when enabled. This avoids accessing swa_kv_pool in unified mode while preserving the original non-unified path.

python/sglang/srt/speculative/dflash_utils.py
Adds HIP Torch fallbacks for top-k and top-p probability renormalization, since ROCm builds may not register the corresponding sgl_kernel ops. CUDA/MUSA behavior remains unchanged.

python/sglang/srt/speculative/dspark_components/dspark_kv_inject.py
Adds unified-KV injection addressing using request state slots and ring positions. Applies SWA-window skipping and commit masking via negative locations, while preserving the existing full-to-SWA mapping path for non-unified KV.

python/sglang/srt/speculative/dspark_components/dspark_verify.py
Passes state-slot information into verify commit injection and selects unified commit-inject layouts when unified KV is enabled. Non-unified DSpark verify injection continues using the original layout builder.

python/sglang/srt/speculative/dspark_components/dspark_worker_v2.py
Computes per-token state slots and final positions during prefill injection for unified KV. This lets the injector keep only SWA-window tokens and avoid ring-slot collisions.

python/sglang/srt/speculative/dspark_components/kernels/dspark_verify_window.py
Adds a unified-KV commit-inject layout builder. It computes ring rows from request slots and token positions, marks uncommitted tokens as skipped, and returns the existing layout result type.

Accuracy Tests

Parallel	Backend	Spec mode	Accuracy
TP8	triton	off	0.924
TP8	triton	dspark	0.923
TP8	unified_kv_triton	off	0.938
TP8	unified_kv_triton	dspark	0.940
TP8DP8	triton	off	0.926
TP8DP8	triton	dspark	0.929
TP8DP8	unified_kv_triton	off	0.936
TP8DP8	unified_kv_triton	dspark	0.944

ci results by test/registered/amd/test_deepseek_v4_pro_fp4_dspark.py

[2026-07-18 17:06:05] INFO:     127.0.0.1:60712 - "POST /generate HTTP/1.1" 200 OK
[2026-07-18 17:06:05] INFO:     127.0.0.1:60626 - "POST /generate HTTP/1.1" 200 OK
[2026-07-18 17:06:05] INFO:     127.0.0.1:46568 - "POST /generate HTTP/1.1" 200 OK
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [00:32<00:00, 41.00it/s]
Accuracy: 0.948
Invalid: 0.000
Latency: 32.701 s
Output throughput: 3552.634 token/s
metrics={'accuracy': np.float64(0.9476876421531463), 'invalid': np.float64(0.0), 'latency': 32.70108681358397, 'output_throughput': 3552.634218619949}
[2026-07-18 17:06:05] INFO:     127.0.0.1:60816 - "GET /server_info HTTP/1.1" 200 OK
avg_spec_accept_length=3.7011230219499747
.
----------------------------------------------------------------------
Ran 3 tests in 305.374s

OK

Speed Tests and Profiling

Serving Bench - TP8 - 8k/1k

Backend	Spec mode	Conc	NP	TTT (tok/s)	TTT speedup	Mean TTFT (ms)	TTFT speedup	Mean TPOT (ms)	TPOT speedup
triton	off	1	4	548.65	1.00x	883.43	1.00x	15.55	1.00x
triton	dspark	1	4	1607.02	2.93x	888.85	0.99x	4.73	3.29x
triton	mtp	1	4	1289.22	2.35x	893.74	0.99x	6.11	2.55x
triton	off	8	32	2825.23	1.00x	3956.19	1.00x	21.63	1.00x
triton	dspark	8	32	5155.19	1.82x	2260.16	1.75x	11.04	1.96x
triton	mtp	8	32	4460.77	1.58x	1835.41	2.16x	13.97	1.55x
triton	off	16	64	4121.51	1.00x	7416.41	1.00x	27.71	1.00x
triton	dspark	16	64	6434.69	1.56x	3104.05	2.39x	18.61	1.49x
triton	mtp	16	64	5681.03	1.38x	3128.92	2.37x	21.84	1.27x
triton	off	32	128	5666.09	1.00x	14217.42	1.00x	36.96	1.00x
triton	dspark	32	128	7440.74	1.31x	5036.95	2.82x	32.93	1.12x
triton	mtp	32	128	6953.73	1.23x	5005.69	2.84x	35.92	1.03x
unified_kv_triton	off	1	4	643.97	1.00x	364.10	1.00x	13.63	1.00x
unified_kv_triton	dspark	1	4	1758.66	2.73x	369.11	0.99x	4.75	2.87x
unified_kv_triton	mtp	1	4	1107.36	1.72x	398.36	0.91x	7.74	1.76x
unified_kv_triton	off	8	32	3801.30	1.00x	1660.70	1.00x	17.33	1.00x
unified_kv_triton	dspark	8	32	7796.98	2.05x	1016.54	1.63x	7.51	2.31x
unified_kv_triton	mtp	8	32	5356.22	1.41x	819.82	2.03x	11.97	1.45x
unified_kv_triton	off	16	64	6154.82	1.00x	3027.38	1.00x	20.45	1.00x
unified_kv_triton	dspark	16	64	10826.17	1.76x	1365.23	2.22x	11.19	1.83x
unified_kv_triton	mtp	16	64	7953.92	1.29x	1234.07	2.45x	16.21	1.26x
unified_kv_triton	off	32	128	9164.07	1.00x	5676.91	1.00x	25.89	1.00x
unified_kv_triton	dspark	32	128	14753.85	1.61x	2132.05	2.66x	16.65	1.55x
unified_kv_triton	mtp	32	128	11496.34	1.25x	2064.64	2.75x	22.30	1.16x

Serving Bench - TP8DP8 - 8k/1k

Backend	Spec mode	Conc	NP	TTT (tok/s)	TTT speedup	Mean TTFT (ms)	TTFT speedup	Mean TPOT (ms)	TPOT speedup
triton	off	256	1024	25656.63	1.00x	23353.01	1.00x	66.37	1.00x
triton	dspark	256	1024	31616.95	1.23x	9551.34	2.44x	61.28	1.08x
triton	mtp	256	1024	30916.10	1.20x	8549.94	2.73x	64.50	1.03x
unified_kv_triton	off	256	1024	29294.55	1.00x	18427.91	1.00x	60.05	1.00x
unified_kv_triton	dspark	256	1024	40089.16	1.37x	7607.62	2.42x	47.18	1.27x
unified_kv_triton	mtp	256	1024	35258.91	1.20x	6852.99	2.69x	56.63	1.06x

Accept length(4.84) as Expected(5) at TP8DP8 conc256 unified_kv_triton

============ Serving Benchmark Result ============
Backend:                                 sglang    
Traffic request rate:                    inf       
Max request concurrency:                 256       
Successful requests:                     1024      
Benchmark duration (s):                  235.40    
Total input tokens:                      8388608   
Total input text tokens:                 8388608   
Total generated tokens:                  1048576   
Total generated tokens (retokenized):    1048493   
Request throughput (req/s):              4.35      
Input token throughput (tok/s):          35634.81  
Output token throughput (tok/s):         4454.35   
Peak output token throughput (tok/s):    17657.00  
Peak concurrent requests:                347       
Total token throughput (tok/s):          40089.16  
Concurrency:                             243.06    
Accept length:                           4.84      
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   55876.00  
Median E2E Latency (ms):                 53257.66  
P90 E2E Latency (ms):                    80128.68  
P95 E2E Latency (ms):                    88712.88  
P99 E2E Latency (ms):                    97474.48  
---------------Time to First Token----------------
Mean TTFT (ms):                          7607.62   
Median TTFT (ms):                        2861.56   
P90 TTFT (ms):                           24705.18  
P95 TTFT (ms):                           31722.26  
P99 TTFT (ms):                           37251.77  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          47.18     
Median TPOT (ms):                        46.70     
P90 TPOT (ms):                           66.93     
P95 TPOT (ms):                           72.46     
P99 TPOT (ms):                           86.07     
---------------Inter-Token Latency----------------
Mean ITL (ms):                           47.18     
Median ITL (ms):                         14.61     
P90 ITL (ms):                            160.60    
P95 ITL (ms):                            209.16    
P99 ITL (ms):                            411.06    
Max ITL (ms):                            26673.19  
==================================================

Server cmd

export SGLANG_DEFAULT_THINKING=1
export SGLANG_DSV4_REASONING_EFFORT=max
export SGLANG_OPT_DEEPGEMM_HC_PRENORM=false
export SGLANG_USE_AITER=1
export SGLANG_USE_ROCM700A=${SGLANG_USE_ROCM700A:-0}
export SGLANG_OPT_USE_FUSED_COMPRESS=true
export SGLANG_HACK_FLASHMLA_BACKEND=${SGLANG_HACK_FLASHMLA_BACKEND:-unified_kv_triton}
export SGLANG_OPT_FP8_WO_A_GEMM=false
export SGLANG_OPT_USE_JIT_INDEXER_METADATA=false
export SGLANG_OPT_USE_TOPK_V2=false
export SGLANG_OPT_USE_AITER_INDEXER=true
export SGLANG_OPT_USE_TILELANG_INDEXER=false
export SGLANG_OPT_USE_TILELANG_MHC_PRE=false
export SGLANG_OPT_USE_TILELANG_MHC_POST=false
export SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1
export SGLANG_OPT_USE_FUSED_COMPRESS_TRITON=true
export SGLANG_OPT_USE_MULTI_STREAM_OVERLAP=false
export SGLANG_ROCM_USE_MULTI_STREAM=false
export AITER_BF16_FP8_MOE_BOUND=0
export SGLANG_EAGER_INPUT_NO_COPY=true
export SGLANG_SHARED_EXPERT_TP1=1
export SGLANG_DP_SHARED_EXPERT_LOCAL=1
export SGLANG_DP_USE_GATHERV=1
export SGLANG_DP_USE_REDUCE_SCATTER=1
export GPU_MAX_HW_QUEUES=5

DSPARK=${DSPARK:-1}
if [ "${DSPARK}" = "1" ]; then
    echo "DSpark is enabled"
    MODEL=${MODEL:-/mnt/raid0/pretrained_model/deepseek-ai/DeepSeek-V4-Pro-DSpark}
    export SGLANG_RAGGED_VERIFY_MODE=${SGLANG_RAGGED_VERIFY_MODE:-static}
    export SGLANG_DSPARK_ENABLE_SPS_ONLINE_PROFILE=${SGLANG_DSPARK_ENABLE_SPS_ONLINE_PROFILE:-0}
    SPEC_ARGS="--speculative-algorithm DSPARK --speculative-dspark-block-size 5"
    if [ -n "${SPS_TABLE}" ]; then
        SPEC_ARGS="${SPEC_ARGS} --speculative-dspark-sps-table-path ${SPS_TABLE}"
    fi
else
    echo "DSpark is disabled, MTP is enabled"
    MODEL=${MODEL:-/mnt/data/pretrained_model/deepseek-ai/DeepSeek-V4-Pro}
    MTP_NUM_STEPS=${MTP_NUM_STEPS:-3}
    MTP_TOPK=${MTP_TOPK:-1}
    MTP_NUM_DRAFT_TOKENS=${MTP_NUM_DRAFT_TOKENS:-4}
    SPEC_ARGS="--speculative-algorithm EAGLE --speculative-num-steps ${MTP_NUM_STEPS} --speculative-eagle-topk ${MTP_TOPK} --speculative-num-draft-tokens ${MTP_NUM_DRAFT_TOKENS}"
fi

DP=${DP:-8}
if [ "${DP}" -gt 1 ]; then
    DP_ARGS="--dp ${DP} --enable-dp-attention --enable-dp-lm-head"
    PD_ARGS="--enable-prefill-delayer"
    CHUNKED_PREFILL=65536
else
    DP_ARGS=""
    PD_ARGS=""
    CHUNKED_PREFILL=8192
fi

MAX_RUNNING=${MAX_RUNNING:-$((64 * DP))}

echo "DP_ARGS: ${DP_ARGS}"
echo "PD_ARGS: ${PD_ARGS}"
echo "CHUNKED_PREFILL: ${CHUNKED_PREFILL}"
echo "MAX_RUNNING: ${MAX_RUNNING}"
echo "MODEL: ${MODEL}"
echo "SGLANG_HACK_FLASHMLA_BACKEND: ${SGLANG_HACK_FLASHMLA_BACKEND}"
echo "SPEC_ARGS: ${SPEC_ARGS}"

sglang serve \
    --model-path ${MODEL} \
    --trust-remote-code \
    --tp 8 \
    ${DP_ARGS} \
    ${PD_ARGS} \
    --disable-radix-cache \
    --attention-backend dsv4 \
    --page-size 256 \
    --mem-fraction-static 0.9 \
    --swa-full-tokens-ratio 0.15 \
    --disable-shared-experts-fusion \
    --tool-call-parser deepseekv4 \
    --reasoning-parser deepseek-v4 \
    --kv-cache-dtype fp8_e4m3 \
    --chunked-prefill-size ${CHUNKED_PREFILL} \
    --cuda-graph-max-bs 512 \
    --max-running-requests ${MAX_RUNNING} \
    ${SPEC_ARGS}

Client cmd

#!/bin/bash

# ===== Default parameters =====
INPUT_LEN=${1:-8192}
OUTPUT_LEN=${2:-1024}
ENABLE_PROFILE=${3:-0}   # 1 = enable profile, 0 = disable

# ===== Timestamp =====
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")

echo "INPUT_LEN=${INPUT_LEN}"
echo "OUTPUT_LEN=${OUTPUT_LEN}"
echo "PROFILE=${ENABLE_PROFILE}"
echo "TIMESTAMP=${TIMESTAMP}"

python3 /sgl-workspace/sglang/benchmark/gsm8k/bench_sglang.py --num-questions 1319 --parallel 512

for concurrency in 256
do
    prompt=$((concurrency * 4))
    warmup=$((concurrency * 2))
    LOG_FILE="mi355_${INPUT_LEN}_${OUTPUT_LEN}_tp8_c-${concurrency}_${TIMESTAMP}.log"

    CMD="python3 -m sglang.bench_serving \
        --dataset-name random \
        --random-input ${INPUT_LEN} \
        --random-output ${OUTPUT_LEN} \
        --random-range-ratio 1 \
        --max-concurrency ${concurrency} \
        --num-prompt ${prompt} \
        --warmup-requests ${warmup}"

    # ===== Optional profile =====
    if [ "${ENABLE_PROFILE}" -eq 1 ]; then
        CMD="${CMD} --profile --profile-num-steps 4"
    fi

    echo "Running: ${CMD}"
    echo "Log: ${LOG_FILE}"

    eval ${CMD} 2>&1 | tee ${LOG_FILE}
done

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ✅ Run #31089093009
Latest PR Test (Extra): ❌ Run #31089091879

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[michaelzhang-ai]

@HaiShaw @At1a8 I dispatched the nightly suite on our MI35x CI runners so we get a CI-side signal on top of the author's local run.

Run: https://github.com/sgl-project/sglang/actions/runs/31291406552

• Suite: nightly-amd-8-gpu-mi35x-deepseek-v4-pro-dspark (test/registered/amd/test_deepseek_v4_pro_fp4_dspark.py)
• Runner: linux-mi35x-gpu-8, ROCm 7.2 container, SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton
• Code under test: current main (i.e. post-merge of this PR)
• Gates: full GSM8K (1319 questions) accuracy > 0.92 and DSpark avg_spec_accept_length > 3.0

One gap worth flagging: the suite registered here isn't referenced by any job in .github/workflows/nightly-test-amd.yml or nightly-test-amd-rocm720.yml, so as of main today it never actually runs on the nightly schedule — the register_amd_ci(..., nightly=True) call alone doesn't get it scheduled. To dispatch it I had to add a nightly-8-gpu-mi35x-deepseek-v4-pro-dspark-rocm720 job (sibling of the existing ...-deepseek-v4-pro-rocm720 job), which I've sent as #34147. Until that lands, the coverage this PR added stays dormant.

I'll report accuracy and acceptance-length numbers back here once the run finishes.

[At1a8]

@HaiShaw @At1a8 I dispatched the nightly suite on our MI35x CI runners so we get a CI-side signal on top of the author's local run.

Run: https://github.com/sgl-project/sglang/actions/runs/31291406552

• Suite: nightly-amd-8-gpu-mi35x-deepseek-v4-pro-dspark (test/registered/amd/test_deepseek_v4_pro_fp4_dspark.py)
• Runner: linux-mi35x-gpu-8, ROCm 7.2 container, SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton
• Code under test: current main (i.e. post-merge of this PR)
• Gates: full GSM8K (1319 questions) accuracy > 0.92 and DSpark avg_spec_accept_length > 3.0

One gap worth flagging: the suite registered here isn't referenced by any job in .github/workflows/nightly-test-amd.yml or nightly-test-amd-rocm720.yml, so as of main today it never actually runs on the nightly schedule — the register_amd_ci(..., nightly=True) call alone doesn't get it scheduled. To dispatch it I had to add a nightly-8-gpu-mi35x-deepseek-v4-pro-dspark-rocm720 job (sibling of the existing ...-deepseek-v4-pro-rocm720 job), which I've sent as #34147. Until that lands, the coverage this PR added stays dormant.

I'll report accuracy and acceptance-length numbers back here once the run finishes.

@HaiShaw @michaelzhang-ai
Accuracy: 0.951
avg_spec_accept_length=3.7301708747768427
https://github.com/sgl-project/sglang/actions/runs/31291406552/job/93189109485

[michaelzhang-ai]

Results are in — the nightly suite passed on our MI35x CI runners. @HaiShaw @At1a8

Run: 31291406552 → nightly-8-gpu-mi35x-deepseek-v4-pro-dspark-rocm720, success. Test Summary: 1/1 passed, Ran 3 tests in 2906.133s — the two unified-KV kernel unit tests plus the full GSM8K run.

Metric	Value	Gate
GSM8K accuracy (1319 questions, 5-shot)	0.9515	> 0.92
DSpark avg_spec_accept_length	3.730	> 3.0
Invalid outputs	0.0	—
Eval latency	39.45s	—
Output throughput	2998 tok/s	—

Configuration was linux-mi35x-gpu-8, ROCm 7.2, SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton, against main at f6cbdc1, i.e. post-merge of this PR. That covers the merge-verdict item asking for the suite to be run on MI35x with unified_kv_triton and GSM8K accuracy plus DSpark acceptance confirmed.

Still open is the scheduling gap: this only ran because I added the job on a branch. #34147 lands that job on main so the suite runs nightly instead of on demand.