Add AMD support for DeepSeek V4

[AgainstEntropy]

Docker image

lmsysorg/sglang:v0.5.8-rocm700-mi35x-dsk

To build a docker image yourself, please reference to the comment from @yushengsu-thu below.

Launch command

export CUDA_VISIBLE_DEVICES=0,1,2,3

export SGLANG_OPT_USE_FUSED_COMPRESS=false #use PyTorch implemented compressor
export SGLANG_OPT_USE_OLD_COMPRESSOR=true #use old compressor
export SGLANG_OPT_USE_TILELANG_SWA_PREPARE=false #use old prepare
export SGLANG_OPT_USE_JIT_KERNEL_FUSED_TOPK=false #use old topk
export SGLANG_OPT_USE_FUSED_HASH_TOPK=false #AMD: hash_topk JIT needs CUDA toolchain

export SGLANG_HACK_FLASHMLA_BACKEND=torch
export SGLANG_OPT_DEEPGEMM_HC_PRENORM=false #use old prenorm

export SGLANG_OPT_USE_TILELANG_MHC_PRE=false #use torch hc_pre
export SGLANG_OPT_USE_TILELANG_MHC_POST=false #use torch hc_post

export SGLANG_ENABLE_THINKING=1
export SGLANG_USE_AITER=1
export SGLANG_USE_ROCM700A=1
export SGLANG_TOPK_TRANSFORM_512_TORCH=1
export SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1

export SGLANG_DSV4_FP4_EXPERTS=false

export SGLANG_OPT_DPSK_V4_RADIX=0
export SGLANG_OPT_USE_OVERLAP_STORE_CACHE=false #non-radix backend has no store_cache method
export SGLANG_OPT_USE_FUSED_STORE_CACHE=false #fused_store_cache JIT needs CUDA toolchain

export SGLANG_FORCE_TRITON_MOE_FP8=1  # this is required to apply swiglu_limit clamp in fused_moe_triton

python3 -m sglang.launch_server \
    --model-path sgl-project/DeepSeek-V4-Flash-FP8 \
    --trust-remote-code \
    --tp 4 \
    --dp 4 \
    --enable-dp-attention \
    --disable-radix-cache \
    --attention-backend compressed \
    --max-running-request 256 \
    --page-size 256 \
    --chunked-prefill-size 8192 \
    --port 30010 \
    --disable-shared-experts-fusion \
    --disable-cuda-graph \
    --tool-call-parser deepseekv4 \
    --reasoning-parser deepseek-v4

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[HaiShaw]

Integration branch:
https://github.com/sgl-project/sglang/tree/amd/deepseek_v4

[HaiShaw]

SGLang daily image for AMD deepseek_v4 is posted at:
https://hub.docker.com/r/rocm/sgl-dev/tags
TAG ending with -DSv4

[tawan0109]

@HaiShaw @AgainstEntropy @jhinpan any plan to support mi300x? we have a good amount of mi300x and want to use them for dsv4.