DeepSeek-V4 support disaggregation-decode-enable-radix-cache and support MTP

[zhangxiaolei123456]

Motivation

Co-authored-by: TobyMint

Modifications

Prefill
MC_SLICE_SIZE=262144 MC_NUM_QP_PER_EP=2 MC_WORKERS_PER_CTX=2 SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 SGLANG_DISAGGREGATION_QUEUE_SIZE=4 NCCL_SOCKET_IFNAME=eth0  NCCL_IB_DISABLE=0 SGLANG_DSV4_FP4_EXPERTS=1 SGLANG_OPT_USE_ONLINE_COMPRESS=1 GLOO_SOCKET_IFNAME=eth0 python3 -m sglang.launch_server --model-path /data00/models/DeepSeek-V4-Flash --tokenizer-path /data00/models/DeepSeek-V4-Flash --host 0.0.0.0 --port 30000 --trust-remote-code --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.9 --max-running-requests 64 --chunked-prefill-size 8192 --max-prefill-tokens 16384 --tp-size 2 --pp-size 4 --attn-cp-size 2 --attention-backend dsv4 --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4 --disable-overlap-schedule --disable-piecewise-cuda-graph --enable-nsa-prefill-context-parallel --disaggregation-mode prefill --disaggregation-transfer-backend mooncake --nsa-prefill-cp-mode round-robin-split --enable-metrics --disaggregation-ib-device mlx5_1,mlx5_2,mlx5_3,mlx5_4 --moe-runner-backend flashinfer_mxfp4 --tokenizer-worker-num 8

Decode
MC_SLICE_SIZE=262144 MC_NUM_QP_PER_EP=2 MC_WORKERS_PER_CTX=2   SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 SGLANG_EXPERIMENTAL_DSV4_DECODE_RADIX_CACHE=1 SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600 SGLANG_DISAGGREGATION_QUEUE_SIZE=4 SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=256 SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 SGLANG_OPT_FIX_MEGA_MOE_MEMORY=1 SGLANG_OPT_USE_ONLINE_COMPRESS=1 SGLANG_EXPERIMENTAL_ONLINE_C128_MTP=1 SGLANG_DSV4_FP4_EXPERTS=0 SGLANG_DEFAULT_THINKING=1 SGLANG_OPT_DEEPGEMM_HC_PRENORM=1 SGLANG_OPT_USE_TILELANG_MHC_PRE=1 SGLANG_JIT_DEEPGEMM_PRECOMPILE=1 GLOO_SOCKET_IFNAME=eth0 NCCL_MIN_NCHANNELS=24 NCCL_IB_QPS_PER_CONNECTION=8  sglang serve --trust-remote-code --model-path /data00/models/DeepSeek-V4-Flash-FP8 --tp 8 --dp-size 8 --enable-dp-attention --cuda-graph-max-bs 80 --max-running-requests 640 --enable-metrics --host 0.0.0.0 --port 30000 --mem-fraction-static 0.9 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --moe-runner-backend deep_gemm --moe-a2a-backend deepep --deepep-mode low_latency --disaggregation-mode decode  --disaggregation-ib-device "mlx5_1,mlx5_2,mlx5_3,mlx5_4"  --enable-dp-lm-head --load-balance-method round_robin --disaggregation-transfer-backend mooncake --disaggregation-decode-enable-radix-cache --tokenizer-worker-num 8 --speculative-algo EAGLE --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3

Accuracy Tests

python3 bench_sglang.py --host http://localhost  --port 8090 --data-path /data00 --num-questions 5000 --parallel 100
100%|███████████████████████| 1319/1319 [00:29<00:00, 44.59it/s]
Accuracy: 0.933
Invalid: 0.000
Latency: 29.585 s
Output throughput: 4091.809 token/s
python bench_sglang.py --parallel 128 --backend srt --host http://127.0.0.1 --port 8090 --data_dir /data00/mmlu
100%|█████████████████████| 14042/14042 [04:27<00:00, 52.56it/s]
subject: abstract_algebra, #q:100, acc: 0.870
subject: anatomy, #q:135, acc: 0.889
subject: astronomy, #q:152, acc: 0.947
subject: business_ethics, #q:100, acc: 0.830
subject: clinical_knowledge, #q:265, acc: 0.891
subject: college_biology, #q:144, acc: 0.979
subject: college_chemistry, #q:100, acc: 0.680
subject: college_computer_science, #q:100, acc: 0.910
subject: college_mathematics, #q:100, acc: 0.870
subject: college_medicine, #q:173, acc: 0.850
subject: college_physics, #q:102, acc: 0.961
subject: computer_security, #q:100, acc: 0.840
subject: conceptual_physics, #q:235, acc: 0.962
subject: econometrics, #q:114, acc: 0.816
subject: electrical_engineering, #q:145, acc: 0.890
subject: elementary_mathematics, #q:378, acc: 0.958
subject: formal_logic, #q:126, acc: 0.762
subject: global_facts, #q:100, acc: 0.770
subject: high_school_biology, #q:310, acc: 0.965
subject: high_school_chemistry, #q:203, acc: 0.877
subject: high_school_computer_science, #q:100, acc: 0.950
subject: high_school_european_history, #q:165, acc: 0.897
subject: high_school_geography, #q:198, acc: 0.960
subject: high_school_government_and_politics, #q:193, acc: 0.995
subject: high_school_macroeconomics, #q:390, acc: 0.933
subject: high_school_mathematics, #q:270, acc: 0.811
subject: high_school_microeconomics, #q:238, acc: 0.971
subject: high_school_physics, #q:151, acc: 0.868
subject: high_school_psychology, #q:545, acc: 0.960
subject: high_school_statistics, #q:216, acc: 0.912
subject: high_school_us_history, #q:204, acc: 0.936
subject: high_school_world_history, #q:237, acc: 0.932
subject: human_aging, #q:223, acc: 0.865
subject: human_sexuality, #q:131, acc: 0.931
subject: international_law, #q:121, acc: 0.934
subject: jurisprudence, #q:108, acc: 0.917
subject: logical_fallacies, #q:163, acc: 0.920
subject: machine_learning, #q:112, acc: 0.839
subject: management, #q:103, acc: 0.922
subject: marketing, #q:234, acc: 0.966
subject: medical_genetics, #q:100, acc: 0.990
subject: miscellaneous, #q:783, acc: 0.957
subject: moral_disputes, #q:346, acc: 0.882
subject: moral_scenarios, #q:895, acc: 0.801
subject: nutrition, #q:306, acc: 0.938
subject: philosophy, #q:311, acc: 0.920
subject: prehistory, #q:324, acc: 0.941
subject: professional_accounting, #q:282, acc: 0.855
subject: professional_law, #q:1534, acc: 0.729
subject: professional_medicine, #q:272, acc: 0.938
subject: professional_psychology, #q:612, acc: 0.926
subject: public_relations, #q:110, acc: 0.818
subject: security_studies, #q:245, acc: 0.882
subject: sociology, #q:201, acc: 0.965
subject: us_foreign_policy, #q:100, acc: 0.960
subject: virology, #q:166, acc: 0.578
subject: world_religions, #q:171, acc: 0.924
Total latency: 267.192
Average accuracy: 0.884
 python -m sglang.test.run_eval --port 8090 --eval-name gpqa --num-examples 32 --max-tokens 128000 --repeat 8 --top-p 0.95 --temperature 1.0 --thinking-mode deepseek-v3
Repeat: 8, mean: 0.898
Scores: ['0.938', '0.906', '0.875', '0.938', '0.906', '0.875', '0.844', '0.906']
Mean latency: 871.151 s██▎      | 24/32 [15:23<05:00, 37.50s/it]
====================
Output throughput: 437.961 token/s
[METRIC] gpqa_mean_score=0.8984375 labels={"model": "/data00/models/DeepSeek-V4-Flash-FP8", "eval": "gpqa", "repeat": 8}
Writing report to /tmp/gpqa__data00_models_DeepSeek-V4-Flash-FP8.html
{'chars': np.float64(1435.46875), 'chars:std': np.float64(446.60056149028463), 'score:std': np.float64(0.2914805954090255), 'scores': ['0.938', '0.906', '0.875', '0.938', '0.906', '0.875', '0.844', '0.906'], 'mean_score': np.float64(0.8984375), 'latency': 871.1512180131394, 'output_throughput': 437.9608466486073}
Writing results to /tmp/gpqa__data00_models_DeepSeek-V4-Flash-FP8.json

Speed Tests and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[dongyibo]

@zhangxiaolei123456 Hello, does this feature support dspark? Or is there a plan to support it?