[Deepseek V3.2] Support Overlap Spec + NSA

[b8zhong]

Motivation

Part of V3.2 Roadmap #15025

Enable overlap spec and EAGLE + NSA backend.

SGLANG_ENABLE_SPEC_V2=1 python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3.2 --trust-remote-code --tp 8 --speculative-algorithm EAGLE

Modifications

In EAGLE V1, we had (with python3 -m sglang.test.send_one --stream --max-new-tokens 1024)

+-------------+--------+------------+-----------------+
| Latency (s) | Tokens | Acc Length | Speed (token/s) |
+-------------+--------+------------+-----------------+
|    7.922    |  1024  |   2.960    |     129.26      |
+-------------+--------+------------+-----------------+

After simply adding in the guards for include_v2=True, we had a slowdown to:

+-------------+--------+------------+-----------------+
| Latency (s) | Tokens | Acc Length | Speed (token/s) |
+-------------+--------+------------+-----------------+
|    9.015    |  1024  |   2.960    |     113.59      |
+-------------+--------+------------+-----------------+

After profiling, we were able to find the root cause:

EAGLE V1

EAGLE V2 (before code change)

When we use extend_seq_lens_cpu, it will cause an unneeded sync.

EAGLE V2 (after code change)

It will increase to:

+-------------+--------+------------+-----------------+
| Latency (s) | Tokens | Acc Length | Speed (token/s) |
+-------------+--------+------------+-----------------+
|    7.306    |  1024  |   2.960    |     140.17      |
+-------------+--------+------------+-----------------+

So it is around 8%.

Accuracy Tests

python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319

Before:

python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [02:35<00:00,  8.46it/s]
Accuracy: 0.946
Invalid: 0.000
Latency: 156.114 s
Output throughput: 817.299 token/s

python3 benchmark/gsm8k/bench_sglang.py --num-shots 20 --num-questions 1319 --parallel 1319
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [02:23<00:00,  9.20it/s]
Accuracy: 0.955
Invalid: 0.000
Latency: 147.938 s
Output throughput: 868.237 token/s

After:

python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [01:56<00:00, 11.31it/s]
Accuracy: 0.948
Invalid: 0.000
Latency: 116.809 s
Output throughput: 1112.259 token/s

python3 benchmark/gsm8k/bench_sglang.py --num-shots 20 --num-questions 1319 --parallel 1319
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [01:33<00:00, 14.16it/s]
Accuracy: 0.954
Invalid: 0.000
Latency: 93.615 s
Output throughput: 1390.591 token/s

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[b8zhong]

/tag-and-rerun-ci again?

[Fridge003]

@b8zhong That's really good result!
Can you also test some accuracy benchmarks, like gsm8k or gpqa?
https://docs.sglang.io/basic_usage/deepseek_v32.html#accuracy-test-with-gpqa-diamond

[Fridge003]

Nice job!

[hzh0425]

@b8zhong

Hi, could you please share your launch command?

I get an error when setting export SGLANG_ENABLE_SPEC_V2=1.

[b8zhong]

@hzh0425 Sure. It's:

SGLANG_ENABLE_SPEC_V2=1 python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3.2 --trust-remote-code --tp 8 --speculative-algorithm EAGLE

By the way, I did not use DP attention.