[amosyou]

Motivation

Adds support for HArmonized Speculative Sampling (HASS), ICLR 2025 paper: https://arxiv.org/abs/2408.15766

Modifications

• Add HASS speculative method to server args
• Inherit draft model architecture from EAGLE
• Test for HASSEngine

Benchmarks

using test script from #2150

Autoregressive (normal)

python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --mem-fraction 0.7 --dtype float16 --port 30000
speed: 50.70 token/s

EAGLE-2

python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --speculative-algo EAGLE --speculative-draft lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --cuda-graph-max-bs 1 --mem-fraction 0.7 --dtype float16 --port 30000
speed: 84.80 token/s

HASS

python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --speculative-algo HASS --speculative-draft amosyou/sglang-HASS-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --cuda-graph-max-bs 1 --mem-fraction 0.7 --dtype float16 --port 30000
speed: 91.10 token/s

---

running Engine unit tests under test/

EAGLE-3

acc_length=2.7675675675675677

HASS

acc_length=3.710144927536232

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[merrymercy]

if the implementation is the same, we do not need to change any code