[AMD][SPEC]Fix DeepSeek-V4 benchmark accuracy gap by supporting temperature sampling in EAGLE verify

[At1a8]

This PR co-work with @yuttian1 and @amd-danli103

Motivation

On ROCm, EAGLE verify silently ignores the requested temperature.

In eagle_sample, _is_hip and is_all_greedy use the same path, so every HIP request takes the argmax branch — a workaround for the unregistered tree_speculative_sampling_target_only kernel that became a behaviour change with no error or warning.

Greedy then falls into repetition loops on open-ended tasks: on Simple-QA 1323 of 4326 requests (30.6%) end without a parseable answer, versus 39 (0.9%) once temperature is honoured — most of the 8.97-point gap below.

Modifications

Add select_target_predict(), which returns the target model's own token at each verify position:

• feature disabled by default, or the request is greedy → argmax(logits) (unchanged behaviour)
• otherwise(enable by SGLANG_SPEC_TEMPERATURE_SAMPLING_TARGET_VERIFY, hip gated) → y ~ softmax(logits / temperature)

verify_tree_greedy_func accepts a draft token only when it equals this choice, so the committed token is always the target's own — argmax reproduces greedy exactly, sampling reproduces temperature sampling exactly. Draft probabilities are never used.

Gated by SGLANG_SPEC_TEMPERATURE_SAMPLING_TARGET_VERIFY (default off) and _is_hip; other backends(nv) unchanged. top_p/top_k fall back to argmax with a warning (renorm ops unregistered on ROCm). Accept length stays at 2.5+/4.

Accuracy Tests

Benchmark	Official	MTP314 (target greedy)	Gap to official (target greedy)	MTP314 (target temp sampling)	Gap to official (target temp sampling)	Gain
GSM8K (1319)	92.6	96.51	−3.91	96.97	−4.37	+0.46
AA-LCR (100)	66.3	70.00	−3.70	75.00	−8.70	+5.00
GPQA-Diamond (198)	90.1	84.34	5.76	87.88	2.22	+3.54
Simple-QA (4326)	57.9	48.61	9.29	57.58	0.32	+8.97
LiveCodeBench (1055)	93.5	87.20	6.30	91.09	2.41	+3.89
MMLU-Pro (12032)	87.5	86.34	1.16	87.41	0.09	+1.07
HLE (2158)	37.7	30.26	7.44	33.78	3.92	+3.52

Official data copy from: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro
Gap to official (target greedy) = Official - MTP314 (target greedy)
Gap to official (target temp sampling) = Official - MTP314 (target temp sampling)
These data measured on gfx950.

Speed Tests and Profiling

No significant performance difference between two different sampling way.

metric	greedy sample by default	enable this PR by SGLANG_SPEC_TEMPERATURE_SAMPLING_TARGET_VERIFY=1	Δ
Total token throughput (tok/s)	14199.94	14090.88	−0.77%
Accept length	2.8633	2.8485	−0.52%
Mean TTFT (ms)	1994.81	1994.83	+0.00%
Mean TPOT (ms)	17.76	17.90	+0.79%
Mean ITL (ms)	17.76	17.90	+0.79%

Server command

SPEC=${SPEC:-MTP}
case "${SPEC}" in
    DSPARK)
        echo "SPEC=DSPARK: DSpark is enabled"
        MODEL=${MODEL:-/mnt/raid0/pretrained_model/deepseek-ai/DeepSeek-V4-Pro-DSpark}
        export SGLANG_RAGGED_VERIFY_MODE=${SGLANG_RAGGED_VERIFY_MODE:-static}
        export SGLANG_DSPARK_ENABLE_SPS_ONLINE_PROFILE=${SGLANG_DSPARK_ENABLE_SPS_ONLINE_PROFILE:-0}
        SPEC_ARGS="--speculative-algorithm DSPARK --speculative-dspark-block-size 5"
        if [ -n "${SPS_TABLE}" ]; then
            SPEC_ARGS="${SPEC_ARGS} --speculative-dspark-sps-table-path ${SPS_TABLE}"
        fi
        ;;
    MTP)
        echo "SPEC=MTP: MTP is enabled"
        MODEL=${MODEL:-/mnt/data/pretrained_model/deepseek-ai/DeepSeek-V4-Pro}
        MTP_NUM_STEPS=${MTP_NUM_STEPS:-3}
        MTP_TOPK=${MTP_TOPK:-1}
        MTP_NUM_DRAFT_TOKENS=${MTP_NUM_DRAFT_TOKENS:-4}
        SPEC_ARGS="--speculative-algorithm EAGLE --speculative-num-steps ${MTP_NUM_STEPS} --speculative-eagle-topk ${MTP_TOPK} --speculative-num-draft-tokens ${MTP_NUM_DRAFT_TOKENS}"
        ;;
    None|NONE|none)
        echo "SPEC=None: speculative decoding is disabled"
        MODEL=${MODEL:-/mnt/data/pretrained_model/deepseek-ai/DeepSeek-V4-Pro}
        unset SGLANG_RAGGED_VERIFY_MODE
        SPEC_ARGS=""
        ;;
    *)
        echo "Unsupported SPEC=${SPEC}. Expected one of: DSPARK, MTP, None"
        exit 1
        ;;
esac

TBO_ARGS=""
[ "${TBO:-0}" = "1" ] && TBO_ARGS="--enable-two-batch-overlap"

echo "MODEL: ${MODEL}"
echo "SPEC: ${SPEC}"
echo "SPEC_ARGS: ${SPEC_ARGS}"
echo "SGLANG_RAGGED_VERIFY_MODE: ${SGLANG_RAGGED_VERIFY_MODE}"
echo "SGLANG_SPEC_TEMPERATURE_SAMPLING_TARGET_VERIFY: ${SGLANG_SPEC_TEMPERATURE_SAMPLING_TARGET_VERIFY:-<unset>}"
echo "TBO_ARGS: ${TBO_ARGS}"

SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_SHARED_EXPERT_TP1=1 \
SGLANG_DP_SHARED_EXPERT_LOCAL=1 \
SGLANG_DP_USE_GATHERV=1 \
SGLANG_DP_USE_REDUCE_SCATTER=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
AITER_BF16_FP8_MOE_BOUND=0 \
SGLANG_OPT_USE_AITER_BATCHED_GEMM=true \
sglang serve \
  --trust-remote-code \
  --model-path ${MODEL} \
  --tp 8 \
  --dp 8 \
  --enable-dp-attention \
  --enable-dp-attention-local-control-broadcast \
  --tokenizer-worker-num 8 \
  --stream-interval 20 \
  --prefill-decode-interval 10 \
  ${TBO_ARGS} \
  --attention-backend dsv4 \
  --page-size 256 \
  --mem-fraction-static 0.9 \
  --swa-full-tokens-ratio 0.15 \
  --enforce-shared-experts-fusion \
  --kv-cache-dtype fp8_e4m3 \
  --chunked-prefill-size 65536 \
  --enable-dp-lm-head \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --disable-radix-cache \
  --disable-custom-all-reduce \
  ${SPEC_ARGS} \
  --host 0.0.0.0 \
  --port 30000 \
  ${EXTRA_ARGS:-}

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): 🚫 Run #34818468430
Latest PR Test (Extra): ❌ Run #34818468496
Latest PR Test (AMD ROCm 10): ❌ Run #34818468812

[HaiShaw]

What traffic is used for tree_speculative_sampling_target_only?
Able to do a single kernel solution?

[HaiShaw]

Please do:

elif _is_hip:
    target_predict = select_target_predict(
        next_token_logits, sampling_info, verify_input.draft_token_num
    )

[At1a8]

Done.

[At1a8]

What traffic is used for tree_speculative_sampling_target_only? Able to do a single kernel solution?

Hi @HaiShaw , as far as I know, tree_speculative_sampling_target_only is not yet supported on HIP. We plan to add full support, but implementing the complete solution will take some time.

For now, I'd like to introduce a simpler implementation(this PR) as an initial step. Once the functionality is in place, we can further optimize and improve the implementation in follow-up work.

[At1a8]

@amd-bot ci-status

[bingxche]

@At1a8

CI Status for PR #39253

Merge verdict: Do not merge on green alone. PR CI is incomplete (NVIDIA base-c B200/B300 jobs were cancelled, XPU is still queued, and the Extra suites were gated off) and — more importantly — this PR's actual new behavior is not exercised by any PR-CI test. The 7 executed test failures (5 AMD, 1 NPU, 1 MLX) are all in unrelated subsystems and none touch the changed EAGLE verify path.

Caution

The new code path is not verified by CI. The change in eagle_utils.py is double-gated: HIP-only and behind the default-off env var SGLANG_SPEC_TEMPERATURE_SAMPLING_TARGET_VERIFY. No test or workflow sets that env var (verified by grep), so the new torch.softmax(logits/temperature) + multinomial verify path never runs in CI. With the env var off, select_target_predict() returns torch.argmax(...) — behaviorally identical to the old _is_hip branch — so even a green AMD EAGLE run only re-tests the old behavior. Before merge, the author must run the DeepSeek-V4 / Simple-QA accuracy check with SGLANG_SPEC_TEMPERATURE_SAMPLING_TARGET_VERIFY=1 on a ROCm box to prove the fix, since PR CI cannot.

Caution

Required NVIDIA downstream jobs did not complete: base-c-test-4-gpu-b200 (0–4) and base-c-test-8-gpu-b300 (1) were cancelled (run 34818468430), and stage-a-test-1-gpu-xpu is still queued. Those paths are untested — rerun/refresh the branch and confirm they finish before merge.

Changed files: python/sglang/srt/speculative/eagle_utils.py (+33/-1)

Executed CI failure attribution: AMD: 5 failures (0 related) · Others: 2 failures (0 related) · plus cancelled B200/B300 + queued XPU (incomplete, not counted)

AMD Executed Failures

Job	Test File	Test Function	Error	Related?	Why
stage-b-1gpu-small-mi35x	test/registered/amd/test_vattn_segplan_mi35x.py	test_* (vattn segplan)	AssertionError: torch.isnan(o_plan) / torch.equal false	🟢	vAttn segment-plan kernel numerics; diff only touches EAGLE spec-decode sampling
stage-b-large-8gpu-mi35x-disagg	test/registered/amd/disaggregation/test_disaggregation_pp.py, test_nixl_transfer_engine_e2e.py	setUp/e2e	KeyError: 'answer'; nixlBackendError NIXL_ERR_BACKEND, server exit -9	🟢	PD-disaggregation / NIXL transport; unrelated to verify path
stage-c-dsv4-flash-fp4-fp8-mi35x	test/registered/amd/test_deepseek_v4_flash_fp8.py, hicache/test_hicache_storage_umbp_backend.py	setUpClass	ValueError: SGLANG_OPT_FP8_WO_A_GEMM enabled but wo_a.weight is bf16, expected float8_e4m3fn	🟢	fp8 checkpoint/env mismatch at weight load — not a sampling change
stage-c-large-8gpu-mi35x	test/registered/amd/perf/mi35x/test_qwen35_fp8_ar_fusion_mi35x.py	perf	RuntimeError: vattn_asm hipModuleLaunchKernel failed: 709 (context is destroyed)	🟢	vAttn ASM kernel launch / context teardown; unrelated
stage-b-1gpu-small-amd (mi300, 11)	test/registered/attention/test_deterministic.py	test_*	AssertionError: result == 1; return_sampling_mask top_k errors	🟢	deterministic sampling-mask path; not EAGLE verify

Other Executed Failures

Job	Test File	Test Function	Error	Related?	Why
base-c-test-acc-16-npu-a3	test/registered/npu/accuracy/glm5_top64_pruned/test_npu_glm5_top64_pruned_bf16_8p_gsm8k.py	assert_metrics	AssertionError: 0.46 not >= 0.4752 (GSM8K accuracy)	🟢	NPU GLM5 accuracy threshold; NPU never hits the HIP-gated code
stage-a-unit-test-mlx	test/registered/unit/hardware_backend/mlx/test_scheduler_mixin.py	test_loop_exits_after_shutdown_req	Error in event_loop_overlap_mlx / ingest_requests mock	🟢	MLX (arm64) scheduler mixin unit test; unrelated to eagle_utils.py

The six *-finish failures (pr-test-finish, pr-test-npu-finish, pr-test-amd-finish, pr-test-mlx-finish, pr-test-extra-finish, pr-test-amd-extra-finish) are aggregation jobs cascading from the above; the two call-gate / pr-gate failures gated the Extra suites off (not run). None are independent failures.

Details / what to do before merge

• Verify the fix manually (highest priority): run the DeepSeek-V4 Simple-QA accuracy comparison with SGLANG_SPEC_TEMPERATURE_SAMPLING_TARGET_VERIFY=1 on ROCm. CI green proves nothing about this PR — the new sampling branch is unreachable without the env var. Consider adding a small registered AMD EAGLE test that sets the env var so the path gets real coverage.
• Complete CI signal: re-run / update-branch to get base-c B200/B300 (currently cancelled) and stage-a-test-1-gpu-xpu (queued) to a conclusion. For full signal despite unrelated early red, the bypass-fastfail label is an option (uses more CI, use sparingly).
• Ignore the red for attribution: all 7 executed failures are pre-existing/infra in vAttn, PD-disagg/NIXL, fp8-checkpoint, deterministic-mask, NPU-GLM5-accuracy, and MLX-scheduler subsystems — none reachable from the diff, and the default code path is behaviorally identical to before.

---

Generated by amd-bot using Claude Code CLI

[HaiShaw]

Please check #37134
It is largely an overlap, let me know.

[At1a8]

It is largely an overlap, let me know.

Confirmed that #37134 also fixes the accuracy issue in DeepSeek-V4 MTP. Thanks @HaiShaw @xiaobochen-amd

dataset	offcial(https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro)	#37134
GSM8K	92.6	96.44
AA-LCR	66.3	76.00
GPQA-Diamond	90.1	89.39
LongBench v2	51.5	53.28
Simple-QA	57.9	57.33
LiveCodeBench	93.5	90.43
MMLU-Pro	87.5	87.35
HLE	37.7	35.35