Repository navigation
[AMD][Fix] AgentX HIP TPOT regression when SGLANG_SIMULATE_ACC_LEN is set - #40598
Merged
ch-wan merged 1 commit intoSep 22, 2026
Merged
Conversation
…ATE_ACC_LEN is on
1am9trash
approved these changes
Sep 22, 2026
1am9trash
left a comment
Collaborator
There was a problem hiding this comment.
LGTM.
Skip sampling with fixed SGLANG_SIMULATE_ACC_LEN case (agentic workload).
Only impact on is_hip() code path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
This PR fixes an AgentX TPOT regression on HIP after #37134.
Agent mode (InferenceMax AgentX) sets
SGLANG_SIMULATE_ACC_LEN=3.39for Qwen3.5 (match-expected/real-draft-token). The server still runs a full EAGLE verify, then overwrites accept length. #37134 auto-enables--speculative-use-rejection-samplingon HIP, so every decode step also runs the Triton chain sampler; that result is then thrown away. Extra verify work shows up as ~14% worse AgentX TPOT (TP4 conc=4) vs pre-#37134 / a 0915 baseline.Fixed-length MTP does not set
SGLANG_SIMULATE_ACC_LEN, and the client is greedy (temperature=0), so verify stays onargmaxand does not hit this regression.Skip the HIP auto-enable when
SGLANG_SIMULATE_ACC_LEN > 0. Real eval / serving (env unset or< 0) still auto-enables as in #37134.Modifications
_should_auto_enable_hip_rejection_samplingnow takessimulate_acc_lenand returns false whensimulate_acc_len > 0._handle_eagle_familypassesenvs.SGLANG_SIMULATE_ACC_LENinto that gate.--speculative-use-rejection-samplingis unchanged (the helper already no-ops when the flag is already on).test_simulate_acc_len_stays_offintest/registered/unit/spec/test_eagle_gate_routing.py.Accuracy Tests
No new EVAL_ONLY / quality run. The
EVAL_ONLY=trueAgentX path does not setSGLANG_SIMULATE_ACC_LEN, so HIP still auto-enables rejection sampling as in #37134. This change only affects the simulated-accept throughput path, whose accept length is overwritten after verify. The new unit test covers the auto-enable predicate forsimulate_acc_len=3.39vs-1.0.Benchmarking and Profiling
InferenceX arm
qwen3.5-fp4-mi355x-sglang-agentic-mtp, recipebenchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh, TP4, conc=4,DURATION=1200. Same container (rocm/sgl-dev:v0.5.19-rocm720-mi35x-20260915), weights, GPUs, and recipe; only the SGLang SHA / this patch changes.832ec39cc(pre-#37134-era good)8d08dfdab(main, #37134 in tree)8d08dfdab+ this patchFirst-bad commit for the AgentX drop was
7eedd57ab(#37134). Parent2f7d04da5was still good.Checklist
Review Process
/tag-run-ci-label,/rerun-failed-ci,/tag-and-rerun-ciCI States
Latest PR Test (Base): 🚫 Run #35624210597
Latest PR Test (Extra): ✅ Run #35676026555
Latest PR Test (AMD ROCm 10): ⏳ Run #35624210574