[AMD] Gate DFLASH non-greedy verify on the target-only kernel being registered - #33694
Merged
Merged
Conversation
yctseng0211
requested review from
Qiaolin-Yu,
Ying1123,
hnyls2002 and
merrymercy
as code owners
August 5, 2026 10:10
Collaborator
Author
|
[stage-b-test-1-gpu-small-amd]
verified |
Collaborator
Author
|
@HaiShaw this PR only stops the crash by falling back to greedy verify on ROCm; the follow-up would be getting the target-only sampling kernel into the ROCm sgl-kernel build. |
Collaborator
|
@Raiden-Makoto is building the tree spec sampling kernel |
Collaborator
Author
@HaiShaw since this PR gates on the op being registered rather than on is_hip(), that kernel will flip DFLASH non-greedy verify back on automatically. |
HaiShaw
approved these changes
Aug 6, 2026
kfhfar
pushed a commit
to kfhfar/sglang
that referenced
this pull request
Aug 7, 2026
…registered (sgl-project#33694) Co-authored-by: HaiShaw <hixiao@gmail.com>
Xia-Weiwen
pushed a commit
to Xia-Weiwen/sglang
that referenced
this pull request
Aug 10, 2026
…registered (sgl-project#33694) Co-authored-by: HaiShaw <hixiao@gmail.com>
saturn-acc
pushed a commit
to saturn-acc/sglang
that referenced
this pull request
Aug 16, 2026
…registered (sgl-project#33694) Co-authored-by: HaiShaw <hixiao@gmail.com>
jakki-amd
pushed a commit
to jakki-amd/sglang
that referenced
this pull request
Sep 9, 2026
…registered (sgl-project#33694) Co-authored-by: HaiShaw <hixiao@gmail.com>
Atituiset
pushed a commit
to Atituiset/sglang
that referenced
this pull request
Sep 10, 2026
…registered (sgl-project#33694) Co-authored-by: HaiShaw <hixiao@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Motivation
test/registered/spec/dflash/test_dflash.pycrashes the scheduler on the AMDstage-brunner:#32541 added an
elif is_hip():branch that sets_DFLASH_SAMPLING_VERIFY_AVAILABLE = Truebut never bindstree_speculative_sampling_target_onlyin that branch. The kernel genuinely does not exist on ROCm:csrc/speculative/speculative_sampling.cuis listed only in the CUDACMakeLists.txt, whilesetup_rocm.pycompiles justeagle_utils.cuandcommon_extension_rocm.ccregisters onlyverify_tree_greedy.With the flag unconditionally
True, any request withtemperature > 0entered the non-greedy verify path and hit the unbound name. That matches the CI signal:test_early_stopandtest_greedy_determinismusetemperature=0and pass, whiletest_eos_handlingusestemperature=0.1and kills the scheduler, turning every later request intoConnection refused.Modifications
Bind the symbol in the
is_hip()branch and derive_DFLASH_SAMPLING_VERIFY_AVAILABLEfrom whether the op is actually registered.An
ImportErrorguard would not work:sgl_kernel/__init__.pyunconditionally re-exports the pure-Python wrapper, so the import always succeeds on ROCm and only thetorch.ops.sgl_kernel.*dispatch inside it fails. Probing the op table is the reliable check, andhasattr(torch.ops.<ns>, ...)is already the idiom used insrt/utils/custom_op.pyanddevice_communicators/cpu_communicator.py.Scope: the diff is confined to the
elif is_hip():branch. The CUDA/MUSA branch and theelsebranch are byte-for-byte unchanged, so there is no path by which this can affect NVIDIA or MUSA behavior.On ROCm the probe returns
False, sodflash_worker_v2.pytakes its existing greedy-argmax fallback and emits the warning that path already logs. This restores the pre-#32541 ROCm behavior and matcheseagle_utils.py, which puts_is_hipin the greedy branch for the same reason.Follow-up
This PR stops the crash; it does not add non-greedy verification on ROCm, where
temperature > 0still verifies againstargmax. Two options for a real fix, both orthogonal to this PR:setup_rocm.py(setup_musa.pyalready does exactly this for the same file), then add the source and register the op. The open question is whetherflashinfer/sampling.cuhsurvives hipify; encouragingly, AMD's ROCm/flashinfer port already ships a HIPifiedsampling.cuh.eagle_utils.pyalready treatschain_speculative_sampling_tritonandtree_speculative_sampling_target_onlyas interchangeable same-signature functions, so this needs no call-site change. DFLASH verification is a pure chain (retrieve_next_siblingis all-1), which collapses the tree walk to a linear scan. Note thatchain_speculative_sampling_tritonitself is not a drop-in substitute: it is classic rejection sampling, and with the all-zerodraft_probsDFLASH passes, itscoin * q < ptest accepts unconditionally.Once either lands, the op probe added here flips to
Trueon its own with no further change to this file.Speed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): 🚫 Run #31079763648
Latest PR Test (Extra): ❌ Run #31079763058