Remove unnecessary FlashInfer top-k fallback (environment issue, not a CUDA 13 bug) - #2
Merged
Conversation
…issue The try/except fallback in _topk was added because FlashInfer's radix top-k failed to JIT-compile on CUDA 13. Root cause turned out to be the CPATH-based include setup pulling in a mismatched host_runtime.h; with curand*.h symlinked into the system include path instead, FlashInfer compiles and works fine on CUDA 13. The fallback is unnecessary and masks real errors, so drop it and use FlashInfer directly. Also types the config parameter as Qwen3Config to match the base class signature, aligning this file with vllm-project#52883. Co-authored-by: pi coding agent <agent@fieldio.com> Signed-off-by: Andreas Echavez <oceanplexian@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Drops the
try/excepttorch.topkfallback in_topkfrom #1, and types theconfigparameter asQwen3Configto match the base class signature.Why
#1 added the fallback because FlashInfer 0.6.17's radix top-k failed to
JIT-compile on the ai-5xx server (CUDA 13). Root-caused since: the failure was
an environment issue, not a CUDA 13 incompatibility. The venv was using
so FlashInfer's JIT picked up a mismatched
host_runtime.hfrom that includetree. Symlinking
curand*.hinto the system include path instead (dropping theCPATH override) lets the radix top-k compile and run correctly on CUDA 13 —
verified on the same dual-3090 box.
Since FlashInfer works once the include path is fixed, the fallback is
unnecessary — and actively harmful, since it catches bare
Exceptionandpermanently disables FlashInfer top-k based on an environment misconfiguration,
masking real errors.
Note on the upstream PRs
UnquantizedLinearMethodfor aParallelLMHead) is a real bug and is upstreamed as[Bugfix][Spec Decode] DFlash2: accept unquantized linear LM heads in the candidate selector vllm-project/vllm#52883, stacked on the DFlash2 base PR
[Spec Decode] DFlash2: local convolution + candidate selector vllm-project/vllm#52816.
PR aligns this fork's file with what is actually being proposed upstream.
change, not a code fallback.
Test plan
ruff check/ruff format --checkon the touched file → cleanthe server picks up the fixed FlashInfer kernel