Skip to content

Remove unnecessary FlashInfer top-k fallback (environment issue, not a CUDA 13 bug) - #2

Merged
oceanplexian merged 1 commit into
mainfrom
remove-unnecessary-flashinfer-fallback
Aug 19, 2026
Merged

oceanplexian merged 1 commit into
mainfrom
remove-unnecessary-flashinfer-fallback

Conversation

@oceanplexian

Copy link
Copy Markdown
Owner

What

Drops the try/except torch.topk fallback in _topk from #1, and types the
config parameter as Qwen3Config to match the base class signature.

Why

#1 added the fallback because FlashInfer 0.6.17's radix top-k failed to
JIT-compile on the ai-5xx server (CUDA 13). Root-caused since: the failure was
an environment issue, not a CUDA 13 incompatibility. The venv was using

export CPATH="$VENV/lib/python3.10/site-packages/nvidia/cu13/include"

so FlashInfer's JIT picked up a mismatched host_runtime.h from that include
tree. Symlinking curand*.h into the system include path instead (dropping the
CPATH override) lets the radix top-k compile and run correctly on CUDA 13 —
verified on the same dual-3090 box.

Since FlashInfer works once the include path is fixed, the fallback is
unnecessary — and actively harmful, since it catches bare Exception and
permanently disables FlashInfer top-k based on an environment misconfiguration,
masking real errors.

Note on the upstream PRs

Test plan

…issue

The try/except fallback in _topk was added because FlashInfer's radix
top-k failed to JIT-compile on CUDA 13. Root cause turned out to be the
CPATH-based include setup pulling in a mismatched host_runtime.h; with
curand*.h symlinked into the system include path instead, FlashInfer
compiles and works fine on CUDA 13. The fallback is unnecessary and
masks real errors, so drop it and use FlashInfer directly.

Also types the config parameter as Qwen3Config to match the base class
signature, aligning this file with vllm-project#52883.

Co-authored-by: pi coding agent <agent@fieldio.com>
Signed-off-by: Andreas Echavez <oceanplexian@gmail.com>
@oceanplexian
oceanplexian merged commit 7790d27 into main Aug 19, 2026
@oceanplexian
oceanplexian deleted the remove-unnecessary-flashinfer-fallback branch August 19, 2026 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant