Bump flashinfer version to 0.6.14

[AmeenP]

Purpose

Pick up flashinfer-ai/flashinfer#3615, which fixes the multi-CTA radix top-k sampler stream hang (flashinfer-ai/flashinfer#3610): the kernel's epilogue resets the software barrier's arrival counter with no sync against peer CTAs still polling it, so a peer can spin forever and permanently wedge the stream (GPU pinned at 100% util / low power, no Xid, engine frozen). 0.6.14 is the first release containing the fix; current pin (0.6.13) is affected.

Originally reported on SM120/SM121 (#43885, worked around by #44405), but it is not Blackwell-specific — we hit it in production on H100 serving Qwen models (top_k+top_p are Qwen's defaults, so every request launches this kernel) and can reproduce on Hopper: a __nanosleep widening the wait_ge poll gap wedges 0.6.11.post2/0.6.13-era kernels on the first launch, while a #3615 backport survives 50k launches under the same perturbation. See #43885 (comment).

With this in, the SM120/121 default-off from #44405 can be revisited.

Test Plan

Version bump only (same file set as #46683). Wheels verified available: flashinfer-python/flashinfer-cubin 0.6.14 on PyPI, flashinfer-jit-cache 0.6.14+cu130 on https://flashinfer.ai/whl.

Fix validated at the kernel level on H100: stock kernel + widened barrier-poll window = deterministic wedge on first launch; 0.6.14's reset scheme under identical conditions = 50k launches clean with bit-identical top-k masks.

Test Result

See above.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3bc28e653a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bkryu]

@AmeenP, can you check the latest from flashinfer-ai/flashinfer#3808 and see how the flashinfer-cubin wheel's index should be updated to correctly download the FlashInfer 0.6.14 cubin wheel?

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[AmeenP]

Claude Test Result: validated on Hopper (H100) and Blackwell (GB300)

Ran vLLM integration + sampler-kernel validation for the 0.6.13 -> 0.6.14 bump on both
architectures (CI covers the x86 kernel/unit suites; this adds e2e serving on the H100 we hit
the original wedge on, plus GB300/Blackwell aarch64 which CI does not exercise).

Hopper — 1x H100, vLLM main @ ba2215209, torch 2.11.0+cu130

Check	Result
Greedy e2e, FLASHINFER backend, 0.6.13 vs 0.6.14	bitwise identical (0/8 streams differ)
Sampled e2e, Qwen defaults top_k=20,top_p=0.8	32/32 complete, full-length, distinct
Sampler soak, 2000 req (~16k top-k/top-p launches)	no hang, 29s
Direct multi-CTA radix top-k soak, 20k launches (incl all--inf/flat rows)	no hang, 18.3s, steady throughput
tests/v1/sample/test_topk_topp_sampler.py	130 passed, 1 skipped
tests/kernels/attention/test_flashinfer.py	114 passed, 48 skipped
tests/kernels/moe/test_flashinfer{,_moe}.py	14 passed, 13 skipped (skips are sm>=100/nvfp4-only)
flashinfer FP8 + NVFP4 GEMM kernel tests	skip on Hopper (require cc>=100), covered on GB300 below

Blackwell — 1x GB300 NVL72 (SM103, aarch64), torch 2.11.0+cu130

Check	Result
aarch64 wheels (flashinfer-python/-cubin 0.6.14, flashinfer-jit-cache-0.6.14+cu130 aarch64)	present, install + import OK
Greedy e2e, 0.6.13 vs 0.6.14 (exact PR delta)	bitwise identical (0/8)
Determinism control, 0.6.14 vs 0.6.14	bitwise identical (0/8)
Sampled e2e, Qwen defaults	32/32 complete, full-length, distinct
Sampler soak, 2000 req	no hang, 77s
Direct multi-CTA radix top-k soak, 20k launches (incl degenerate rows)	no hang, 11.9s, steady throughput
flashinfer FP8 + NVFP4 GEMM (test_flashinfer_scaled_mm.py + test_flashinfer_nvfp4_scaled_mm.py)	174 passed, 54 skipped, 0 failed

Greedy output is bitwise-identical across 0.6.13 -> 0.6.14 on both architectures, so the bump
is numerically neutral on the attention path; the sampler kernel that #3615 fixes survives 20k
direct launches + ~16k e2e launches per platform with no wedge, including degenerate-logit rows;
and the NVFP4/FP8 GEMM kernels (Blackwell-only cutedsl path) pass their full matrix on GB300.

All three 0.6.14 wheels (flashinfer-python, flashinfer-cubin, flashinfer-jit-cache+cu130
including the aarch64 build) install and import cleanly on both x86_64/H100 and aarch64/GB300.

[AmeenP]

@mgoin, @pavanimajety can I get this one marked ready

[AmeenP]

Followed the 0.6.14 release-notes install instructions from flashinfer-ai/flashinfer#3808 in b1aac62:

• docker/Dockerfile now installs flashinfer-cubin==${FLASHINFER_VERSION} --index-url https://flashinfer.ai/whl, same pattern as the existing flashinfer-jit-cache install, in both stages that previously got cubin from requirements (base and vllm-base). One global FLASHINFER_VERSION ARG drives both; versions.json regenerated.
• requirements/cuda.txt keeps only flashinfer-python==0.6.14 (still on PyPI). The flashinfer-cubin pin had to come out entirely rather than gain an extra index: that file also feeds the wheel's install_requires (setup.py strips --extra-index-url option lines, and PEP 508 metadata can't carry an index URL), so any cubin pin would break pip install vllm / source installs once cubin is off PyPI — while Docker-based CI stayed green. Without the cubin package, flashinfer-python falls back to fetching cubins at runtime (vllm/utils/flashinfer.py), so non-Docker installs keep working.

Verified flashinfer-cubin 0.6.14 resolves from https://flashinfer.ai/whl (PyPI tops out at 0.6.13).

[mgoin]

Can we add the index url for https://flashinfer.ai/whl here for flashinfer-cubin in the requirements file directly?

[AmeenP]

Done flashinfer-cubin==0.6.14 is pinned here with `--extra-index-url https://flashinfer.ai/whl/

I think this is cleaner overall, still requires excluding it from setup.py

One addition to make this shape safe: setup.py now excludes flashinfer-cubin from install_requires (same pattern as the existing vllm-flash-attn skip). setup.py drops -- option lines when reading requirements, so without the exclusion the published wheel would pin a package that no longer exists on PyPI (flashinfer is off PyPI for cubin permanently per flashinfer-ai/flashinfer#3808 — project size limit), breaking pip install vllm / pip install -e . while Docker CI stays green. Without the cubin package installed, flashinfer falls back to fetching cubins at runtime (vllm/utils/flashinfer.py::has_flashinfer_cubin).

[AmeenP]

Claude Test Result: lm-eval-large-models-8xh200 failures are not this PR (reproduced on both flashinfer 0.6.13 and 0.6.14)

The DeepSeek-R1-TP / DeepSeek-V3.2-TP GSM8K 0.0000 failures in build 76930 reproduce outside CI and are independent of the flashinfer version. Repro on an idle 8xH200 node using this PR's exact CI test image (vllm-ci-test-repo:d3fc933...) and the exact DeepSeek-R1-TP.yaml config (TP8 + EP + MTP num_speculative_tokens=3 + enforce-eager):

• flashinfer 0.6.14 (this PR): workers die ~4 min into serving with a device-side assert — IndexKernel.cu:111: Assertion '-sizes[i] <= index && index < sizes[i] && "index out of bounds"' failed — immediately after eagle_step_slot_mapping_metadata_kernel JIT-compiles on the first MTP step. CUDA context dies, engine goes EngineDead, all subsequent requests fail, GSM8K scores 0.0000. Same externally-visible signature as CI ("Worker proc VllmWorker-1 died unexpectedly (exit code: None)").
• flashinfer 0.6.13 control (same pod, same image, only flashinfer-python/flashinfer-cubin/flashinfer-jit-cache downgraded): identical IndexKernel.cu:111 assert, identical outcome.

Corroborating points:

• The crash is in a PyTorch indexing kernel on the MTP/Eagle spec-decode slot-mapping path, not in any flashinfer kernel.
• CI's DP variants of the same models (DeepSeek-R1-DP, DeepSeek-V3.2-DP — identical config except --data-parallel-size 8) passed in the same build, so the bug is TP-sharding-specific.
• A V2-Lite FP8 MLA canary (same per-rank head count as R1-TP8, no MTP) passes GSM8K above threshold on 0.6.14.

So the failing job is a main-side MTP/spec-decode regression that this branch inherited via merges; likely-suspect area is the spec-decode/MTP changes merged 07-06/07-07. Happy to file a separate issue with full logs if useful.

[AmeenP]

@mgoin Anything else blocking on getting this one in?

[mergify]

Hi @AmeenP, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[hermia-ai]

+1 — and one correction to the Test Plan

A second, independent reason to land this. Beyond the radix top-k sampler wedge: on sm_120 the
FLASHINFER_MLA_SPARSE_SM120 backend already passes a 0.6.14-only kwarg, so the current 0.6.13 pin is not just
suboptimal there, it's broken at engine init:

TypeError: trtllm_batch_decode_with_kv_cache_mla() got an unexpected keyword argument 'kv_scale_format'

Repro'd serving GLM-5.2 (GlmMoeDsaForCausalLM) on 4× RTX PRO 6000 Blackwell, stock vLLM 0.25.1 with its pinned
0.6.13. kv_scale_format first appears in 0.6.14's signature. See #48054 (same root cause on the DeepSeek-V4
sparse-SWA path). So this bump fixes at least two unrelated breakages.

---

Correction: flashinfer-cubin 0.6.14 is not on PyPI

The Test Plan says:

Wheels verified available: flashinfer-python/flashinfer-cubin 0.6.14 on PyPI

flashinfer-python 0.6.14 is there, but flashinfer-cubin's latest release is 0.6.13 — there is no 0.6.14.
Checked against the PyPI JSON API just now:

import json, urllib.request
for pkg in ("flashinfer-python", "flashinfer-cubin"):
    d = json.loads(urllib.request.urlopen(f"https://pypi.org/pypi/{pkg}/json").read())
    print(pkg, "latest:", d["info"]["version"], "| has 0.6.14:", "0.6.14" in d["releases"])

# flashinfer-python latest: 0.6.14 | has 0.6.14: True
# flashinfer-cubin  latest: 0.6.13 | has 0.6.14: False

(Also reproducible with pip index versions flashinfer-cubin.)

Why it matters for this PR: it's not cosmetic. With flashinfer-python at 0.6.14 and flashinfer-cubin stuck
at 0.6.13, flashinfer/jit/env.py's guard aborts at import:

RuntimeError: flashinfer-cubin version (0.6.13) does not match flashinfer version (0.6.14).
Please install the same version of both packages. Set FLASHINFER_DISABLE_VERSION_CHECK=1 to bypass this check.

I hit exactly this bumping 0.6.13 → 0.6.14 by hand. FLASHINFER_DISABLE_VERSION_CHECK=1 works fine in practice
(FlashInfer JITs the kernels anyway, and the model serves correctly at 250K context with needle recall verified to
233,809 tokens) — but it's an unobvious extra step, and anyone following this PR's Test Plan literally will hit the
guard and assume the bump is broken.

Might be worth either waiting for a matching flashinfer-cubin release, or documenting the
FLASHINFER_DISABLE_VERSION_CHECK=1 requirement alongside the bump. Not blocking — just flagging so the claim
matches reality.

Full context: https://github.com/hermia-ai/glm-5.2-sm120-stock-vllm

[AmeenP]

Post-merge heads-up: FlashInfer 0.6.14 has a separate SM103/GB300 NVFP4 MoE hang reported in sglang#30919 and isolated in flashinfer#3971 to the TRTLLM_GEN_BMM artifact batched_gemm-dd6d23e-721ae60.

This affects a path vLLM can auto-select for Blackwell NVFP4 MoE. A replacement artifact merged in flashinfer#3973, and the FlashInfer team is targeting a 0.6.15 release for Thursday/Friday.

Rather than reverting to 0.6.13, which restores the sampler hang fixed here, we should update to 0.6.15 as soon as it is published and validate the affected GB300 path.