Bump FlashInfer to 0.6.16.post4 - #33092
elvischenv wants to merge 5 commits into
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
0dd32ad to
56fadcb
Compare
|
@elvischenv please change it to .post2. thanks |
56fadcb to
4c5596f
Compare
|
/tag-and-rerun-ci |
|
The main NVIDIA failures are real and caused by PR #33092. Rebasing onto current SGLang main will not fix them. PR #33092 upgrades FlashInfer from 0.6.15.post1 to 0.6.16.post2. |
|
We probably want to target post4 instead of post2 in this PR. Here is the finished post4 build: https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.6.16.post4 |
|
@elvischenv could you change it to .post4? thanks |
9d19972 to
a58be8d
Compare
https://github.com/sgl-project/sglang/actions/runs/31452962311/job/93662822354?pr=33092 |
Bump `flashinfer_python` (and the Docker `FLASHINFER_VERSION` arg that drives `flashinfer-cubin` / `flashinfer-jit-cache`) from 0.6.15.post1 to 0.6.16, plus the runtime `assert_pkg_version` pin and the docstring example version. FlashInfer 0.6.16 (flashinfer-ai/flashinfer#4142) needs no source changes on the SGLang side: every public API SGLang touches is either unchanged or extended with defaulted keyword arguments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
flashinfer-jit-cache 0.6.16 was built against apache-tvm-ffi 0.1.13, and
all 959 of its prebuilt modules carry an undefined reference to
`TVMFFIGetCustomAllocator` — a symbol that first appears in tvm-ffi
0.1.13 and is absent from the `apache-tvm-ffi==0.1.11` this repo pins.
Because tvm-ffi loads these modules with lazy binding, the happy path
still works and the mismatch stays hidden. The symbol is referenced only
by the `tvm::ffi::Error` constructors, so it is reached the first time a
kernel actually reports an error. At that point the lazy PLT resolution
fails and the process dies with
symbol lookup error: .../norm.so: undefined symbol: TVMFFIGetCustomAllocator
instead of raising a catchable exception — a recoverable validation error
takes down the whole server.
FlashInfer 0.6.16.post1 rebuilds the companion wheels against tvm-ffi
0.1.13-post0, which drops that dependency: none of the 959 prebuilt
modules reference the symbol, and the same failing call now raises a
clean `RuntimeError: Check failed: input.size(1) == weight.size(0)`.
`flashinfer_python` itself is byte-identical between 0.6.16 and
0.6.16.post1 apart from the version string, so this only moves the
prebuilt `flashinfer-cubin` / `flashinfer-jit-cache` artifacts that the
Dockerfile and CI resolve from these pins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0.6.16.post2 rebuilds the prebuilt wheels against apache-tvm-ffi
0.1.13-post2, which restores the pointer-sized ABI for object optionals
that 0.1.13/-post0/-post1 had widened to an Any-backed representation:
sizeof(tvm::ffi::Optional<tvm::ffi::ObjectRef>)
apache-tvm-ffi 0.1.11 -> 8 (this repo's pin)
apache-tvm-ffi 0.1.13-post0 -> 16 (0.6.16.post1 built against)
apache-tvm-ffi 0.1.13-post2 -> 8 (0.6.16.post2 built against)
That layout change never reaches SGLang, because it does not cross the
module boundary. Of the 959 prebuilt modules, 954 talk to libtvm_ffi.so
purely over the C ABI, and the remaining 5 import only
`ModuleObj::ClearImports()` and `ModuleObj::ImportModule(Module const&)`
— both exported by 0.1.11, and neither taking an Optional. `Module` (8)
and `ModuleObj` (48) are identical in all three versions. The complete
undefined-symbol sets of post1 and post2 differ by one libstdc++ entry
(`std::basic_iostream::basic_iostream()`) and nothing from tvm-ffi.
So post2 is not a correctness fix for this repo the way post1 was over
0.6.16 — it aligns us with the build upstream recommends and removes a
latent mismatch should any Optional-carrying symbol ever cross that
boundary. `flashinfer_python` is again byte-identical to post1 apart
from the version string, so only the prebuilt `flashinfer-cubin` /
`flashinfer-jit-cache` artifacts move.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Unlike post1/post2, which only rebuilt the companion wheels, post3 and post4 carry real source changes. post3 reverts the SM90 CUTLASS MoE backend (flashinfer-ai/flashinfer#3738 plus dependents sgl-project#4025 and sgl-project#4080). That work first shipped in 0.6.16 — it is not in 0.6.15.post1 — so the revert returns those kernels to roughly the state this repo already runs against today, rather than regressing anything. It drops six public symbols and one parameter: preprocess_moe_weights_for_sm90_mixed_gemm_humming sm90_mixed_gemm_humming_weight_preprocess_trace_dispatch bits_from_float / float_from_bits / quant_to_fp4_val / dequant_fp4_val use_wfp4afp8_humming= on cutlass_fused_moe / cutlass_fused_moe_workspace_size SGLang uses none of them. `interleave_moe_{scales,weights}_for_sm90_mixed_gemm` move back from `fused_moe/prepare.py` to `fused_moe/core.py` but keep their signatures and stay exported from `flashinfer.fused_moe`, which is where mxfp4.py and mxfp4_flashinfer_cutlass_moe.py import them from. post4 adds `from __future__ import annotations` to `flashinfer/comm/fd_exchange.py`. That module is new in 0.6.16 and annotates `array.array[int]`, which is only subscriptable on Python 3.12+. `flashinfer/comm/mnnvl.py` imports it unconditionally, so on Python 3.10/3.11 — both allowed by our `requires-python = ">=3.10"` — importing `flashinfer.comm.mnnvl` raised TypeError: type 'array.array' is not subscriptable taking out the MNNVL comm-fusion and MoE all-to-all paths. Verified on a real 3.11 interpreter: post2's fd_exchange raises, post4's imports fine. The published Docker image is Python 3.12 and was unaffected; this only bit source installs on older interpreters. Checked against 0.6.15.post1 with the full symbol sweep: no API SGLang uses is removed, and every changed signature is purely additive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a58be8d to
3c6d972
Compare
|
/rerun-failed-ci |
Looks like real failure? |
|
The fix of python/sglang/srt/layers/attention/flashinfer_backend.py is in #33997 |
|
@elvischenv let's close this. we will use #33997 instead |
Motivation
Bump
flashinfer_pythonfrom0.6.15.post1to0.6.16.post4.The intermediate 0.6.16 releases each fix something that matters here, so the target moved as they landed:
0.6.160.6.16.post10.6.16.post20.6.16.post30.6.16.post4fd_exchange.pyimportable on Python 3.10/3.11Modifications
Produced by
python scripts/release/bump_flashinfer_version.py 0.6.16.post4:python/pyproject.toml—flashinfer_python[cu13]==0.6.16.post4docker/Dockerfile—ARG FLASHINFER_VERSION=0.6.16.post4, which also drives theflashinfer-cubinandflashinfer-jit-cacheinstallspython/sglang/srt/entrypoints/engine.py— runtimeassert_pkg_version("flashinfer_python", ...)pinpython/sglang/srt/utils/common.py— docstring example versionCI (
scripts/ci/cuda/ci_install_dependency.sh,ci_download_flashinfer_jit_cache.sh) derives the required version frompython/pyproject.toml, so it follows automatically.apache-tvm-ffistays at0.1.11.Intentionally not bumped:
docker/kimi_k3/kimi_k3_cu{12,13}.Dockerfilepin FlashInfer to0.6.15.post1independently of the sharedFLASHINFER_VERSIONarg, because they apply the version-specific CuTeDSL MLA DCP patchdocker/kimi_k3/flashinfer-perkz-dcp-0.6.15.txt. Moving those forward requires rebasing that patch and is left as separate work.Why plain 0.6.16 is unusable here
TVMFFIGetCustomAllocatorwas introduced inapache-tvm-ffi0.1.13 and does not exist in the 0.1.11 this repo pins.flashinfer-jit-cache==0.6.16was built against 0.1.13, and all 959 of its prebuilt modules carry an undefined reference to it. From0.6.16.post1onward, 0 of 959 do.The failure is latent. tvm-ffi loads these modules with lazy binding, so the module loads and the happy path runs fine. Disassembly shows the symbol is referenced only from the
tvm::ffi::Errorconstructors, so it is first reached when a kernel actually reports an error — and lazy PLT resolution then fails fatally. Reproduced on an H100 withapache-tvm-ffi==0.1.11, callingrmsnormwith a mismatched weight length directly against the prebuilt module:A recoverable validation error inside any FlashInfer kernel would otherwise take down the whole server process.
What post3 and post4 change
Unlike post1/post2 — which only rebuilt the companion wheels, leaving
flashinfer_pythonbyte-identical — post3 and post4 carry real source changes.post3 reverts the SM90 CUTLASS MoE backend (flashinfer-ai/flashinfer#3738 plus dependents #4025, #4080). That work merged 2026-07-15 and first shipped in 0.6.16 — it is not in 0.6.15.post1 — so the revert returns those kernels to roughly the state this repo already runs against, rather than regressing anything. It drops six public symbols and one parameter:
preprocess_moe_weights_for_sm90_mixed_gemm_hummingsm90_mixed_gemm_humming_weight_preprocess_trace_dispatchbits_from_float,float_from_bits,quant_to_fp4_val,dequant_fp4_valuse_wfp4afp8_humming=oncutlass_fused_moeandcutlass_fused_moe_workspace_sizeSGLang uses none of them.
interleave_moe_{scales,weights}_for_sm90_mixed_gemmmove back fromfused_moe/prepare.pytofused_moe/core.py, but keep their signatures and stay exported fromflashinfer.fused_moe— which is wheremxfp4.py,mxfp4_flashinfer_cutlass_moe.pyandtest_mxfp4_sm90_cutlass.pyimport them from.post4 adds
from __future__ import annotationstoflashinfer/comm/fd_exchange.py. That module is new in 0.6.16 and annotatesarray.array[int], which only became subscriptable in Python 3.12.flashinfer/comm/mnnvl.pyimports it unconditionally, so on Python 3.10/3.11 — both allowed by ourrequires-python = ">=3.10"— importingflashinfer.comm.mnnvlraisedTypeError: type 'array.array' is not subscriptable, taking out the MNNVL comm-fusion and MoE all-to-all paths. Verified on a real 3.11 interpreter:The published Docker image is Python 3.12 and was unaffected; this only bit source installs on older interpreters.
API compatibility review
The 0.6.15.post1 and 0.6.16.post4 wheels were extracted and diffed directly, sweeping every FlashInfer symbol referenced anywhere in the repo (124 of them) plus the methods of every FlashInfer class SGLang uses — re-checked against the rebased tree.
Nothing SGLang uses is removed, and every changed signature is purely additive:
Two relocations are transparent because the original modules still re-export: the SM90 helpers above, and
CommBackend/TorchDistBackend/MpiComm, which moved to the newflashinfer/comm/abstractions.pyandcomm_backend.pybut remain re-exported fromflashinfer.comm.mnnvl.Three details worth recording:
fast_decode_planinsertsq_len_per_req=1beforeglobal_override_indptr_cpu, andcudnn_batch_prefill_with_kv_cacheinsertsbatch_offsets_unitsmid-signature. Both are safe only because the SGLang call sites (flashinfer_backend.py,vision.py) pass these purely by keyword.ActivationTypegainedSitu = 10, shiftingInvalidTypefrom 10 to 11. SGLang only referencesSwiglu/Geglu/Silu/Gelu/Relu2/Identity(values 0–9, unchanged).CommBackendchanged from anabc.ABCto a@runtime_checkableProtocol. SGLang'sTorchDistributedCommBackendsubclasses stay valid; the change is strictly more permissive.get_trtllm_gen_multi_ctas_kv_counter_bytesis byte-identical between the releases, so the TRTLLM-MLA multi-CTA counter-buffer sizing introduced in #31927 remains correct.New opt-in parameters are deliberately left unadopted to keep this a pure version bump.
Accuracy Tests
No model-output-affecting source changes. Validated on an H100 with
apache-tvm-ffi==0.1.11:test/registered/attention/—test_chunk_gated_delta_rule,test_create_kvindices,test_normal_decode_set_metadata,test_verify_splitkv: 55 passed, 24 subtests passed.rmsnormnumerical check against a PyTorch reference through the prebuilt module, plus the error-path check shown above.Full model-level accuracy runs were not performed locally.
Speed Tests and Profiling
Not run locally. Performance thresholds in registered tests are left unchanged, since there is no 0.6.16.post4 benchmark data to justify moving them. Note that post3's SM90 MoE revert removes a perf optimization that only ever existed in 0.6.16 — it is not a regression against current
main, but SM90 MXFP4 MoE will not gain it either.Validation
main(8d050dd880); resolved a Dockerfile conflict where upstream addedTRTLLM_GEN_MOE_CUBIN_*args adjacent toFLASHINFER_VERSION— both sides keptpre-commithooks over all changed files — passflashinfer-cubin==0.6.16.post4, andflashinfer-jit-cache==0.6.16.post4forcu128,cu129,cu130Checklist
🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #31470843979
Latest PR Test (Extra): ❌ Run #31470843814