Skip to content

Bump FlashInfer to 0.6.16.post4 - #33092

Closed
elvischenv wants to merge 5 commits into
sgl-project:mainfrom
elvischenv:dev/bump-flashinfer-0.6.16
Closed

elvischenv wants to merge 5 commits into
sgl-project:mainfrom
elvischenv:dev/bump-flashinfer-0.6.16

Conversation

@elvischenv

@elvischenv elvischenv commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Motivation

Bump flashinfer_python from 0.6.15.post1 to 0.6.16.post4.

The intermediate 0.6.16 releases each fix something that matters here, so the target moved as they landed:

release what it changes relevance
0.6.16 unusable: prebuilt jit-cache needs a tvm-ffi symbol we don't have
0.6.16.post1 rebuild against tvm-ffi 0.1.13-post0 fixes the above
0.6.16.post2 rebuild against tvm-ffi 0.1.13-post2 ABI alignment, no functional change for us
0.6.16.post3 reverts the SM90 CUTLASS MoE backend returns those kernels to ~0.6.15.post1 state
0.6.16.post4 fd_exchange.py importable on Python 3.10/3.11 fixes MNNVL import on our minimum Python

Modifications

Produced by python scripts/release/bump_flashinfer_version.py 0.6.16.post4:

  • python/pyproject.tomlflashinfer_python[cu13]==0.6.16.post4
  • docker/DockerfileARG FLASHINFER_VERSION=0.6.16.post4, which also drives the flashinfer-cubin and flashinfer-jit-cache installs
  • python/sglang/srt/entrypoints/engine.py — runtime assert_pkg_version("flashinfer_python", ...) pin
  • python/sglang/srt/utils/common.py — docstring example version

CI (scripts/ci/cuda/ci_install_dependency.sh, ci_download_flashinfer_jit_cache.sh) derives the required version from python/pyproject.toml, so it follows automatically. apache-tvm-ffi stays at 0.1.11.

Intentionally not bumped: docker/kimi_k3/kimi_k3_cu{12,13}.Dockerfile pin FlashInfer to 0.6.15.post1 independently of the shared FLASHINFER_VERSION arg, because they apply the version-specific CuTeDSL MLA DCP patch docker/kimi_k3/flashinfer-perkz-dcp-0.6.15.txt. Moving those forward requires rebasing that patch and is left as separate work.

Why plain 0.6.16 is unusable here

TVMFFIGetCustomAllocator was introduced in apache-tvm-ffi 0.1.13 and does not exist in the 0.1.11 this repo pins. flashinfer-jit-cache==0.6.16 was built against 0.1.13, and all 959 of its prebuilt modules carry an undefined reference to it. From 0.6.16.post1 onward, 0 of 959 do.

The failure is latent. tvm-ffi loads these modules with lazy binding, so the module loads and the happy path runs fine. Disassembly shows the symbol is referenced only from the tvm::ffi::Error constructors, so it is first reached when a kernel actually reports an error — and lazy PLT resolution then fails fatally. Reproduced on an H100 with apache-tvm-ffi==0.1.11, calling rmsnorm with a mismatched weight length directly against the prebuilt module:

# flashinfer-jit-cache 0.6.16
happy path: OK
triggering C++ error path ...
python3: symbol lookup error: .../norm/norm.so: undefined symbol: TVMFFIGetCustomAllocator   # process dies

# flashinfer-jit-cache 0.6.16.post2 / post4
happy path: OK
triggering C++ error path ...
RESULT: clean exception -> RuntimeError: Check failed: input.size(1) == weight.size(0) (512 vs. 999)

A recoverable validation error inside any FlashInfer kernel would otherwise take down the whole server process.

What post3 and post4 change

Unlike post1/post2 — which only rebuilt the companion wheels, leaving flashinfer_python byte-identical — post3 and post4 carry real source changes.

post3 reverts the SM90 CUTLASS MoE backend (flashinfer-ai/flashinfer#3738 plus dependents #4025, #4080). That work merged 2026-07-15 and first shipped in 0.6.16 — it is not in 0.6.15.post1 — so the revert returns those kernels to roughly the state this repo already runs against, rather than regressing anything. It drops six public symbols and one parameter:

  • preprocess_moe_weights_for_sm90_mixed_gemm_humming
  • sm90_mixed_gemm_humming_weight_preprocess_trace_dispatch
  • bits_from_float, float_from_bits, quant_to_fp4_val, dequant_fp4_val
  • use_wfp4afp8_humming= on cutlass_fused_moe and cutlass_fused_moe_workspace_size

SGLang uses none of them. interleave_moe_{scales,weights}_for_sm90_mixed_gemm move back from fused_moe/prepare.py to fused_moe/core.py, but keep their signatures and stay exported from flashinfer.fused_moe — which is where mxfp4.py, mxfp4_flashinfer_cutlass_moe.py and test_mxfp4_sm90_cutlass.py import them from.

post4 adds from __future__ import annotations to flashinfer/comm/fd_exchange.py. That module is new in 0.6.16 and annotates array.array[int], which only became subscriptable in Python 3.12. flashinfer/comm/mnnvl.py imports it unconditionally, so on Python 3.10/3.11 — both allowed by our requires-python = ">=3.10" — importing flashinfer.comm.mnnvl raised TypeError: type 'array.array' is not subscriptable, taking out the MNNVL comm-fusion and MoE all-to-all paths. Verified on a real 3.11 interpreter:

### flashinfer 0.6.16.post2  fd_exchange.py on Python 3.11
  FAILED -> TypeError: type 'array.array' is not subscriptable
### flashinfer 0.6.16.post4  fd_exchange.py on Python 3.11
  IMPORT OK

The published Docker image is Python 3.12 and was unaffected; this only bit source installs on older interpreters.

API compatibility review

The 0.6.15.post1 and 0.6.16.post4 wheels were extracted and diffed directly, sweeping every FlashInfer symbol referenced anywhere in the repo (124 of them) plus the methods of every FlashInfer class SGLang uses — re-checked against the rebased tree.

Nothing SGLang uses is removed, and every changed signature is purely additive:

chunk_gated_delta_rule              added=['state_indices']
cudnn_batch_prefill_with_kv_cache   added=['batch_offsets_units']
cutlass_fused_moe                   added=['profile_ids', 'workspace_buffer']
e2m1_and_ufp8sf_scale_to_float      added=['is_sf_8x4_layout']
fast_decode_plan                    added=['q_len_per_req']
moe_a2a_get_workspace_size_per_rank added=['eplb_stats_num_experts']
recurrent_kda                       added=['beta_is_logit', 'initial_state_indices', 'initial_state_source']
trtllm_batch_decode_with_kv_cache   added=['enable_block_sparse_attention']

Two relocations are transparent because the original modules still re-export: the SM90 helpers above, and CommBackend / TorchDistBackend / MpiComm, which moved to the new flashinfer/comm/abstractions.py and comm_backend.py but remain re-exported from flashinfer.comm.mnnvl.

Three details worth recording:

  • fast_decode_plan inserts q_len_per_req=1 before global_override_indptr_cpu, and cudnn_batch_prefill_with_kv_cache inserts batch_offsets_units mid-signature. Both are safe only because the SGLang call sites (flashinfer_backend.py, vision.py) pass these purely by keyword.
  • ActivationType gained Situ = 10, shifting InvalidType from 10 to 11. SGLang only references Swiglu/Geglu/Silu/Gelu/Relu2/Identity (values 0–9, unchanged).
  • CommBackend changed from an abc.ABC to a @runtime_checkable Protocol. SGLang's TorchDistributedCommBackend subclasses stay valid; the change is strictly more permissive.

get_trtllm_gen_multi_ctas_kv_counter_bytes is byte-identical between the releases, so the TRTLLM-MLA multi-CTA counter-buffer sizing introduced in #31927 remains correct.

New opt-in parameters are deliberately left unadopted to keep this a pure version bump.

Accuracy Tests

No model-output-affecting source changes. Validated on an H100 with apache-tvm-ffi==0.1.11:

  • All 16 SGLang modules that touch FlashInfer import cleanly.
  • test/registered/attention/test_chunk_gated_delta_rule, test_create_kvindices, test_normal_decode_set_metadata, test_verify_splitkv: 55 passed, 24 subtests passed.
  • Direct rmsnorm numerical check against a PyTorch reference through the prebuilt module, plus the error-path check shown above.

Full model-level accuracy runs were not performed locally.

Speed Tests and Profiling

Not run locally. Performance thresholds in registered tests are left unchanged, since there is no 0.6.16.post4 benchmark data to justify moving them. Note that post3's SM90 MoE revert removes a perf optimization that only ever existed in 0.6.16 — it is not a regression against current main, but SM90 MXFP4 MoE will not gain it either.

Validation

  • Rebased onto upstream main (8d050dd880); resolved a Dockerfile conflict where upstream added TRTLLM_GEN_MOE_CUBIN_* args adjacent to FLASHINFER_VERSION — both sides kept
  • API compatibility re-checked against the rebased tree
  • Repository pre-commit hooks over all changed files — pass
  • Verified the artifacts are published: flashinfer-cubin==0.6.16.post4, and flashinfer-jit-cache==0.6.16.post4 for cu128, cu129, cu130

Checklist

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #31470843979
Latest PR Test (Extra): ❌ Run #31470843814

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@github-actions github-actions Bot added the dependencies Pull requests that update a dependency file label Jul 31, 2026
@elvischenv elvischenv changed the title Bump FlashInfer to 0.6.16 Bump FlashInfer to 0.6.16.post1 Aug 6, 2026
@elvischenv
elvischenv force-pushed the dev/bump-flashinfer-0.6.16 branch from 0dd32ad to 56fadcb Compare August 6, 2026 03:31
@nvpohanh

nvpohanh commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

@elvischenv please change it to .post2. thanks

@elvischenv elvischenv changed the title Bump FlashInfer to 0.6.16.post1 Bump FlashInfer to 0.6.16.post2 Aug 7, 2026
@elvischenv
elvischenv force-pushed the dev/bump-flashinfer-0.6.16 branch from 56fadcb to 4c5596f Compare August 7, 2026 07:31
@nvpohanh

nvpohanh commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci label Aug 7, 2026
@nvpohanh

Copy link
Copy Markdown
Collaborator

The main NVIDIA failures are real and caused by PR #33092. Rebasing onto current SGLang main will not fix them.

PR #33092 upgrades FlashInfer from 0.6.15.post1 to 0.6.16.post2.
FlashInfer 0.6.16.post2 contains this Python 3.10-incompatible annotation:
array.array[int]
Importing FlashInfer consequently raises:
TypeError: 'type' object is not subscriptable
This single error caused the CPU, JIT, diffusion-unit, and Pi0.5 failures. Many subsequent NVIDIA job failures were fast-fail/cascade failures. See representative CPU, JIT, and diffusion jobs.
As checked at SGLang main commit ee3ee83, main still pins FlashInfer 0.6.15.post1. Therefore main passes by avoiding the affected release; it does not contain a fix for 0.6.16.post2. The problematic annotation also remains on FlashInfer main, and I found no existing SGLang or FlashInfer fix PR.

@leejnau

leejnau commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

We probably want to target post4 instead of post2 in this PR. Here is the finished post4 build:

https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.6.16.post4

@nvpohanh

Copy link
Copy Markdown
Collaborator

@elvischenv could you change it to .post4? thanks

@elvischenv elvischenv changed the title Bump FlashInfer to 0.6.16.post2 Bump FlashInfer to 0.6.16.post4 Aug 11, 2026
@elvischenv
elvischenv force-pushed the dev/bump-flashinfer-0.6.16 branch from 9d19972 to a58be8d Compare August 11, 2026 02:38
@nvpohanh

Copy link
Copy Markdown
Collaborator
  File "/actions-runner/_work/sglang/sglang/python/sglang/srt/speculative/eagle_worker_v2.py", line 1193, in forward_batch_generation
    self.draft_worker._draft_extend_for_decode(batch, batch_output)
  File "/actions-runner/_work/sglang/sglang/python/sglang/srt/speculative/eagle_worker_v2.py", line 923, in _draft_extend_for_decode
    draft_logits_output = self.cuda_graph_runner_for_draft_extend.execute(
  File "/actions-runner/_work/sglang/sglang/python/sglang/srt/speculative/eagle_draft_extend_cuda_graph_runner.py", line 590, in execute
    self.draft_extend_attn_backend.init_forward_metadata_out_graph(fb_view)
  File "/actions-runner/_work/sglang/sglang/python/sglang/srt/layers/attention/flashinfer_backend.py", line 759, in init_forward_metadata_out_graph
    self.indices_updater_prefill.update(
  File "/actions-runner/_work/sglang/sglang/python/sglang/srt/layers/attention/flashinfer_backend.py", line 1830, in update_single_wrapper
    self.call_begin_forward(
  File "/actions-runner/_work/sglang/sglang/python/sglang/srt/layers/attention/flashinfer_backend.py", line 2196, in call_begin_forward
    wrapper_paged.begin_forward(
  File "/actions-runner/_work/sglang/sglang/python/sglang/srt/layers/attention/flashinfer_backend.py", line 285, in fast_prefill_plan
    self._plan_info = self._cached_module.plan(*args)
  File "python/tvm_ffi/cython/function.pxi", line 968, in tvm_ffi.core.Function.__call__
TypeError: Mismatched number of arguments when calling: `plan(0: DLTensor*, 1: DLTensor*, 2: DLTensor*, 3: DLTensor*, 4: DLTensor*, 5: DLTensor*, 6: int, 7: int, 8: int, 9: int, 10: int, 11: bool, 12: int, 13: int, 14: bool, 15: int, 16: int, 17: bool, 18: int, 19: int) -> Array<int>`. Expected 20 but got 19 arguments

https://github.com/sgl-project/sglang/actions/runs/31452962311/job/93662822354?pr=33092
@elvischenv to fix this.

elvischenv and others added 4 commits August 11, 2026 00:53
Bump `flashinfer_python` (and the Docker `FLASHINFER_VERSION` arg that
drives `flashinfer-cubin` / `flashinfer-jit-cache`) from 0.6.15.post1 to
0.6.16, plus the runtime `assert_pkg_version` pin and the docstring
example version.

FlashInfer 0.6.16 (flashinfer-ai/flashinfer#4142) needs no source
changes on the SGLang side: every public API SGLang touches is either
unchanged or extended with defaulted keyword arguments.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
flashinfer-jit-cache 0.6.16 was built against apache-tvm-ffi 0.1.13, and
all 959 of its prebuilt modules carry an undefined reference to
`TVMFFIGetCustomAllocator` — a symbol that first appears in tvm-ffi
0.1.13 and is absent from the `apache-tvm-ffi==0.1.11` this repo pins.

Because tvm-ffi loads these modules with lazy binding, the happy path
still works and the mismatch stays hidden. The symbol is referenced only
by the `tvm::ffi::Error` constructors, so it is reached the first time a
kernel actually reports an error. At that point the lazy PLT resolution
fails and the process dies with

    symbol lookup error: .../norm.so: undefined symbol: TVMFFIGetCustomAllocator

instead of raising a catchable exception — a recoverable validation error
takes down the whole server.

FlashInfer 0.6.16.post1 rebuilds the companion wheels against tvm-ffi
0.1.13-post0, which drops that dependency: none of the 959 prebuilt
modules reference the symbol, and the same failing call now raises a
clean `RuntimeError: Check failed: input.size(1) == weight.size(0)`.

`flashinfer_python` itself is byte-identical between 0.6.16 and
0.6.16.post1 apart from the version string, so this only moves the
prebuilt `flashinfer-cubin` / `flashinfer-jit-cache` artifacts that the
Dockerfile and CI resolve from these pins.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0.6.16.post2 rebuilds the prebuilt wheels against apache-tvm-ffi
0.1.13-post2, which restores the pointer-sized ABI for object optionals
that 0.1.13/-post0/-post1 had widened to an Any-backed representation:

    sizeof(tvm::ffi::Optional<tvm::ffi::ObjectRef>)
      apache-tvm-ffi 0.1.11        ->  8   (this repo's pin)
      apache-tvm-ffi 0.1.13-post0  -> 16   (0.6.16.post1 built against)
      apache-tvm-ffi 0.1.13-post2  ->  8   (0.6.16.post2 built against)

That layout change never reaches SGLang, because it does not cross the
module boundary. Of the 959 prebuilt modules, 954 talk to libtvm_ffi.so
purely over the C ABI, and the remaining 5 import only
`ModuleObj::ClearImports()` and `ModuleObj::ImportModule(Module const&)`
— both exported by 0.1.11, and neither taking an Optional. `Module` (8)
and `ModuleObj` (48) are identical in all three versions. The complete
undefined-symbol sets of post1 and post2 differ by one libstdc++ entry
(`std::basic_iostream::basic_iostream()`) and nothing from tvm-ffi.

So post2 is not a correctness fix for this repo the way post1 was over
0.6.16 — it aligns us with the build upstream recommends and removes a
latent mismatch should any Optional-carrying symbol ever cross that
boundary. `flashinfer_python` is again byte-identical to post1 apart
from the version string, so only the prebuilt `flashinfer-cubin` /
`flashinfer-jit-cache` artifacts move.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Unlike post1/post2, which only rebuilt the companion wheels, post3 and
post4 carry real source changes.

post3 reverts the SM90 CUTLASS MoE backend (flashinfer-ai/flashinfer#3738
plus dependents sgl-project#4025 and sgl-project#4080). That work first shipped in 0.6.16 — it
is not in 0.6.15.post1 — so the revert returns those kernels to roughly
the state this repo already runs against today, rather than regressing
anything. It drops six public symbols and one parameter:

    preprocess_moe_weights_for_sm90_mixed_gemm_humming
    sm90_mixed_gemm_humming_weight_preprocess_trace_dispatch
    bits_from_float / float_from_bits / quant_to_fp4_val / dequant_fp4_val
    use_wfp4afp8_humming= on cutlass_fused_moe / cutlass_fused_moe_workspace_size

SGLang uses none of them. `interleave_moe_{scales,weights}_for_sm90_mixed_gemm`
move back from `fused_moe/prepare.py` to `fused_moe/core.py` but keep
their signatures and stay exported from `flashinfer.fused_moe`, which is
where mxfp4.py and mxfp4_flashinfer_cutlass_moe.py import them from.

post4 adds `from __future__ import annotations` to
`flashinfer/comm/fd_exchange.py`. That module is new in 0.6.16 and
annotates `array.array[int]`, which is only subscriptable on Python 3.12+.
`flashinfer/comm/mnnvl.py` imports it unconditionally, so on Python
3.10/3.11 — both allowed by our `requires-python = ">=3.10"` — importing
`flashinfer.comm.mnnvl` raised

    TypeError: type 'array.array' is not subscriptable

taking out the MNNVL comm-fusion and MoE all-to-all paths. Verified on a
real 3.11 interpreter: post2's fd_exchange raises, post4's imports fine.
The published Docker image is Python 3.12 and was unaffected; this only
bit source installs on older interpreters.

Checked against 0.6.15.post1 with the full symbol sweep: no API SGLang
uses is removed, and every changed signature is purely additive.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@elvischenv
elvischenv force-pushed the dev/bump-flashinfer-0.6.16 branch from a58be8d to 3c6d972 Compare August 11, 2026 07:53
@nvpohanh

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@nvpohanh

Copy link
Copy Markdown
Collaborator
  File "/actions-runner/_work/sglang/sglang/python/sglang/kernels/ops/attention/flash_attn/cute/flash_fwd_sm100.py", line 2767, in if_region_49
    if const_expr(not self.is_split_kv) or n_block_min < n_block_max:
  File "/usr/local/lib/python3.10/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/base_dsl/ast_helpers.py", line 306, in if_executor
    return executor.if_execute(
  File "/usr/local/lib/python3.10/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/base_dsl/ast_helpers.py", line 155, in if_execute
    return self._if_dynamic(
  File "/usr/local/lib/python3.10/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/cutlass_dsl/cutlass_ast_decorators.py", line 535, in _if_execute_dynamic
    return scf_gen.scf_execute_dynamic(
  File "/usr/local/lib/python3.10/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/cutlass_dsl/cutlass_ast_decorators.py", line 231, in scf_execute_dynamic
    ScfGenerator._check_region_result(
  File "/usr/local/lib/python3.10/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/cutlass_dsl/cutlass_ast_decorators.py", line 174, in _check_region_result
    raise DSLUserCodeError(
cutlass.base_dsl.common.DSLUserCodeError: 
error[TYPE_UNSTABLE_JOIN]: `n_block_first` has type `None` on one path and `Int32` on another. Where the paths come back together, both versions must be the same type.
    --> /actions-runner/_work/sglang/sglang/python/sglang/kernels/ops/attention/flash_attn/cute/flash_fwd_sm100.py:2767
     |
2765 |                         dummy_first_bias_load or (bias_max_idx0 > bias_max_idx1)
2766 |                     ) and const_expr(self.q_stage == 2)
2767 |                 if const_expr(not self.is_split_kv) or n_block_min < n_block_max:
     |                 ^
  suggestion: Make every assignment to `n_block_first` produce the same type.
  suggestion: If you need a conversion, convert it explicitly inside the branch so the result is
              consistently one type.

Looks like real failure?

@b8zhong

b8zhong commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

The fix of python/sglang/srt/layers/attention/flashinfer_backend.py is in #33997

@nvpohanh

Copy link
Copy Markdown
Collaborator

@elvischenv let's close this. we will use #33997 instead

@nvpohanh nvpohanh closed this Aug 12, 2026
@elvischenv
elvischenv deleted the dev/bump-flashinfer-0.6.16 branch August 12, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file run-ci

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants