Bump FlashInfer to 0.6.15.post1 - #31927
Conversation
github-actions
Bot
added
dependencies
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the FlashInfer dependency to version 0.6.15.post1 across the Dockerfile, pyproject.toml, and version assertions. It adapts the CuteDSL MoE runner to use the FlashInfer ActivationType enum, removes a workaround for correction_bias_dtype in DeepSeek-V2, and increases the speed threshold in a GLM-5.2 test. Feedback is provided to use a dictionary mapping instead of multiple conditional statements when resolving activation types to improve code maintainability.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
/rerun-test test/registered/models_e2e/test_dsa_glm52_nvfp4_tp_mtp.py |
|
Results for 🚀 |
flashinfer's trtllm_batch_decode_with_kv_cache_mla instantiates a fresh runner per call, so its internal counter-buffer cache never survives across decode steps and it torch.zeros() a new one every call by default. Own a persistent buffer in TRTLLMMLABackend and DeepseekSparseAttnBackend and pass it via multi_ctas_kv_counter_buffer= instead. DSA's trtllm sparse-indexer path flattens every token in a prefill/ extend/verify chunk into its own decode-shaped batch entry, so its batch dimension can exceed max_running_requests; grow the buffer on demand there instead of assuming a fixed bound. Cherry-picked from #31607 (squashed from 0042fd9, 121c901, a84b529, ed135a8).
|
/rerun-test test/registered/models_e2e/test_dsa_glm52_nvfp4_tp_mtp.py |
|
Results for 🚀 |
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Seem main is broken |
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
3 participants
Summary
flashinfer_python,flashinfer-cubin, and the optional JIT cache from 0.6.14 to 0.6.15.post1Why
The original 0.6.15 bump was reverted in #31625 after host-side regressions reduced long-context serving throughput. FlashInfer 0.6.15.post1 includes both follow-up fixes needed to recover performance:
TuningConfigobjects for equivalent inputs. This prevents fresh, unequal initializer closures from growing a single_find_nearest_profilehash-collision chain, which made per-call host overhead O(n) and total overhead O(n²) in long-running MLA decode workloads. It also improves persisted autotuner-cache validation and healing.This release therefore addresses the issues that led to the revert while retaining the 0.6.15 compatibility changes.
Validation
python/pyproject.tomlTOML parsing0.6.15.post1 > 0.6.15git diff --checkGPU model E2E tests were not run locally.
CI States
Latest PR Test (Base): ❌ Run #29940919826
Latest PR Test (Extra): ❌ Run #29940919610