build: update FlashInfer to 0.6.16 - #869
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b3e4b7a8f0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| nvidia-cutlass-dsl-libs-cu13==4.6.0 | ||
| torch==2.11.0 | ||
| flashinfer-python==0.6.16.dev20260730 | ||
| flashinfer-python==0.6.16 |
There was a problem hiding this comment.
This pin is also consumed by test/ci_system/install_deps.sh during CUDA setup: that script derives FLASHINFER_VERSION=0.6.16 from this file and then force-installs https://github.com/flashinfer-ai/flashinfer/releases/download/v0.6.16/flashinfer_cubin-0.6.16-py3-none-any.whl. The FlashInfer cubin indexes currently do not publish flashinfer_cubin-0.6.16 (they list rc5 and older stable builds), so any CUDA CI path that reaches step 8 will retry and fail even though flashinfer-python itself resolves. Please pin a version whose cubin wheel exists or update the installer to use an available cubin source.
Useful? React with 👍 / 👎.
b3e4b7a to
dbcf175
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dbcf175468
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| nvidia-cutlass-dsl-libs-cu13==4.6.0 | ||
| torch==2.11.0 | ||
| flashinfer-python==0.6.16.dev20260730 | ||
| flashinfer-python==0.6.16 |
There was a problem hiding this comment.
For Kimi-K3 on B300, the packaged tuning-cache lookup includes the installed flashinfer-python version in the filename (flashinfer=0.6.16.json after this pin), but the repo still only ships tactic tables named and metadata-pinned for 0.6.16.dev20260730. That means the pre-swept tables can no longer be found/loaded and these deployments fall back to slow lazy autotuning despite shipping cache data; regenerate or update the packaged tactics alongside the stable FlashInfer pin.
Useful? React with 👍 / 👎.
Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com>
dbcf175 to
54551bc
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 54551bc959
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| apache-tvm-ffi==0.1.13 | ||
| tokenspeed-flashmla==1.0.0.post20260615 | ||
| tokenspeed-mla==0.2.1 | ||
| tokenspeed-mla==0.2.2 |
There was a problem hiding this comment.
CUDA installs include this file via setup.py's install_requires, and CI runs pip install tokenspeed-kernel/python/ before any optional in-tree MLA override. PyPI's release history currently stops at 0.2.1 (Jul 21, 2026), so tokenspeed-mla==0.2.2 has no matching wheel/sdist and the CUDA install resolver fails before tests can run; publish 0.2.2 first or keep this at an available version.
Useful? React with 👍 / 👎.
… on stale tables The flashinfer 0.6.16 pin bump (#869) silently orphaned the packaged K3 tactic tables: their filenames embed the swept flashinfer version (0.6.16.dev20260730), so the loader — correctly — refuses them and every K3 launch falls back to the startup autotune window, whose native picks lose 16-83% on prefill-sized buckets. Tactic indices do not survive version changes (the trtllm-gen kernel enumeration reorders between builds: 17/21 ep=8 buckets and 12/21 tp=8 buckets map to different indices for the same tile families), so tables must be re-swept on every pin bump, never renamed. - Re-sweep both layouts (ep=8,tp=1 and ep=1,tp=8) on flashinfer 0.6.16 stable, B300, tune_max=8192; load-verified on current main. - Warn on stale tables: a lookup miss that finds a same-model/layout/ device table swept on a different flashinfer version now logs a WARNING naming the orphaned file and the re-sweep command, instead of the generic INFO. A pin bump that forgets the tables becomes visible in serving logs rather than a silent perf regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: dongjiyingdjy <87510204+dongjiyingdjy@users.noreply.github.com>
3 participants
FlashInfer 0.6.16 JIT-cache binaries require
TVMFFIGetCustomAllocator, first exported by apache-tvm-ffi 0.1.13. The matchingtokenspeed-mla0.2.2 andtokenspeed-cutedsl-kda0.1.0.post20260801 releases let TokenSpeed adopt that ABI through normal dependency resolution on both x86_64 and aarch64.tokenspeed-flashkdaremains at 0.0.1.post20260729 because it is already the latest release and upstream FlashKDA has no newer commit.Validation: pre-commit; focused CI-system tests; dependency resolution with the released packages; direct aarch64 wheel resolution.