Repository navigation
[Dependency] Upgrade FlashInfer to 0.7.1 - #60948
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
wzhao18
added
the
ready-run-all-tests
94d1d68 to
da3cf3a
Compare
|
/ci run nightly |
|
✅ Triggered Buildkite CI #93957 for commit |
|
This pull request has merge conflicts that must be resolved before it can be |
Update the shared Python pin for standard CUDA and Rubin builds, preserving the common download-kernels installation flow. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Avoid the automatic small-BH path, whose packed metadata cache reads version counters unavailable on inference tensors. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
da3cf3a to
58d5922
Compare
|
/ci run nightly |
|
✅ Triggered Buildkite CI #94016 for commit |
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
|
/ci run nightly |
|
✅ Triggered Buildkite CI #94017 for commit |
|
/ci retry |
|
✅ Queued 6 failed job(s) for retry in Buildkite CI #94017. |
|
/ci retry |
|
✅ Queued 6 failed job(s) for retry in Buildkite CI #94017. |
|
@zyongye @jeejeelee The lm-eval-humming tests are failing in main as well. |
|
/ci run nightly |
|
✅ Triggered Buildkite CI #94519 for commit |
|
/ci retry |
|
✅ Queued 12 failed job(s) for retry in Buildkite CI #94519. |
|
/ci retry |
|
✅ Queued 5 failed job(s) for retry in Buildkite CI #94519. |
|
/ci retry |
|
✅ Queued 10 failed job(s) for retry in Buildkite CI #94519. |
CI failure triage: nightly #94519None of the confirmed failures come from the FlashInfer 0.7.1 bump. Only Snapshot E2E is still unconfirmed.
|
|
/ci retry |
|
✅ Queued 4 failed job(s) for retry in Buildkite CI #94519. |
|
/ci retry |
|
✅ The previous CI build is still running: https://buildkite.com/vllm/ci/builds/94519 |
|
retrying to see if Initialized snapshot can pass, since we don't have much data about it on main |
|
/ci run |
|
✅ Triggered Buildkite CI #94647 for commit |
|
@khluu Ready for merge? |
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Overview
Upgrade FlashInfer from
0.7.0.post1to0.7.1for standard CUDA and Rubin builds. Explicitly select the CuTe DSL KDA prefill backend to avoid the inference-tensor crash observed in the B200 Kimi K3 CI job.Claims
flashinfer-pythonpin to0.7.1, retaining upstream's commonflashinfer download-kernelsinstallation flow for CUDA and Rubin.Validation
On an allocated NVIDIA GB300 node with FlashInfer 0.7.1, PyTorch 2.13.0+cu130, and nvidia-cutlass-dsl 4.8.0, using an isolated checkout and the existing environment:
pytest -v tests/models/kimi_k3 --ignore=tests/models/kimi_k3/test_prefix_cache.py tests/kernels/attention/test_kimi_k3_mla_fused_epilogue.py— 260 passed, 154 skipped. This is the B200 Kimi K3 CI test selection, executed on GB300. Tests used the environment's.venv/bin/python -m pytest..venv/bin/python -m pre_commit run --files requirements/cuda.txt vllm/models/kimi_k3/nvidia/kda.py— passed after rebasing.git diff --check— passed.GPU tests were run before the final rebase; the rebase did not change Kimi implementation or test files. B200 execution, full model accuracy evaluations, and standard/Rubin container builds have not been performed for this revision. The earlier MI250 EAGLE3 acceptance failure remains unresolved and should be reassessed in CI; it is not established to be an existing main failure.
Details
FlashInfer's automatic small-BH prefill path caches packed sequence metadata using
cu_seqlens._version. Inference tensors have no version counter, so this raises before kernel launch. Explicitbackend="cute-dsl"avoids that path without changing PyTorch inference mode.This bypasses the small-BH optimization. GB300 CUPTI microbenchmarks (BF16, D128, bounded gate, cold L2, eager GPU timing; median of three blocks of 20 samples) measured:
Auto was measured with ordinary metadata tensors to avoid the crash. Explicit CuTe DSL was faster than FlashKDA for these cases, but 12–26% slower than working auto selection for the two longer H12 cases. These are microbenchmarks, not end-to-end serving measurements. Output and final-state correctness were checked before timing.
Rebased onto upstream main
838526c7e5. Upstream removed separate cubin/JIT-cache pins in favor offlashinfer download-kernels; this PR preserves that change. Rubin continues to use the shared FlashInfer pin.The initial duplicate-work search found no other open PR implementing this version upgrade; this updates the existing PR rather than opening a duplicate. No associated issue. AI assistance (Codex) was used for implementation, investigation, and validation.
Pull Request Checklist
/pr-checklistskill.