Synchronize distributed FlashInfer autotuning and persist cache

[jiahanc]

Need flashinfer version update

Purpose

Distributed FlashInfer autotuning currently disables the persistent cache and reruns tuning on every server startup. Ranks also profile tactics independently, so GPU timing noise can cause them to select different tactics.

flashinfer-ai/flashinfer#3187 adds set_autotune_process_group, which averages tactic timings across ranks so all ranks select the same tactic.

This PR wires that API into vLLM:

• Broadcast and load an existing cache before distributed autotuning.
• Run autotuning on every rank with synchronized timings over the world CPU process group.
• Save the resulting cache on rank 0.
• Broadcast and load the final cache on every rank.
• Clear the autotune process group in a finally block.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

[nehashah-eng]

Hey @jiahanc, do you intend to continue working on this PR. This is something which would be really useful for improving vllm start engine time ... and help with autoscaling etc. Thank you for working on it.

Nevermind you mentioned you need flash infer bump

[wzhao18]

@nehashah-eng Yes, we will reopen this with flashinfer 0.16.0, which has the feature needed.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @jiahanc.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork