Add option to use torch._inductor.standalone_compile

[zou3519]

This PR adds the option to use torch._inductor.standalone_compile to perform compilation instead of compile_fx. The goal of standalone_compile is to remove the hacks around vLLM's usage of compile_fx, we want to migrate to using it in PyTorch 2.8.

standalone_compile replaces how vLLM interacts with the torch.compile caches. Instead of vLLM trying to redirect them into its torch_compile_cache folder, vLLM can pass standalone_compile a filepath that is inside of the torch_compile_cache folder and standalone_compile will write the full precompiled artifact to it.

Right now this option is hidden behind an envvar (VLLM_TEST_STANDALONE_COMPILE). It is also not tested in vLLM CI (vLLM CI only tests against PyTorch 2.6). This option also needs more testing before we turn it on by default for PyTorch 2.8+. I am putting this PR out so that we can merge something that we can keep developing on top of.

Test Plan:

• Run https://gist.github.com/zou3519/aebb622714e80f4cd4c369472f2372cd with or without VLLM_TEST_STANDALONE_COMPILE

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zou3519.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[houseroad]

Overall looks reasonable to me.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zou3519.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[houseroad]

Let's rebase?

[tlrmchlsmth]

@zou3519 Could you explain this comment? Would like to understand the sketchiness

[zou3519]

This is a pre-existing problem, the pre-existing InductorAdaptor also has this code in it.

If a function being compiled returns a single tensor, e.g. f(x) = x.sin(), and we compile this, then Inductor is always passed a graph that returns a (Tensor,) and Inductor returns a compiled artifact that returns a (Tensor,). Dynamo is responsible for unpacking this back into a single tensor via the bytecode it generates.

vLLM takes the bytecode that Dynamo generates and turns it into some Python code that wraps the compiled artifact. However, since we also need to manually do the unpacking here, I suspect that vLLM is not doing that process correctly.

[tlrmchlsmth]

Makes sense, thanks for the explanation.

[youkaichao]

since we also need to manually do the unpacking here, I suspect that vLLM is not doing that process correctly

what does this mean? as you mentioned, we have special handling logic for the case when the original graph returns a single tensor, and I think vLLM is correct here.

[zou3519]

what does this mean? as you mentioned, we have special handling logic for the case when the original graph returns a single tensor, and I think vLLM is correct here.

In torch.compile, the handling logic for what happens when the original graph returns a single tensor is in the Dynamo-produced bytecode. In vLLM, the handling logic is in the InductorAdaptor. I would expect it to be in the Dynamo-produced bytecode.

[zou3519]

@youkaichao mentioned to me that vLLM does use the Dynamo-produced bytecode directly so... this needs more investigation

[houseroad]

Let's track it in the following PR? (like creating some issue?)

[youkaichao]

this is a user-facing config, and I don't think we should add it here. If you want to switch the behavior, you can have an env var like VLLM_TEST_XXX , that's less user-facing and shows it is not intended for external use cases.

[zou3519]

Thanks for pointing that out, I'll update this.

[zou3519]

Updated

[youkaichao]

I think it should be the compiler's responsibility to summarize the graph and generate the key. The compiler manager just calls the compiler. It should not provide the graph index.

[zou3519]

@youkaichao The "key" is the file that the compiled artifact gets saved to. I don't think it's the compiler's responsibility to specify the key. Consider gcc - by default it will write a file to a.out, otherwise it's the user's responsibility to pick a name for their binary.

From the Inductor side, the inductor hash key computation is not public API and shouldn't be.

[zou3519]

@youkaichao The "key" is the file that the compiled artifact gets saved to. I don't think it's the compiler's responsibility to specify the key. Consider gcc - by default it will write a file to a.out, otherwise it's the user's responsibility to pick a name for their binary.

From the Inductor side, the inductor hash key computation is not public API and shouldn't be.

[houseroad]

Btw, do we need to use the key for hash computation to validate the cache?

[zou3519]

Btw, do we need to use the key for hash computation to validate the cache?

We do not use this key to validate the vLLM cache. The "key" here is just a file path to the compiled artifact.

The "key" that is returned by InductorAdapator has semantic meaning to torch.compile. But vLLM manages its own cache and compilation, so this key has no semantic meaning to vLLM.

[houseroad]

we should add a unittest for this adaptor. Feel free to do it in the following PR.

[zou3519]

In the following PR I will turn on InductorStandaloneAdaptor for PyTorch >= 2.8, which will make it so that it gets tested in the "torch nightly" vLLM CI.

[houseroad]

Do we need to consider the none pytorch source code in the hash?

[zou3519]

What do you mean? The compute_hash function here is the same as the compute_hash function in InductorAdaptor. I can put this into a helper function for better code reuse.

[houseroad]

Offline synced, it will be called by vLLM to compute the overall cache key.

[houseroad]

Looks good to me.