[Frontend][torch.compile] CompilationConfig Overhaul (#20283): Set up -O infrastructure

[morrison-turnansky]

CompilationConfig Overhaul and Optimization Levels

Overview

Description of changes to VLLMConfg and CompilationConfig. The introduction of meaningful optimization levels (-O0, -O1, -O2, -O3) from GitHub Issue #20283. This change aims to improve user experience by providing intuitive optimization levels that trade startup time for performance, while consolidating and simplifying the compilation configuration system. We also have changed defaults to help users get desired out of the box performance. These defaults are determined by optimization-level. Importantly defaults are purely defaults; explicit user settings will not be overwritten.

Key Changes

1. Repurposing -O for Optimization Levels

The -O<n> flags now represent meaningful optimization levels that trade startup time for performance:

-O0: No Optimization

• Startup: Fastest startup time
• Performance: No compilation, no cudagraphs, no optimizations
• Equivalent to: --enforce-eager (deprecated)
• Use case: Development, debugging, or when startup time is critical

# CLI usage
python -m vllm.entrypoints.api_server --model microsoft/DialoGPT-medium -O0

# Python API usage
from vllm.entrypoints.llm import LLM
from vllm.config.vllm import OptimizationLevel

llm = LLM(
    model="microsoft/DialoGPT-medium",
    optimization_level=OptimizationLevel.O0
)

-O1: Quick Optimizations

• Startup: Moderate startup time
• Performance: Inductor compilation, CUDAGraphMode.PIECEWISE
• Use case: Balance for most development scenarios

# CLI usage
python -m vllm.entrypoints.api_server --model microsoft/DialoGPT-medium -O1

# Python API usage
from vllm.entrypoints.llm import LLM
from vllm.config.vllm import OptimizationLevel

llm = LLM(
    model="microsoft/DialoGPT-medium",
    optimization_level=OptimizationLevel.O1
)

-O2: Full Optimizations (Default)

• Startup: Longer startup time
• Performance: -O1 + CUDAGraphMode.FULL_AND_PIECEWISE
• Use case: Production workloads where performance is important. This is the default use case. It is also very similar to the previous default. The primary difference is that noop & fusion flags are enabled.

# CLI usage (default, so optional)
python -m vllm.entrypoints.api_server --model microsoft/DialoGPT-medium -O2

# Python API usage
from vllm.entrypoints.llm import LLM
from vllm.config.vllm import OptimizationLevel

llm = LLM(
    model="microsoft/DialoGPT-medium",
    optimization_level=OptimizationLevel.O2  # This is the default
)

-O3: Full Optimization

Still in development. Added infrastructure to prevent changing API in future
release. Currently behaves the same O2.

Troubleshooting

Common Issues

1. Startup Time Too Long: Use -O0 or -O1 for faster startup
2. Compilation Errors: Use debug_dump_path for additional debugging information
3. Performance Issues: Ensure using -O2 for production

Added functions

Added functionality determining if a model is quantized and if a model is MOE. This will be relevant for future work. Added lambda's to easily get information about configuration.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[mergify]

Documentation preview: https://vllm--26847.org.readthedocs.build/en/26847/

[morrison-turnansky]

@ProExpertProg Tracking a list of keys for determining if a model is moe/sequential here.

[morrison-turnansky]

@ProExpertProg let us know what the defaults for each mode need to be.

[ProExpertProg]

Some initial thoughts

[ProExpertProg]

Here we should ask the platform for an opinion on the defaults.

[ProExpertProg]

3 options:

• we pass default to platform, it makes modifications
• platform returns default, can use these global defaults as a starting point
• each platform owns its own defaults

Let's chat about this at the end of release meeting maybe

[morrison-turnansky]

@ProExpertProg Are we still including platform specific modifications? Are we pushing that until inductor partition is on by default?

[ProExpertProg]

I think for now we can just do this on is_cuda_alike() platforms. But we're gonna need a more robust approach here

[ProExpertProg]

This was actually changed recently; we should still keep pass config separate, but this reason is no longer true. Can be done in follow up cc @ilmarkov

[ProExpertProg]

Why this? Aren't you giving it 2 strings?

[morrison-turnansky]

cudagraph mode could be None (not the enum). I didn't want to write None to the user in the log, but yes should clean it up. At one point I had moved this after the defaults were applied, and then I saw a failure in Example Test. I wanted to keep everything exactly the same, but I am now more convinced that the test is just very flaky.