[Feature][Perf] Support Selective CPU Weight Offloading

[wzhao18]

Purpose

This PR adds support to selectively offload parameters to CPU based on name matching. One use case is to only offload experts weights for MoE weights, which is useful for low-concurrency settings. This is turned on by passing argument --cpu-offload-params

Test Plan

Tested offloading Kimi K2 NVFP4 on one GB300.

VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1 python3 -m vllm.entrypoints.openai.api_server \
	--model nvidia/Kimi-K2-Thinking-NVFP4 \
	--trust-remote-code \
        --cpu-offload-gb 350 \
        --load-format dummy \
        --cpu-offload-params w13_weight w2_weight

Benchmarking single-user throughput:

vllm bench serve \
  --backend vllm \
  --endpoint /v1/completions \
  --num-prompts 5 \
  --dataset-name random \
  --input-len 100 \
  --output-len 100 \
  --max-concurrency 1 \
  --trust-remote-code \
  --num-warmups 2

Test Result

Before: 15 tok/s

============ Serving Benchmark Result ============
Successful requests:                     5         
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  32.17     
Total input tokens:                      500       
Total generated tokens:                  500       
Request throughput (req/s):              0.16      
Output token throughput (tok/s):         15.54     
Peak output token throughput (tok/s):    17.00     
Peak concurrent requests:                2.00      
Total token throughput (tok/s):          31.08     
---------------Time to First Token----------------
Mean TTFT (ms):                          302.71    
Median TTFT (ms):                        326.77    
P99 TTFT (ms):                           327.42    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          61.94     
Median TPOT (ms):                        61.93     
P99 TPOT (ms):                           61.95     
---------------Inter-token Latency----------------
Mean ITL (ms):                           61.94     
Median ITL (ms):                         61.90     
P99 ITL (ms):                            62.72     
==================================================

Offloading MoE weights only: 31 tok/s

============ Serving Benchmark Result ============
Successful requests:                     5         
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  15.81     
Total input tokens:                      500       
Total generated tokens:                  500       
Request throughput (req/s):              0.32      
Output token throughput (tok/s):         31.62     
Peak output token throughput (tok/s):    34.00     
Peak concurrent requests:                2.00      
Total token throughput (tok/s):          63.24     
---------------Time to First Token----------------
Mean TTFT (ms):                          244.12    
Median TTFT (ms):                        272.78    
P99 TTFT (ms):                           273.59    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          29.48     
Median TPOT (ms):                        29.48     
P99 TPOT (ms):                           29.49     
---------------Inter-token Latency----------------
Mean ITL (ms):                           29.48     
Median ITL (ms):                         29.47     
P99 ITL (ms):                            29.85     
==================================================

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces a useful feature for selectively offloading model parameters to the CPU, which can significantly improve performance in memory-constrained scenarios, as demonstrated by the provided benchmarks. The implementation is clear and follows existing patterns in the codebase. The changes to the configuration and model loading logic are well-integrated. The parameter name matching logic, while a bit subtle, appears correct and robust for its intended purpose. Overall, this is a solid contribution that enhances the flexibility and performance of vLLM.

[mgoin]

Nice UX, LGTM!

[ehfd]

@wzhao18 Is it possible to use regex syntax like in e.g., .ffn_.*_exps. (possible in llama.cpp)?