[Feature] Decode Context Parallel support for GPU model runner v2

[yewentao256]

Purpose

Part of the #32455

Enable DCP for V2, with support of cuda graph

Performance slightly slower than V1, but we haven't really optimized the performance, both for V2 and DCP, we can optimize performance later in following up PRs

Test

export MODEL="deepseek-ai/DeepSeek-V2-lite"
export VLLM_USE_V2_MODEL_RUNNER=1
vllm serve $MODEL -tp 4  --port 9256 --enable-expert-parallel --max_num_seqs 128 -dcp 4

ACC

lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:9256/v1/completions,model=$MODEL,num_concurrent=1024" --tasks gsm8k
# export VLLM_USE_V2_MODEL_RUNNER=1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.3844|±  |0.0134|
|     |       |strict-match    |     5|exact_match|↑  |0.3806|±  |0.0134|

# export VLLM_USE_V2_MODEL_RUNNER=0
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.3768|±  |0.0133|
|     |       |strict-match    |     5|exact_match|↑  |0.3730|±  |0.0133|

Perf

vllm bench serve --model $MODEL  --dataset-name random --host 127.0.0.1 --port 9256 --random-input-len 2 --random-output-len 512 --request-rate inf --num-prompts 128 --num-warmups 16
# V2
============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  5.84      
Total input tokens:                      128       
Total generated tokens:                  65536     
Request throughput (req/s):              21.90     
Output token throughput (tok/s):         11214.20  
Peak output token throughput (tok/s):    11648.00  
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          11236.10  
---------------Time to First Token----------------
Mean TTFT (ms):                          187.06    
Median TTFT (ms):                        188.50    
P99 TTFT (ms):                           200.01    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          11.04     
Median TPOT (ms):                        11.04     
P99 TPOT (ms):                           11.14     
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.04     
Median ITL (ms):                         11.00     
P99 ITL (ms):                            12.88     
==================================================

# V1
============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  5.50      
Total input tokens:                      128       
Total generated tokens:                  65536     
Request throughput (req/s):              23.28     
Output token throughput (tok/s):         11917.34  
Peak output token throughput (tok/s):    12313.00  
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          11940.62  
---------------Time to First Token----------------
Mean TTFT (ms):                          138.40    
Median TTFT (ms):                        149.05    
P99 TTFT (ms):                           167.96    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.44     
Median TPOT (ms):                        10.43     
P99 TPOT (ms):                           10.50     
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.44     
Median ITL (ms):                         10.43     
P99 ITL (ms):                            13.39     
==================================================

[gemini-code-assist]

Code Review

This pull request introduces Decode Context Parallel (DCP) support for the v2 GPU model runner. The changes involve updating attention utilities, block table management, and the model runner itself to handle sharded KV caches for DCP. The core logic is implemented in a Triton kernel for slot mapping. My review focuses on code structure and maintainability. I've identified an opportunity to reduce code duplication for better long-term maintenance.

[WoosukKwon]

LGTM. Thanks for the PR!
I will follow up with simple refactoring.