[MM][Perf][CG] Support ViT full CUDA graph for glm4_1v image and video inference

[grYe99]

Purpose

Following #38175, this PR implements ViT CUDA graph support for glm4_1v models image and video inference . The implementation draws references from #35963 (image) and #38061 (video)

1. Functional Test
2. Benchmark in some scenarios:

• no DP VIT + eager vs no DP VIT + cuda graph.
• DP VIT + eager vs DP VIT + cuda graph.

3. Bench Serve

Test Plan

1. Functional Test

python examples/generate/multimodal/vision_language_offline.py -m glm4_1v --enable_vit_cuda_graph --modality image
python examples/generate/multimodal/vision_language_offline.py -m glm4_1v --enable_vit_cuda_graph --modality video
pytest tests/models/multimodal/generation/test_vit_cudagraph.py -k "glm4_1v"

2. Benchmark

# Image
# Single GPU:
vllm bench mm-processor \
--model zai-org/GLM-4.1V-9B-Thinking \
--max-model-len 4096 \
--dataset-name random-mm \
--random-mm-base-items-per-request 1 \
--random-mm-num-mm-items-range-ratio 0.0 \
--random-mm-bucket-config '{(224, 224, 1): 1.0}' \
--random-mm-limit-mm-per-prompt '{"image": 1, "video": 0}' \
--num-prompts 1000 \
--seed 42 \
--mm-encoder-attn-backend FLASH_ATTN \
--compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [128, 256, 512, 1024, 1536, 2048], "encoder_cudagraph_max_vision_items_per_batch": 4, "encoder_cudagraph_max_frames_per_batch": 32}'

# Multi GPU:
vllm bench mm-processor \
--model zai-org/GLM-4.1V-9B-Thinking \
--max-model-len 4096 \
--dataset-name random-mm \
--random-mm-base-items-per-request 1 \
--random-mm-num-mm-items-range-ratio 0.0 \
--random-mm-bucket-config '{(224, 224, 1): 1.0}' \
--random-mm-limit-mm-per-prompt '{"image": 2, "video": 0}' \
--num-prompts 1000 \
--seed 42 \
--tensor-parallel-size 2 \
--mm-encoder-tp-mode data \
--mm-encoder-attn-backend FLASH_ATTN \
--compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [128, 256, 512, 1024, 1536, 2048], "encoder_cudagraph_max_vision_items_per_batch": 4, "encoder_cudagraph_max_frames_per_batch": 32}'

# Video
# Single GPU:
vllm bench mm-processor \
--model zai-org/GLM-4.6V-Flash \
--max-model-len 4096 \
--dataset-name random-mm \
--random-mm-base-items-per-request 1 \
--random-mm-num-mm-items-range-ratio 0.0 \
--random-mm-bucket-config '{(224, 224, 8): 1.0}' \
--random-mm-limit-mm-per-prompt '{"image": 0, "video": 1}' \
--num-prompts 1000 \
--seed 42 \
--mm-encoder-attn-backend FLASH_ATTN \
--compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [128, 256, 512, 1024, 1536, 2048], "encoder_cudagraph_max_vision_items_per_batch": 4, "encoder_cudagraph_max_frames_per_batch": 32}'

# Multi GPU:
vllm bench mm-processor \
--model zai-org/GLM-4.6V-Flash \
--max-model-len 4096 \
--dataset-name random-mm \
--random-mm-base-items-per-request 1 \
--random-mm-num-mm-items-range-ratio 0.0 \
--random-mm-bucket-config '{(224, 224, 8): 1.0}' \
--random-mm-limit-mm-per-prompt '{"image": 0, "video": 2}' \
--num-prompts 1000 \
--seed 42 \
--tensor-parallel-size 2 \
--mm-encoder-tp-mode data \
--mm-encoder-attn-backend FLASH_ATTN \
--compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [128, 256, 512, 1024, 1536, 2048], "encoder_cudagraph_max_vision_items_per_batch": 4, "encoder_cudagraph_max_frames_per_batch": 32}'

3. Bench Serve

python -m vllm.entrypoints.cli.main serve zai-org/GLM-4.6V-Flash \
--max-model-len 4096 \
--trust-remote-code \
--tensor-parallel-size 2 \
--mm-encoder-tp-mode data  \
--mm-encoder-attn-backend FLASH_ATTN \
--compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [128, 256, 512, 1024, 1536, 2048], "encoder_cudagraph_max_vision_items_per_batch": 4, "encoder_cudagraph_max_frames_per_batch": 32}'

vllm bench serve   \
--backend openai-chat   \
--model zai-org/GLM-4.6V-Flash  \
 --base-url http://localhost:8000   \
--endpoint /v1/chat/completions   \
--dataset-name random-mm \
--random-mm-base-items-per-request 1 \
--random-mm-num-mm-items-range-ratio 0.0 \
--random-mm-bucket-config '{(224, 224, 8): 1.0}' \
--random-mm-limit-mm-per-prompt '{"image": 0, "video": 2}' \
--num-prompts 100 \
--seed 42 

Test Result

1.Functional Test

# python examples/generate/multimodal/vision_language_offline.py -m glm4_1v --enable_vit_cuda_graph --modality image
--------------------------------------------------
<think>Got it, let's look at the image. There are cherry blossom trees with pink flowers, and in the background, there's a tall tower, which is the Tokyo Skytree, I think. The sky is clear blue, and the cherry blossoms are in the foreground, framing the tower. So the content
--------------------------------------------------
<think>Got it, let's look at the image. There are cherry blossom trees with pink flowers, and in the background, there's a tall tower, which is the Tokyo Skytree, right? The sky is clear blue, and the cherry blossoms are in the foreground, framing the tower. So the content includes
--------------------------------------------------
<think>Got it, let's look at the image. There are cherry blossom trees with pink flowers, and in the background, there's a tall tower, which is the Tokyo Skytree, right? The sky is clear blue, and the cherry blossoms are in full bloom, framing the tower. So the content includes
--------------------------------------------------
<think>Got it, let's look at the image. There are cherry blossom trees with pink flowers, and in the background, there's a tall tower, which is the Tokyo Skytree, right? The sky is clear blue, and the cherry blossoms are in the foreground, framing the tower. So the content includes
--------------------------------------------------

# python examples/generate/multimodal/vision_language_offline.py -m glm4_1v --enable_vit_cuda_graph --modality video
--------------------------------------------------
<think>Got it, let's figure out why this video is funny. First, look at the elements: a young child wearing oversized glasses, which are way too big for their face. That's a common humorous setup—kids in adult-sized items look comical. Then, the child is engrossed in a book
--------------------------------------------------
<think>Got it, let's figure out why this video is funny. First, look at the elements: a baby wearing oversized glasses, which are way too big for a baby's face. Babies are usually cute, but the mismatch between the big glasses and the baby's small face creates a humorous contrast. Also, the
--------------------------------------------------
<think>Got it, let's figure out why this video is funny. First, look at the elements: a young child with oversized glasses, which are probably not meant for them, so that's a funny contrast. The child is "reading" a book, but maybe the glasses make them look like an adult trying to
--------------------------------------------------
<think>Got it, let's figure out why this video is funny. First, look at the elements: a young child wearing oversized glasses, which are way too big for their face. That's a common humorous setup—kids in adult accessories. Then, the child is engrossed in a book, maybe pretending to
--------------------------------------------------

2. Benchmark (encoder_forward_ms)

• Image Inference:

Single GPU (zai-org/GLM-4.1V-9B-Thinking, 1xRTX4090, random-mm, 1000 prompts):

Backend	Mean	P99
FLASH_ATTN	+2.88% (4.17ms -> 4.05ms)	+48.34% (9.33ms -> 4.82ms)

Multi GPU (zai-org/GLM-4.1V-9B-Thinking, 2xRTX4090, random-mm, 1000 prompts):

Backend	Mean	P99
FLASH_ATTN	+62.73% (6.01ms -> 2.24ms)	+62.43% (24.73ms -> 9.29ms)

• Video Inference:

Single GPU (zai-org/GLM-4.6V-Flash,, 1xRTX4090, random-mm, 1000 prompts):

Backend	Mean	P99
FLASH_ATTN	+37.47% (7.26ms -> 4.54ms)	+63.07% (25.78ms -> 9.52ms)

Multi GPU (zai-org/GLM-4.6V-Flash, 2xRTX4090, random-mm, 1000 prompts):

Backend	Mean	P99
FLASH_ATTN	+66.77% (9.51ms -> 3.16ms)	+62.44% (26.17ms -> 9.83ms)

3. Bench Serve
   eager

============ Serving Benchmark Result ============
Successful requests:                     100       
Failed requests:                         0         
Benchmark duration (s):                  28.34     
Total input tokens:                      110200    
Total generated tokens:                  12800     
Request throughput (req/s):              3.53      
Output token throughput (tok/s):         451.65    
Peak output token throughput (tok/s):    4100.00   
Peak concurrent requests:                100.00    
Total token throughput (tok/s):          4340.10   
---------------Time to First Token----------------
Mean TTFT (ms):                          18770.72  
Median TTFT (ms):                        19287.97  
P99 TTFT (ms):                           25433.90  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          71.09     
Median TPOT (ms):                        67.74     
P99 TPOT (ms):                           187.21    
---------------Inter-token Latency----------------
Mean ITL (ms):                           70.53     
Median ITL (ms):                         24.00     
P99 ITL (ms):                            236.64    
==================================================

cuda graph

============ Serving Benchmark Result ============
Successful requests:                     100       
Failed requests:                         0         
Benchmark duration (s):                  26.45     
Total input tokens:                      110200    
Total generated tokens:                  12800     
Request throughput (req/s):              3.78      
Output token throughput (tok/s):         483.91    
Peak output token throughput (tok/s):    4200.00   
Peak concurrent requests:                100.00    
Total token throughput (tok/s):          4650.05   
---------------Time to First Token----------------
Mean TTFT (ms):                          16884.68  
Median TTFT (ms):                        17274.28  
P99 TTFT (ms):                           23563.19  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          71.04     
Median TPOT (ms):                        68.08     
P99 TPOT (ms):                           170.80    
---------------Inter-token Latency----------------
Mean ITL (ms):                           70.48     
Median ITL (ms):                         23.95     
P99 ITL (ms):                            239.59    
==================================================

Note

Glm4vVisionAttention not support --mm-encoder-attn-backend FLASHINFER yet, thus only test in FLASH_ATTN. It will be supported in another PR.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[grYe99]

@claude review

[gemini-code-assist]

Code Review

This pull request implements CUDA graph support for the GLM-4V model by introducing a fused Triton kernel for position-embedding interpolation and refactoring the vision encoder's metadata preparation. Key changes include the addition of a native PyTorch fallback for interpolation, the implementation of the SupportsEncoderCudaGraph protocol, and optimizations to rotary position ID generation using lru_cache. Review feedback identified a potential regression in model accuracy due to the switch from bicubic to bilinear interpolation and highlighted a lack of error handling for empty input lists in the metadata preparation logic.

[shen-shanshan]

Please also update this model to:

Doc: https://docs.vllm.ai/en/latest/design/cuda_graphs_multimodal/#model-integration-via-supportsencodercudagraph

Example: https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language.py#L2466-L2469 (if needed)

CI test: https://github.com/vllm-project/vllm/blob/main/tests/models/multimodal/generation/test_vit_cudagraph.py#L44-L58

[grYe99]

@shen-shanshan done

[mergify]

Documentation preview: https://vllm--40576.org.readthedocs.build/en/40576/

[grYe99]

@DarkLight1337 Hi, could you give this PR a 'ready' label to run CI tests? Thanks!

[grYe99]

@shen-shanshan @b-mu Hi, could you help review this PR when you have time? I recently updated code to support "auto-infer compilation-config" and passed the following tests:

python examples/generate/multimodal/vision_language_offline.py -m glm4_1v --enable_vit_cuda_graph --modality image
python examples/generate/multimodal/vision_language_offline.py -m glm4_1v --enable_vit_cuda_graph --modality video
pytest tests/models/multimodal/generation/test_vit_cudagraph.py -k "glm4_1v"

and serve with --compilation-config '{"cudagraph_mm_encoder": true}'
Any suggestions are welcome!

[Isotr0py]

Can you double check if Qwen3-VL and GLM-4.1V's VisiomEmbed implementation are fully equivalent with converged numeric results?

I think GLM-4.1V uses cubic interpolation instead of Qwen3-VL's bilinear interpolation.

[grYe99]

@Isotr0py Thanks for spotting this. Indeed the original GLM-4.1V uses bicubic interpolation for vision position embeddings. I have updated code and passed functional tests.