[MM][CG] Support ViT full CUDA graph for Ernie-4.5-VL image inference by qyYue1389 · Pull Request #45254 · vllm-project/vllm

qyYue1389 · 2026-06-11T07:51:42Z

Purpose

Adds encoder (ViT) CUDA graph support for Ernie4_5_VLMoeForConditionalGeneration (image inputs), under the ViT Full CUDA Graph tracker #38175. Follows the Qwen3-VL SupportsEncoderCudaGraph reference.

Key Changes

Splits the inline rotary / cu_seqlens / max_seqlen computation out of Ernie4_5_VisionTransformer.forward into prepare_encoder_metadata(), so it can be precomputed on the host and fed into the captured graph via fixed buffers. Ernie consumes a single rotary freqs tensor, so the buffer set is 4 keys (pixel_values, rotary_pos_emb, cu_seqlens, max_seqlen)
Capture boundary (ViT only) Ernie's spatial-merge & projection live in a separate VariableResolutionResamplerModel whose temporal path does host-side work (grid_thw.cpu().numpy() & dynamic index_select) that can't be captured. The resampler therefore runs eagerly in postprocess_encoder_output (graph path) and encoder_eager_forward (eager fallback)
Image only Video has use_temporal_conv in the resampler that changes the output token count (code), which needs separate handling. A separate PR is needed for video and will be raised after this one is merged.

Test

1. Functional

Added ernie45_vl (image, load_format=dummy) to tests/models/multimodal/generation/test_vit_cudagraph.py. pytest tests/models/multimodal/generation/test_vit_cudagraph.py::test_vit_cudagraph_image[ernie45_vl] - E2E encoder CG capture & replay & eager-resampler postprocess

# pytest tests/models/multimodal/generation/test_vit_cudagraph.py::test_vit_cudagraph_image[ernie45_v
l]
============================================================ test session starts ============================================================
platform linux -- Python 3.12.3, pytest-9.0.3, pluggy-1.6.0
rootdir: /workspace/vllm
configfile: pyproject.toml
plugins: anyio-4.13.0, timeout-2.4.0, asyncio-1.4.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 1 item                                                                                                                            

tests/models/multimodal/generation/test_vit_cudagraph.py .                                                                            [100%]

============================================================= warnings summary ==============================================================
...
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================= 1 passed, 17 warnings in 78.43s (0:01:18) =================================================

2. Benchmark (H100 80GB)

# CUDA graph
vllm bench mm-processor \
  --model baidu/ERNIE-4.5-VL-28B-A3B-PT \
  --trust-remote-code \
  --max-model-len 4096 \
  --dataset-name random-mm \
  --random-mm-base-items-per-request 1 \
  --random-mm-num-mm-items-range-ratio 0.0 \
  --random-mm-bucket-config '{(224, 224, 1): 1.0}' \
  --random-mm-limit-mm-per-prompt '{"image": 1, "video": 0}' \
  --num-prompts 1000 --seed 42 \
  --compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [128, 256, 512, 1024, 1536, 2048], "encoder_cudagraph_max_vision_items_per_batch": 4}'

# eager baseline
vllm bench mm-processor \
  --model baidu/ERNIE-4.5-VL-28B-A3B-PT \
  --trust-remote-code \
  --max-model-len 4096 \
  --dataset-name random-mm \
  --random-mm-base-items-per-request 1 \
  --random-mm-num-mm-items-range-ratio 0.0 \
  --random-mm-bucket-config '{(224, 224, 1): 1.0}' \
  --random-mm-limit-mm-per-prompt '{"image": 1, "video": 0}' \
  --num-prompts 1000 --seed 42 \

encoder_forward_ms	Mean	Median	Std	P99
eager	20.19	19.89	1.69	30.88
CUDA graph	5.33	5.31	0.22	5.77
speedup	3.79×	3.74×	—	5.35×

3. Accuracy (CG on vs off)

CG toggled via compilation_config={"cudagraph_mm_encoder": ...} on the same PR branch with lmms-eval and greedy decoding

benchmark	CG ON	CG OFF	Δ
AI2D	0.6548 ± 0.0086	0.6564 ± 0.0085	≈0.2σ
MMStar	0.1241 ± 0.0088	0.1284 ± 0.0089	≈0.5σ

Both within stderr and show enabling the ViT CUDA graph does not change accuracy.

Output parity (comparing raw outputs directly): greedy-decoded 50 images with the encoder CUDA graph ON vs OFF: 47/50 (94%) byte-identical outputs. The 3 diffs are single-letter flips on near-tie questions, expected since the graph replays from padded fixed buffers (numerically close, not bit-identical). The ViT CUDA graph does not change the model's outputs.

(MMStar's absolute is low for both arms, but that's the benchmark, not a vision/setup issue. The same model scores AI2D 0.65 in this setup, so vision is fine. MMStar's fine-grained multiple-choice is hard for Ernie here. It affects eager and CG identically, so it doesn't affect the accuracy comparison.)

mergify · 2026-06-11T07:52:18Z

Documentation preview: https://vllm--45254.org.readthedocs.build/en/45254/

qyYue1389 · 2026-06-12T01:59:38Z

Hi @shen-shanshan, could you also review when getting a chance? Thanks!

Isotr0py · 2026-06-12T09:41:04Z

Can you verify the multimodal accuracy with ViT CG as well?

qyYue1389 · 2026-06-14T01:14:11Z

Thanks @Isotr0py!

Finished verification on ERNIE-4.5-VL-28B-A3B on H100 with benchmarks AI2D and MMStar, toggling the ViT CUDA graph via compilation_config={"cudagraph_mm_encoder": ...}

Both within stderr and show enabling the ViT CUDA graph does not change accuracy
Updated details in the PR description

qyYue1389 · 2026-06-16T17:46:59Z

Hi @shen-shanshan @Isotr0py, could you review the PR when getting a chance? Thanks!

mergify · 2026-06-16T19:35:27Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @qyYue1389.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

shen-shanshan

I left some comments and questions.

shen-shanshan · 2026-06-18T06:30:54Z

 | Architecture | Models | CG for Image | CG for Video | Dual-Path Graph |
 | ------------ | ------ | ------------ | ------------ | --------------- |
 | `DeepseekOCRForCausalLM` | `DeepSeek-OCR` | ✅︎ | ❌︎ | ✅︎ |
+| `Ernie4_5_VLMoeForConditionalGeneration` | `ERNIE-4.5-VL` | ✅︎ | ❌︎ | ❌︎ |


As far as I know, this model supports video inference, does this PR support?

Yes it supports video inference, but this PR supports image only for now

ERNIE's video path applies a use_temporal_conv in the resampler that changes the output token count (code), which needs separate handling. Thus I plan to work on it as a follow-up PR

This PR only adds the image encoder CUDA graph, and Eager video inference is unaffected. Thus ❌ in the Video column for now.

Does the follow-up PR plan sound good to you or any suggestions? Thanks!

shen-shanshan · 2026-06-18T06:37:49Z

+    def get_input_modality(self, mm_kwargs: dict[str, Any]) -> str:
+        if "image_grid_thw" in mm_kwargs:
+            return "image"
+        raise AssertionError("This line should be unreachable.")


If it only supports image modality, this method is not needed, FYI, #44484.

Thank you for pointing out! Removed

shen-shanshan · 2026-06-18T06:38:31Z

+    def _get_pixel_values_by_modality(self, mm_kwargs: dict[str, Any]) -> torch.Tensor:
+        return mm_kwargs["pixel_values"]
+
+    def _get_grid_thw_by_modality(self, mm_kwargs: dict[str, Any]) -> list[list[int]]:
+        grid_thw = mm_kwargs["image_grid_thw"]
+        if not isinstance(grid_thw, list):
+            grid_thw = grid_thw.tolist()
+        return grid_thw


Thank you for pointing out!

shen-shanshan · 2026-06-18T06:55:28Z

+        grid_thw_list = self._get_grid_thw_by_modality(batch_mm_kwargs)
+        num_valid = sum(t * h * w for t, h, w in grid_thw_list)
+        grid_thw = torch.tensor(grid_thw_list, device=output.device)
+        image_embeds = self.resampler_model(output[:num_valid], grid_thw)


In VariableResolutionResamplerModel, if use_temporal_conv=False, I suppose it's CUDA graph compatible? Since it only contains linear layer and activation layer.

Yes. If use_temporal_conv=False the resampler is just linear and activation and would be CUDA graph compatible.

However ERNIE-4.5-VL-28B-A3B ships with use_temporal_conv=True from Ernie4_5_VLMoEConfig.__init__ in the model's bundled configuration_ernie4_5_vl.py
With that flag, VariableResolutionResamplerModel.forward always takes the fwd_placeholder path. Hence only ViT is put into the CUDA graph in this PR and the resampler stays on the eager path

Including the resampler in the CUDA graph for use_temporal_conv=False could be done in the follow-up PR

Does it sound good or any suggestions?

qyYue1389 · 2026-06-19T02:14:34Z

Hi @shen-shanshan, thank you for your feedback. I finished adding changes and reply. Could you review again when getting a chance? Thank you!

mergify · 2026-06-29T05:20:15Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @qyYue1389.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Qiuyang Yue <yueqiuyang1389@gmail.com>

…iuyang1389@gmail.com> Signed-off-by: Qiuyang Yue <yueqiuyang1389@gmail.com>

Signed-off-by: Qiuyang Yue <yueqiuyang1389@gmail.com>

Signed-off-by: qyYue1389 <yueqiuyang1389@gmail.com>

Signed-off-by: Qiuyang Yue <yueqiuyang1389@gmail.com>

qyYue1389 · 2026-07-13T23:22:26Z

Thanks for the approval @Isotr0py!
The remaining failing CI is due to infra, and I can't seem to clear it by pushing from my fork - builds failed immediately at bootstrap with:
ValueError: Invalid branch name: qyYue1389:ernie45vl-encoder-cudagraph. Contains disallowed characters.

It looks like the fork PR's owner:branch ref (the :) trips the bootstrap branch-name check, so retriggering by pushing to my fork just re-hits this.
Could you help kick off a run from Buildkite (or maybe send it through the merge queue) when getting a chance? Thanks a lot!

qyYue1389 · 2026-07-20T04:07:56Z

Hi @Isotr0py, thanks for re-triggering the CI. Looks both CI failures are in test_can_initialize_large_subset on GLM-4.7-Flash / Glm4MoeLite models, which should be unrelated to this PR changes

Could you please rerun the two failing initialization jobs or maybe merge when getting a chance? Thanks a lot!

qyYue1389 · 2026-07-27T06:26:11Z

Hi @Isotr0py, sorry looks the CI failed again. Both failures are in Mamba and sampling test areas and involve pfnet/plamo-2-1b / facebook/opt-125m, none of which is related to this PR. Probably they are flaky tests?

Could you please retry the two failing jobs (or re-run after a merge with main)?

Pls let me know if we need to rebase this PR from main

Thank you so much!

qyYue1389 · 2026-08-04T04:07:05Z

Hi @shen-shanshan, sorry the CI failed again. Both failures are in Mamba and sampling test areas and involve pfnet/plamo-2-1b / facebook/opt-125m, looks none of which is related to this PR. Probably they are flaky tests?

Could you please retry the two failing jobs and help merge (or re-run after a merge with main)?

Pls let me know if we need to rebase this PR from main

Thank you so much!

Isotr0py · 2026-08-05T13:26:51Z

/ci retry

github-actions · 2026-08-05T13:27:11Z

✅ No earlier CI build exists for this PR. Use /ci run first.

qyYue1389 · 2026-08-05T16:23:10Z

/ci run

github-actions · 2026-08-05T16:23:30Z

✅ Triggered Buildkite CI #82516 for commit 729d0ba7a726.

qyYue1389 · 2026-08-05T19:42:28Z

/ci retry

github-actions · 2026-08-05T19:42:50Z

✅ Queued 2 failed job(s) for retry in Buildkite CI #82516.

qyYue1389 · 2026-08-05T23:35:41Z

/ci run

github-actions · 2026-08-05T23:35:57Z

✅ Triggered Buildkite CI #82578 for commit 1404260e950e.

qyYue1389 · 2026-08-06T04:35:28Z

/ci retry

github-actions · 2026-08-06T04:35:44Z

✅ Queued 4 failed job(s) for retry in Buildkite CI #82578.

mergify Bot added documentation Improvements or additions to documentation multi-modality Related to multi-modality (#4194) nvidia labels Jun 11, 2026

github-project-automation Bot added this to NVIDIA Jun 11, 2026

qyYue1389 marked this pull request as ready for review June 11, 2026 09:04

qyYue1389 requested review from AndreasKaratzas, DarkLight1337 and ywang96 as code owners June 11, 2026 09:04

shen-shanshan mentioned this pull request Jun 12, 2026

[RFC]: Support ViT Full CUDA Graph (Tracker) #38175

Open

41 tasks

mergify Bot added the needs-rebase label Jun 16, 2026

qyYue1389 force-pushed the ernie45vl-encoder-cudagraph branch from 59d5051 to 2f0cc33 Compare June 17, 2026 06:11

mergify Bot removed the needs-rebase label Jun 17, 2026

qyYue1389 force-pushed the ernie45vl-encoder-cudagraph branch 2 times, most recently from c44a606 to 54f466a Compare June 17, 2026 06:16

shen-shanshan reviewed Jun 18, 2026

View reviewed changes

mergify Bot added the needs-rebase label Jun 29, 2026

qyYue1389 added 5 commits June 29, 2026 18:25

[MM][CG] Support ViT full CUDA graph for Ernie-4.5-VL image inference

5e4927a

Signed-off-by: Qiuyang Yue <yueqiuyang1389@gmail.com>

[MM][CG] Clarify prepare_encoder_metadata docstring

a37e8a3

Signed-off-by: Qiuyang Yue <yueqiuyang1389@gmail.com>

[MM][CG] Add ernie45_vl to example VIT CUDA graph list

42afdb2

Signed-off-by: Qiuyang Yue <yueqiuyang1389@gmail.com>

[MM][CG] resolve conflicts and rebase Signe-off-by: Qiuyang Yue <yueq…

8ff5722

…iuyang1389@gmail.com> Signed-off-by: Qiuyang Yue <yueqiuyang1389@gmail.com>

resolve comments

7a82531

Signed-off-by: Qiuyang Yue <yueqiuyang1389@gmail.com>

qyYue1389 force-pushed the ernie45vl-encoder-cudagraph branch from fd2210f to 7a82531 Compare June 29, 2026 18:32

Merge branch 'main' into ernie45vl-encoder-cudagraph

e2f4d2e

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 13, 2026

qyYue1389 added 5 commits July 13, 2026 07:31

Retrigger CI

449e8c4

Signed-off-by: Qiuyang Yue <yueqiuyang1389@gmail.com>

Retrigger CI

1b6c634

Signed-off-by: qyYue1389 <yueqiuyang1389@gmail.com>

Retrigger CI

5e2700d

Signed-off-by: qyYue1389 <yueqiuyang1389@gmail.com>

Retrigger CI

1eaab20

Signed-off-by: qyYue1389 <yueqiuyang1389@gmail.com>

Retrigger CI

2eef3fb

Signed-off-by: Qiuyang Yue <yueqiuyang1389@gmail.com>

Merge branch 'main' into ernie45vl-encoder-cudagraph

d4132d9

Merge branch 'main' into ernie45vl-encoder-cudagraph

5ebad91

shen-shanshan and others added 2 commits August 4, 2026 12:45

Merge branch 'main' into ernie45vl-encoder-cudagraph

f0af99a

Merge branch 'main' into ernie45vl-encoder-cudagraph

729d0ba

Merge branch 'main' into ernie45vl-encoder-cudagraph

1404260

Isotr0py merged commit 777b01d into vllm-project:main Aug 6, 2026
98 checks passed

github-project-automation Bot moved this from Ready to Done in NVIDIA Aug 6, 2026

vllm-agent mentioned this pull request Aug 6, 2026

Revert "[MM][CG] Support ViT full CUDA graph for Ernie-4.5-VL image inference" (#45254) #51263

Closed

qyYue1389 mentioned this pull request Aug 7, 2026

[MM][CG][BugFix] Fix Ernie-4.5-VL encoder CG postprocess for multi-path outputs #51461

Merged

Uh oh!

Uh oh!

Conversation

qyYue1389 commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Key Changes

Test

1. Functional

2. Benchmark (H100 80GB)

3. Accuracy (CG on vs off)

Uh oh!

mergify Bot commented Jun 11, 2026

Uh oh!

qyYue1389 commented Jun 12, 2026

Uh oh!

Isotr0py commented Jun 12, 2026

Uh oh!

qyYue1389 commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qyYue1389 commented Jun 16, 2026

Uh oh!

mergify Bot commented Jun 16, 2026

Uh oh!

shen-shanshan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shen-shanshan Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

qyYue1389 Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shen-shanshan Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

qyYue1389 Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

shen-shanshan Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

qyYue1389 Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

shen-shanshan Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

qyYue1389 Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

qyYue1389 commented Jun 19, 2026

Uh oh!

mergify Bot commented Jun 29, 2026

Uh oh!

qyYue1389 commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qyYue1389 commented Jul 20, 2026

Uh oh!

qyYue1389 commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qyYue1389 commented Aug 4, 2026

Uh oh!

Isotr0py commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

qyYue1389 commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

qyYue1389 commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

qyYue1389 commented Jun 11, 2026 •

edited

Loading

qyYue1389 commented Jun 14, 2026 •

edited

Loading

shen-shanshan left a comment •

edited

Loading

qyYue1389 Jun 19, 2026 •

edited

Loading

qyYue1389 commented Jul 13, 2026 •

edited

Loading

qyYue1389 commented Jul 27, 2026 •

edited

Loading