[Bugfix] fix recompile in qwen3 vl

[narutolhy]

Motivation

Qwen3-VL automatically injects input_deepstack_embeds into the language model inputs when requests contain mm_inputs (multimodal inputs). The current implementation does not handle this injected tensor consistently, so the model execution path differs between multimodal and text-only requests. In mixed traffic, this leads to unnecessary TorchDynamo recompilations at runtime (e.g., repeatedly specializing on the presence/absence of deepstack inputs), increasing latency and reducing throughput stability.

This PR makes the deepstack injection path explicit and stable to avoid redundant recompilation for Qwen3-VL multimodal inference.

Modifications

Standardized the Qwen3-VL deepstack injection behavior so that the language model forward path remains stable when mm_inputs is present.

Ensured input_deepstack_embeds is handled in a consistent way across requests to prevent runtime recompilation churn (especially under mixed multimodal/text workloads).

Refactored the relevant code paths to make the deepstack injection explicit and easier to reason about.

Accuracy Tests

The update in test/manual/nightly/test_vlms_vit_cuda_graph.py case covers the accuracy.

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[yuan-luo]

Could you help to verify with SGLANG_VIT_ENABLE_CUDA_GRAPH=1? Qwen3-VL ViT CUDA Graph considered injecting deepstack output into LLM layers, just want make sure the full ViT+LLM CUDA Graph path working correctly.
The issue also exists when SGLANG_VIT_ENABLE_CUDA_GRAPH is set to 0.
#15320

[yuan-luo]

/rerun-failed-ci

[narutolhy]

Could you help to verify with SGLANG_VIT_ENABLE_CUDA_GRAPH=1? Qwen3-VL ViT CUDA Graph considered injecting deepstack output into LLM layers, just want make sure the full ViT+LLM CUDA Graph path working correctly.
The issue also exists when SGLANG_VIT_ENABLE_CUDA_GRAPH is set to 0.

Hi @yuan-luo Thank you for your review.
I tested with VIT CUDA Graph before. It has some error. This is the issue #16883
I think this is due to the latest optimizations, but after I removed them, it resulted in a CUDA address access error during graph replay. I think it's because different graphs are being used for the same seq_len length, and your planned refactoring should solve this problem. #16902
I'll try test it with your new pr. Thank you

[Oasis-Git]

Previously when I run this model I noticed the recompilation is caused by grad mode. So could you please share which part introduce forward with gradient?

[yuan-luo]

Could you help to verify with SGLANG_VIT_ENABLE_CUDA_GRAPH=1? Qwen3-VL ViT CUDA Graph considered injecting deepstack output into LLM layers, just want make sure the full ViT+LLM CUDA Graph path working correctly.
The issue also exists when SGLANG_VIT_ENABLE_CUDA_GRAPH is set to 0.

Hi @yuan-luo Thank you for your review. I tested with VIT CUDA Graph before. It has some error. This is the issue #16883 I think this is due to the latest optimizations, but after I removed them, it resulted in a CUDA address access error during graph replay. I think it's because different graphs are being used for the same seq_len length, and your planned refactoring should solve this problem. #16902 I'll try test it with your new pr. Thank you

@narutolhy Did #16902 fix your problem?

[narutolhy]

Could you help to verify with SGLANG_VIT_ENABLE_CUDA_GRAPH=1? Qwen3-VL ViT CUDA Graph considered injecting deepstack output into LLM layers, just want make sure the full ViT+LLM CUDA Graph path working correctly.
The issue also exists when SGLANG_VIT_ENABLE_CUDA_GRAPH is set to 0.

Hi @yuan-luo Thank you for your review. I tested with VIT CUDA Graph before. It has some error. This is the issue #16883 I think this is due to the latest optimizations, but after I removed them, it resulted in a CUDA address access error during graph replay. I think it's because different graphs are being used for the same seq_len length, and your planned refactoring should solve this problem. #16902 I'll try test it with your new pr. Thank you

@narutolhy Did #16902 fix your problem?

Hi @yuan-luo I still got the error with VIT CUDA Graph in your branch. I will try to debug it. Thank you.
File "/opt/tiger/search-llm-inference/qwen/sglang/python/sglang/srt/managers/mm_utils.py", line 1091, in general_mm_embed_routine
input_embeds, other_info = embed_mm_inputs(
^^^^^^^^^^^^^^^^
File "/opt/tiger/search-llm-inference/qwen/sglang/python/sglang/srt/managers/mm_utils.py", line 983, in embed_mm_inputs
embedding, mask, input_ids = get_embedding_and_mask(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tiger/search-llm-inference/qwen/sglang/python/sglang/srt/managers/mm_utils.py", line 903, in get_embedding_and_mask
special_multimodal_mask = _get_multimodal_mask(input_ids, placeholder_tensor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tiger/search-llm-inference/qwen/sglang/python/sglang/srt/managers/mm_utils.py", line 820, in _get_multimodal_mask
return torch.isin(input_ids, placeholder_tensor).unsqueeze(-1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[narutolhy]

Previously when I run this model I noticed the recompilation is caused by grad mode. So could you please share which part introduce forward with gradient?

Hi @Oasis-Git This part forward with gradient. And my latest code only added @torch.no_grad() on this part.
hidden_states = general_mm_embed_routine( input_ids=input_ids, forward_batch=forward_batch, language_model=self.model, multimodal_model=self, positions=positions, use_deepstack=self.use_deepstack, pp_proxy_tensors=pp_proxy_tensors, )

[narutolhy]

Hi @yuan-luo
A new discovery is that when I use --disable-custom-all-reduce along with your pull request, the ViT CUDA graph works correctly. I'm trying to understand the logic of custom-all-reduce. Thank you
SGLANG_MM_FEATURE_CACHE_MB=4096
SGLANG_USE_CUDA_IPC_TRANSPORT=1
SGLANG_VLM_CACHE_SIZE_MB=0
SGLANG_VIT_ENABLE_CUDA_GRAPH=1
python3 -m sglang.launch_server --host 127.0.0.1 --mem-fraction-static 0.7 --port 30000 --max-running-requests 64 --chunked-prefill-size 8192 --attention-backend fa3 --mm-attention-backend fa3 --enable-multimodal --model Qwen/Qwen3-VL-8B-Instruct --disable-radix-cache --piecewise-cuda-graph-max-tokens 4096 --enable-piecewise-cuda-graph --piecewise-cuda-graph-compiler eager --tp-size 4 --disable-custom-all-reduce --enable-torch-symm-mem

[narutolhy]

#17255
Hi @yuan-luo I found the custom-all-reduce related failure and submitted a fix. Please review it, thank you.

[Oasis-Git]

LGTM from the pcg side. @yuan-luo Could you take a next step look please? Thanks

[yuan-luo]

Will take a look at it soon.

[narutolhy]

Hi @yuan-luo , Do you have time to take a look? Or can you take a look when you come back from your holiday?
Thank you

[narutolhy]

/tag-and-rerun-ci