Your current environment
Environment summary (full collect_env.py output to follow in a comment)
vLLM: v0.27.1 (official image vllm/vllm-openai:v0.27.1-aarch64-cu129-ubuntu2404)
GPU: 4x GH200 (SM90), one node
Driver: 565.57.01 (CUDA 12.9 userspace via NVIDIA forward-compat libraries)
Platform: aarch64 (Grace), Linux
Model: deepseek-ai/DeepSeek-V4-Flash
Parallelism: --data-parallel-size 4 --enable-expert-parallel
KV cache: --kv-cache-dtype fp8, --max-model-len 128000
Compilation config: default (FULL_AND_PIECEWISE)
🐛 Describe the bug
Serving DeepSeek-V4-Flash with the default compilation config crashes during engine start, at the beginning of Capturing CUDA graphs (decode, FULL), with CUDA error: an illegal memory access was encountered. Reproduces 2/2 on different workers with:
vllm serve deepseek-ai/DeepSeek-V4-Flash \
--data-parallel-size 4 --enable-expert-parallel \
--kv-cache-dtype fp8 --max-model-len 128000 --trust-remote-code
(Worker_DP1_EP1) ERROR ... File ".../vllm/models/deepseek_v4/attention.py", line 369, in forward
(Worker_DP1_EP1) ERROR ... self.attn_gemm_parallel_execute(hidden_states)
(Worker_DP1_EP1) ERROR ... File ".../vllm/models/deepseek_v4/attention.py", line 449, in attn_gemm_parallel_execute
(Worker_DP1_EP1) ERROR ... qr_kv, (kv_score, indexer_weights, indexer_kv_score) = execute_in_parallel(
(Worker_DP1_EP1) ERROR ... File ".../vllm/utils/multi_stream_utils.py", line 121, in execute_in_parallel
(Worker_DP1_EP1) ERROR ... start_event.record()
(Worker_DP1_EP1) ERROR ... torch.AcceleratorError: CUDA error: an illegal memory access was encountered
That stack is where the sticky async error surfaces, not where it originates. With CUDA_LAUNCH_BLOCKING=1 the faulting call is get_paged_mqa_logits_metadata, reached from DeepseekV32IndexerMetadataBuilder._build_attention_metadata (indexer.py:951), before any model kernel runs.
The fault is an out-of-bounds shared-memory read in DeepGEMM's SM90 paged-MQA metadata scheduler. When every context length is zero, the kernel's per-SM binary search over the segment prefix sums saturates to q_idx == batch_size, and num_segs_q = prefix_sum[q_idx] - prefix_sum[q_idx - 1] reads one int past the allocation whenever batch_size is a multiple of 32 (the smem buffer holds align(batch_size, 32) ints). DeepGEMM merged the equivalent guard for its generic scheduler variant in June; the SM90 variant is still unguarded, and we understand a fix for it is in review there.
vLLM feeds the kernel exactly this input on every DSv4-Flash FULL capture: dummy decode batches use seq_len = 1, the indexer builder divides by compress_ratio (4 for V4-Flash) so every length becomes 1 // 4 = 0, and capture sizes are multiples of 32. DeepSeek-V3.2 has compress_ratio = 1 and can never produce all-zero lengths, so the path is well-tested there. The same mechanism fires from execute_dummy_batch on older releases, and may explain other reported FULL-graph symptoms on other SM architectures.
Standalone repro, no model needed (subprocess per case because the IMA poisons the CUDA context):
import os, subprocess, sys
if case := os.environ.get("CASE"):
import torch
from vllm.utils.deep_gemm import get_paged_mqa_logits_metadata
B, fill = map(int, case.split(","))
num_sms = torch.cuda.get_device_properties(0).multi_processor_count
ctx = torch.full((B, 1), fill, dtype=torch.int32, device="cuda")
get_paged_mqa_logits_metadata(ctx, 64, num_sms)
torch.cuda.synchronize()
sys.exit(0)
for b, fill in [(256, 0), (64, 0), (32, 0), (33, 0), (250, 0), (8, 0), (256, 1), (256, 8192)]:
r = subprocess.run([sys.executable, __file__], env={**os.environ, "CASE": f"{b},{fill}"})
print(f"B={b:4d} fill={fill:5d}: {'FAIL' if r.returncode else 'ok'}")On GH200 (SM90): zero-filled lengths fail at B = 32, 64, 256 and pass at B = 8, 33, 250; nonzero fills pass at every B.
Suggested fix
Clamp the scheduler-metadata input in the indexer builder. The schedule is a work-distribution hint and the logits kernel re-derives per-token work from the true context_lens, so the clamp is semantics-preserving:
--- a/vllm/v1/attention/backends/mla/indexer.py
+++ b/vllm/v1/attention/backends/mla/indexer.py
@@ (DeepseekV32IndexerMetadataBuilder, CUDA/DeepGEMM branch)
self.scheduler_metadata_buffer[:] = get_paged_mqa_logits_metadata(
- seq_lens,
+ torch.clamp(seq_lens, min=1),
self.kv_cache_spec.storage_block_size,
self.num_sms,
)Verified on the failing config above: all 83 FULL decode graphs capture, serving is healthy through an OSL=1 benchmark and a 10-minute idle watch. A kernel-side bounds guard is the more fundamental fix, but the clamp also protects every released wheel, since wheels vendor DeepGEMM's JIT sources at build time. PR to follow.
Before submitting a new issue...
Your current environment
Environment summary (full
collect_env.pyoutput to follow in a comment)🐛 Describe the bug
Serving DeepSeek-V4-Flash with the default compilation config crashes during engine start, at the beginning of
Capturing CUDA graphs (decode, FULL), withCUDA error: an illegal memory access was encountered. Reproduces 2/2 on different workers with:That stack is where the sticky async error surfaces, not where it originates. With
CUDA_LAUNCH_BLOCKING=1the faulting call isget_paged_mqa_logits_metadata, reached fromDeepseekV32IndexerMetadataBuilder._build_attention_metadata(indexer.py:951), before any model kernel runs.The fault is an out-of-bounds shared-memory read in DeepGEMM's SM90 paged-MQA metadata scheduler. When every context length is zero, the kernel's per-SM binary search over the segment prefix sums saturates to
q_idx == batch_size, andnum_segs_q = prefix_sum[q_idx] - prefix_sum[q_idx - 1]reads oneintpast the allocation wheneverbatch_sizeis a multiple of 32 (the smem buffer holdsalign(batch_size, 32)ints). DeepGEMM merged the equivalent guard for its generic scheduler variant in June; the SM90 variant is still unguarded, and we understand a fix for it is in review there.vLLM feeds the kernel exactly this input on every DSv4-Flash FULL capture: dummy decode batches use
seq_len = 1, the indexer builder divides bycompress_ratio(4 for V4-Flash) so every length becomes1 // 4 = 0, and capture sizes are multiples of 32. DeepSeek-V3.2 hascompress_ratio = 1and can never produce all-zero lengths, so the path is well-tested there. The same mechanism fires fromexecute_dummy_batchon older releases, and may explain other reported FULL-graph symptoms on other SM architectures.Standalone repro, no model needed (subprocess per case because the IMA poisons the CUDA context):
On GH200 (SM90): zero-filled lengths fail at B = 32, 64, 256 and pass at B = 8, 33, 250; nonzero fills pass at every B.
Suggested fix
Clamp the scheduler-metadata input in the indexer builder. The schedule is a work-distribution hint and the logits kernel re-derives per-token work from the true
context_lens, so the clamp is semantics-preserving:Verified on the failing config above: all 83 FULL decode graphs capture, serving is healthy through an OSL=1 benchmark and a 10-minute idle watch. A kernel-side bounds guard is the more fundamental fix, but the clamp also protects every released wheel, since wheels vendor DeepGEMM's JIT sources at build time. PR to follow.
Before submitting a new issue...