Repository navigation
[Bugfix] Skip CUDA-graph padded positions in eagle draft step kernel - #43682
kshitij2605 wants to merge 1 commit into
Conversation
During speculative decoding draft iterations, the eagle_step_slot_mapping_metadata_kernel increments seq_lens for all positions in the batch. CUDA-graph padded positions (beyond the actual request count) have seq_lens == 0 and block_table entries filled with -1. Incrementing their seq_lens from 0 to 1 causes the attention kernel to index into block_table with the -1 entry, triggering a cudaErrorIllegalAddress / illegal memory access. The fix checks seq_lens before incrementing: if seq_len == 0, the position is treated as padding (PADDING_SLOT_ID for slot_mapping, 0 for clamped_positions) and seq_lens is left at 0. This manifests under concurrent load with async scheduling enabled, when CUDA-graph batch padding creates positions that the original kernel would incorrectly activate. Signed-off-by: Kshitij <kshitij7156@gmail.com>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the eagle_step_slot_mapping_metadata_kernel in vllm/v1/spec_decode/utils.py to handle CUDA-graph padded positions where seq_len is 0. By checking for seq_len == 0 early and returning, it avoids incrementing sequence lengths for padded positions, which would otherwise trigger an illegal memory access on block_table[-1]. There are no review comments to address, and I have no additional feedback to provide.
2 participants
Summary
Fixes
cudaErrorIllegalAddress/ CUDA illegal memory access crash during speculative decoding (Eagle, Eagle3, MTP) under concurrent load with CUDA graphs enabled.Root cause:
eagle_step_slot_mapping_metadata_kernelincrementsseq_lensfor all positions in the batch, including CUDA-graph padded positions. Padded positions haveseq_lens == 0andblock_tableentries filled with-1. When theirseq_lensis incremented from 0 to 1, the attention kernel readsblock_table[-1], triggering an out-of-bounds GPU memory access that kills the engine.Fix: Check
seq_lensat the start of the kernel. Ifseq_len == 0(indicating a CUDA-graph padded position), writePADDING_SLOT_IDforslot_mapping, zero forclamped_positions, and return early — leavingseq_lensat 0 so the attention kernel correctly skips these positions.This is a one-line-logic fix in the Triton kernel with no API changes.
Reproduction
The crash requires:
--enforce-eager)Typical error:
Related issues: #39295, #40756
Test plan
pytest tests/v1/spec_decode/)