Repository navigation
[Model] Voxtral Realtime: add support for CUDAGraphMode.FULL_DECODE_ONLY - #51167
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
9795db4 to
44ae565
Compare
mergify
Bot
added
multi-modality
|
/ci run |
|
✅ Triggered Buildkite CI #82495 for commit |
|
/ci retry |
|
✅ Buildkite CI #82495 has failed jobs without stable step keys, so they cannot be retried on a new commit. Use |
|
/ci run |
|
✅ Triggered Buildkite CI #82835 for commit |
|
Thanks for moving this forward @NickLucche 🙏 coauthored-by: was not parsed correctly in commit. Appreciate it! |
b463408 to
4768754
Compare
|
/ci run |
|
✅ Triggered Buildkite CI #85769 for commit |
|
/ci run |
1 similar comment
|
/ci run |
|
✅ Triggered Buildkite CI #85981 for commit |
|
/ci run |
1 similar comment
|
/ci run |
|
✅ Triggered Buildkite CI #88286 for commit |
|
/ci run |
|
✅ Triggered Buildkite CI #88348 for commit |
|
This pull request has merge conflicts that must be resolved before it can be |
a026d74 to
cd745ed
Compare
|
/ci run |
|
✅ Triggered Buildkite CI #88857 for commit |
…ONLY` (vllm-project#51167) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: Jorge Gallego Feliciano <jorge.gallego@mistral.ai>
…ONLY` (vllm-project#51167) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: Jorge Gallego Feliciano <jorge.gallego@mistral.ai>
3 participants
This PR allows running Voxtral Realtime full CG graph for decode-only batches.
Currently decode steps are stuck on piecewise capture and pay the per-step launch overhead.
The blocker is the whisper encoder's block-pooling attention builder: its
builddid acopy.deepcopyof the commonattention metadata and allocated a fresh
slot_mappingevery step, so the tensor addresses baked in at capture time were dead by replay.This makes that builder capture-safe for uniform single-token decode and relaxes the assert accordingly.
Prefill/mixed batches still fall back to piecewise.
With CUDAGraphMode.FULL_DECODE_ONLY (single invocation per step):
Single H200, Voxtral-Mini-4B-Realtime-2602,
max_num_seqs=4,max_model_len=8192, 209 output tokens, 3 iterations averaged.Single H200, Voxtral-Mini-4B-Realtime-2602,
max_model_len=16384,max_num_batched_tokens=16384, 3 iterations averaged.With low batch size gaining the most by kernel launch time slashing, as expected.
cc @LucasWilkinson