Gate spec_decoding off multi_node_tp_pp - #895
namgyu-youn wants to merge 2 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates several model configuration YAML files to restrict speculative decoding strategies. Specifically, it gates speculative decoding features (such as Eagle3 and Multi-Token Prediction) off the multi_node_tp_pp strategy because these draft models do not yet support pipeline parallelism. The supported strategies are now explicitly listed for each model. There are no review comments, so I have no feedback to provide.
MTP/dspark/eagle3 draft models don't implement SupportsPP, so PP + spec_decoding fails at startup. Same fix already applied to Kimi-K3; extends it to the 20 other recipes offering both. Signed-off-by: namgyu-youn <namgyu.dev@gmail.com>
3b3be3f to
00a8aa9
Compare
Signed-off-by: namgyu-youn <namgyu.dev@gmail.com>
|
cc @esmeetu PTAL, thanks! |
|
Hi @namgyu-youn, Thanks for this. But it seems we just support it. can u help confirm? vllm-project/vllm#50514 |
Thanks for letting me know, RC (v0.29.1rc0) works well. Maybe we can close #875 too? |
2 participants
Summary
multi_node_tp_pp+spec_decodingcrashes at startup: MTP/dspark/eagle3 draft models don't implementSupportsPPupstream (vllm-project/vllm#50098), so PP forwarded to the draft model's config hits vLLM's PP-capability gate even though the target model supports PP fine.Similar to Kimi-K3
recipes/models/moonshotai/Kimi-K3.yaml
Lines 62 to 68 in 742a1ea
, this PR adds a
strategies:allowlist to speculative decoding configuration on the 21 affected recipes, excludingmulti_node_tp_ppfor the same upstream issue, extended here to the rest.