Skip to content

[V1][Spec Decode] Make Eagle compatible with pipeline parallelism - #16568

Closed
xyang16 wants to merge 1 commit into
vllm-project:mainfrom
xyang16:specdec1
Closed

[V1][Spec Decode] Make Eagle compatible with pipeline parallelism#16568
xyang16 wants to merge 1 commit into
vllm-project:mainfrom
xyang16:specdec1

Conversation

@xyang16

@xyang16 xyang16 commented Apr 14, 2025 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

This PR aims to make speculative decoding compatible with pipeline parallelism.

  • Introducing draft_pipeline_parallel_size config. Currently drafter are only loaded in the last pp stage, which essentially means draft_pipeline_parallel_size=1.
  • EagleLlamaForCausalLM already inherits SupportsPP from LlamaForCausalLM.

Commands:

export VLLM_USE_V1=1
vllm serve unsloth/llama-3-8b-Instruct \
    --tensor-parallel-size 2 \
    --pipeline-parallel-size 2 \
    --distributed-executor-backend ray \
    --speculative-config '{"method": "eagle", "model":"yuhuili/EAGLE-LLaMA3-Instruct-8B", "num_speculative_tokens": 3, "draft_pipeline_parallel_size": 2}'

@github-actions

github-actions Bot commented Apr 14, 2025

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify Bot added the v1 label Apr 14, 2025
@xyang16
xyang16 force-pushed the specdec1 branch 3 times, most recently from 1deef32 to cc071e1 Compare April 14, 2025 17:55
@xyang16 xyang16 changed the title [V1][Spec Decode] Make speculative decoding compatible with pipeline parallelism [V1][Spec Decode] Make Eagle compatible with pipeline parallelism Apr 14, 2025
@xyang16
[V1][Spec Decode] Make speculative decoding compatible with pipeline …
20f7eba 
…parallelism

Signed-off-by: Xin Yang <xyangx@amazon.com>
@xyang16
xyang16 marked this pull request as ready for review April 14, 2025 23:30
luyuzhe111
luyuzhe111 reviewed Apr 15, 2025
View reviewed changes
Comment thread vllm/v1/spec_decode/eagle.py
target_layer_num = self.vllm_config.model_config.get_num_layers(
self.vllm_config.parallel_config)
self.vllm_config.parallel_config
) * self.vllm_config.parallel_config.pipeline_parallel_size

@luyuzhe111 luyuzhe111 Apr 15, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this code assumes the layers are always equally distributed between nodes. Wondering whether it's always the case?

@mergify mergify Bot added the llama Related to Llama models label Jun 11, 2025
@mergify

mergify Bot commented Jun 11, 2025

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @xyang16.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jun 11, 2025
@xyang16 xyang16 closed this Jul 18, 2025
@xyang16
xyang16 deleted the specdec1 branch August 31, 2025 01:54
@AliceChenyy AliceChenyy mentioned this pull request Jul 10, 2026
Merged
functionstackx added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Jul 28, 2026
@functionstackx @claude
fix: verify DSpark draft against a pp=1 parallel-config view
cbb1f44 
Engine init died with "NotImplementedError: Pipeline parallelism is not
supported for this model" — SpeculativeConfig verifies the draft model
against a parallel config that unconditionally inherits the target's
pipeline_parallel_size (2), and the Inferact/Kimi-K3-DSpark draft head
does not implement SupportsPP. At runtime V1 drafters load ONLY on the
final pipeline stage (vllm-project/vllm#16568), effectively
draft_pipeline_parallel_size=1, and upstream has no config knob for it —
so extend the in-container patch script to verify the draft against a
pp=1 view of the parallel config. Idempotent; refuses to patch if the
call-site shape changed.

中文：引擎初始化报 "Pipeline parallelism is not supported for this model"——
SpeculativeConfig 用无条件继承目标 pipeline_parallel_size(2) 的并行配置校验
草稿模型，而 Inferact/Kimi-K3-DSpark 草稿头未实现 SupportsPP。V1 运行时草稿
模型仅加载在最后一个流水线阶段（vllm#16568），等效
draft_pipeline_parallel_size=1，且上游无相应配置项——故扩展容器内补丁脚本，
以 pp=1 视图校验草稿模型。补丁幂等，调用点形态变化时拒绝执行。

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
functionstackx added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Jul 28, 2026
@functionstackx @claude
fix: verify DSpark draft against a pp=1 parallel-config view
41dd5a9 
Engine init died with "NotImplementedError: Pipeline parallelism is not
supported for this model" — SpeculativeConfig verifies the draft model
against a parallel config that unconditionally inherits the target's
pipeline_parallel_size (2), and the Inferact/Kimi-K3-DSpark draft head
does not implement SupportsPP. At runtime V1 drafters load ONLY on the
final pipeline stage (vllm-project/vllm#16568), effectively
draft_pipeline_parallel_size=1, and upstream has no config knob for it —
so extend the in-container patch script to verify the draft against a
pp=1 view of the parallel config. Idempotent; refuses to patch if the
call-site shape changed.

中文：引擎初始化报 "Pipeline parallelism is not supported for this model"——
SpeculativeConfig 用无条件继承目标 pipeline_parallel_size(2) 的并行配置校验
草稿模型，而 Inferact/Kimi-K3-DSpark 草稿头未实现 SupportsPP。V1 运行时草稿
模型仅加载在最后一个流水线阶段（vllm#16568），等效
draft_pipeline_parallel_size=1，且上游无相应配置项——故扩展容器内补丁脚本，
以 pp=1 视图校验草稿模型。补丁幂等，调用点形态变化时拒绝执行。

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon
@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat
@njhill njhill Awaiting requested review from njhill njhill is a code owner
@ywang96 ywang96 Awaiting requested review from ywang96
@comaniac comaniac Awaiting requested review from comaniac
@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat
1 more reviewer
@luyuzhe111 luyuzhe111 luyuzhe111 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

llama Related to Llama models needs-rebase speculative-decoding v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@xyang16 @luyuzhe111