[BugFix] Support setting tp=1 for the Eagle draft model to take effect by zhaomingyu13 · Pull Request #5519 · vllm-project/vllm-ascend

zhaomingyu13 · 2025-12-30T08:39:44Z

What this PR does / why we need it?

According to the official documentation, the parameter "draft_tensor_parallel_size": 1 is supposed to be applied to the Eagle3 model. However, based on actual debugging, it was found that the number of tensor parallelisms (tp) of the Eagle model is consistent with that of the target model. The setting of tp for the draft model did not take effect as expected.

Note: This feature has not been superimposed and tested with sp and dp. It will be adapted later

Does this PR introduce any user-facing change?

No

How was this patch tested?

from vllm import LLM, SamplingParams

def main():
    prompts = [
        "The future of AI is",
    ]

    # Create a sampling params object.
    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
    # Create an LLM.
    llm = LLM(
            model="meta-llama/Llama-3.1-8B-Instruct",
            tensor_parallel_size=4,
            gpu_memory_utilization=0.9,
            enforce_eager=True,
            speculative_config={
                "method": "eagle3",
                "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B"
                "draft_tensor_parallel_size": 1,
                "num_speculative_tokens": 3,
            },
        )

    # Generate texts from the prompts.
    outputs = llm.generate(prompts, sampling_params)
    print(f"Outputs: {outputs}")
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

vLLM version: v0.13.0
vLLM main: vllm-project/vllm@45c1ca1

Fixes vllm-project/vllm#31345

gemini-code-assist

Code Review

This pull request addresses a bug where draft_tensor_parallel_size=1 was not being respected for the Eagle3 draft model. The proposed solution correctly uses patch_tensor_parallel_group to temporarily set the tensor parallelism to 1 when loading the draft model. My review includes a suggestion to refactor duplicated code for better maintainability and points out a critical issue in a seemingly unrelated change that could break handling of different KV cache types. Please address the critical issue and consider the refactoring suggestion.

github-actions · 2025-12-30T10:57:04Z

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

A PR should do only one thing, smaller PRs enable faster reviews.
Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

github-actions · 2026-01-06T08:52:09Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

…to take effect (#5804) ### What this PR does / why we need it? According to the official documentation, the parameter "draft_tensor_parallel_size": 1 is supposed to be applied to the Eagle3 model. However, based on actual debugging, it was found that the number of tensor parallelisms (tp) of the Eagle model is consistent with that of the target model. The setting of tp for the draft model did not take effect as expected. **Note:** This feature has not been superimposed and tested with `sp` and `dp`. It will be adapted later ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ```python from vllm import LLM, SamplingParams def main(): prompts = [ "The future of AI is", ] # Create a sampling params object. sampling_params = SamplingParams(temperature=0.8, top_p=0.95) # Create an LLM. llm = LLM( model="meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=4, gpu_memory_utilization=0.9, enforce_eager=True, speculative_config={ "method": "eagle3", "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B" "draft_tensor_parallel_size": 1, "num_speculative_tokens": 3, }, ) # Generate texts from the prompts. outputs = llm.generate(prompts, sampling_params) print(f"Outputs: {outputs}") for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ``` pick-from: #5519 Fixes vllm-project/vllm#31345 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com> Co-authored-by: drslark <slarksblood@qq.com>

…to eplb_refactor * 'main' of https://github.com/vllm-project/vllm-ascend: [CI] Unblock 4-cards test (vllm-project#5831) [Refactor] Provide a framework to accommodate operators for different hardware devices (vllm-project#5735) [Refactor] Modify the binding logic to allocate CPU cores for each NPU card (vllm-project#5555) [BugFix] Support setting tp=1 for the Eagle draft model to take effect (vllm-project#5519) support triton of mrope (vllm-project#5664) [bugfix] A2 Environment Pooling for Memcache Compatibility (vllm-project#5601) [Doc] Update community contributors and versioning naming to follow vLLM (vllm-project#5820) [Refactor] Add comments for Metadata classes in attention module (vllm-project#5789) [Bugfix] bugfix for the order of dummy run pad and sync (vllm-project#5777) [CI] Move nightly-a2 test to hk (vllm-project#5807) [CI] Show disk usage for CI shared volume (vllm-project#5821) Bump actions/checkout from 4 to 6 (vllm-project#5795) Bump actions/github-script from 7 to 8 (vllm-project#5796) [bugfix](cp) align max_context_chunk to cp_virtual_block_size (vllm-project#5767) [bugfix]limit graph replay sync (vllm-project#5761) [CI]Add Kimi k2 nightly test (vllm-project#5682) [Doc] add tls check to pd disaggregation readme (vllm-project#5638) [CI] adpat v0.13.0 change (vllm-project#5793)

vllm-project#5519) ### What this PR does / why we need it? According to the official documentation, the parameter "draft_tensor_parallel_size": 1 is supposed to be applied to the Eagle3 model. However, based on actual debugging, it was found that the number of tensor parallelisms (tp) of the Eagle model is consistent with that of the target model. The setting of tp for the draft model did not take effect as expected. **Note:** This feature has not been superimposed and tested with `sp` and `dp`. It will be adapted later ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ```python from vllm import LLM, SamplingParams def main(): prompts = [ "The future of AI is", ] # Create a sampling params object. sampling_params = SamplingParams(temperature=0.8, top_p=0.95) # Create an LLM. llm = LLM( model="meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=4, gpu_memory_utilization=0.9, enforce_eager=True, speculative_config={ "method": "eagle3", "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B" "draft_tensor_parallel_size": 1, "num_speculative_tokens": 3, }, ) # Generate texts from the prompts. outputs = llm.generate(prompts, sampling_params) print(f"Outputs: {outputs}") for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ``` - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@45c1ca1 Fixes vllm-project/vllm#31345 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com> Co-authored-by: drslark <slarksblood@qq.com>

…ke effect (#5519)"(#5902) This reverts commit d886b81. it breaks pd function - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@bde38c1 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com>

vllm-project#5519) ### What this PR does / why we need it? According to the official documentation, the parameter "draft_tensor_parallel_size": 1 is supposed to be applied to the Eagle3 model. However, based on actual debugging, it was found that the number of tensor parallelisms (tp) of the Eagle model is consistent with that of the target model. The setting of tp for the draft model did not take effect as expected. **Note:** This feature has not been superimposed and tested with `sp` and `dp`. It will be adapted later ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ```python from vllm import LLM, SamplingParams def main(): prompts = [ "The future of AI is", ] # Create a sampling params object. sampling_params = SamplingParams(temperature=0.8, top_p=0.95) # Create an LLM. llm = LLM( model="meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=4, gpu_memory_utilization=0.9, enforce_eager=True, speculative_config={ "method": "eagle3", "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B" "draft_tensor_parallel_size": 1, "num_speculative_tokens": 3, }, ) # Generate texts from the prompts. outputs = llm.generate(prompts, sampling_params) print(f"Outputs: {outputs}") for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ``` - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@45c1ca1 Fixes vllm-project/vllm#31345 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com> Co-authored-by: drslark <slarksblood@qq.com>

…ke effect (vllm-project#5519)"(vllm-project#5902) This reverts commit d886b81. it breaks pd function - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@bde38c1 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com>

…ke effect(#5519) (#5926) This reverts commit d886b81. Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com>

…to take effect (#6095) According to the official documentation, the parameter "draft_tensor_parallel_size": 1 is supposed to be applied to the Eagle3 model. However, based on actual debugging, it was found that the number of tensor parallelisms (tp) of the Eagle model is consistent with that of the target model. The setting of tp for the draft model did not take effect as expected. **Note:** This feature has not been superimposed and tested with `sp` and `dp`. It will be adapted later pick-from: #5519 Fixes vllm-project/vllm#31345 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com> Co-authored-by: drslark <slarksblood@qq.com>

…ke effect (vllm-project#5519)"(vllm-project#5902) This reverts commit d886b81. it breaks pd function - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@bde38c1 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com>

…ke effect(vllm-project#5519) (vllm-project#5926) This reverts commit d886b81. Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com>

…to take effect (vllm-project#6095) According to the official documentation, the parameter "draft_tensor_parallel_size": 1 is supposed to be applied to the Eagle3 model. However, based on actual debugging, it was found that the number of tensor parallelisms (tp) of the Eagle model is consistent with that of the target model. The setting of tp for the draft model did not take effect as expected. **Note:** This feature has not been superimposed and tested with `sp` and `dp`. It will be adapted later pick-from: vllm-project#5519 Fixes vllm-project/vllm#31345 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com> Co-authored-by: drslark <slarksblood@qq.com>

…ke effect (vllm-project#5519)"(vllm-project#5902) This reverts commit d886b81. it breaks pd function - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@bde38c1 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com>

…to take effect (vllm-project#6095) According to the official documentation, the parameter "draft_tensor_parallel_size": 1 is supposed to be applied to the Eagle3 model. However, based on actual debugging, it was found that the number of tensor parallelisms (tp) of the Eagle model is consistent with that of the target model. The setting of tp for the draft model did not take effect as expected. **Note:** This feature has not been superimposed and tested with `sp` and `dp`. It will be adapted later pick-from: vllm-project#5519 Fixes vllm-project/vllm#31345 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com> Co-authored-by: drslark <slarksblood@qq.com>

vllm-project#5519) ### What this PR does / why we need it? According to the official documentation, the parameter "draft_tensor_parallel_size": 1 is supposed to be applied to the Eagle3 model. However, based on actual debugging, it was found that the number of tensor parallelisms (tp) of the Eagle model is consistent with that of the target model. The setting of tp for the draft model did not take effect as expected. **Note:** This feature has not been superimposed and tested with `sp` and `dp`. It will be adapted later ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ```python from vllm import LLM, SamplingParams def main(): prompts = [ "The future of AI is", ] # Create a sampling params object. sampling_params = SamplingParams(temperature=0.8, top_p=0.95) # Create an LLM. llm = LLM( model="meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=4, gpu_memory_utilization=0.9, enforce_eager=True, speculative_config={ "method": "eagle3", "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B" "draft_tensor_parallel_size": 1, "num_speculative_tokens": 3, }, ) # Generate texts from the prompts. outputs = llm.generate(prompts, sampling_params) print(f"Outputs: {outputs}") for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ``` - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@45c1ca1 Fixes vllm-project/vllm#31345 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com> Co-authored-by: drslark <slarksblood@qq.com> Signed-off-by: zrj026 <zhangrunjiang026@gmail.com>

…ke effect (vllm-project#5519)"(vllm-project#5902) This reverts commit d886b81. it breaks pd function - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@bde38c1 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com> Signed-off-by: zrj026 <zhangrunjiang026@gmail.com>

vllm-project#5519) ### What this PR does / why we need it? According to the official documentation, the parameter "draft_tensor_parallel_size": 1 is supposed to be applied to the Eagle3 model. However, based on actual debugging, it was found that the number of tensor parallelisms (tp) of the Eagle model is consistent with that of the target model. The setting of tp for the draft model did not take effect as expected. **Note:** This feature has not been superimposed and tested with `sp` and `dp`. It will be adapted later ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ```python from vllm import LLM, SamplingParams def main(): prompts = [ "The future of AI is", ] # Create a sampling params object. sampling_params = SamplingParams(temperature=0.8, top_p=0.95) # Create an LLM. llm = LLM( model="meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=4, gpu_memory_utilization=0.9, enforce_eager=True, speculative_config={ "method": "eagle3", "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B" "draft_tensor_parallel_size": 1, "num_speculative_tokens": 3, }, ) # Generate texts from the prompts. outputs = llm.generate(prompts, sampling_params) print(f"Outputs: {outputs}") for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ``` - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@45c1ca1 Fixes vllm-project/vllm#31345 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com> Co-authored-by: drslark <slarksblood@qq.com>

…ke effect (vllm-project#5519)"(vllm-project#5902) This reverts commit d886b81. it breaks pd function - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@bde38c1 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com>

vllm-project#5519) ### What this PR does / why we need it? According to the official documentation, the parameter "draft_tensor_parallel_size": 1 is supposed to be applied to the Eagle3 model. However, based on actual debugging, it was found that the number of tensor parallelisms (tp) of the Eagle model is consistent with that of the target model. The setting of tp for the draft model did not take effect as expected. **Note:** This feature has not been superimposed and tested with `sp` and `dp`. It will be adapted later ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ```python from vllm import LLM, SamplingParams def main(): prompts = [ "The future of AI is", ] # Create a sampling params object. sampling_params = SamplingParams(temperature=0.8, top_p=0.95) # Create an LLM. llm = LLM( model="meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=4, gpu_memory_utilization=0.9, enforce_eager=True, speculative_config={ "method": "eagle3", "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B" "draft_tensor_parallel_size": 1, "num_speculative_tokens": 3, }, ) # Generate texts from the prompts. outputs = llm.generate(prompts, sampling_params) print(f"Outputs: {outputs}") for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ``` - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@45c1ca1 Fixes vllm-project/vllm#31345 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com> Co-authored-by: drslark <slarksblood@qq.com> Signed-off-by: zrj026 <zhangrunjiang026@gmail.com>

…ke effect (vllm-project#5519)"(vllm-project#5902) This reverts commit d886b81. it breaks pd function - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@bde38c1 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com> Signed-off-by: zrj026 <zhangrunjiang026@gmail.com>

vllm-project#5519) ### What this PR does / why we need it? According to the official documentation, the parameter "draft_tensor_parallel_size": 1 is supposed to be applied to the Eagle3 model. However, based on actual debugging, it was found that the number of tensor parallelisms (tp) of the Eagle model is consistent with that of the target model. The setting of tp for the draft model did not take effect as expected. **Note:** This feature has not been superimposed and tested with `sp` and `dp`. It will be adapted later ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ```python from vllm import LLM, SamplingParams def main(): prompts = [ "The future of AI is", ] # Create a sampling params object. sampling_params = SamplingParams(temperature=0.8, top_p=0.95) # Create an LLM. llm = LLM( model="meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=4, gpu_memory_utilization=0.9, enforce_eager=True, speculative_config={ "method": "eagle3", "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B" "draft_tensor_parallel_size": 1, "num_speculative_tokens": 3, }, ) # Generate texts from the prompts. outputs = llm.generate(prompts, sampling_params) print(f"Outputs: {outputs}") for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ``` - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@45c1ca1 Fixes vllm-project/vllm#31345 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com> Co-authored-by: drslark <slarksblood@qq.com>

…ke effect (vllm-project#5519)"(vllm-project#5902) This reverts commit d886b81. it breaks pd function - vLLM version: v0.13.0 - vLLM main: vllm-project/vllm@bde38c1 Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com>

gemini-code-assist bot reviewed Dec 30, 2025

View reviewed changes

Comment thread vllm_ascend/worker/model_runner_v1.py Outdated

Comment thread vllm_ascend/worker/model_runner_v1.py Outdated

pjgao reviewed Dec 30, 2025

View reviewed changes

Comment thread vllm_ascend/worker/model_runner_v1.py Outdated

zhaomingyu13 force-pushed the main branch 3 times, most recently from 039592f to a36e8ed Compare December 30, 2025 09:29

zhaomingyu13 force-pushed the main branch from a36e8ed to 20986f7 Compare December 31, 2025 01:26

drslark reviewed Dec 31, 2025

View reviewed changes

Comment thread vllm_ascend/worker/model_runner_v1.py Outdated

zhaomingyu13 force-pushed the main branch 6 times, most recently from 496383c to 0524b9f Compare January 4, 2026 07:41

wangxiyuan approved these changes Jan 4, 2026

View reviewed changes

Comment thread vllm_ascend/spec_decode/eagle_proposer.py Outdated

zzzzwwjj approved these changes Jan 4, 2026

View reviewed changes

zhaomingyu13 force-pushed the main branch 9 times, most recently from 4efac44 to 6e8285b Compare January 6, 2026 07:59

github-actions bot added merge-conflicts and removed merge-conflicts labels Jan 6, 2026

zhaomingyu13 changed the title ~~[BugFix] Support setting tp=1 for the Eagle3 draft model to take effect~~ [BugFix] Support setting tp=1 for the Eagle draft model to take effect Jan 6, 2026

zhaomingyu13 force-pushed the main branch from 22b6f4e to 0bb76e4 Compare January 6, 2026 12:16

zhaomingyu13 force-pushed the main branch from 5750874 to 8d4d2d6 Compare January 12, 2026 01:23

wangxiyuan approved these changes Jan 12, 2026

View reviewed changes

wangxiyuan merged commit d886b81 into vllm-project:main Jan 13, 2026
16 checks passed

This was referenced Jan 14, 2026

Revert "[BugFix] Support setting tp=1 for the Eagle draft model to ta… #5902

Merged

Revert "[BugFix] Support setting tp=1 for the Eagle draft model to ta… #5903

Merged

zhaomingyu13 mentioned this pull request Jan 15, 2026

Revert "[BugFix] Support setting tp=1 for the Eagle draft model to take effect(#5519) #5926

Merged

wangxiyuan pushed a commit that referenced this pull request Jan 16, 2026

Revert "[BugFix] Support setting tp=1 for the Eagle draft model to ta…

a8d5af3

…ke effect(#5519) (#5926) This reverts commit d886b81. Signed-off-by: zhaomingyu <zhaomingyu13@h-partners.com>

zhaomingyu13 mentioned this pull request Jan 21, 2026

[cherry-pick][BugFix] Support setting tp=1 for the Eagle draft model to take effect #6095

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix] Support setting tp=1 for the Eagle draft model to take effect#5519

[BugFix] Support setting tp=1 for the Eagle draft model to take effect#5519
wangxiyuan merged 1 commit intovllm-project:mainfrom
zhaomingyu13:main

zhaomingyu13 commented Dec 30, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

zhaomingyu13 commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zhaomingyu13 commented Dec 30, 2025 •

edited

Loading