feat(cpu): add CPU support for draft model speculative decoding

[ganeshr10]

Purpose

This PR enables speculative decoding with draft models on CPU by adding PyTorch fallbacks for Triton kernels.

Benchmark Test

Command:
vllm serve Qwen/Qwen3-32B --dtype=bfloat16 --trust_remote_code --host 0.0.0.0 --port 8008 --max-model-len 20000 --speculative_config '{"model": "Qwen/Qwen3-1.7B", "num_speculative_tokens": 3, "method": "draft_model", "dtype": "bfloat16", "max-model-len": 20000}'

vllm bench serve --dataset-name hf --dataset-path philschmid/mt-bench --model Qwen/Qwen3-8B --host 0.0.0.0 --port 8008 --num-prompts 80 --max-concurrency (1/100) --temperature 0.0 --top-p 1.0

Results - Qwen 3 metrics

• Draft Model : Qwen/Qwen3-1.7B
• Target Model : Qwen/Qwen3-32B
• Hardware Used: AMD EPYC 9654 96-Core Processor

Performance Highlights:

• Equivalent Time-To-First-Token (TTFT) except in high batch sizes.
• Better Time-Per-Output-Token (TPOT) in SD as compared to Autoregressive decoding even at higher concurrencies.
• Higher output and total token throughput across all concurrency levels with speculative decoding.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[gemini-code-assist]

Code Review

This pull request introduces CPU support for speculative decoding with draft models by implementing PyTorch fallbacks for Triton-specific kernels. The changes are well-structured, adding HAS_TRITON checks to conditionally execute either the optimized Triton kernels on CUDA devices or the new PyTorch-based CPU implementations. The CPU model runner is also appropriately updated to handle speculative decoding logic without relying on CUDA-specific features. I've identified a performance optimization opportunity in one of the new PyTorch fallback functions.

[mergify]

Hi @ganeshr10, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

Comment @cursor review or bugbot run to trigger another review on this PR

[mergify]

Hi @ganeshr10, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @ganeshr10, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[benchislett]

I am in the process of reworking the way we do merge_toks_kernel into a new kernel that can handle parallel drafting. Stay tuned for updates, but in the meantime (few days at most) there's not much value in maintaining this.

[ganeshr10]

@benchislett I tested your #32887 (Unified Parallel Drafting) on CPU.

I updated my code locally to add CPU fallbacks for the new kernels:

• Removed merge_toks_kernel (no longer needed with parallel draft)
• Added PyTorch implementation for copy_and_expand_eagle_inputs_kernel
• Added PyTorch implementations for eagle_prepare_inputs_padded_kernel and eagle_prepare_next_token_padded_kernel

Test Setup:

• Draft Model: amd/PARD-Llama-3.2-1B
• Target Model: unsloth/Meta-Llama-3.1-8B-Instruct
• Speculative tokens: 8
• Temperature: 0

I obtained the same acceptance length (3.56) as mentioned in your benchmark comment.

[mergify]

Hi @ganeshr10, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @ganeshr10, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @ganeshr10, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[xuechendi]

Suggested change

	if HAS_TRITON and device.type == "cuda":
	if HAS_TRITON:

[xuechendi]

Please remove device.type == "cuda":

[ganeshr10]

Removed.

[xuechendi]

Suggested change

	if HAS_TRITON and device.type == "cuda":
	if HAS_TRITON:

[ganeshr10]

Removed.

[ganeshr10]

@bigPYJ1151 Refactored the code to follow the pattern mentioned in #37987

[bigPYJ1151]

Thanks @ganeshr10 ! It looks good now. Just have some nits, please check :)

[bigPYJ1151]

Some compilation errors in the build image with the headers. Please use cpu_types.hpp, which already includes required headrs.

[mergify]

Hi @ganeshr10, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[bigPYJ1151]

@ganeshr10 looks like there are some format issues, please check :)

[mergify]

Deprecation notice: This pull request comes from a fork and was rebased using bot_account impersonation. This capability will be removed on July 1, 2026. After this date, the rebase action will no longer be able to rebase fork pull requests with this configuration. Please switch to the update action/command to ensure compatibility going forward.

After the refactor this PR implemented SD on CPU via a plugin pattern without heavy change in the triton implementation. And will not increase the maintenance effort of the triton SD. The concern should be resolved.

[bigPYJ1151]

Hi @benchislett I think the new implementation has resolved your concern. I would move the PR forward. Please let me know if you have further thoughts, thanks! :)