[Feature] add draft_model spec decode for mrv2

[HF-001]

What this PR does / why we need it?

refer to : vllm-project/vllm#43091

How was this patch tested?

export VLLM_USE_V2_MODEL_RUNNER=1
#export CUDA_VISIBLE_DEVICES=8
export ASCEND_RT_VISIBLE_DEVICES=6

vllm serve /nas/disk1/Qwen3-14B/Qwen3-14B/Qwen3-14B_20250506_01 --port 8898 --dtype bfloat16  \
--tensor-parallel-size 1 --gpu-memory-utilization 0.85 \
--max-model-len 32768 --trust-remote-code \
--no-enable-prefix-caching \
--async-scheduling \
--speculative_config '{"method":"draft_model", "model":"/nas/disk1/Qwen3-1.7B","num_speculative_tokens":3,"draft_tensor_parallel_size":1}'

• vLLM main: vllm-project/vllm@ced6857

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces support for speculative decoding using an external draft model within the Model Runner V2 (MRv2) architecture on Ascend devices. By implementing a custom speculator that adapts the upstream standalone AR speculator, the changes allow for efficient draft model execution while adhering to specific hardware constraints. The implementation includes necessary validation for model configurations, integration with Ascend-specific attention metadata builders, and robust test coverage to ensure parity between target and draft model outputs.

Highlights

• Feature Implementation: Added support for draft_model speculative decoding for Model Runner V2 (MRv2) on Ascend hardware.
• Integration: Implemented AscendStandaloneARSpeculator to interface with the upstream StandaloneARSpeculator API, enabling eager drafting for MRv2.
• Testing: Added comprehensive end-to-end tests for draft model execution and probabilistic sampling, along with unit tests for the new speculator implementation.
• Documentation: Updated the speculative decoding feature guide with configuration instructions and requirements for MRv2 Ascend integration.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

• A PR should do only one thing, smaller PRs enable faster reviews.
• Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
• Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

---

Tip

💡 Consider Linking a Related Issue or RFC

Your PR title contains the [Feature] tag, indicating a bug fix or new feature.

Linking a related issue or RFC in the PR description is strongly encouraged — it gives reviewers helpful context and speeds up the review. You can use any of these keywords:

• Fixes #<issue_number>
• Closes #<issue_number>
• Resolves #<issue_number>
• Refs #<rfc_or_issue_number> (for RFCs)

🙏 Thanks for helping us keep the project well-organized!

[gemini-code-assist]

Code Review

Suggested PR Title:\n\nmarkdown\n[Attention][Feature] Support external draft models with Model Runner V2 on Ascend\n\n\nSuggested PR Summary:\n\nmarkdown\n### What this PR does / why we need it?\nThis PR introduces support for external draft models with Model Runner V2 (MRv2) on Ascend, targeting the upstream `standalone_ar.StandaloneARSpeculator` API. It implements `AscendStandaloneARSpeculator` to handle independent draft LM execution with exact FIA lengths and upstream eager drafting.\n\nFeedback:\n- In `_prefill`, explicitly cast both `input_batch.seq_lens` and `num_rejected` to `int32` before stacking to prevent potential dtype mismatch errors on Ascend NPUs.\n- In `_build_attn_metadata`, safely retrieve `_target_seq_lens_cpu` and `_num_rejected_cpu` using `getattr` to prevent `AttributeError` when `_prefill` is skipped or called with `skip_attn=True`.\n\n### Does this PR introduce _any_ user-facing change?\nYes, users can now enable external draft models with Model Runner V2 on Ascend by setting `VLLM_USE_V2_MODEL_RUNNER=1` and configuring `speculative_config` with `method="draft_model"`.\n\n### How was this patch tested?\n- Added unit tests in `tests/ut/worker/v2/test_standalone_ar_speculator.py` and `tests/ut/worker/v2/test_spec_decode.py`.\n- Added end-to-end tests in `tests/e2e/pull_request/one_card/model_runner_v2/test_qwen3_draft_model.py`.\n

[gemini-code-assist]

To prevent potential dtype mismatch errors or unnecessary type conversions on Ascend NPUs, both input_batch.seq_lens and num_rejected should be explicitly cast to int32 (using .int()) before being stacked. On Ascend, int32 is the preferred integer type for attention metadata and indexing operations.

Suggested change

	lengths = torch.stack((input_batch.seq_lens[:num_reqs], num_rejected[:num_reqs].int())).cpu()
	lengths = torch.stack((input_batch.seq_lens[:num_reqs].int(), num_rejected[:num_reqs].int())).cpu()

[gemini-code-assist]

If _prefill is skipped or called with skip_attn=True (e.g., during profiling, dry-runs, or early initialization), self._target_seq_lens_cpu and self._num_rejected_cpu will not be initialized. This will cause an AttributeError when _build_attn_metadata is called. To make this robust and prevent crashes, retrieve these attributes safely using getattr and fallback to seq_lens_cpu_upper_bound and a zero tensor respectively if they are not set.

        target_seq_lens_cpu = getattr(self, "_target_seq_lens_cpu", None)
        num_rejected_cpu = getattr(self, "_num_rejected_cpu", None)
        if target_seq_lens_cpu is None:
            target_seq_lens_cpu = seq_lens_cpu_upper_bound
            num_rejected_cpu = torch.zeros_like(target_seq_lens_cpu)

        if step == 0:
            # Expanded prefill includes rejected placeholders; causal attention
            # aligns queries using target length + one correction slot.
            seq_lens_cpu = target_seq_lens_cpu + 1
            seq_lens_cpu_upper_bound = seq_lens_cpu
            positions = self.expanded_positions
            attn_state = AscendAttentionState.ChunkedPrefill
        else:
            # Upstream passes step + 1 to account for the correction token.
            seq_lens_cpu = target_seq_lens_cpu - num_rejected_cpu + step
            seq_lens_cpu_upper_bound = target_seq_lens_cpu - num_rejected_cpu
            positions = self.input_buffers.positions
            attn_state = AscendAttentionState.DecodeOnly

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.