[eagle3][pcp] fix acceptance rate for eagle3 and pcp enabled

[lilinsiman]

What this PR does / why we need it?

fix the position 3 acceptance rate for eagle3 and pcp enabled

detail:
In the merged graph of eagle_proposer, the code logic was changed from updating the code once before the forward pass of the draft model to updating all three positions of common_attn_metadata in the merged graph before performing the forward pass of the model. As a result, the update of position 2 and position 3 affected the update of position 1.

For example, in the following field:
common_attn_metadata.block_table_tensor[:batch_size] = common_attn_metadata.block_table_tensor[block_indices]

When updating the block_table_tensor at position 2, the modification of this field occurred at the original address of common_attn_metadata. As a result, the parameter at position 1 was also modified, but the forward pass at position 1 had not been performed. Therefore, a copy of the address of block_table_tensor needs to be made, and the modification needs to be performed on the new address to ensure complete isolation between positions.

Does this PR introduce any user-facing change?

no

How was this patch tested?

tests and ut

• vLLM version: v0.18.0
• vLLM main: vllm-project/vllm@8b63257

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical bug affecting the acceptance rate of the Eagle3 speculative decoding mechanism when used in conjunction with PCP. The fix ensures that the attention metadata, which is crucial for correct token generation and validation, is properly isolated for each processing step by cloning relevant tensors. This prevents data corruption due to shared mutable state, leading to accurate speculative decoding and improved model performance under these specific configurations.

Highlights

• Acceptance Rate Fix: Corrected an issue impacting the acceptance rate for Eagle3 speculative decoding when PCP (Parallel Compute Pipeline) is enabled, specifically addressing position 3.
• Attention Metadata Cloning: Introduced explicit cloning of block_table_tensor, seq_lens, seq_lens_cpu, num_computed_tokens_cpu, and positions within the attention metadata to prevent unintended modifications and ensure data integrity during speculative decoding operations.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request addresses a data corruption bug in the speculative decoding logic for the Eagle proposer, specifically when Prefill Context Parallelism (PCP) is enabled. The issue stems from in-place modifications of tensors within the attention metadata across different speculative decoding steps, which could lead to incorrect behavior and a reduced token acceptance rate. The fix involves cloning several tensors (block_table_tensor, seq_lens, seq_lens_cpu, num_computed_tokens_cpu, and positions) before they are modified within the proposal and attention update logic. This ensures that each step of the speculative decoding process operates on an independent copy of the metadata, preventing side effects and ensuring correctness. The changes are well-targeted and correctly resolve the potential for data corruption.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

• A PR should do only one thing, smaller PRs enable faster reviews.
• Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
• Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

[yiz-liu]

This should be fine in eager mode, but it will definitely cause issues in ACL Graph.

1. Put the new block tables in a persistent buffer, maybe as a self attribute in the proposer.
2. Make sure both dummy_run and propose read from that same buffer.