[Cherry-pick][releases/v0.26.0rc][Bugfix][MRV2][Spec Decode] Fix draft token offset in bad words kernel (from #14433) - #14526
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request cherry-picks a fix from the upstream repository to address an off-by-one error occurring during speculative decoding. By adjusting the memory offset when loading draft tokens, the kernel now correctly identifies tokens, ensuring accurate filtering of bad words. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
Code Review
Suggested PR Title:
[Ops][BugFix] Correct draft token offset in bad_words_kernelSuggested PR Summary:
### What this PR does / why we need it?
This PR fixes an indexing issue in `_bad_words_kernel` when loading draft tokens. It adds `+ 1` to the offset calculation because the token at local position 0 is the last committed token, and draft tokens start at local position 1.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Not specified in the patch.I have reviewed the changes and have no additional feedback to provide as there are no review comments.
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. Tip 💡 Consider Linking a Related Issue or RFCYour PR title contains the [BugFix] tag, indicating a bug fix or new feature. Linking a related issue or RFC in the PR description is strongly encouraged — it gives reviewers helpful context and speeds up the review. You can use any of these keywords:
🙏 Thanks for helping us keep the project well-organized! |
…t token offset in bad words kernel (from vllm-project#14433) (vllm-project#14526) Cherry-pick of PR vllm-project#14433 onto `releases/v0.26.0rc`. Original PR: vllm-project#14433 Original author: @SOMEONEUNSEEN --- ### What this PR does / why we need it? 同步上游更改 vllm-project/vllm#52311 ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@d02df74 Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Cherry-pick of PR #14433 onto
releases/v0.26.0rc.Original PR: #14433
Original author: @SOMEONEUNSEEN
What this PR does / why we need it?
同步上游更改
vllm-project/vllm#52311
Does this PR introduce any user-facing change?
How was this patch tested?