Skip to content

generate invoking doesn't require detokenization for beam search - #33563

Closed
gameofdimension wants to merge 102 commits into
vllm-project:mainfrom
gameofdimension:patch-1
Closed

generate invoking doesn't require detokenization for beam search#33563
gameofdimension wants to merge 102 commits into
vllm-project:mainfrom
gameofdimension:patch-1

Conversation

@gameofdimension

@gameofdimension gameofdimension commented Feb 2, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown

Purpose

vllm/vllm/entrypoints/openai/engine/serving.py

Line 504 in b398e5c

beam.text = tokenizer.decode(tokens)

The code here demonstrates detokenization isn't required for generate(), and skipping it offers minor latency benefits.

Test Plan

Test Result

model Qwen3-1.7B
GPU NVIDIA A30
dataset An internal dataset containing 106 samples,
with sequence lengths ranging from 958 to 1,230 tokens
max_tokens 5
beam_width 20

latency improvement

mean p99
baseline 0.178 0.208
this pr 0.201 0.232
Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@mergify mergify Bot added the frontend label Feb 2, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Feb 2, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a performance optimization to the beam_search method. By setting detokenize=False for the generate calls within the beam search loop, it avoids unnecessary work since the intermediate text is not used. The final sequences are correctly decoded at the end of the function. This is a good and safe optimization.

@gameofdimension

gameofdimension commented Feb 2, 2026

Copy link
Copy Markdown
Author

@aarnphm @chaunceyjiang
Would you mind reviewing this?

njhill
njhill approved these changes Feb 2, 2026
View reviewed changes

@njhill njhill left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gameofdimension.

This is actually also included in https://github.com/vllm-project/vllm/pull/29133/files which stalled. Maybe you could also include an explanatory comment similar to that one.

@mergify mergify Bot added the qwen Related to Qwen models label Feb 3, 2026
gameofdimension
gameofdimension commented Feb 3, 2026
View reviewed changes
Comment thread vllm/entrypoints/openai/engine/serving.py Outdated
@@ -347,6 +347,7 @@ async def beam_search(
logprobs=logprobs_num,
max_tokens=1,
temperature=temperature,
detokenize=False,

@gameofdimension gameofdimension Feb 3, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the review bot pointed out, we never use generate()'s returned text - the real detokenization happens later in this function.

@gameofdimension gameofdimension Feb 3, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@njhill

@njhill njhill Feb 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I meant adding a comment in the code next to this line, as is in the other PR I referenced.

@gameofdimension gameofdimension changed the title generate API doesn't require detokenization for beam search generate invoking doesn't require detokenization for beam search Feb 3, 2026
gameofdimension and others added 18 commits February 5, 2026 09:54
@gameofdimension felix01.yu
detokenize is unnecessary
b98ce64 
Signed-off-by: G.O.D <32255912+gameofdimension@users.noreply.github.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@DarkLight1337 felix01.yu
[Model] Use explicit types in get_generation_prompt (vllm-project#3…
4b2d503 
…3551)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@gameofdimension felix01.yu
offline beam search doesn't require detokenization either
dfd512f 
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@rabi felix01.yu
fix[ROCm]: Remove unconditional aiter import (vllm-project#32902)
9878396 
Signed-off-by: rabi <ramishra@redhat.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@danielafrimi felix01.yu
[MoE] Enable Shared/Routed Overlap For Latent MoE (Nemotron-H) (vllm-…
5b6bc9b 
…project#32790)

Signed-off-by: dafrimi <dafrimi@nvidia.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@Isotr0py felix01.yu
[Bugfix] Enable Kimi k25 processor test (vllm-project#33562)
f5dc8c0 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@shanjiaz felix01.yu
move spec decode slow test to test_areas.yaml (vllm-project#33365)
525b318 
Signed-off-by: shanjiaz <zsjwpianpian@gmail.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@kebe7jun @tvegas1
@youkaichao felix01.yu
[Feature][Core] Support Fabric detection to adapt the MNNVL protocol …
f56f36b 
…for the GB series (vllm-project#33540)

Signed-off-by: Kebe <mail@kebe7jun.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Thomas Vegas <tvegas@nvidia.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@DarkLight1337 felix01.yu
[Refactor] Move profiling methods to MM budget (vllm-project#33559)
8c818aa 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@MatthewBonanni felix01.yu
[CI] Add DeepSeek V3.2 nightly eval (vllm-project#33566)
5941e46 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@KKSK-DON felix01.yu
[Model] Use mm_position to compute mrope positions for GLM-4.xV (vllm…
a89e73a 
…-project#33039)

Signed-off-by: Yang <lymailforjob@gmail.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@hmellor felix01.yu
Remove incorrect tokenizer info test (vllm-project#33565)
35a1923 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@hmellor felix01.yu
Update huggingface-hub again (vllm-project#33567)
2588853 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@yugong333 felix01.yu
Reduce the kernel overhead when num of active loras is smaller than …
a37cdde 
…max loras. Multiple cuda graphs are captured for each num of active-loras. (vllm-project#32005)

Signed-off-by: Yu Gong <yu3.gong@gmail.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@MatthewBonanni felix01.yu
[UX] Format attention backend log line (vllm-project#33570)
49a44bb 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@vkuzo felix01.yu
fix memory for online fp8 quantization with streaming weight load (vl…
1b9e3ff 
…lm-project#31914)

Signed-off-by: vasiliy <vasiliy@fb.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@IwakuraRein @pavanimajety felix01.yu
fix cutlass_3x_gemm_fp8_blockwise on sm103a (vllm-project#32224)
3807f65 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
Co-authored-by: Pavani Majety <pmajety@nvidia.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@patrickvonplaten @gemini-code-assist felix01.yu
[Voxtral Realtime] Introduce global log mel max (vllm-project#33574)
5c2d374 
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: felix01.yu <felix01.yu@vipshop.com>
@mergify

mergify Bot commented Feb 5, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--33563.org.readthedocs.build/en/33563/

@mergify mergify Bot added documentation Improvements or additions to documentation ci/build deepseek Related to DeepSeek models llama Related to Llama models multi-modality Related to multi-modality (#4194) new-model Requests to new models performance Performance-related issues nvidia rocm Related to AMD ROCm labels Feb 5, 2026
@mergify mergify Bot added the cpu Related to CPU backends label Feb 5, 2026
@github-project-automation github-project-automation Bot moved this to Ready in NVIDIA Feb 5, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Feb 5, 2026
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Feb 5, 2026
@github-project-automation github-project-automation Bot moved this from Ready to Done in NVIDIA Feb 5, 2026
@gameofdimension

gameofdimension commented Feb 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

The commit history is messed up. closed

wait PR 29133 to be merged

@gameofdimension
gameofdimension deleted the patch-1 branch June 22, 2026 06:49
This was referenced Jun 22, 2026
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@njhill njhill njhill approved these changes
@aarnphm aarnphm Awaiting requested review from aarnphm aarnphm is a code owner
@chaunceyjiang chaunceyjiang Awaiting requested review from chaunceyjiang chaunceyjiang is a code owner
@patrickvonplaten patrickvonplaten Awaiting requested review from patrickvonplaten
@sighingnow sighingnow Awaiting requested review from sighingnow
@russellb russellb Awaiting requested review from russellb
@noooop noooop Awaiting requested review from noooop
@tjtanaa tjtanaa Awaiting requested review from tjtanaa
@gshtras gshtras Awaiting requested review from gshtras
@tdoublep tdoublep Awaiting requested review from tdoublep
@jikunshang jikunshang Awaiting requested review from jikunshang
@bigPYJ1151 bigPYJ1151 Awaiting requested review from bigPYJ1151
@hmellor hmellor Awaiting requested review from hmellor
@markmc markmc Awaiting requested review from markmc
@yewentao256 yewentao256 Awaiting requested review from yewentao256
@ApostaC ApostaC Awaiting requested review from ApostaC
@orozery orozery Awaiting requested review from orozery
@jeejeelee jeejeelee Awaiting requested review from jeejeelee
@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon
@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat
@ywang96 ywang96 Awaiting requested review from ywang96
@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat
@heheda12345 heheda12345 Awaiting requested review from heheda12345
@mgoin mgoin Awaiting requested review from mgoin
@pavanimajety pavanimajety Awaiting requested review from pavanimajety
@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337 DarkLight1337 is a code owner
@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth
@NickLucche NickLucche Awaiting requested review from NickLucche
@benchislett benchislett Awaiting requested review from benchislett
@luccafong luccafong Awaiting requested review from luccafong
@zhuohan123 zhuohan123 Awaiting requested review from zhuohan123
@youkaichao youkaichao Awaiting requested review from youkaichao
@LucasWilkinson LucasWilkinson Awaiting requested review from LucasWilkinson
@houseroad houseroad Awaiting requested review from houseroad
@ProExpertProg ProExpertProg Awaiting requested review from ProExpertProg
@zou3519 zou3519 Awaiting requested review from zou3519
@22quinn 22quinn Awaiting requested review from 22quinn
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build cpu Related to CPU backends deepseek Related to DeepSeek models documentation Improvements or additions to documentation frontend kv-connector llama Related to Llama models multi-modality Related to multi-modality (#4194) new-model Requests to new models nvidia performance Performance-related issues qwen Related to Qwen models rocm Related to AMD ROCm speculative-decoding structured-output v1

Projects

AMD
Status: Done
NVIDIA
Status: Done
Structured Output
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.