[Perf] Skip detokenization in online beam search

[GuyStone]

What & why

Online beam search builds an internal per-step SamplingParams requesting
logprobs = 2 * beam_width. The engine then detokenizes every one of those
2 * beam_width logprob token ids into strings on every decode step
(LogprobsProcessor → tokenizer.decode). Beam search never uses those
strings — it ranks candidates by cum_logprob and detokenizes only the final
selected sequences itself (tokenizer.decode(tokens) near the end of
beam_search).

vllm/vllm/entrypoints/generate/beam_search/online.py

Line 208 in 837d5b3

beam.text = tokenizer.decode(tokens)

The per-step detokenization is therefore pure overhead that
grows with beam width, and it starves the GPU between decode steps.

This sets detokenize=False on the internal per-step SamplingParams in
vllm/entrypoints/generate/beam_search/online.py. Final output text is
unchanged (still produced by the explicit tokenizer.decode on the winning
beams). detokenize=False is only invalid alongside stop strings, and these
per-step params set none, so it is safe here.

         logprobs_num = 2 * beam_width
         sampling_params = SamplingParams(
             logprobs=logprobs_num,
             max_tokens=1,
             temperature=temperature,
+            detokenize=False,
         )

Scope is the online path only. The offline path
(beam_search/offline.py) has the identical pattern and would benefit from the
same one-liner as a follow-up.

Benchmark — before / after

vllm bench serve, Qwen3-1.7B, single RTX PRO 6000 Blackwell, --backend openai-chat, random dataset 512-in / 32-out, 20 prompts, --max-concurrency 4, --ignore-eos, fixed --seed 12345. Beam search triggered via
--extra-body '{"use_beam_search": true, "n": <bw>, "temperature": 1.0}'.
Beam search is non-streaming, so TTFT ≈ E2EL and TPOT/ITL are ~0 — the
meaningful metrics are end-to-end latency and throughput.

Beam width	Mean E2EL (baseline → after)	Speedup	Output tok/s (before → after)
20	1542.7 ms → 855.3 ms	1.80×	1652 → 2985
60	6611.7 ms → 2615.3 ms	2.53×	1161 → 2913

The win grows with beam width, as expected for an O(beam_width) per-step cost.

GPU utilization (nvidia-smi dmon, during the runs)

The GPU was being starved by CPU-side detokenization; removing it roughly
doubles SM utilization:

Beam width	SM util, active samples (before → after)	Peak SM
20	34% → 62%	40% → 73%
60	22% → 42%	54% → 74%

nsys profiling (before / after)

Nsight Systems trace bracketing an identical focused bw60 run (12 prompts,
conc 2, seed 777, captured via nsys start/stop):

	Baseline	After
Focused-run wall (bench duration)	21.13 s	10.08 s
Total GPU kernel time (cuda_gpu_kern_sum)	3.311 s	3.319 s
GPU busy fraction (kernel time / wall)	15.7%	32.9%
Mean E2EL (focused run)	3521.8 ms	1679.1 ms

The total GPU kernel time is identical before and after (~3.31 s) — the fix
removes no GPU work, only CPU work. Same compute in ~half the wall clock, so the
GPU busy fraction doubles (cross-validates the dmon numbers). Host-side CUDA API
time stays dominated by cudaEventSynchronize (the engine waiting on the GPU),
consistent with the bottleneck being host-side detok between GPU bursts.
(CPU IP/backtrace sampling was unavailable in this environment, so the trace
attributes the saved time to reduced wall clock / higher GPU duty cycle rather
than naming tokenizers.decode directly.)

Correctness

Identical deterministic beam-search request (n=4, temperature=0) against the
server before and after the change returned byte-identical output for all
beams — confirming detokenize=False does not alter results.

Not a duplicate

No open PR addresses this. Upstream PR #33563 implemented the
same idea and was approved by a maintainer ("a good and safe optimization")
but was closed by its author over commit-history issues, deferring to
#29133, which has since also closed. Both touched the older
file layout (entrypoints/llm.py, entrypoints/openai/engine/serving.py); the
current entrypoints/generate/beam_search/online.py still lacks the
optimization. This PR applies it to the current code path.

Test commands run

# build (editable, precompiled wheel pinned to a published commit)
VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_COMMIT=6cc2c9ba… \
  uv pip install -e . --torch-backend=auto
# server
vllm serve Qwen/Qwen3-1.7B --port 8000 --max-logprobs 256
# benchmark (run per beam width, before and after)
vllm bench serve --backend openai-chat --endpoint /v1/chat/completions \
  --model Qwen/Qwen3-1.7B --dataset-name random --random-input-len 512 \
  --random-output-len 32 --num-prompts 20 --max-concurrency 4 --seed 12345 \
  --ignore-eos --num-warmups 3 \
  --extra-body '{"use_beam_search": true, "n": 20, "temperature": 1.0}'

Lint: ruff check and ruff format --check pass on the changed file.

AI assistance

This change was developed with AI assistance (Claude Code). The human submitter
reviewed every changed line and ran the benchmarks and correctness check above.

[mgoin]

Thanks for catching this forgotten bit