[qgallouedec]

Purpose

closes #52398

serve_tokens sets sampling_params.output_kind only for streaming requests, so non-streaming ones keep the SamplingParams default, CUMULATIVE. With n > 1, serve_tokens_full_generator keeps only the last streamed RequestOutput and reads final_res.outputs, which under CUMULATIVE holds just the sequences updated in that step,
the other completions are silently dropped. No error is raised; the response is simply short.

CompletionRequest.to_sampling_params already uses FINAL_ONLY when not streaming, so this makes the
token-in-token-out path match it.

Test Plan

10 non-streaming requests per n (2, 4, 8) against vllm serve Qwen/Qwen2.5-1.5B on an H100, before and after the patch, counting the choices returned by /inference/v1/generate. Same check with multimodal input (/v1/chat/completions/render → token_ids + features, Qwen2.5-VL-3B). Streaming, logprobs, prompt_logprobs and n=1 re-checked after the patch. Adds a regression test asserting the response carries indices 0..n-1.

Test Result

Before:

n=2 -> {1: 6, 2: 4}
n=4 -> {2: 5, 3: 3, 4: 2}
n=8 -> {5: 2, 7: 5, 3: 1, 6: 2}
multimodal, n=4 -> 3 choices on 6/6 runs

After:

n=2 -> {2: 10}
n=4 -> {4: 10}
n=8 -> {8: 10}
multimodal, n=4 -> 4 choices on 6/6 runs

n=1, streaming, logprobs and prompt_logprobs behave as before. The measurements were taken on v0.27.1 with this
patch applied to the installed package; the added test follows the existing fixtures in tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py and will get its first full run in CI.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[DarkLight1337]

/ci run

[github-actions]

✅ Triggered Buildkite CI #84001 for commit fcbd83beada7.

[DarkLight1337]

PTAL at the failing test: https://buildkite.com/vllm/ci/builds/84001/canvas?jid=01a009b0-abda-47a4-88a8-30180757cba6&tab=output

[qgallouedec]

Strange, I can't reproduce this. On main (0.27.2rc1.dev171+gc1e438728), with the test's payload and the fixture's server args, 5 trials each:

vllm serve Qwen/Qwen3-0.6B --dtype bfloat16 --max-model-len 1024 \
  --enforce-eager --no-enable-prefix-caching --port 8123

for i in 1 2 3 4 5; do
  curl -s localhost:8123/inference/v1/generate -H 'Content-Type: application/json' -d '{
    "model": "Qwen/Qwen3-0.6B", "token_ids": [1,2,3], "stream": false,
    "sampling_params": {"max_tokens": 5, "temperature": 1.0, "n": 4}}' \
  | python -c 'import json,sys; print(sorted(c["index"] for c in json.load(sys.stdin)["choices"]))'
done

with this patch:   [0,1,2,3] [0,1,2,3] [0,1,2,3] [0,1,2,3] [0,1,2,3]
without it:        [1,2,3]   [3]       [2,3]     [1,2,3]   [2,3]

So the bug reproduces on main and the patch fixes it. The CI result is what the unpatched path returns.

Could we re-run the job? If it fails again I'll log the SamplingParams the server actually receives (n, output_kind) to see whether the request reaches the engine with the patched values. Thanks

[DarkLight1337]

/ci run

[github-actions]

✅ Triggered Buildkite CI #84337 for commit e646b1d7db16.

[qgallouedec]

ok I get it: the failing job runs this suite with VLLM_USE_RUST_FRONTEND=1, where /inference/v1/generate is served by the Rust frontend rather than serve_tokens, so the fix here doesn't apply.

The Rust frontend doesn't implement parallel sampling: its OpenAI routes reject n > 1 explicitly, and the raw generate route silently ignores n (the field isn't even part of its SamplingParams) and always returns a single choice.

Skipped the regression test under the Rust frontend, following the existing test_generate_sampling_mask precedent.

Follow-up PR: make the Rust generate route reject n > 1 loudly, matching its OpenAI routes. Opening a tracking issue for this: #52843

[DarkLight1337]

/ci run

[github-actions]

✅ Triggered Buildkite CI #84531 for commit 5aaed3d1e7f9.

[qgallouedec]

Alright, it's green now