Skip to content

fix(scale-out): return all n choices from /inference/v1/generate #52507

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hclsys wants to merge 1 commit into from
Closed

fix(scale-out): return all n choices from /inference/v1/generate #52507

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -462,3 +462,24 @@ async def test_generate_with_lora_adapter(client, tokenizer, messages):
completions_res = completions_data["choices"][0]["message"]["content"]

assert generate_res == completions_res


@pytest.mark.asyncio
@pytest.mark.parametrize("n", [2, 4])
async def test_generate_endpoint_returns_all_n_choices(client, n):
# Regression test for #52398: a non-streaming /inference/v1/generate
# request with n > 1 must return exactly n choices. The handler previously
# left output_kind at the SamplingParams default (CUMULATIVE) for
# non-streaming requests, so sequences that finished before the final
# streamed step were dropped and fewer than n choices came back.
payload = {
"model": MODEL_NAME,
"token_ids": [9707, 1879, 11],
"sampling_params": {"max_tokens": 5, "temperature": 1.0, "n": n},

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Force choices to finish on different decoding steps

With max_tokens=5, all choices normally hit the length limit in the same engine iteration; the RequestOutputCollector then merges those same-step cumulative outputs, so the pre-fix implementation can still return all n choices and pass this test. An unseeded, temperature-1 sample only exposes the regression if a choice happens to emit EOS early, making coverage probabilistic and potentially flaky. Use a deterministic setup or mocked generator that finishes choices on different steps so reverting the production fix reliably fails.

AGENTS.md reference: AGENTS.md:L82-L93

Useful? React with 👍 / 👎.

"stream": False,
}
resp = await client.post(GEN_ENDPOINT, json=payload)
resp.raise_for_status()
data = resp.json()
assert len(data["choices"]) == n
assert sorted(c["index"] for c in data["choices"]) == list(range(n))
10 changes: 8 additions & 2 deletions vllm/entrypoints/scale_out/token_in_token_out/serving.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,8 +218,14 @@ async def serve_tokens(

if self.force_no_detokenize:
sampling_params.detokenize = False
if request.stream:
sampling_params.output_kind = RequestOutputKind.DELTA
# Match the OpenAI serving path (CompletionRequest.to_sampling_params):
# non-streaming must use FINAL_ONLY. Under the SamplingParams default
# (CUMULATIVE) serve_tokens_full_generator keeps only the last streamed
# RequestOutput, and sequences that finished earlier are already pruned
# from it, so with n > 1 they silently never reach `choices`.
sampling_params.output_kind = (
RequestOutputKind.DELTA if request.stream else RequestOutputKind.FINAL_ONLY
)

self._log_inputs(
request_id,
Expand Down
Loading