[richard-for-bugen]

Motivation

Python serves n > 1 on /v1/chat/completions and /v1/completions through parallel sampling: AsyncLLM wraps the request in a ParentRequest (vllm/v1/engine/parallel_sampling.py), renders and tokenizes the prompt once, and submits one engine request per sample with the child ID {index}_{request_id} and, when seeded, seed + index. The serving layer gives each choice its own parser, streams chunks tagged with the choice index, and reports usage with the prompt counted once and completion (and reasoning) tokens summed. Per-request metrics are suppressed for n > 1, since they describe one sample.

The Rust frontend rejected any n > 1 with 400 Only n=1 is supported. (and the chat DTO capped n at 10, where Python allows up to VLLM_MAX_N_SEQUENCES, 16384 by default). n > 1 is a common OpenAI parameter (best-of style sampling, RL rollouts), and it blocks the n cases of test_chat.py and test_completion.py::test_parallel_* under the Rust frontend.

Supersedes inactive vllm-project#58914, which fans out by calling chat()/generate() n times in the server and so renders the prompt (multimodal processing included) n times.

What this PR does

The prompt is rendered and tokenized once; the fan-out happens where Python does it, and each layer above returns one stream per sample instead of pushing a choice index through every stream type:

• vllm-llm: Llm::generate_n(request, n) submits the n child engine requests like ParentRequest: internal IDs {i}_{request_id}, seed + i when seeded (wrapping; -1, Python's unseeded sentinel, stays -1 on every child, which the engine reads as unseeded, like Python normalizing it to None before the fan-out), the caller's external ID kept on every child, so an abort by that ID, or dropping the streams on client disconnect, reaches every sample. A shared ParallelSamplingStats observes vllm:request_params_n and vllm:request_max_num_generation_tokens once per request, when the last child finishes (max over the children), while the other request histograms stay per child, as in Python. n = 1 submits the request unchanged; generate is generate_n(.., 1).
• vllm-text: TextLlm::generate_n lowers once and decodes each child stream. It owns Python's SamplingParams rule that greedy sampling needs n = 1, checked against the resolved temperature (after generation-config defaults): 400 n must be 1 when using greedy sampling, got N. Python first raises a positive temperature below 0.01 to 0.01, so only temperature = 0 is greedy.
• vllm-chat: ChatLlm::chat_n renders once and builds one output processor per sample (parsers are stateful); the output grammar is built once and applied to the shared request.
• vllm-server: the routes resolve n like Python's _verify_num_sequences (n must be at least 1, got 0.; n must be at most {VLLM_MAX_N_SEQUENCES}, ...; the limit is read once at startup through a hidden --max-n-sequences arg bound to VLLM_MAX_N_SEQUENCES and the --args-json env fallbacks, parsed like Python's int() so an invalid value fails startup where Python fails each request, and carried in ApiServerOptions and RenderConfig), run the existing per-choice chunk stream or collector for each sample, and merge: streamed chunks interleave as the samples produce them, stamped with their choice index; the per-choice final usage chunks fold into one (Usage::add_choice); non-streaming responses append the choices in index order with prompt-level fields from the first choice. Per-request metrics are kept only for n = 1. n = 1 takes the same path and is unchanged on the wire.
• Render routes forward n into the rendered generate request, whose route still accepts only n = 1, so n > 1 keeps failing there explicitly instead of being dropped.

Ownership and layering: the fan-out (IDs, seeds, request-level metrics) sits in vllm-llm, the engine-client facade that mirrors AsyncLLM; the greedy rule sits in vllm-text, which owns SamplingParams semantics; per-sample parsing in vllm-chat; wire DTO checks and response merging in vllm-server. n is passed as an argument rather than added to the shared request types, since only the fan-out and the greedy rule read it.

Behavior notes

• Chunk order across choices follows arrival; Python sends every choice's role chunk before any content chunk. Clients key on index, so this is not a wire contract.
• /inference/v1/generate and gRPC still accept only n = 1 (unchanged).
• test_completion.py::test_parallel_streaming asserts at most max_tokens chunks per choice; Rust sends the finish reason in a separate final chunk (the recorded test_completion.py relax-test divergence), so a length-limited choice gets one extra chunk. test_parallel_no_streaming and the chat n tests in test_chat.py should pass; not run on GPU in this PR.

Validation

• New route tests against the mock engine (rust/src/server/src/routes/tests.rs): chat_completions_n_returns_one_choice_per_seeded_sample (child IDs and seeds 7/8 under one external ID, choices hi/stop and yo/length at indices 0/1, usage 22 + 5, request-level histograms observed once with n = 2 and max 3, per-sample generation histogram twice), chat_completions_stream_n_indexes_chunks_and_sends_one_usage_chunk, completions_n_returns_one_choice_per_sample (with seed: -1, both children unseeded), completions_stream_n_indexes_chunks_and_sends_one_usage_chunk; the two stream tests and the completions test send temperature: 0.000001, which is not greedy, n_is_checked_like_python_sampling_params (Python's messages for n = 0, n > 16384, greedy), render_forwards_n_to_the_generate_request_check.
• Red-green: on the base every new test except the render one fails (n > 1 rejected); the render test passes on the base too (its old Only n=1 check) and pins that removing that check keeps the explicit error. With targeted reverts: dropping seed + index fails the seeded test, offsetting -1 fails the completions test, yielding every per-choice usage chunk fails both stream tests, dropping the greedy check fails the validation test, a < 1e-5 greedy threshold fails the two tiny-temperature tests, observing the request-level histograms per child fails the metrics assertions (2.0 vs 1.0), and dropping n in render fails the render test.
• Rust CI command set (run-rust-frontend-cargo-ci.sh style-clippy and test): fmt, cargo sort, cargo deny check bans, workspace clippy -D warnings, cargo nextest run --workspace --all-features --locked: all green, 2435 passed, 13 skipped.
• 2026-11-04: rebased onto 39dbf51839 (the vllm-text/vllm-chat conflict with [Rust Frontend] Add the stateless Responses API (streaming and non-streaming) vllm-project/vllm#63902's generate_prepared resolved as generate_prepared_n), plus the review commit that moves VLLM_MAX_N_SEQUENCES into a hidden startup arg (n_limit_follows_max_n_sequences_option for chat, completions and both render routes; CLI default, env, --args-json and invalid-value cases). Same CI set on the head: all green, 2517 passed, 13 skipped.

Codex review

gpt-6.1-sol, high effort: changes needed, then fixed.

• P2, tiny positive temperatures rejected as greedy: fixed (only temperature = 0 is greedy, matching Python's 0.01 floor; regression on both routes).
• P2, seed: -1 offset into seeded children: fixed (the sentinel stays unseeded on every child; regression on completions).
• P3, render test described as red-green: fixed in this body (it is preservation coverage; passes on the base).

🤖 Generated with Claude Code

https://claude.ai/code/session_01CJckC9t9pN9Pi3BkC8tpAv

[BugenZhao]

Is it valuable to do this concurrently?

[richard-for-bugen]

I don't think so. submit → EngineCoreClient::call only registers the output stream and pushes the request onto the engine socket; there is no engine round trip to overlap, so submitting the children concurrently would save nothing measurable. Doing it sequentially also keeps the children's arrival order deterministic, and Python's AsyncLLM adds them the same way, in a plain for idx in range(n) loop (v1/engine/async_llm.py:558).

[BugenZhao]

Shall we cache it or make it a hidden clap arg (like some other configs)

[richard-for-bugen]

Done in 4841673: VLLM_MAX_N_SEQUENCES is now a hidden --max-n-sequences arg bound to the env var (and read in the --args-json env fallbacks), parsed once at startup like Python's int() (an invalid value fails startup), carried in ApiServerOptions / RenderConfig, and checked by the chat, completions and render handlers.

[BugenZhao]

Can we merge them or do you think it's okay to leave them apart

[richard-for-bugen]

I'd leave them apart. resolve_n runs inside the request lowering (prepare_chat_request / prepare_completion_request, also used by the render routes), which is pure: it sees only the request, never the server config, and it has to produce the NonZeroU32 that the rest of the lowering uses. The upper bound needs max_n_sequences from ApiServerOptions / RenderState, so it sits in the handler right after lowering, next to validate_max_output_tokens, the other config-dependent check placed the same way. Merging would mean threading the server limit into every lowering call (and the tests that call them) just for this check.

[richard-for-bugen]

Promoted as vllm-project#65625.