[BugenZhao]

Overview

Support n > 1 on /v1/chat/completions and /v1/completions in the Rust frontend, following Python's parallel sampling: render and tokenize the prompt once, then submit one engine request per sample. Today the Rust frontend rejects any n > 1 with 400 Only n=1 is supported.

Supersedes inactive #58914, which fans out by calling chat()/generate() n times in the server and so renders the prompt (multimodal processing included) n times.

Claims

• Chat and completions accept n > 1, streaming and non-streaming. Each choice carries its index; usage counts the prompt once and sums completion and reasoning tokens across choices.
• Like Python's ParentRequest, the prompt is rendered once, child requests use IDs {index}_{request_id} and seed + index when seeded, and the caller's external request ID stays on every child, so an abort or a client disconnect reaches every sample.
• n is validated like Python's SamplingParams: at least 1, at most VLLM_MAX_N_SEQUENCES (16384 by default; the chat DTO's old cap of 10 is gone), and 1 under greedy sampling, with Python's error messages.
• Metrics follow Python: vllm:request_params_n and vllm:request_max_num_generation_tokens are observed once per request, the other request histograms per sample, and per-request metrics in the response only for n = 1.
• n = 1 takes the same path and is unchanged on the wire. /inference/v1/generate and gRPC still accept only n = 1; the render routes forward n into the rendered generate request, so n > 1 keeps failing there explicitly instead of being dropped.

Validation

Commands from .buildkite/scripts/run-rust-frontend-cargo-ci.sh (style-clippy and test), run from the repo root on this branch:

cargo fmt --manifest-path rust/Cargo.toml --all -- --check   # ok
cargo sort --workspace --check rust                          # ok
cargo deny --manifest-path rust/Cargo.toml --config rust/deny.toml check bans   # ok
cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --all-features --locked -- -D warnings   # ok
cargo nextest run --manifest-path rust/Cargo.toml --workspace --all-features --locked --no-fail-fast
# 2550 tests run: 2550 passed, 13 skipped

The first commit also passes clippy and nextest on its own (2549 passed, 13 skipped).

New route tests against the mock engine (rust/src/server/src/routes/tests.rs):

• chat_completions_n_returns_one_choice_per_seeded_sample: child IDs and seeds 7/8 under one external ID; choices hi/stop and yo/length at indices 0/1; usage 22 + 5; the request-level histograms observed once (n = 2, max 3), the per-sample generation histogram twice.
• chat_completions_stream_n_indexes_chunks_and_sends_one_usage_chunk and completions_stream_n_indexes_chunks_and_sends_one_usage_chunk.
• completions_n_returns_one_choice_per_sample, with seed: -1: both children stay unseeded.
• n_is_checked_like_python_sampling_params: Python's messages for n = 0, n > 16384, and greedy sampling.
• n_limit_follows_max_n_sequences_option: the configured limit on chat, completions and both render routes; CLI tests cover the default, the env var, --args-json and an invalid value.
• render_forwards_n_to_the_generate_request_check: the render routes keep n and the generate route still rejects n > 1.

Red-green: on main, every new test except the render one fails (n > 1 is rejected); the render test passes on main too and pins that removing the old check keeps the explicit error. With targeted reverts: dropping seed + index fails the seeded test, offsetting -1 fails the completions test, yielding every per-choice usage chunk fails both stream tests, dropping the greedy check fails the validation test, a < 1e-5 greedy threshold fails the tiny-temperature cases, observing the request-level histograms per sample fails the metrics assertions, and dropping n in render fails the render test.

Not run end to end on GPU in this PR. test_completion.py::test_parallel_* and the n cases in test_chat.py remain deselected in the Rust frontend CI step; test_parallel_streaming asserts at most max_tokens chunks per choice, while Rust sends the finish reason in a separate final chunk, so a length-limited choice gets one extra chunk.

Details

The fan-out happens where Python does it, and each layer above returns one stream per sample instead of pushing a choice index through every stream type:

• vllm-llm: generate_n(request, n) is the required generation method of the Llm trait, and generate a provided method calling generate_n(.., 1), since how n is carried out (child IDs, routing, request-level metrics) belongs to the implementation. EngineCoreLlm::generate_n submits the n child engine requests like ParentRequest. Seeds wrap on overflow, and -1, Python's unseeded sentinel, stays -1 on every child (Python normalizes it to None before the fan-out). A shared ParallelSamplingStats observes the request-level histograms once, when the last child finishes (max over the children). n = 1 submits the request unchanged.
• vllm-text: TextLlm::generate_n lowers once and decodes each child stream. It owns the greedy rule, checked against the resolved temperature (after generation-config defaults). Python first raises a positive temperature below 0.01 to 0.01, so only temperature = 0 is greedy. Submission goes through the existing submit step, which now takes n, so the --skip-tokenizer-init structured-output check covers every sample.
• vllm-chat: ChatLlm::chat_n renders once and builds one output processor per sample, since parsers are stateful; the output grammar is built once and applied to the shared request.
• vllm-server: the routes resolve n like Python's _verify_num_sequences, run the existing per-choice chunk stream or collector for each sample, and merge. Streamed chunks interleave as the samples produce them, stamped with their choice index; the per-choice final usage chunks fold into one (Usage::add_choice); non-streaming responses append the choices in index order, with prompt-level fields from the first choice.
• VLLM_MAX_N_SEQUENCES is read once at startup through a hidden --max-n-sequences argument bound to the env var (and to the --args-json env fallbacks), parsed like Python's int(), and carried in ApiServerOptions and RenderConfig. An invalid value fails startup, where Python fails each request.

Ownership and layering: the fan-out (IDs, seeds, request-level metrics) sits in vllm-llm's EngineCoreLlm, the Llm implementation that mirrors AsyncLLM; the greedy rule in vllm-text, which owns SamplingParams semantics; per-sample parsing in vllm-chat; wire DTO checks and response merging in vllm-server. n is passed as an argument rather than added to the shared request types, since only the fan-out and the greedy rule read it. The n range check stays in request lowering, which is configuration-free and shared with the render routes, separate from the configured upper limit.

Behavior notes:

• Chunk order across choices follows arrival; Python sends every choice's role chunk before any content chunk. Clients key on index, so this is not a wire contract.

Not a duplicate: the only open PR implementing n in the Rust frontend is #58914, inactive since 2026-10-12 and built on a different design (see Overview); searched open PRs for n choices, parallel sampling, n > 1 and Only n=1. The open beam-search PRs (#47705, #64862) cover a different feature. AI assistance (Claude Code) was used to prepare this change; the human submitter reviewed every changed line.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CJckCAkBsrWKV6peRwSdih

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @BugenZhao.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #100165 for commit e8fc583be6d7.

ℹ️ This PR is 11 commits behind upstream main. Merged upstream main into this branch as e8fc583be6d7 before starting CI. Pull the branch before pushing more changes.