[richard-for-bugen]

Summary

First e2e run PR: step 1 of the backfill that pins Rust parity with Python's own server-level tests. Based on upstream main at aa31fa038f. It goes through every -k exclusion and commented-out command in .buildkite/test_areas/rust_frontend.yaml, runs each one locally against the Rust frontend (and against Python for every failure), and enables the ones that pass. 10 exclusions are dropped (one of them with a test relaxation); the remaining ones now say why they exist. Step 2 adds 11 commands from server-level test files that were not in the Rust CI yet (four with a relaxation).

Only tests/ and .buildkite/test_areas/rust_frontend.yaml change. A test is relaxed only where both behaviors are semantically acceptable; no assertion on the behavior under test is weakened.

Items

Item	CI step	Python test	Finding	Action
invalid structured-output inputs	OpenAI Coverage	test_chat_completion.py::test_invalid_json_schema, ::test_invalid_regex	pass: Rust returns the 400	drop exclusion
streamed logprobs comparison	OpenAI Coverage	test_return_token_ids.py::test_comparison_with_prompt_logprobs_and_logprobs	Rust streams a leading prompt_token_ids chunk and a trailing finish_reason chunk with logprobs: null (nullable in the OpenAI schema); the test dereferenced logprobs.tokens on every chunk	enable; the loop skips null logprobs, the token-id equality assertion is unchanged
stale shutdown clause	OpenAI Coverage	test_shutdown.py	test_abort_timeout_exits_quickly was renamed (test_abort_timeout_exits_within_cleanup_grace), so the clause matched nothing	drop the clause
benchmark CLI	OpenAI Coverage	benchmarks/test_serve_cli.py insecure, test_bench_serve	gated Llama 3.2 1B cannot be downloaded locally; with Qwen2.5-1.5B swapped in, all pass under Rust and Python	drop exclusion; CI is the final check
server load	Serve/Admin	test_basic.py server_load	15/15 across three fresh servers; not flaky	drop exclusion
generate endpoint	Serve/Admin	test_serving_tokens.py stream, test_generate_logprobs, stop_string_workflow	pass; the top_logprobs truncation comment was stale. Four stream cases self-skip under Rust	keep only -k "not lora"
metrics	Serve/Admin	test_metrics.py show, test_metrics_counts, multimodal variants	pass (multimodal test_metrics_counts self-skips)	-k "not run_batch and not test_metrics_exist"
in-process selection	OpenAI Coverage	test_serving_chat.py three -k-selected tests	they run over a MockEngine in-process and never reach Rust	drop the line
DP load balancing	Distributed	v1/distributed/test_{internal,external,hybrid}_lb_dp.py 4 cases (--api-server-count 4)	pass: the Rust frontend ignores the count and serves them from one multithreaded process	-k "not server_info"
tool use	Tool Use	test_response_format_with_tool_choice_required, test_parallel_tool_calls_false, test_tool_call_and_choice	gated Llama 3.2 3B; the ungated mirror unsloth/Llama-3.2-3B-Instruct with the same config passes 5/5 each	drop exclusions; CI is the final check

Step 2, new commands (each passes locally under Rust with --gpu-memory-utilization capped to the CI slice size):

CI step	Command	Note
OpenAI Coverage	chat_completion/test_chat_echo.py	relaxed: the test sent "logprobs": "true" (a string); the OpenAI schema says boolean, and Rust only accepts a boolean, so it sends True
OpenAI Coverage	chat_completion/test_logprob_token_ids.py
OpenAI Coverage	chat_completion/test_thinking_token_budget.py -k "not qwen35"	the Qwen3.5-35B FP8 MTP case needs two GPUs
OpenAI Coverage	completion/test_tensorizer_entrypoint.py	relaxed: Rust also sends completion_tokens_details, so the usage check compares the three counts instead of the whole object
OpenAI Coverage	test_chunked_prompt.py	relaxed: Rust ends the stream with a finish-only chunk and sends the role chunk without content; every chunk that carries tokens is still checked
Serve/Admin	serve/middleware/test_optional_middleware.py -k "not test_v2_endpoint_accepts_valid_api_token"	the excluded test calls /v2/embed (pooling)
Serve/Admin	launchers/api_server/test_multi_api_servers.py	Rust ignores --api-server-count and runs one server, so this only checks the flag is accepted
Serve/Admin	rl/return_routed_experts/test_scale_out_api.py
Tool Use	chat_completion/test_completion_with_function_calling.py -k "not test_no_args_tool_call"	relaxed: a tool-call-only message has content: null in Rust and "" in Python. The empty-arguments history case is a Rust bug (500); the {} case passes but shares the name
Tool Use	tool_parsers/test_granite4_tool_parser.py
Tool Use	tool_parsers/test_hermes_tool_parser.py -k granite4	the llama cases use --lora-modules

Tests that need a pending Rust fix (the LoRA request field, the gpt-oss parser selection, a single structured-outputs backend, ORCA headers, and others) are tracked in the parity ledger and land in a later run.

Still excluded, each with a comment (Rust fixes tracked in the parity ledger):

• test_kv_transfer_prompt_token_ids_*: Rust re-renders messages instead of reusing kv_transfer_params["prompt_token_ids"].
• completion/test_prompt_validation.py (stays commented out, per review): test_empty_prompt gets a body-deserialization 400 for prompt: null instead of Python's "Either prompt or prompt_embeds must be provided and non-empty." The Rust fix (completions.prompt_missing_message) comes first; the test then runs unrelaxed.
• test_chat_logit_bias_multiple_non_integer_keys: enabled together with its fix in [Rust Frontend] Report every invalid logit_bias key like Python vllm-project/vllm#63313 ([Rust Frontend] Report every invalid logit_bias key like Python #32); this PR leaves that line alone.
• test_serving_tokens.py lora: the Rust LoraRequest still encodes tensorizer_config_dict, removed by [LoRA] Remove tensorizer vllm-project/vllm#60024, so --lora-modules fails at startup; the fix is [Bugfix][Rust Frontend] LoraRequest: drop tensorizer_config_dict to match the Python wire layout vllm-project/vllm#63295.
• test_metrics.py test_metrics_exist: vllm:admission_rejections (open [Rust Frontend] Support --max-num-queued-reqs and --max-num-queued-tokens vllm-project/vllm#61296) and image inputs.
• test_tokenization.py tokenizer_info: Rust returns 501 (the old comment said no-op).
• tool_use/mistral: no mistral tokenizer mode or renderer in Rust.
• test_shutdown_on_engine_failure: launches the Python API server entry directly.
• test_*_lb_dp.py server_info: /server_info reports the Rust frontend's own config, which has no parallel_config; the api-server-count=4 variants also expect four API processes.

Step timeouts, NVIDIA (AMD): OpenAI Coverage 30→50 (35→55), Serve/Admin 25→45 (30→50), Tool Use 25→35 (30→40). Locally, step 1 added about 10 minutes each to OpenAI Coverage and Serve/Admin (test_metrics.py alone took 6 and test_basic.py with server_load 4), and step 2 adds about 10, 7 and 9 minutes to the three steps.

Validation

Local, GB200, VLLM_USE_RUST_FRONTEND=1, vllm-rs built from this base with VLLM_RS_BUILD_VERSION set to the vLLM version; --gpu-memory-utilization capped through a pytest plugin (another job shares the GPUs), no test-file change. Each changed command, exactly as in the YAML:

benchmarks/test_serve_cli.py                      # gated model; substitute run: 3 passed (Rust and Python)
test_chat_completion.py -k "not test_kv_transfer_prompt_token_ids_round_trip and not ..._streaming"   # 6 passed, 2 deselected
launchers/test_shutdown.py -k "not engine_failure"                                                    # 7 passed, 1 deselected
test_return_token_ids.py                                                                              # 6 passed
instrumentator/test_basic.py                                                                          # 6 passed
token_in_token_out/test_serving_tokens.py -k "not lora"                                               # 10 passed, 16 skipped, 1 deselected
instrumentator/test_metrics.py -k "not run_batch and not test_metrics_exist"                          # 9 passed, 3 skipped, 7 deselected
tool_use --ignore=tool_use/mistral --models llama3.2                                                  # gated model; mirror run: 365 passed, 84 skipped

Step 2, under Rust, with memory capped to an 18 GB slice (35 GB for Tool Use):

chat_completion/test_chat_echo.py                                                     # 4 passed
chat_completion/test_logprob_token_ids.py                                             # 9 passed
chat_completion/test_thinking_token_budget.py -k "not qwen35"                         # 6 passed, 1 deselected
completion/test_tensorizer_entrypoint.py --timeout=900 --timeout-method=thread        # 1 passed
test_chunked_prompt.py                                                                # 2 passed
serve/middleware/test_optional_middleware.py -k "not test_v2_..."                     # 8 passed, 1 deselected
launchers/api_server/test_multi_api_servers.py                                        # 2 passed
rl/return_routed_experts/test_scale_out_api.py                                        # 1 passed
chat_completion/test_completion_with_function_calling.py -k "not test_no_args_tool_call"  # 28 passed, 2 deselected
tool_parsers/test_granite4_tool_parser.py                                             # 1 passed
tool_parsers/test_hermes_tool_parser.py -k granite4                                   # 4 passed, 4 deselected

Every relaxed test also passes under the Python frontend.

Distributed step, on a separate clean 4-GPU node, the previously excluded cases only (-k "4 or server_info"):

TP_SIZE=1 DP_SIZE=4 v1/distributed/test_internal_lb_dp.py   # Rust: 4 passed, 2 failed (server_info); Python: 6 passed
TP_SIZE=1 DP_SIZE=2 v1/distributed/test_external_lb_dp.py   # Rust: 2 passed, 3 failed (server_info); Python: 5 passed
TP_SIZE=1 DP_SIZE=4 v1/distributed/test_hybrid_lb_dp.py     # Rust: 2 passed, 2 failed (server_info); Python: 4 passed
``` `ruff check`/`ruff format --check` (0.14.0), `check_buildkite_step_keys.py` and `check_test_tethering.py` pass.

## Queued test work (daily run 2026-10-22)

Three test-side items from the coverage ledger. All runs on a GB200 node through Slurm (`--exclusive`), from `tests/`, `VLLM_WORKER_MULTIPROC_METHOD=spawn`, Rust runs with `VLLM_USE_RUST_FRONTEND=1` and a `vllm-rs` built from this branch.

| Commit | Change | Rust | Python |
|---|---|---|---|
| d3892a9038 | `test_completion.py`: `facebook/opt-125m` → `Qwen/Qwen3-0.6B` (prompt token counts 6/11 → 5/10); two relaxations; added to OpenAI Coverage with `-k "not test_single_completion and not test_prompt_logprobs_completion and not test_parallel and not test_batch_completions"` | CI selection 17 passed, 8 deselected; full file 18 passed, 7 failed (the excluded ones) | full file 25 passed |
| 2eec280b5c | `test_lora_adapters.py`: `test_multiple_lora_adapters` and `test_beam_search_with_lora_adapters` await their tasks with `asyncio.gather` (they could not fail); the beam test gets its own adapter names, since it collided with the first test's adapters on the shared server once it could fail. Not added to Rust CI (needs the `LoraRequest` fix, vllm-project/vllm#63295) | red-green with the #29 binary: old file 2 passed, new file 2 failed (batched prompts, a known gap) | full file 13 passed |
| 7c0dae1a14 | `test_custom_online.py` serves `Qwen/Qwen3-0.6B` (per review, as `test_completion.py`); added to OpenAI Coverage with `-k "not test_invalid_custom_logitsproc_arg"`. The offline tests keep opt-125m | CI selection 2 passed; `test_invalid_custom_logitsproc_arg` fails (`chat+completions.custom_logitsprocs_arg_validation`) | file 4 passed |

Relaxations in `test_completion.py` (both behaviors acceptable, runbook §2 rule 5):

- `test_completion_streaming`: the last chunk may carry only the `finish_reason`; the joined streamed text is still compared with the non-streaming output (same relaxation as `test_chunked_prompt.py`).
- `test_completion_stream_options`: with `stream=false`, `{"include_usage": null}` and `{"continuous_usage_stats": null}` may be accepted as unset (Rust) or rejected (Python); the non-null cases still require a 400.

Why the offline logits-processor tests keep opt-125m: with SmolLM2-135M (bf16; Instruct and base), `test_custom_logitsprocs` failed in 2 of 3 runs of the file because a request that does not use the processor flipped a near-tied greedy token between the two `LLM` instances. Only the online test reaches the Rust frontend.

## Review follow-up (2026-10-23)

- Dropped the `test_prompt_validation.py` commit (rebased); the command stays commented out as on `main`, and `completions.prompt_missing_message` is queued as the fix.
- `test_custom_online.py` uses `Qwen/Qwen3-0.6B` instead of SmolLM2-135M (Slurm job 38739: Rust 2 passed, Python 4 passed).
- The comment above the `test_chat_logit_bias_validation.py` line is gone, since vllm-project/vllm#63313 drops that exclusion.

Standing note: this PR only enables tests; it changes no behavior.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01XTSySQLs9bSuTKD9ScKhDb

[BugenZhao]

I think Py's error reporting is better. Remove this entry from the PR and queue it to fix later

[richard-for-bugen]

Removed: the test_prompt_validation.py commit is dropped from this PR (rebased, now 7c0dae1), so the command stays commented out as on main and the test keeps Python's message unrelaxed. completions.prompt_missing_message stays queued as the fix: once Rust returns Python's "Either prompt or prompt_embeds must be provided and non-empty." for prompt: null, the test runs unchanged.

[BugenZhao]

Can we also use Qwen3 0.6B?

[richard-for-bugen]

Yes, switched to Qwen/Qwen3-0.6B (folded into the same commit). On one GB200 GPU each (Slurm job 38739): Rust -k "not test_invalid_custom_logitsproc_arg" 2 passed, Python the whole file 4 passed. The target tokens 128 and 67 are ordinary byte-level tokens in Qwen3's vocabulary, not EOS, so the assertion keeps its meaning.

[richard-for-bugen]

Merged upstream as vllm-project#63320.