[BugenZhao]

Motivation

A router frontend that keeps rendering, tokenization, detokenization, and parsing local needs to reach vllm-rs serve workers at the Llm boundary, without losing anything on the way. The existing public interfaces do not fit:

• Inference.Generate and /inference/v1/generate take user-facing sampling parameters, so a request the frontend has already lowered would have to be mapped back, losing fields such as the structured-output backend, and the worker would apply its own defaults again.
• Mirroring GenerateRequest/GenerateOutput in vllm-proto would turn engine-core internals (_all_stop_token_ids, _backend, ...) into a published schema that needs a release for every change.

Changes

• New crate vllm-remote-llm with RemoteLlm: Llm.
• vllm.internal.Llm: a gRPC service defined with tonic_build::manual, without a .proto, with a MessagePack codec that encodes fields by name, like the engine-core ZMQ wire. Generate is server-streaming; Info returns the server Llm's LlmInfo; Abort takes external request IDs.
• LlmInfo, GenerateRequest, GenerateOutput, and the types they carry derive serde, skipping None fields.
• vllm-server serves vllm.internal.Llm on the gRPC port next to Inference and Control; its health status follows readiness, like vllm.Inference.
• RemoteLlm uses only vllm.internal.Llm and health, so the server may run any Llm, not only an engine-core one (Control is served only with an engine core). It adopts the server's LlmInfo, including the model dtype, so multimodal inputs are prepared on the calling side; it aborts through Abort, which rejects an empty list, since aborting every request of a shared server is not one caller's to do; and it reports the server's vllm.internal.Llm status as its readiness. Its health stays true: a failing server makes it unready, not failed, since restarting the caller would not fix it.

The protocol is internal and unversioned. Adding an Option or #[serde(default)] field to a struct with named fields stays compatible between adjacent revisions. Engine-core types encoded as tuples, such as LoraRequest and the tensor types, are positional, so adding a field there does not, nor do renames, type changes, and new enum variants. The crate docs state this rule.

Stack

1. [Rust Frontend] Generate through an Llm trait vllm-project/vllm#64695 ([Rust Frontend] Generate through an Llm trait #82): Llm trait and error contract, merged
2. [Rust Frontend] Let the server run over any Llm vllm-project/vllm#64729 ([Rust Frontend] Let the server run over any Llm #83): the server over any Llm
3. [Rust Frontend] Remote Llm over an internal MessagePack gRPC service #84: internal MessagePack gRPC remote Llm (this PR)

Validation

• cargo clippy --workspace --all-targets --all-features, cargo fmt --check: clean.
• cargo nextest run on llm, text, chat, server, remote-llm, proto, engine-core-client, cmd, bench: 1802 passed, on main 9901ffcb4a. The other 4 (vllm-chat roundtrip and grammar replay) download real Hugging Face models and fail locally only because the shared model cache is not readable on this machine.
• Round-trip tests run the same request through an EngineCoreLlm and through a RemoteLlm against a served EngineCoreLlm, and compare the request the engine receives and the outputs the caller gets, covering stop strings, logprobs, and raw outputs, against a server that serves no Control. Another test checks that RemoteLlm adopts the server's LlmInfo and aborts by external ID; it fails if the server's Info drops the dtype or its Abort does nothing. On the original branch, dropping frequency_penalty or kv_transfer_params on the server side made them fail. Another test pauses the server and checks that the RemoteLlm turns unready and back while staying healthy; it fails if the server stops publishing vllm.internal.Llm readiness or if RemoteLlm reports readiness as health.
• Mock end to end, two vllm-rs serve workers (DP=2 each, mock engines) behind a downstream router frontend built on RemoteLlm. On this revision: the router adopted the workers' LlmInfo with the model dtype, and chat, 40 concurrent chats, and a router-side stop string all succeeded. On the previous revision (after the rebase onto 9901ffcb4a): after one worker was killed the router's /health stayed 200 and 20 concurrent requests all succeeded on the other worker, and a chat request with n: 2 got a 400 from the router, while the worker serves it directly.
• End to end on the original branch: a downstream router frontend driving 1P1D prefill/decode over NIXL produced output identical to calling the decode worker directly (short and 2.9k-token prompts, stop strings, logprobs), with 32 concurrent requests all succeeding.

Open questions

• Whether this belongs in-tree. The client could live downstream, but the worker side needs either this service in vllm-rs serve or an extension hook plus a custom worker binary.
• No revision check yet; connect has a TODO.
• RemoteLlm rejects n > 1 as unsupported. Carrying n would need the client to split one call into a stream per sample; a version that does is on bz/remote-llm-n, kept until something needs it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CJckCBxv8Gtnq23avKQwau