[BugenZhao]

Upstream: vllm-project#64695. This fork PR stays as the base of the stack below.

Motivation

TextLlm owns a concrete Llm, the token-level facade over the engine-core client, so the server, chat, and text layers can only run over engine-core. Frontends that put a network hop between the API layer and the engine, such as a router frontend in front of vllm-rs serve workers or a tokens-in/tokens-out gateway (vllm-project#63887), want to reuse rendering, tokenization, detokenization, and parsing while supplying their own generation backend.

This PR makes the Llm layer an interface: a lowered GenerateRequest goes in, raw GenerateOutputs come out. Detokenization (with per-token attribution) and stop strings stay with the caller, so the parsers keep their full input regardless of where the engine runs.

Changes

• Rename the existing struct Llm to EngineCoreLlm (first commit, pure rename), so Llm can name the interface. In the Rust frontend, "engine" means engine-core; naming the interface EngineClient after Python would read as a synonym of EngineCoreClient.
• vllm_llm::Llm: the generate/abort subset of Python's EngineClient, plus info(), health, and shutdown(). EngineCoreLlm implements it.
• LlmInfo: model and engine facts fixed at creation that the upper layers read (effective context length, model dtype, LoRA support, vLLM version). model_dtype is optional; without it, multimodal inputs are rejected.
• The trait requires Any and does not expose the engine-core client. The server's admin and utility routes downcast to EngineCoreLlm to reach it, so they keep working unchanged.
• vllm_llm::Error gains Unsupported, Unavailable, RemoteInvalidRequest, and Remote. The existing variants all describe engine-core failures, so an implementation backed by something else had nothing to return. vllm-text classifies the first and third as request validation errors and adds is_unavailable(); vllm-chat forwards it.
• TextLlm holds a Box<dyn Llm> and exposes it as llm(), replacing engine_core_client(); ChatLlm and the server read facts from LlmInfo. ChatLlm::engine_core_client() is removed.

No behavior changes: the server still always builds an EngineCoreLlm.

Stack

1. [Rust Frontend] Generate through an Llm trait #82: Llm trait and error contract (this PR)
2. [Rust Frontend] Let the server run over any Llm #83: HTTP API over any Llm
3. [Rust Frontend] Remote Llm over an internal MessagePack gRPC service #84: internal MessagePack gRPC remote Llm

Validation

• cargo clippy --workspace --all-targets --all-features, cargo fmt --check: clean, on both commits.
• cargo nextest run -p vllm-llm -p vllm-text -p vllm-chat -p vllm-server -p vllm-cmd -p vllm-bench: 1417 passed.

Open questions

• Error shape: by HTTP semantics there are three classes (invalid request, unavailable, other remote failure). Unsupported carries the field name for the OpenAI error param; it could fold into the invalid-request class.
• Implementing the trait still compiles the ZMQ engine-core client in. Splitting the protocol types from the transport would let implementors depend on types only.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CJckCBxv8Gtnq23avKQwau