[BugenZhao]

Overview

Make the Rust frontend's token-level generation layer an interface, so TextLlm, ChatLlm, and the server can run over implementations that are not backed by a local engine-core client, such as an HTTP client to a tokens-in/tokens-out gateway (#63887, llm-d/llm-d-router#3617) or a remote Rust frontend.

Claims

• New vllm_llm::Llm trait: info(), subscribe_health, subscribe_readiness, generate, abort, and shutdown. A lowered GenerateRequest goes in, raw GenerateOutputs come out.
• Health is two signals, as on the engine-core client: subscribe_health turns false only on a permanent failure (liveness), and subscribe_readiness follows whether new requests can be taken now, such as not while engines are paused or asleep. /health and the gRPC health services ([Rust Frontend][gRPC] Separate liveness from inference readiness #63870) read them from the Llm.
• The existing Llm struct is renamed EngineCoreLlm (first commit, pure rename) and implements the trait.
• TextLlm holds a Box<dyn Llm>; ChatLlm and the server read engine facts from LlmInfo instead of the engine-core client.
• New vllm_llm::Error variants (Unsupported, Unavailable, RemoteInvalidRequest, Remote) for implementations not backed by engine-core. vllm-text classifies Unsupported and RemoteInvalidRequest as request validation errors.
• No behavior change: the server still always runs an EngineCoreLlm.

Validation

In rust/, on each commit:

• cargo clippy --workspace --all-targets --all-features and cargo fmt --all -- --check: clean.
• cargo nextest run -p vllm-llm -p vllm-text -p vllm-chat -p vllm-server -p vllm-cmd -p vllm-bench: 1615 passed, 13 skipped, on main 5c97556318. The other 3 (vllm-chat roundtrip and grammar replay) download real Hugging Face models and fail locally only because the shared model cache is not readable on this machine; this PR does not touch them.

This is a refactor of internal Rust APIs, so it adds no tests; generated tokens are unaffected and no model evaluation was run.

Details

• Why this layer. Detokenization, which attributes text to tokens, and stop strings stay with the caller, so the parsers keep their full input wherever the engine runs. The trait mirrors the generate/abort subset of Python's EngineClient.
• Naming. In the Rust frontend, "engine" means engine-core (EngineCoreClient, vllm-managed-engine), so the interface takes the layer's name, Llm, and the engine-core implementation becomes EngineCoreLlm.
• No control methods on the trait. Llm requires Any; the server downcasts to EngineCoreLlm for the admin and utility routes that drive the engine-core client. A separate control trait can come once a remote implementation needs those operations.
• Relation to [Rust Frontend] engine client interface #63887. Same goal. [Rust Frontend] engine client interface #63887 currently places an EngineClient trait beneath Llm that mirrors the full EngineCoreClient API (about 45 methods, including the utility calls), which every implementation, including an HTTP client, would have to provide. This PR abstracts the Llm layer itself, as [Rust Frontend] engine client interface #63887 originally proposed, and is co-authored with its author; the HTTP client can build on it.
• Follow-up. Let the server run over an Llm without an engine-core client: register engine-core routes only for EngineCoreLlm, and map Unavailable to 503.

AI assistance (Claude) was used for this change.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CJckCBxv8Gtnq23avKQwau

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[sagearc]

towards abstracting tokens-in/tokens-out, this is clean!
rust/src/llm/src/lib.rs is basically all we need

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @BugenZhao.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #100109 for commit 2aaa88428e94.

ℹ️ This PR is 75 commits behind upstream main. Merged upstream main into this branch as 2aaa88428e94 before starting CI. Pull the branch before pushing more changes.