[BugenZhao]

Upstream: vllm-project#64729 (stacked on vllm-project#64695).

Motivation

With the Llm trait (vllm-project#64695), a frontend can build TextLlm and ChatLlm over an Llm that is not backed by an engine-core client, but it cannot run the server over one:

• serve() always connects to engine-core and builds an EngineCoreLlm. A router frontend embedding the server has to re-assemble AppState, the router, listeners, TLS, gRPC, and graceful shutdown itself; a frontend-only vllm-rs mode (such as the HTTP generation client planned in [Rust Frontend] engine client interface vllm-project/vllm#63887) has no place to plug its Llm in.
• Admin, LoRA lifecycle, checkpoint, /kv_event_sources routes and the gRPC Control service call the engine-core client directly.
• A request that cannot be taken right now surfaces as a 500, so callers cannot tell that a retry may succeed.

Changes

1. Split engine-core settings out of the server config (pure refactor). The transport and coordinator modes, stream interval, stats logging, request ID randomization, engine lifecycle, static LoRA adapters, and profiler move to EngineCoreConfig; the rest is renamed ServerConfig. Starting the server goes through a Server builder instead of serve and serve_with_router_extension, with LlmSource naming where the server's Llm comes from:

   Server::new(config, LlmSource::EngineCore(engine_core))      // vllm-cmd
   Server::new(config, LlmSource::Provided(Box::new(routed)))   // a router frontend
       .router_extension(f)
       .serve(shutdown)
       .await

   vllm-cmd builds both halves as a FrontendConfig; /server_info still reports one flat rust_frontend_config.

2. Serve over a caller-built Llm.

   • LlmSource::Provided(llm) builds the state around the given Llm instead of connecting to engine-core; model backends, listeners, TLS, gRPC, and graceful shutdown are shared. TextLlm::from_boxed keeps the box as is, so the server still recognizes an EngineCoreLlm by downcasting.
   • A provided Llm is opaque. Routes and services that drive the engine-core client are only served over an EngineCoreLlm: dev-mode admin routes, runtime LoRA loading, checkpoints, /kv_event_sources, and gRPC Control. gRPC health follows the Llm and covers only the served services.
   • An EngineCoreLlm passed as Provided is rejected at startup. It would otherwise get the engine-core routes but none of the settings in EngineCoreConfig (server info, profiler, static LoRA, engine shutdown), and its request histograms would already be built with the default buckets, since the server applies --custom-histogram-buckets while building its state.
   • Unavailable maps to HTTP 503 and gRPC UNAVAILABLE.

AppState and build_router stay private, in line with vllm-project#43774.

Stack

1. [Rust Frontend] Generate through an Llm trait vllm-project/vllm#64695 (upstream draft): Llm trait and error contract
2. [Rust Frontend] Let the server run over any Llm #83: this PR
3. [Rust Frontend] Remote Llm over an internal MessagePack gRPC service #84: internal MessagePack gRPC remote Llm (not yet restacked onto this version)

Validation

• cargo clippy --workspace --all-targets --all-features, cargo fmt --check: clean.
• cargo nextest run -p vllm-llm -p vllm-text -p vllm-chat -p vllm-server -p vllm-cmd -p vllm-bench: 1505 passed. The vllm-cmd config snapshots change only by nesting fields under server (ServerConfig) and engine_core.
• New llm_without_engine_core_serves_generation_routes_only: over a fake Llm with no engine-core client and dev mode plus runtime LoRA enabled, engine-core routes return 404, /health follows the Llm, and an unavailable Llm gives 503. It fails without the route gating (the handler panics on the missing engine-core client) and without the mapping (500).
• New provided_engine_core_llm_is_rejected: an EngineCoreLlm is rejected as a provided Llm and other Llms pass; it fails when the check is removed.
• Fit check with a downstream router frontend, on a local restack of [Rust Frontend] Remote Llm over an internal MessagePack gRPC service #84: its custom AppState + build_router + axum::serve + drain loop became Server::new(config, LlmSource::Provided(..)).router_extension(..).serve(shutdown) (checked with the earlier Server::llm spelling of the same entry), and it no longer loads model backends itself. Against two vllm-rs serve workers over mock engines (DP=2 each): chat, stop strings, streaming usage, 40 concurrent requests, /kv_event_sources → 404 on the router, and SIGTERM draining an in-flight request to 200 all work.

Open questions

• Building a ServerConfig by hand needs all 30 fields; the router fit spelled out ~30 lines of defaults. A Default or a small constructor would help embedders, at the cost of a second set of defaults next to the CLI's.
• gRPC over a caller-built Llm (Inference and health only) is not exercised end to end yet.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CJckCBxv8Gtnq23avKQwau