[JulienDarve]

Overview

The Rust frontend's gRPC health service reports the same status for every service name, so a Kubernetes probe or a sidecar cannot tell an engine that is paused for an RL weight update from one that must be restarted. This adds liveness and readiness health services and leaves the existing names unchanged. Motivation: ai-dynamo/dynamo#14897.

Claims

• New liveness service: SERVING unless the engine has failed. Pause, sleep, weight updates, and orderly shutdown do not affect it until the listener closes; a prior engine failure keeps it NOT_SERVING.
• New readiness service: NOT_SERVING while a pause, sleep (including a partial wake-up), or weight update sent through this frontend's HTTP or gRPC control APIs is in effect, and on engine failure or shutdown.
• On shutdown, NOT_SERVING is published before the gRPC listener stops accepting connections.
• vllm.Inference, vllm.Control, and "" behave as before. No protobuf changes.

Engine state	liveness (new)	readiness (new)	vllm.Inference, vllm.Control, "" (unchanged)
Serving	SERVING	SERVING	SERVING
Paused, asleep, partially awake, or weight update open	SERVING	NOT_SERVING	SERVING
Engine failed	NOT_SERVING	NOT_SERVING	NOT_SERVING
Orderly shutdown	SERVING until the listener closes	NOT_SERVING, published before the listener closes	NOT_SERVING, published before the listener closes

Validation

Unit tests (run from rust/): cargo nextest run -p vllm-server -p vllm-engine-core-client -p vllm-mock-engine: 662 passed. cargo clippy -p vllm-server -p vllm-engine-core-client --tests -- -D warnings, cargo fmt --all -- --check, and pre-commit on the changed files passed. CI's existing cargo nextest run --workspace step runs the new tests.

New tests use the scripted engine in vllm_engine_core_client::mock_engine. Each fails when its behavior is reverted in the implementation:

Test	Protects
grpc_pause_withdraws_readiness_only	Pause makes readiness NOT_SERVING while liveness, vllm.Inference, vllm.Control, and "" stay SERVING; resume restores it
grpc_shutdown_closes_health_watch_and_server (extended)	Orderly shutdown fails readiness and closes the liveness watch without a NOT_SERVING update
grpc_health_transitions_to_not_serving_when_engine_becomes_unhealthy (extended)	Engine failure fails liveness and readiness too
availability_returns_only_after_full_wake_up	A partial wake-up keeps the engines unavailable
cancelled_resume_confirms_availability_with_engines	A cancelled call still ends with the engines' confirmed state
resume_keeps_available_engines_available	An unneeded resume does not flap readiness
overlapping_pause_and_resume_confirm_with_engines	Overlapping calls are confirmed with the engines instead of using the last result
weight_update_availability_follows_the_worker_session	Readiness stays down through an open weight update, a full wake-up, and a failed finish, and returns after a successful finish or a failed load

Live run on 1x RTX 6000 Ada with Qwen/Qwen3-0.6B, using this branch's Rust binary (debug build) and its Python package (editable install with precompiled kernels):

VLLM_USE_FLASHINFER_SAMPLER=0 vllm-rs serve Qwen/Qwen3-0.6B --host 127.0.0.1 --port 8000 --grpc-port 50051 \
  -- --enable-sleep-mode --gpu-memory-utilization 0.5

VLLM_USE_FLASHINFER_SAMPLER=0 works around the test machine's CUDA 11.5 nvcc, which cannot JIT-compile FlashInfer's sampler. A probe script polled grpc.health.v1.Health/Check every 0.5 s on a new connection per check, as kubelet does, kept one Watch per service open, and drove vllm.Control and /v1/completions:

Step	readiness	liveness, vllm.Inference, vllm.Control, ""
After startup, then a completion	SERVING	SERVING
PauseGeneration in abort, wait, and keep modes	NOT_SERVING	SERVING
ResumeGeneration after each, then a completion	SERVING	SERVING
Sleep(level=1), then WakeUp(tags=["weights"])	NOT_SERVING	SERVING (liveness and vllm.Inference checked)
Full WakeUp, then a completion	SERVING	SERVING
keep pause held 180 s, checked every 10 s (18 checks)	NOT_SERVING throughout	liveness SERVING throughout
Resume after the long pause	SERVING, same instance_id, completion succeeds	SERVING
SIGTERM	Watch receives NOT_SERVING, then closes	liveness watch closes without NOT_SERVING; vllm.Inference watch receives NOT_SERVING; new connections are refused 0.4 s later

All 27 checks passed on this revision. The kill -9 check ran on the previous revision, and this revision does not change the engine-failure path: killing the EngineCore process made the readiness watch receive NOT_SERVING and close, and new connections were refused within 50 ms. The managed-engine supervisor shut the server down before the engine client reported the failure, so probes saw the listener close rather than a liveness NOT_SERVING status; the unit test above covers the latter.

Not validated: real weight-update sessions (they need a trainer; covered by the scripted test), worker failure with TP > 1 (one GPU available), the Python-supervised vllm serve --grpc-port path, and Kubernetes probes in a cluster. No model evaluation: the change does not affect generated tokens.

Details

How readiness is derived. Every pause, sleep, resume, wake-up, and weight-update call, whether from HTTP routes or gRPC Control, goes through EngineCoreClient. The client now keeps an availability watch next to its existing health watch:

• pause_generation and sleep make the engines unavailable before the call is sent; their success means the engines are paused.
• wake_up reports whether every engine is fully awake, and the engine resumes scheduling exactly then (EngineCore.wake_up), so its result decides availability. A partial wake-up keeps the engines unavailable.
• When a result does not decide the state (after resume_scheduler, a failed or cancelled call, or calls that overlapped), a background task asks every engine is_sleeping, which covers both a paused scheduler and sleeping executor memory. Until it answers, the published value is unchanged, so an unneeded resume does not flap readiness. An epoch discards answers that span a later call.
• A weight update is open from start_weight_update until a successful finish_weight_update. A failed start or update_weights closes it, matching the workers, which abandon the update in those cases; a failed finish leaves it open.

The gRPC server maps that watch to readiness. monitor_health cancels a new withdrawn token after publishing shutdown statuses, and the listener stops accepting on that token instead of the shutdown token.

Compatibility. No wire change, and no existing health name changes behavior: vllm.Inference and vllm.Control keep meaning that their API is up, and during a pause that API still answers, with a retryable rejection or a queued request depending on the pause mode. Readiness probes and clients opt in by naming readiness. Older builds do not register liveness or readiness and answer NOT_FOUND, so probe configurations should switch with the image.

Limitations.

• Pauses that bypass these client methods (EngineCoreClient::call_utility, the dev-mode /collective_rpc route) are not observed. The Rust frontend is the only API server for its engines, so other frontends cannot pause them.
• A non-data-parallel engine accepts resume_scheduler while a wait-mode pause or sleep is still draining. If that pause's caller was cancelled, the late sleep can still apply after readiness is restored. This is an engine-side ordering issue; data-parallel engines reject the resume.
• A failed is_sleeping confirmation keeps the engines unavailable until the next pause, sleep, resume, or wake-up call.
• The lib.rs shutdown ordering is not covered by a unit test (the gRPC test harness builds its own server); the live run below checks it.
• #60505 adds an engine-side readiness broadcast that the Rust client decodes and ignores. If it lands, the is_sleeping confirmation can switch to that broadcast.

Related work and duplicates. Checked open and merged PRs and issues on 2026-10-22 and 2026-10-23. No PR splits Rust gRPC liveness from readiness. #60505 and #59179 add Python HTTP readiness endpoints. #56886 (draft) adds a gRPC shutdown grace period without separating liveness; its grace period would sit between publishing NOT_SERVING and cancelling withdrawn. #56364 (draft RFC) splits vllm.Control into several services.

AI assistance (Claude) was used to write and review this change.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code