[JulienDarve]

Update: stacked on #64265, readiness reported by the engines

Stacked on #64265 ([Rust Frontend] Run utility calls to completion when the caller is cancelled). The base branch bz/rust-uncancellable-utility mirrors #64265's head; this PR will be retargeted to main once #64265 merges. Its three commits:

• f2ed549 (Julien): this PR's net change, squashed from the previous history (last head 21feaf4, which also merged main).
• c7c67d8: holds the readiness guard until the engines reply.
• 5ac1ff2: withdraws readiness before the gRPC listener stops, including when the server is aborted.

This revision keeps the gRPC service mapping and changes how the frontend learns the engine state. The original revision inferred it in the client from the results of the control calls it sent (client/availability.rs: per-call facts, resolve, epochs, and a confirm loop asking is_sleeping plus a worker RPC). EngineCore already owns that state, so it now reports it:

• EngineCore.is_inference_ready() (Python, 8 lines): not paused (including for a checkpoint) or asleep, and no worker reports has_incomplete_weight_update().
• The client (client/readiness.rs) withdraws readiness before a call that stops serving is sent (pause, sleep, checkpoint prepare, start weight update), and rechecks after every control call ends by asking the engines is_inference_ready. An epoch discards answers that predate a call, so readiness becomes SERVING only from an engine answer. The public collective_rpc also rechecks, which covers weight updates and the dev-mode /collective_rpc route.
• A control call passes its readiness guard to call_utility, whose task (from [Rust Frontend] Run utility calls to completion when the caller is cancelled #64265) holds it until the engines reply. A caller cancelled after sending a pause, for example by a client disconnecting from /pause, therefore cannot recheck before the engines have handled the pause.
• The client asks only while someone subscribes (the gRPC server), so HTTP-only deployments add no worker RPC after control calls.
• With sleep mode, the client asks once at connect: only such engines can start prepared for a checkpoint (--checkpoint-prepare-on-startup), and an engine without the utility fails startup with an error naming it.
• gRPC wiring: EngineHealth groups the client's liveness and readiness watches, and HealthMonitor registers the services before the listener accepts connections and then follows the watches.
• Shutdown order: HealthMonitor::serve runs the gRPC server, and both the signal that stops its listener and the forced abort (immediate with the default --shutdown-timeout 0) wait until every service except liveness reports NOT_SERVING. A Health/Watch subscriber therefore receives the withdrawal before its connection closes. A Check on a new connection after shutdown starts gets NOT_SERVING, an unknown service, or a refused connection; probes treat all three as not ready. lib.rs and the gRPC test server share this wiring.
• finish_weight_update clears the worker's incomplete-update flag only after the draft model's quantized LM heads are restored, so a failed restore keeps readiness withdrawn.

The service mapping is the same as in the original revision:

Service	Reports NOT_SERVING when
liveness	The engine has failed.
readiness and vllm.Inference	The engines report not ready, a call that stops serving is in flight, the engine has failed, or shutdown has started.
vllm.Control and ""	The engine has failed or shutdown has started.

readiness always equals vllm.Inference; both are kept. Kubernetes probes and the Dynamo sidecar use liveness and readiness (ai-dynamo/dynamo#14897), while vllm.Inference and vllm.Control remain the per-service statuses.

Validation of this revision:

• cargo nextest run -p vllm-server -p vllm-engine-core-client -p vllm-mock-engine: 701 passed. cargo clippy --workspace --tests --examples -- -D warnings, cargo fmt --check, and pre-commit passed.
• New tests: unit tests for the readiness state (an answer that predates a call is discarded, an in-flight withdrawal hides answers, no asks without subscribers); scripted-engine client tests (pause and resume, weight update, starting prepared with sleep mode, and a pause whose caller is cancelled must not be rechecked before the engine replies); gRPC tests (a pause withdraws only readiness and vllm.Inference, orderly shutdown closes the liveness watch without NOT_SERVING, engine failure fails all five services); a worker test where a failed draft finish keeps the weights incomplete; HealthMonitor tests where the listener stop and a forced abort both find readiness already withdrawn (300 consecutive runs). Each fails when its fix is reverted.
• pytest tests/v1/worker/test_gpu_worker_weight_transfer.py: passes; one existing test needs a visible GPU and was not run.
• Live GPU runs: the table below replays the original revision's table on c7c67d8. On 5ac1ff2, three SIGTERM runs repeated the SIGTERM row: every watch except liveness received NOT_SERVING within 3 ms and then closed.

Live runs of c7c67d8 on one GB200 GPU with Qwen/Qwen3-0.6B, --enable-sleep-mode --weight-transfer-config '{"backend":"ipc"}', and VLLM_SERVER_DEV_MODE=1. Control steps use the vllm.Control RPCs, except the HTTP checkpoint and draft routes. Each check opens a new connection, as kubelet does, and Health/Watch streams on the services record every transition.

Step	readiness, vllm.Inference	liveness, vllm.Control, ""
After startup, then a completion	SERVING	SERVING
PauseGeneration in abort, wait, and keep modes	NOT_SERVING	SERVING
ResumeGeneration after each, then a completion	SERVING	SERVING
Sleep(level=1), then WakeUp(tags=["weights"])	NOT_SERVING	SERVING
Full WakeUp, then a completion	SERVING	SERVING
keep pause held 180 s, checked every 10 s	NOT_SERVING throughout, no transition on the watch	liveness SERVING throughout
Resume after the long pause	SERVING, same instance_id	SERVING
POST /start_draft_weight_update without a draft model (the workers refuse), then a completion	NOT_SERVING for 6 ms, then SERVING	SERVING
StartWeightUpdate	NOT_SERVING	SERVING
FinishWeightUpdate, then a completion	SERVING	SERVING
StartWeightUpdate, then UpdateWeights with update_info_json set to {} (fails)	NOT_SERVING	SERVING
FinishWeightUpdate after the failed load (fails)	NOT_SERVING, no SERVING on the watch since the start	SERVING
New StartWeightUpdate and FinishWeightUpdate, then a completion	SERVING	SERVING
POST /checkpoint/prepare, until /checkpoint/status returns 200	NOT_SERVING	SERVING
POST /checkpoint/restore, then a completion	SERVING	SERVING
Started with --checkpoint-prepare-on-startup	NOT_SERVING	SERVING
POST /checkpoint/restore, then a completion	SERVING	SERVING
PauseGeneration(wait) during a 3000-token request, with a 0.5 s deadline that cancels the call; checked after the 6 s drain	NOT_SERVING, no SERVING on the watch; IsPaused true	SERVING
ResumeGeneration, then a completion	SERVING	SERVING
Dev-mode POST /collective_rpc with start_weight_update	NOT_SERVING	SERVING
POST /collective_rpc with finish_weight_update, then a completion	SERVING	SERVING
SIGTERM	Watch receives NOT_SERVING, then closes	liveness watch closes without NOT_SERVING; listener closed 4 ms later
kill -9 of the EngineCore process	Watch receives NOT_SERVING, then closes	liveness watch closes without NOT_SERVING; vllm-rs exits with an error

• The cancelled pause is a smoke test: the pause reaches the engine long before the deadline, so it also passes without [Rust Frontend] Run utility calls to completion when the caller is cancelled #64265. The race it guards against, a recheck overtaking the pause, is covered by cancelled_pause_rechecks_only_after_the_engine_replies.
• kill -9: the engine failure path sets liveness last, and the server aborts the gRPC listener at the same time, so the liveness watch closed before receiving NOT_SERVING. This is the race described under the original revision's kill -9 row; probes treat a closed connection as a failure.

AI assistance (Claude) was used for this revision.

---

Original description (previous revision)

Overview

The Rust frontend's gRPC health service reports the same status for every service name, so Kubernetes and sidecars cannot tell an engine paused for an RL weight update from one that needs a restart. This adds two health services: liveness, which fails only when the engine fails, and readiness, which also fails while the engine is paused (including for a process checkpoint) or asleep, or while a weight update has not finished successfully. vllm.Inference now reports the same status as readiness; vllm.Control, "", and the protobufs are unchanged. Motivation: ai-dynamo/dynamo#14897.

Size: about 325 lines of non-test Rust and 8 lines of Python (+336/−34, not counting tests, comments, blank lines, reindentation, or the 39 lines moved unchanged onto ClientInner). The logic is four pieces:

• the engine-state tracker in client/availability.rs, about 170 lines:
  • two facts (every engine is scheduling; every worker's weights are complete) and the rule that readiness is SERVING when both allow serving, keeping its last value while one is being confirmed;
  • a guard each control call holds, which withdraws readiness before a pause or update start is sent and records the fact the call's result settles;
  • a background loop that runs while either fact is unknown (after a resume, a weight-update finish, a failed or cancelled call, or overlapping calls). It asks the engines is_sleeping and the workers has_incomplete_weight_update, asks again if another control call made the answer stale, and retries a failed attempt every second.
• the wiring in client.rs and client/imp.rs, about 90 lines:
  • each pause, sleep, resume, wake-up, weight-update and checkpoint method takes the guard;
  • ClientInner keeps the engine IDs and client index, so the loop can call the engines;
  • subscribe_availability, the loop's task, and one is_sleeping call at connect for sleep-mode engines, which may start prepared for a checkpoint.
• the gRPC mapping in grpc/health.rs and lib.rs, about 70 lines:
  • register liveness, and register readiness and vllm.Inference with the current availability;
  • drive readiness and vllm.Inference from the tracker;
  • on shutdown, publish NOT_SERVING before the listener stops accepting connections.
• the worker method has_incomplete_weight_update() in gpu_worker.py and worker_base.py, 8 lines: tracked per target (base or draft model), set once an update starts, and cleared by that target's successful finish. Because of the second target, a successful finish asks the workers rather than assuming the weights are complete.

Moving the existing call_utility and collective_rpc fan-out onto ClientInner changes no behavior. The rest of the +1119/−162 diff is:

• tests (+575/−63): 7 new tests (5 Rust, 2 Python), the two existing health tests and the existing full-wake-up test extended, gRPC test setup that passes the tracker, and the new worker attribute in hand-built test workers;
• docs (+43 in docs/deployment/k8s.md and rust/README.md);
• doc comments.

Claims

• liveness reports NOT_SERVING only after an engine failure. Pause, sleep, checkpoint prepare, weight updates, and orderly shutdown do not affect it until the listener closes.
• readiness and vllm.Inference report NOT_SERVING in these cases:
  • while a pause, sleep (including a partial wake-up), or process-checkpoint prepare sent through this frontend's HTTP or gRPC control APIs is in effect;
  • while an engine started with --checkpoint-prepare-on-startup is not yet restored;
  • from the start of a weight update until one finishes successfully, including after a failed load;
  • on engine failure or shutdown. On shutdown they are withdrawn before the listener stops accepting connections. With the default --shutdown-timeout 0, the gRPC server is aborted without a grace period, so a Watch can see its connection close before it receives NOT_SERVING; probes treat both as not ready.
• Kubernetes probes: startup and liveness use liveness, readiness uses readiness, as documented in docs/deployment/k8s.md. A liveness probe that names vllm.Inference now fails while the engine stays paused, so it should move to liveness. Likewise, a client that waits for vllm.Inference to be SERVING before it connects now waits while the engine is paused. Dynamo's vLLM sidecar does this at startup (lib/sidecar/vllm/src/engine.rs). To connect to a paused engine, wait on vllm.Control, which reports SERVING while paused in every version; earlier versions do not have liveness.

Validation

Live runs on 1x RTX 6000 Ada with this branch, polling grpc.health.v1.Health/Check every 0.5 s on a new connection, as kubelet does:

VLLM_USE_FLASHINFER_SAMPLER=0 vllm-rs serve Qwen/Qwen3-0.6B --grpc-port 50051 -- --enable-sleep-mode

The weight-update rows add --weight-transfer-config '{"backend":"ipc"}' and VLLM_SERVER_DEV_MODE=1. The failed load sends UpdateWeights with update_info_json set to {}. The last checkpoint rows add --checkpoint-prepare-on-startup.

Step	readiness, vllm.Inference	liveness, vllm.Control, ""
After startup, then a completion	SERVING	SERVING
PauseGeneration in abort, wait, and keep modes	NOT_SERVING	SERVING
ResumeGeneration after each, then a completion	SERVING	SERVING
Sleep(level=1), then WakeUp(tags=["weights"])	NOT_SERVING	SERVING
Full WakeUp, then a completion	SERVING	SERVING
keep pause held 180 s, checked every 10 s	NOT_SERVING throughout	liveness SERVING throughout
Resume after the long pause	SERVING, same instance_id	SERVING
POST /start_draft_weight_update without a draft model (the workers refuse), then a completion	NOT_SERVING for 4 ms, then SERVING	SERVING
StartWeightUpdate	NOT_SERVING	SERVING
FinishWeightUpdate, then a completion	SERVING	SERVING
StartWeightUpdate, then a failed UpdateWeights	NOT_SERVING	SERVING
FinishWeightUpdate after the failed load (fails)	NOT_SERVING	SERVING
New StartWeightUpdate and FinishWeightUpdate, then a completion	SERVING	SERVING
POST /checkpoint/prepare, until /checkpoint/status returns 200	NOT_SERVING	SERVING
POST /checkpoint/restore, then a completion	SERVING	SERVING
Started with --checkpoint-prepare-on-startup	NOT_SERVING	SERVING
POST /checkpoint/restore, then a completion	SERVING	SERVING
SIGTERM	Watch receives NOT_SERVING, then closes	liveness watch closes without NOT_SERVING; listener closed by the next check, under 0.4 s later
kill -9 of the EngineCore process	Watch receives NOT_SERVING then closes, or the connection drops first	Listener closes; vllm-rs exits with an error

Provenance:

• All rows except the 180 s pause and kill -9 rows ran on 5073411. They also cover the head, d42905b, which changes only tests and docs.
• The 180 s pause rows ran on 3c2f1d1.
• With the worker change reverted, the refused draft start left readiness NOT_SERVING.
• The kill -9 row is timing-dependent on both sides of this change: the watch received NOT_SERVING before the connection dropped in 5 of 8 runs on a build of this head without its last two fixes (the finish confirmation and the repeated-status filter), and in 5 of 7 runs on 8ae979f. When the managed engine exits, the server shuts down with no grace period, so aborting the gRPC server races the health monitor's update.

VLLM_USE_FLASHINFER_SAMPLER=0 only works around the test machine's CUDA 11.5 nvcc.

Unit tests:

• cargo nextest run -p vllm-server -p vllm-engine-core-client -p vllm-mock-engine passed (686, with a short TMPDIR for the two Unix-socket listener tests), as did cargo clippy --tests -- -D warnings, cargo fmt --check, and pre-commit.
• The availability and health tests passed 20 consecutive runs.
• pytest tests/v1/worker/test_gpu_worker_weight_transfer.py passed (17), and pytest tests/v1/worker/test_gpu_draft_lm_head.py -k draft_weight_update passed on the GPU (3).
• The new tests use the scripted mock engine or a stub weight-transfer engine. Each fails when its fix is reverted: the re-ask after a stale answer, the retry, the refusal branch of checkpoint prepare, setting the worker flag before the draft-model check, a single flag shared by base and draft, and a finish that does not clear the flag.

Not run:

• a weight transfer from a real trainer;
• a draft-model weight update;
• worker failure with TP > 1;
• probes in a Kubernetes cluster.

No model evaluation: generated tokens are unaffected.

Details

The worker reports has_incomplete_weight_update rather than whether an update session is active because a failed update_weights ends the session without restoring the weights, so generation fails until a later update finishes.

Known limitations:

• Pauses that bypass these client methods, such as the dev-mode /collective_rpc route, are not observed.
• A full wake-up after a level-2 sleep reports SERVING before the weights are reloaded. The documented order (wake weights, reload, then wake kv_cache) keeps readiness down until the weights are back.
• A failed process checkpoint leaves the engine paused, with no way to restore. readiness stays NOT_SERVING and liveness stays SERVING, so whatever drives the checkpoint must restart the pod when /checkpoint/status reports the failure.

The frontend infers engine state from the results of the calls it sends and asks the engines when a result is not conclusive. An engine-reported readiness state, as proposed in #60505, could replace these confirmations.

No open or merged PR splits Rust gRPC liveness from readiness (checked 2026-10-26). #60505 and #59179 add Python HTTP readiness endpoints; #56886 (draft) adds a gRPC shutdown grace period.

AI assistance (Claude) was used to write and review this change.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

Documentation preview: https://vllm--63870.org.readthedocs.build/en/63870/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @JulienDarve.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @JulienDarve.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[JulienDarve]

Thanks for taking this on. Having the engine report readiness, and holding the guard until the engines reply, reads well, and clearing the flag only after the draft LM head restore fixes a real bug.

On the open question: we'd like to keep both. liveness and readiness are the names our Kubernetes probes and the Dynamo sidecar will use (Dynamo's lifecycle DEP, ai-dynamo/dynamo#14897). vllm.Inference and vllm.Control stay as the per-service statuses the gRPC health spec recommends. With this PR, vllm.Inference reports something different from vllm.Control for the first time.

On shutdown: the DEP requires publishing inference NOT_SERVING before the listener closes during an orderly shutdown. With the listener now stopping on shutdown directly, that order isn't guaranteed. The earlier revision gated the graceful stop on the withdrawal but still let --shutdown-timeout 0 abort first. I'd restore it for both paths: HealthMonitor cancels a withdrawn token after it publishes, serve_with_incoming_shutdown uses that token, and the force-abort branch waits for it too. Could you fold that in, along with a test for the ordering?

[BugenZhao]

Thanks @JulienDarve !

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @JulienDarve.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #99166 for commit d03cbd1efe81.

[BugenZhao]

/ci run

[github-actions]

✅ CI is already running for this commit: https://buildkite.com/vllm/ci/builds/99166

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #99198 for commit 66dbdcd9677b.