Repository navigation
[Rust Frontend] Run utility calls to completion when the caller is cancelled - #64265
Merged
BugenZhao merged 3 commits intoNov 2, 2026
Merged
Conversation
Move `call_utility`'s fan-out unchanged from `EngineCoreClient` to `ClientInner`, which now keeps the engine IDs and the client index; `EngineCoreClient::call_utility` delegates. The next commit runs the fan-out in a task that owns the inner client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CJckCCUEjbFWgsUABtnUGD Signed-off-by: Bugen Zhao <i@bugenzhao.com>
…ncelled Dropping a `call_utility` future unregistered its waiters, but requests already handed to the send tasks on the client runtime were still sent, and the engines executed them. Cancellation only abandoned the bookkeeping, so callers could not tie state to a call's completion. Prepare and encode the requests in the caller, then send them and collect every reply in one task on the client runtime; the caller awaits that task and decodes the replies. The task sends directly instead of through `send_to_engine`, which would spawn again per request. Dropping the future now only stops waiting. This removes `UtilityCallGuard`; the task unregisters the waiters of requests that failed to send. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CJckCCUEjbFWgsUABtnUGD Signed-off-by: Bugen Zhao <i@bugenzhao.com>
BugenZhao
force-pushed
the
bz/rust-uncancellable-utility
branch
from
October 28, 2026 11:29
ecab322 to
9c78e6b
Compare
BugenZhao
added a commit
to JulienDarve/vllm
that referenced
this pull request
Oct 28, 2026
Local rebase of vllm-project#63870 onto vllm-project#64265: the PR's net change as of 758a88f, without its two workarounds for cancelled control calls. The utility call fan-out on `ClientInner` now comes from vllm-project#64265. Co-authored-by: Bugen Zhao <i@bugenzhao.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CJckCCUEjbFWgsUABtnUGD Signed-off-by: Julien Darve <jdarve@NVIDIA.com> Signed-off-by: Bugen Zhao <i@bugenzhao.com>
BugenZhao
added a commit
to JulienDarve/vllm
that referenced
this pull request
Oct 28, 2026
Pass a control call's `RecheckGuard` into `ClientInner::call_utility`, whose task (vllm-project#64265) holds it until the engines reply. A caller cancelled after sending a pause, for example by a client disconnecting from `/pause`, then keeps readiness withdrawn until the engines have handled it instead of rechecking at once, which could overtake the pause and publish ready after it took effect. The guard now owns an `Arc<Readiness>`, and `start_weight_update` passes its withdrawal into the `collective_rpc` it runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CJckCCUEjbFWgsUABtnUGD Signed-off-by: Bugen Zhao <i@bugenzhao.com>
6 tasks done
BugenZhao
marked this pull request as ready for review
October 28, 2026 11:51
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Isotr0py
approved these changes
Nov 2, 2026
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #99146 for commit ℹ️ This PR is 312 commits behind upstream |
BugenZhao
added a commit
to JulienDarve/vllm
that referenced
this pull request
Nov 2, 2026
Resolve conflicts after vllm-project#64265 was squash-merged to main: - gpu_worker.py: keep main's finish_weight_update (try/finally around the engine finish, _end_weight_update, KV cache loan release, direct reload handling) and clear the incomplete-weight mark only after _end_weight_update(refresh=True) returns, so a failed finish or a failed draft head restore still reports incomplete weights. - Test helpers: initialize both _incomplete_weight_updates and main's KV cache loan fields. - Rust engine-core-client: keep the readiness guard parameter on call_utility and is_inference_ready on top of the squashed vllm-project#64265; take main's test_utils import list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CJckCCukQgyfESDciCFndy Signed-off-by: Bugen Zhao <i@bugenzhao.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
EngineCoreClient::call_utilityfans a utility request out to every engine and waits for the replies. Dropping its future ranUtilityCallGuard, which unregistered the waiters, but since #46051 each request is sent by a task on the client's ZMQ runtime, so requests already handed off were still sent and the engines executed them. Cancellation only abandoned the client-side bookkeeping, and the replies arrived with no waiter.So a caller cannot tie state to a call's completion: anything it holds is released when its future is dropped, which can be before the engines see the request. #63870 needs exactly that: readiness withdrawn for a pause must stay withdrawn until the engines have handled it, even if the HTTP or gRPC client that sent
/pausedisconnects.This PR makes a utility call run to completion once its requests are prepared. The first commit moves the fan-out unchanged from
EngineCoreClienttoClientInner, which owns the socket; the second changes it:send_to_engine. The task unregisters the waiters of requests that failed to send.'static.Dropping the returned future now only stops waiting.
UtilityCallGuardis removed. A call that fails while preparing still unregisters the waiters it allocated. If the client runtime shuts down during a call, the caller gets aClientClosederror.Not a duplicate: no open PR changes Rust utility call cancellation (searched
rust utility call cancelandcall_utility).Test Plan
cancelled_utility_call_unregisters_waiterbecomescancelled_utility_call_still_collects_the_reply: after the caller is aborted, the waiter stays until the engine replies, and the reply then resolves it. It fails onmain, where the abort unregisters the waiter.cargo nextest run --workspaceinrust/, pluscargo clippy --workspace --tests --examples -- -D warningsandcargo fmt --check.Test Result
cargo nextest run --workspace: 2376 passed, 13 skipped.No model evaluation: generated tokens are unaffected.
AI assistance (Claude) was used for this change.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CJckCCUEjbFWgsUABtnUGD