Repository navigation
[Core] Cheapen the spin while waiting for model output - #65773
Merged
Merged
Conversation
vllm-project#65466 waits for a step's output by looping on future.poll(), which runs several Python calls with the GIL held before each sched_yield(). This delays the engine's input and output threads, and costs some throughput and tail latency, even when no inputs are handled during the wait. Between polls, spin directly on the response queue's flags and the input queue instead, so that an iteration costs about 1us rather than 4us. Inputs are still handled as they arrive, so requests still reach the scheduler about as early as with vllm-project#65466 (median queue wait about 25us, compared with 5-7ms before it). For a remote reader, move the sleep on its socket into this new wait, so that readable() is non-blocking again and a loop iteration sleeps only once. Qwen3-0.6B on GB200, MP executor, Rust frontend. Medians, before vllm-project#65466 -> vllm-project#65466 -> this change: TP=1, 8 repeats, arrivals (128/8 tokens, 256 concurrent): - 3,224 -> 3,133 -> 3,183 req/s; TTFT p99 58.8 -> 65.1 -> 59.7ms TP=4, 3 repeats: - decode (32/512, 256 concurrent): 81,670 -> 74,830 -> 81,500 tok/s - single request (32/256): 685 -> 590 -> 700 tok/s TP=4, 6 repeats, before vllm-project#65466 -> this change: - arrivals: 2,839 -> 2,869 req/s - Poisson at 2,014 req/s: TTFT p99 47.0 -> 43.1ms; ITL p99 10.7 -> 10.3ms These were measured before moving the remote reader's sleep, which doesn't affect them: they used only local readers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xgy3B6JERztsVgxhdteg6e Signed-off-by: Nick Hill <nickhill123@gmail.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #100075 for commit |
WoosukKwon
approved these changes
Nov 5, 2026
2 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#65466 waits for a step's output by looping on future.poll(), which runs several Python calls with the GIL held before each sched_yield(). This delays the engine's input and output threads, and costs some throughput and tail latency, even when no inputs are handled during the wait.
Between polls, spin directly on the response queue's flags and the input queue instead, so that an iteration costs about 1us rather than 4us. Inputs are still handled as they arrive, so requests still reach the scheduler about as early as with #65466 (median queue wait about 25us, compared with 5-7ms before it).
For a remote reader, move the sleep on its socket into this new wait, so that readable() is non-blocking again and a loop iteration sleeps only once.
Qwen3-0.6B on GB200, MP executor, Rust frontend. Medians, before #65466 -> #65466 -> this change:
TP=1, 8 repeats, arrivals (128/8 tokens, 256 concurrent):
TP=4, 3 repeats:
TP=4, 6 repeats, before #65466 -> this change: