Make SGLangApiClient an async worker handle over httpx - #1853
Open
fzyzcjy wants to merge 1 commit into
Open
Conversation
fzyzcjy
requested review from
Shi-Dong,
Zhichenzzz,
maocheng23,
yueming-yuan and
yushengsu-thu
as code owners
July 28, 2026 12:56
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-14
branch
from
August 3, 2026 23:44
b0bbe7b to
abd40de
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-9
branch
2 times, most recently
from
August 4, 2026 00:39
448c30f to
b9946a1
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-14
branch
from
August 4, 2026 06:57
5069435 to
35a8595
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-9
branch
from
August 4, 2026 06:57
b9946a1 to
e10f4c5
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-14
branch
from
August 10, 2026 06:06
35a8595 to
92bf2e8
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-9
branch
2 times, most recently
from
August 10, 2026 16:07
b893f39 to
cf39835
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-14
branch
from
August 10, 2026 16:34
aed4cf1 to
deb8fc0
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-9
branch
from
August 10, 2026 16:34
cf39835 to
015cd40
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-14
branch
from
August 12, 2026 02:48
deb8fc0 to
4c07458
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-9
branch
2 times, most recently
from
August 12, 2026 14:31
49a506d to
b95089c
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-14
branch
from
August 12, 2026 14:31
4c07458 to
7e38f8b
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-9
branch
from
August 12, 2026 23:56
b95089c to
a8e3e02
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-14
branch
from
August 13, 2026 06:01
6965269 to
3703550
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-9
branch
from
August 13, 2026 06:01
a8e3e02 to
16da4ad
Compare
The rollout process is async, and the requests-based client forced every caller onto a thread. Turn the client and the router client into async-only handles: every method is `async def` and goes through a per-event-loop `httpx.AsyncClient`, so hundreds of in-flight requests cost no threads. `httpx.AsyncClient` is loop-affine while `SGLangApiClient` is a frozen, cross-process-serializable dataclass, so the client cannot own one; it is cached in a module-level dict keyed by the running loop. `SGLangEngine`'s methods are Ray actor entry points and stay synchronous: each shell now drives its coroutine on the shared background loop via `async_utils.run`.
fzyzcjy
force-pushed
the
tom/refactor-miles/op3-9
branch
from
August 14, 2026 04:45
16da4ad to
c8fa640
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1837