config: three cache and pool readers take the bags - #36791
Merged
Conversation
ch-wan
requested review from
Fridge003,
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
August 28, 2026 05:15
5 tasks
ch-wan
force-pushed
the
cheng/gc-r3b-widths
branch
from
August 28, 2026 06:02
6767a69 to
ad5f8e3
Compare
ch-wan
force-pushed
the
cheng/gc-r3c-readers
branch
from
August 28, 2026 06:02
aafdb2c to
a17269c
Compare
ch-wan
force-pushed
the
cheng/gc-r3b-widths
branch
from
August 28, 2026 06:11
ad5f8e3 to
c6a9b46
Compare
ch-wan
force-pushed
the
cheng/gc-r3c-readers
branch
from
August 28, 2026 06:11
a17269c to
02f34bf
Compare
ch-wan
force-pushed
the
cheng/gc-r3b-widths
branch
from
August 28, 2026 06:51
c6a9b46 to
8595626
Compare
ch-wan
force-pushed
the
cheng/gc-r3c-readers
branch
from
August 28, 2026 06:51
02f34bf to
1a7b098
Compare
ch-wan
force-pushed
the
cheng/gc-r3b-widths
branch
from
August 28, 2026 07:08
8595626 to
90c7cc1
Compare
ch-wan
force-pushed
the
cheng/gc-r3c-readers
branch
from
August 28, 2026 07:09
1a7b098 to
2b1506c
Compare
ch-wan
force-pushed
the
cheng/gc-r3b-widths
branch
from
August 28, 2026 07:15
90c7cc1 to
ba959c3
Compare
ch-wan
force-pushed
the
cheng/gc-r3c-readers
branch
from
August 28, 2026 07:15
2b1506c to
3bddc80
Compare
ch-wan
force-pushed
the
cheng/gc-r3b-widths
branch
from
August 28, 2026 07:52
ba959c3 to
dcb2776
Compare
ch-wan
force-pushed
the
cheng/gc-r3c-readers
branch
from
August 28, 2026 07:52
3bddc80 to
91f86bf
Compare
ch-wan
force-pushed
the
cheng/gc-r3b-widths
branch
from
August 28, 2026 17:13
dcb2776 to
3e2d584
Compare
ch-wan
force-pushed
the
cheng/gc-r3c-readers
branch
from
August 28, 2026 17:13
91f86bf to
2d675da
Compare
ch-wan
force-pushed
the
cheng/gc-r3b-widths
branch
from
August 28, 2026 17:18
3e2d584 to
4b7ea9b
Compare
`pool_configurator`, `mem_cache/registry` and `unified_radix_cache` read resolved configuration off a `ServerArgs` they were handed. The record holds the operator's raw input, so a resolution-filled field read there answers with the pre-resolution value; the published bag is what resolution decided. Eight reads move: `max_total_tokens` and `page_size` to `get_schedule()`, `enable_hisparse` / `radix_cache_backend` / `hicache_host_memory_mode` / `enable_session_radix_cache` to `get_memory()`, `enable_streaming_session` to `get_serving()`, `extra_metric_labels` to `get_observability()`. `max_speculative_num_draft_tokens` stays on the record: it is a derived property, not a field, so no bag serves it. These three are the first of the eight modules that the launch stand-in's write-through currently props up. Measured by deleting `_apply_fields(server_args, self._fields)` from `override_server_args` and running the 159 registered tests that mention the record: 27 failures across 11 files, of which 7 are the hook's own guards. Converting the readers is what lets that write-through -- and `_internal_write` with it -- go. `test_pool_configurator` read `page_size` off the record too, and now reads the same bag. 2171 tests across `mem_cache/`, `model_executor/` and `observability/` pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ch-wan
force-pushed
the
cheng/gc-r3c-readers
branch
from
August 28, 2026 17:21
2d675da to
e37db74
Compare
saturn-acc
pushed a commit
to saturn-acc/sglang
that referenced
this pull request
Aug 31, 2026
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
nzr-niu
pushed a commit
to nzr-niu/sglang
that referenced
this pull request
Sep 1, 2026
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
pool_configurator,mem_cache/registryandunified_radix_cacheread resolved configurationoff a
ServerArgsthey were handed. The record holds the operator's raw input, so aresolution-filled field read there answers with the pre-resolution value; the published bag is
what resolution decided.
Modifications
Eight reads move:
max_total_tokensandpage_sizetoget_schedule(),enable_hisparse/radix_cache_backend/hicache_host_memory_mode/enable_session_radix_cachetoget_memory(),enable_streaming_sessiontoget_serving(),extra_metric_labelstoget_observability().max_speculative_num_draft_tokensstays on the record: it is a derived property, not a field,so no bag serves it.
test_pool_configuratorreadpage_sizeoff the record too, and now reads the same bag.These three are the first of eight modules that the launch stand-in's write-through currently
props up. The set was measured, not guessed: deleting
_apply_fields(server_args, self._fields)fromoverride_server_argsand running the 159registered tests that mention the record leaves 27 failures across 11 files, of which 7 are the
hook's own guards. Converting these readers is a step toward retiring that write-through, and
the remaining five modules follow the same shape.
One caveat, learned by trying the wide version of this change first: whether a read can move
to a bag depends on whether the reader's process publishes, not on whether the field maps to a
namespace. A mechanical sweep of all 94 such reads produced ~180 test failures, because many
unit tests hand a record in and never publish, and because
HttpServerEngineAdapterconstructs a
ServerArgsin the caller's process and never publishes at all. So this goes onemodule at a time, with its tests.
Accuracy Tests
2171 tests across
mem_cache/,model_executor/andobservability/pass, and the 159registered tests that mention the record report the same failure set as the base.
Speed Tests and Profiling
None. A bag read replaces an attribute read.
Checklist
Review and Merge Process
Eight lines of read, one line of test. The question to ask of each is whether the reader runs
after a publish; all three of these are scheduler-side.
CI States
Latest PR Test (Base): 🚫 Run #33194280480
Latest PR Test (Extra): 🚫 Run #33194280151
Latest PR Test (AMD ROCm 7.2): ❌ Run #33194280257