Skip to content

config: three cache and pool readers take the bags - #36791

Merged
ch-wan merged 1 commit into
mainfrom
cheng/gc-r3c-readers
Aug 28, 2026
Merged

config: three cache and pool readers take the bags#36791
ch-wan merged 1 commit into
mainfrom
cheng/gc-r3c-readers

Conversation

@ch-wan

@ch-wan ch-wan commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Motivation

pool_configurator, mem_cache/registry and unified_radix_cache read resolved configuration
off a ServerArgs they were handed. The record holds the operator's raw input, so a
resolution-filled field read there answers with the pre-resolution value; the published bag is
what resolution decided.

Modifications

Eight reads move: max_total_tokens and page_size to get_schedule(), enable_hisparse /
radix_cache_backend / hicache_host_memory_mode / enable_session_radix_cache to
get_memory(), enable_streaming_session to get_serving(), extra_metric_labels to
get_observability().

max_speculative_num_draft_tokens stays on the record: it is a derived property, not a field,
so no bag serves it.

test_pool_configurator read page_size off the record too, and now reads the same bag.

These three are the first of eight modules that the launch stand-in's write-through currently
props up. The set was measured, not guessed: deleting
_apply_fields(server_args, self._fields) from override_server_args and running the 159
registered tests that mention the record leaves 27 failures across 11 files, of which 7 are the
hook's own guards. Converting these readers is a step toward retiring that write-through, and
the remaining five modules follow the same shape.

One caveat, learned by trying the wide version of this change first: whether a read can move
to a bag depends on whether the reader's process publishes, not on whether the field maps to a
namespace.
A mechanical sweep of all 94 such reads produced ~180 test failures, because many
unit tests hand a record in and never publish, and because HttpServerEngineAdapter
constructs a ServerArgs in the caller's process and never publishes at all. So this goes one
module at a time, with its tests.

Accuracy Tests

2171 tests across mem_cache/, model_executor/ and observability/ pass, and the 159
registered tests that mention the record report the same failure set as the base.

Speed Tests and Profiling

None. A bag read replaces an attribute read.

Checklist

Review and Merge Process

Eight lines of read, one line of test. The question to ask of each is whether the reader runs
after a publish; all three of these are scheduler-side.

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): 🚫 Run #33194280480
Latest PR Test (Extra): 🚫 Run #33194280151
Latest PR Test (AMD ROCm 7.2): ❌ Run #33194280257

@ch-wan
ch-wan force-pushed the cheng/gc-r3b-widths branch from 6767a69 to ad5f8e3 Compare August 28, 2026 06:02
@ch-wan
ch-wan force-pushed the cheng/gc-r3c-readers branch from aafdb2c to a17269c Compare August 28, 2026 06:02
@ch-wan
ch-wan force-pushed the cheng/gc-r3b-widths branch from ad5f8e3 to c6a9b46 Compare August 28, 2026 06:11
@ch-wan
ch-wan force-pushed the cheng/gc-r3c-readers branch from a17269c to 02f34bf Compare August 28, 2026 06:11
@ch-wan
ch-wan force-pushed the cheng/gc-r3b-widths branch from c6a9b46 to 8595626 Compare August 28, 2026 06:51
@ch-wan
ch-wan force-pushed the cheng/gc-r3c-readers branch from 02f34bf to 1a7b098 Compare August 28, 2026 06:51
@ch-wan
ch-wan force-pushed the cheng/gc-r3b-widths branch from 8595626 to 90c7cc1 Compare August 28, 2026 07:08
@ch-wan
ch-wan force-pushed the cheng/gc-r3c-readers branch from 1a7b098 to 2b1506c Compare August 28, 2026 07:09
@ch-wan
ch-wan force-pushed the cheng/gc-r3b-widths branch from 90c7cc1 to ba959c3 Compare August 28, 2026 07:15
@ch-wan
ch-wan force-pushed the cheng/gc-r3c-readers branch from 2b1506c to 3bddc80 Compare August 28, 2026 07:15
@ch-wan
ch-wan force-pushed the cheng/gc-r3b-widths branch from ba959c3 to dcb2776 Compare August 28, 2026 07:52
@ch-wan
ch-wan force-pushed the cheng/gc-r3c-readers branch from 3bddc80 to 91f86bf Compare August 28, 2026 07:52
@ch-wan
ch-wan force-pushed the cheng/gc-r3b-widths branch from dcb2776 to 3e2d584 Compare August 28, 2026 17:13
@ch-wan
ch-wan force-pushed the cheng/gc-r3c-readers branch from 91f86bf to 2d675da Compare August 28, 2026 17:13
@ch-wan
ch-wan force-pushed the cheng/gc-r3b-widths branch from 3e2d584 to 4b7ea9b Compare August 28, 2026 17:18
Base automatically changed from cheng/gc-r3b-widths to main August 28, 2026 17:18
`pool_configurator`, `mem_cache/registry` and `unified_radix_cache` read
resolved configuration off a `ServerArgs` they were handed. The record holds
the operator's raw input, so a resolution-filled field read there answers with
the pre-resolution value; the published bag is what resolution decided.

Eight reads move: `max_total_tokens` and `page_size` to `get_schedule()`,
`enable_hisparse` / `radix_cache_backend` / `hicache_host_memory_mode` /
`enable_session_radix_cache` to `get_memory()`, `enable_streaming_session` to
`get_serving()`, `extra_metric_labels` to `get_observability()`.

`max_speculative_num_draft_tokens` stays on the record: it is a derived
property, not a field, so no bag serves it.

These three are the first of the eight modules that the launch stand-in's
write-through currently props up. Measured by deleting
`_apply_fields(server_args, self._fields)` from `override_server_args` and
running the 159 registered tests that mention the record: 27 failures across
11 files, of which 7 are the hook's own guards. Converting the readers is what
lets that write-through -- and `_internal_write` with it -- go.

`test_pool_configurator` read `page_size` off the record too, and now reads the
same bag.

2171 tests across `mem_cache/`, `model_executor/` and `observability/` pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ch-wan
ch-wan force-pushed the cheng/gc-r3c-readers branch from 2d675da to e37db74 Compare August 28, 2026 17:21
@ch-wan
ch-wan merged commit 7bc3204 into main Aug 28, 2026
26 of 36 checks passed
@ch-wan
ch-wan deleted the cheng/gc-r3c-readers branch August 28, 2026 17:21
saturn-acc pushed a commit to saturn-acc/sglang that referenced this pull request Aug 31, 2026
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
nzr-niu pushed a commit to nzr-niu/sglang that referenced this pull request Sep 1, 2026
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant