Skip to content

Update the serving runtime to SGLang v0.5.21 - #268

Draft
jvmncs wants to merge 1 commit into
mainfrom
codex/reconcile-sglang-v0.5.20-pin
Draft

jvmncs wants to merge 1 commit into
mainfrom
codex/reconcile-sglang-v0.5.20-pin

Conversation

@jvmncs

@jvmncs jvmncs commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Update the cookbook serving runtime to SGLang v0.5.21 and pin the published fork. Keep the existing update flow: prepare inactive weights, pause generation, commit the weights, then resume generation.

The fork has 16 commits marked [Weight Sync n/16]. Nan remains their author; Jason has co-author credit for the port. Tests are included with the implementation they cover. The retained upstream fixes preserve their author credit. The history repair produces exactly the same files and Git modes as measured revision ffbaf2cf907e.

The implementation changes are:

  • Port the model, quantization, and runner hooks to v0.5.21. Of the 14 existing core weight_sync/ files, 11 are unchanged from the published v0.5.20 branch.
  • Bring in that branch’s source-file delta reader and CPU preparation path. These are new relative to Stitch’s older pin, 7686f6b711.
  • Fix preparation when a delta contains base-checkpoint tensors that the native target loader omits. Unknown checkpoint names still fail.
  • Preserve the base image’s native modules, including the Rust cache module, when installing the fork’s Python files.

Keep #34201, #32237, #35804, and #29916. Remove the separate patches already included in v0.5.21: #35204, #40075, #35798, #40932, #40986, #41235, and #40779. #41955 remains excluded. The v0.5.20 and staging branches are unchanged.

Confirmed by probes:

Check Result
Stitch tests 302 passed
Fork CPU tests 491 passed; 90 subtests passed
Registered CPU suite 223 passed; 37 subtests passed; one GPU-only test skipped
Maintained image builder Imports, native modules, and serving passed; new immutable pin checked on CPU
Weight updates Fresh native weights matched; generation continued during preparation; missing targets preserved the active version

For large-model RAM updates, all 12 native-load comparisons passed. The first update measured:

Runtime Prepare (s) Commit (s)
v0.5.20 83.020 ± 109.536 2.844 ± 0.224
v0.5.21 41.241 ± 0.914 2.740 ± 0.001

Values are means ± 95% confidence half-widths from three distinct hosts per runtime and two runs per host. The old runtime varied widely across hosts. These runs do not reproduce the earlier preparation slowdown. Caches were uncontrolled; the results do not establish a general speedup or rule out other regressions.

The README contains the full phase and producer timing tables. Measurements used at most four GPUs per allocation. Large training and eight-GPU cases remain untested. Earlier sampling GPU coverage depends on inspection of later source changes. All validation apps are stopped.

@jvmncs
jvmncs force-pushed the codex/reconcile-sglang-v0.5.20-pin branch from aaffa20 to cc215d6 Compare October 3, 2026 03:56
@jvmncs jvmncs changed the title Pin reconciled SGLang v0.5.20 sampling and pause fixes Pin staged SGLang v0.5.20 sampling and pause fixes Oct 3, 2026
@jvmncs
jvmncs marked this pull request as draft October 6, 2026 03:20
@jvmncs
jvmncs force-pushed the codex/reconcile-sglang-v0.5.20-pin branch from cc215d6 to 2c319a4 Compare October 7, 2026 01:33
@jvmncs jvmncs changed the title Pin staged SGLang v0.5.20 sampling and pause fixes Update the serving runtime to SGLang v0.5.21 Oct 7, 2026
@jvmncs
jvmncs force-pushed the codex/reconcile-sglang-v0.5.20-pin branch from 2c319a4 to f5bf7ad Compare October 7, 2026 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant