Repository navigation
[Diffusion] Rollout API: off-loop serialization, spliced msgpack, timing headers, opt-in uint8 video - #36754
Merged
Conversation
Rockdu
marked this pull request as ready for review
August 28, 2026 20:40
Rockdu
requested review from
AgainstEntropy,
BBuf,
HaiShaw,
mickqian,
ping1jing2 and
yichiche
as code owners
August 28, 2026 20:40
Zhichenzzz
approved these changes
Aug 28, 2026
Zhichenzzz
merged commit Aug 28, 2026
aeb02ff
into
sgl-project:sglang-miles-h3
80 of 90 checks passed
3 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Five standalone commits on the rollout HTTP path (
/rollout/generate):asyncio.to_thread; the safetensors save releases the GIL) and send the pre-built buffers via a fixed-length response with explicitcontent-length. Identity framing, wire bytes unchanged, zero client change.bytesenter the output list by reference instead of being copied through the encoder.b"".join(parts)is byte-identical tomsgspec.msgpack.encode(payload)(unit-tested), and the parts are streamed without re-joining, so a serialized tensor is never copied again after safetensors produces it.x-sgld-timingheader — absolute wall-clock marks (srv_recv→msgpack_end) per request, so the trainer can reconstruct a cross-process waterfall. Header only; msgpack body contract untouched.x-sgld-stagesheader — the engine's per-stage milliseconds (text encode / denoise / decode), distinguishing a slow denoise from a slow VAE decode client-side.rollout_video_dtype="uint8"— engine-side quantization of the decoded [0,1] video to 0..255 uint8 (consumers divide by 255), ~4x smaller video payload. DefaultNoneships unchanged.Why
An H3 rollout response carries hundreds of MB of trajectory tensors. Encoding them inline held the GIL and blocked the uvicorn event loop for the full serialization — stalling health checks, reply handling, and next-request dispatch. With the loop free, serialization of request N overlaps the denoise of request N+1 under
--sglang-server-concurrency >= 2.Verified end-to-end on the MiniMax-H3 2-GPU GRPO recipe (3 rollout steps, wire-identical responses, no regression; unit tests in
test_msgpack_splice.py/test_rollout_api.pyare auto-registered CI).CI States
Latest PR Test (Base): ❌ Run #33128328860
Latest PR Test (Extra): ❌ Run #33128328737
Latest PR Test (AMD ROCm 7.2): ❌ Run #33128328859