Repository navigation
fix(protocol): forward flat output top logprobs for chat completions - #24
Merged
guapisolo merged 1 commit intoOct 14, 2026
Merged
Conversation
SGLang chat completions accept return_flat_raw_output_top_logprobs and return_flat_raw_top_logprobs_b64 (sgl-project/sglang#42890, on sglang-miles as #42838), which return output top-k candidates as base64 [tokens, k] arrays in meta_info instead of [logprob, token_id, text] rows. ChatCompletionRequest did not declare them, so serde dropped both before forwarding and SGLang fell back to the nested rows. The b64 flag is Option<bool> and skipped when unset: SGLang's chat default for it is true, and a #[serde(default)] bool would always forward false. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HJSang
force-pushed
the
hejian/forward-flat-output-top-logprobs
branch
from
October 14, 2026 20:24
3dfad1e to
c6e8955
Compare
HJSang
added a commit
to radixark/miles
that referenced
this pull request
Oct 14, 2026
Score-centering session requests asked SGLang for top_logprobs, which come back as completion_tokens rows of k [logprob, token_id, text] triples. SGLang detokenizes every candidate and the session server parses the rows into Python objects, and JSON cannot carry -inf: a non-finite candidate arrives as null and numpy stores it as NaN in rollout_topk_log_probs. Chat training requests now also set return_flat_raw_output_top_logprobs (sgl-project/sglang#42890, on sglang-miles as #42838), which returns the candidates as float32/int32 [rows, k] arrays, base64 by default on chat, and append_rollout_topk_logprobs decodes them. Without that SGLang or a gateway that forwards the field (radixark/sgl-router-for-miles#24), the response keeps the nested rows, which are still decoded as before. Native /generate requests are unchanged because on-policy distillation reads the nested student rows there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Forward the two SGLang chat-completion extension fields added by sgl-project/sglang#42890 (merged to
main; cherry-picked tosglang-milesas #42838, also merged):return_flat_raw_output_top_logprobsreturn_flat_raw_top_logprobs_b64With these fields, SGLang returns output top-k candidates as base64
[tokens, k]float32/int32 arrays inchoices[].meta_info. Without them it returns[logprob, token_id, text]rows. At 160 tokens × k=128 that's about 0.24 MB instead of about 2.6 MB of JSON, with no per-candidate detokenization, and-infsurvives.ChatCompletionRequestdidn't declare either field. The HTTP router re-serializes the typed request, so serde dropped both and SGLang silently fell back to the nested rows.return_flat_raw_output_top_logprobsfollows the template of #20 (return_sampling_mask): a#[serde(default)]boolean in the SGLang extension block.return_flat_raw_top_logprobs_b64is anOption<bool>left out when unset: SGLang's chat default for it istrue, and always sendingfalsewould override that.Compatibility.
return_flat_raw_output_top_logprobsis always serialized,falsewhen unset; the b64 flag is sent only when the client set it. SGLang'sChatCompletionRequestignores unknown fields, so older SGLang builds accept both.Consumer: miles session server, which requests these fields for score-centering training (radixark/miles#3952).
Tests
tests/spec/chat_completion.rs: the SGLang extension round-trip and default-value tests now cover both fields, including that an omitted b64 flag stays omitted after re-serialization.cargo test --test spec_testgives 98 passed and 0 failed (Linux x86_64, rustc 1.98.1).output_top_logprobs_shape [32, 128]and no nested rows; the released 0.3.2 gateway dropped the flags and the engine returned nested rows. In three miles score-centering runs at k=128, all 1,212 session requests through this gateway carried both flags.cargo fmtwasn't run because rustfmt isn't installed on that host. The edits copy the neighbouring fields' layout.🤖 Generated with Claude Code