Repository navigation
fix(protocol): allow 128 chat top_logprobs and forward sampling_logprobs_mode - #21
Merged
Merged
Conversation
…obs_mode Miles score centering records the sampler's candidate probabilities at generation time, and both of its request shapes were broken by typed routing: - Unfiltered session rollouts send top_logprobs=k to /v1/chat/completions (default k=128). The chat validator rejected any value above 20. - Filtered rollouts send sampling_logprobs_mode="support" on chat and /generate. Neither typed request declared the field, so Serde dropped it before forwarding and SGLang fell back to "selected". Raise the chat top_logprobs bound to 128, the capacity of SGLang's PD logprob metadata, and add sampling_logprobs_mode to ChatCompletionRequest and GenerateRequest so it is forwarded unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 3, 2026
guapisolo
added a commit
that referenced
this pull request
Oct 3, 2026
The Responses route still validated top_logprobs against OpenAI's limit of 20, so requests for 21 to 128 candidates were rejected before routing. The chat route accepts up to 128 since #21, and SGLang's ResponsesRequest passes the value through as top_logprobs_num without an upper bound. Raise the bound to 128, the SGLang PD logprob metadata capacity already used for chat. Values above 128 stay rejected. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Accept 128 chat
top_logprobsand forwardsampling_logprobs_modeon chat and generate requests.Symptom & Reproduction
top_logprobsabove 20 is rejected, whilesampling_logprobs_mode: "support"never reaches SGLang on chat or/generate, so SGLang falls back toselectedscalar sampling log-probs.0e7c1acshow the failures: validating a chat request with"logprobs": true, "top_logprobs": 128returns atop_logprobsrange error (max: 20); deserializing then re-serializing a chat or generate request with"sampling_logprobs_mode": "support"yieldsnullfor that field instead of"support".Root Cause
ChatCompletionRequest::top_logprobsuses#[validate(range(min = 0, max = 20))], the OpenAI limit.ChatCompletionRequestnorGenerateRequestdeclaressampling_logprobs_mode.route_typed_requestforwardsserde_json::to_value(typed_req), so undeclared fields are dropped.Fix
Raise the chat bound to 128, the capacity of SGLang's PD logprob metadata (
max_top_logprobs_numindisaggregation/utils.py), and addsampling_logprobs_mode: Option<String>to both requests, serialized only when set. This follows #20, which fixed the same field loss forreturn_sampling_mask. It ports the functional part of sgl-project/sglang#41373;input_idsandreturn_meta_infoare already typed here. Requests without these fields serialize unchanged.Verification
cargo test --test spec_test: 98 passed, including the newtest_top_logprobs_accepts_up_to_128(20, 21, 128 accepted; 129 rejected) plus the extended chat/generate extension-field tests (round-trip, defaults, omission).cargo test --test routing_tests: 90 passed, 2 failed, both environmental or pre-existing:test_redis_server_start_stopneedsredis-server, andtest_manual_routing_without_headeralso fails 3/3 on0e7c1ac.cargo build --libpassed.cargo fmtwas not run becauserustfmtis not installed on the test host.Not verified: an end-to-end Miles rollout through a rebuilt router wheel.
Review Focus
top_logprobs/src/protocols/chat.rs: 128 exceeds the OpenAI limit; values above 128, beyond SGLang's PD capacity, stay rejected here.sampling_logprobs_mode: kept as a pass-throughString, so SGLang remains the validator of its values.Generated with Claude Code