Skip to content

Allow 128 chat logprob candidates through the model gateway - #41373

Draft
Shi-Dong wants to merge 2 commits into
sgl-project:mainfrom
Shi-Dong:shi/score-centering-router-pr
Draft

Shi-Dong wants to merge 2 commits into
sgl-project:mainfrom
Shi-Dong:shi/score-centering-router-pr

Conversation

@Shi-Dong

@Shi-Dong Shi-Dong commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Allow the OpenAI-compatible chat route to forward up to 128 candidate log probabilities per generated token, and preserve the session extensions input_ids and return_meta_info when forwarding the request to a worker. This enables clients that record candidate probabilities for score centering while using the SGLang router.

The current typed chat request rejects top_logprobs above 20 and discards these extensions. The change retains validation of unrelated fields and rejects candidate counts above the backend limit. It vendors the protocol crate so the extended validation and fields can be represented in the router's typed request. I am open to replacing that vendoring with an upstream protocol-crate release if one is available.

Verification

  • Six focused Rust tests passed on the original source revision. They cover the 128-candidate request, bounds and logprobs validation, and preservation of the session fields.
  • A live probe through the patched router completed 128 concurrent chat requests successfully, followed by 50 successful health checks. The downstream session data retained 128 candidates per token.
  • Rebased the same source changes onto current SGLang main; the rebased tree passes git diff --check. The Rust suite has not been rerun on this rebased tree.

The larger rollout is a separate workload and is not part of this PR's validation.


CI States

Latest PR Test (Base): ✅ Run #36274652162
Latest PR Test (Extra): ❌ Run #36274651967
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation model-gateway

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant