Skip to content

fix(protocol): forward sampling masks for chat completions - #20

Merged
guapisolo merged 1 commit into
radixark:mainfrom
modal-projects:fix/chat-sampling-mask-forwarding
Sep 23, 2026
Merged

guapisolo merged 1 commit into
radixark:mainfrom
modal-projects:fix/chat-sampling-mask-forwarding

Conversation

@nanjiangwill

Copy link
Copy Markdown
Contributor

Summary

  • add return_sampling_mask to ChatCompletionRequest so typed chat routing preserves the SGLang extension
  • cover enabled and default values in the existing SGLang extension serde tests
  • verify the OpenAI router forwards the field to a mock worker
  • initialize the recently added return_indexer_topk field in an existing GenerateRequest test fixture so the routing test target compiles

Why

Miles session rollouts call /v1/chat/completions. The generate protocol already preserves return_sampling_mask, but the chat protocol did not, so Serde discarded the field before forwarding the request to SGLang.

Tests

  • cargo test --test spec_test test_sglang_extension_fields
  • cargo test --test routing_tests test_openai_router_forwards_return_sampling_mask

Both focused test targets passed on the patched tree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants