Repository navigation
[cherrypick from #42349] [RL] Pass return_indexer_topk through chat completions and sessions - #42646
Conversation
This comment has been minimized.
This comment has been minimized.
…ompletions and sessions ChatCompletionRequest had no return_indexer_topk field, so pydantic dropped the flag and chat completions never returned indexer top-k. Session.create_req also built the turn's Req without the flag, so requests that continue a session never returned it either. Neither path reported an error. Add return_indexer_topk to ChatCompletionRequest, reject it without return_meta_info because meta_info is the only place a chat response carries it, pass it into GenerateReqInput, and copy it onto the session turn's Req. The shared fake session request in the session tests sets the field, since Session.create_req now reads it.
This comment has been minimized.
This comment has been minimized.
11fb5be to
26336e6
Compare
This comment has been minimized.
This comment has been minimized.
sglang-miles already returns indexer_topk_num_layers with indexer_topk, but nothing guards it: no unit test covers the scheduler payload, the multi-tokenizer split, or meta_info, and the RL test reshapes the flat int32 bytes with a hardcoded layer count instead of the returned one. Port the tests from #42349 (a4a57b3). sglang-miles imports the capturer lazily inside to_payload, so the output streamer test patches get_global_indexer_capturer at its owning module instead of on output_streamer.
|
Required checks failed on afd6d7e; this PR cannot be merged until they pass. PR title (run)
|
Summary
Port #42349 to
sglang-miles: chat/sessionreturn_indexer_topk, plus its layer-count tests.Symptom & Reproduction
sglang-miles,/v1/chat/completionswithreturn_indexer_topk: trueplusreturn_meta_info: truereturns nometa_info.indexer_topk; a session turn drops the flag the same way, with no error. Miles session rollouts set this flag on chat requests (miles/rollout/session/request_args.py), so indexer replay through sessions silently gets no data.python -m pytest test/registered/unit/entrypoints/openai/test_serving_chat.py -k "internal_request_single or indexer_topk_without_meta_info"expects the flag onGenerateReqInputplus a 400 withoutreturn_meta_info; onsglang-milesat258bc1a8d9both fail, and with this change both pass. The session drop is atSession.create_req(Root Cause step 3).Root Cause
ChatCompletionRequestdeclares noreturn_indexer_topk, so pydantic drops it._convert_to_internal_requesttherefore never setsGenerateReqInput.return_indexer_topk.Session.create_reqcopiesreturn_routed_expertsonto the turn'sReq, notreturn_indexer_topk.Fix
The first commit cherry-picks #42349's two chat/session commits; the hunks are identical. Declare the field on
ChatCompletionRequest, pass it intoGenerateReqInput, and copy it inSession.create_req. Chat rejectsreturn_indexer_topkwithoutreturn_meta_info, matchingreturn_sampling_mask, becausemeta_infois the only place a chat response carries it./generate,/v1/completions, and requests without the flag are unchanged.The second commit ports only the tests of #42349's layer-count commit (
a4a57b3db1).sglang-milesalready returnsindexer_topk_num_layersthrough the scheduler payload, the multi-tokenizer split, andmeta_info, and already has #42349's sequence-relative DSA capture, but nothing tested the layer count. The output-streamer test patchesget_global_indexer_captureratsglang.srt.state_capturer.indexer_topk, becausesglang-milesimports it lazily into_payload. The RL test reshapes by the returnedmeta_info["indexer_topk_num_layers"]instead of a hardcoded count. No production code changes in this commit.Verification
258bc1a8d9, 2 passed with this change.26336e6f98(the first commit, onsglang-miles9d86e9157a), on an H200 devbox whose image tree is9d86e9157awith only that change applied:test_serving_chat.pyandtest_session_token_share_unit.py161 passed, 113 subtests passed.afd6d7e3eb), CPU on macOS withPYTHONPATH=python:test_serving_chat.py,test_session_token_share_unit.py,test_multi_tokenizer_mixin.py,test_output_streamer_customized_info.py, andtest_tokenizer_manager_rid_cleanup.py246 passed, 163 subtests passed, including the 3 new layer-count tests. Withto_payloadforced to sendindexer_topk_num_layers=None, the output-streamer test fails in both IPC codecs.test/registered/rl/test_return_indexer_topk.py(DeepSeek-V3.2, 8 GPUs); onlypy_compileat this head.Review Focus
OpenAIServingChat._validate_request: new 400 forreturn_indexer_topkwithoutreturn_meta_info.Session.create_req: its replay flags must stay in sync with the non-sessionReqbuilt inscheduler.py.TestOutputStreamerIndexerTopk: its capturer patch target must follow whereoutput_streamerimportsget_global_indexer_capturerfrom.CI States
Latest PR Test (Base): ❌ Run #39675164268
Latest PR Test (Extra): ❌ Run #39675163815
Latest PR Test (AMD ROCm 10): ❌ Run #39675163936