Repository navigation
[Refactor] disaggregation: Commit the PD handoff token through the shared per-token commit - #47544
Conversation
…tput-token logprob append; drop the empty top-k row the handoff wrote without a top-k request
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
|
/rerun-test test_disaggregation_wire.py test_batch_result_processor_spec_grammar.py test_flat_raw_top_logprobs.py registered/disaggregation/test_disaggregation_basic.py test_disaggregation_rust_server.py test_srt_endpoint.py test_retract_decode_logprob.py |
|
/tag-and-rerun-ci |
|
🚀 🚀 🚀 |
|
/rerun-test test_srt_endpoint.py |
|
🚀 |
…ared per-token commit (sgl-project#47544)
The decode worker committed the PD handoff token (the token the prefill worker samples) with its own copy of the per-token bookkeeping, separate from the prefill and decode result paths; it has missed a field before (the reasoning count in #32897, the sampling mask in #41235).
SchedulerLogprobResultProcessor.append_output_token_logprobsappends one output token's logprob, top-k and token-ids rows; the conditions ontop_logprobs_numandtoken_ids_logproblive only there.add_logprob_return_values(prefill),_apply_decode_logprobs(each accepted decode token) and the handoff call it.SchedulerBatchResultProcessor.commit_handoff_tokenappends the token tooutput_ids, updates the reasoning count, and appends its logprob and sampling-mask rows.DecodeTransferQueue._commit_transfer_to_reqreads the metadata buffer and calls it; PD-only state (cached tokens, multimodal token counts, speculative seeds) stays there, and a rebootstrap still commits nothing.Behavior change: without a top-k request the handoff no longer appends an empty top-k row, as the prefill and decode paths never did. The Python tokenizer manager ignored that row; the Rust server streamed it as
output_top_logprobs: [null]in the first chunk.Also updates
test_srt_endpointto expect the 400 that #47521 returns for out-of-vocab ids in/detokenize.CI States
Latest PR Test (Base): 🚫 Run #41147764078
Latest PR Test (Extra): ❌ Run #41147764020
Latest PR Test (AMD ROCm 10): 🚫 Run #41147764014