Skip to content

[Refactor] disaggregation: Commit the PD handoff token through the shared per-token commit - #47544

Merged
hnyls2002 merged 3 commits into
mainfrom
lsyin/unify-handoff-token-commit
Oct 31, 2026
Merged

hnyls2002 merged 3 commits into
mainfrom
lsyin/unify-handoff-token-commit

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

The decode worker committed the PD handoff token (the token the prefill worker samples) with its own copy of the per-token bookkeeping, separate from the prefill and decode result paths; it has missed a field before (the reasoning count in #32897, the sampling mask in #41235).

  • SchedulerLogprobResultProcessor.append_output_token_logprobs appends one output token's logprob, top-k and token-ids rows; the conditions on top_logprobs_num and token_ids_logprob live only there. add_logprob_return_values (prefill), _apply_decode_logprobs (each accepted decode token) and the handoff call it.
  • SchedulerBatchResultProcessor.commit_handoff_token appends the token to output_ids, updates the reasoning count, and appends its logprob and sampling-mask rows. DecodeTransferQueue._commit_transfer_to_req reads the metadata buffer and calls it; PD-only state (cached tokens, multimodal token counts, speculative seeds) stays there, and a rebootstrap still commits nothing.

Behavior change: without a top-k request the handoff no longer appends an empty top-k row, as the prefill and decode paths never did. The Python tokenizer manager ignored that row; the Rust server streamed it as output_top_logprobs: [null] in the first chunk.

Also updates test_srt_endpoint to expect the 400 that #47521 returns for out-of-vocab ids in /detokenize.


CI States

Latest PR Test (Base): 🚫 Run #41147764078
Latest PR Test (Extra): ❌ Run #41147764020
Latest PR Test (AMD ROCm 10): 🚫 Run #41147764014

…tput-token logprob append; drop the empty top-k row the handoff wrote without a top-k request

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test test_disaggregation_wire.py test_batch_result_processor_spec_grammar.py test_flat_raw_top_logprobs.py registered/disaggregation/test_disaggregation_basic.py test_disaggregation_rust_server.py test_srt_endpoint.py test_retract_decode_logprob.py

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🚀 ubuntu-latest (3 tests): ✅ View workflow run

cd test/ && python3 registered/unit/disaggregation/test_disaggregation_wire.py
cd test/ && python3 registered/unit/managers/test_batch_result_processor_spec_grammar.py
cd test/ && python3 registered/unit/managers/test_flat_raw_top_logprobs.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_basic.py
cd test/ && python3 registered/disaggregation/test_disaggregation_rust_server.py

🚀 1-gpu-5090 (2 tests): ❌ View workflow run

cd test/ && python3 registered/core/test_srt_endpoint.py
cd test/ && python3 registered/scheduler/test_retract_decode_logprob.py

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 30, 2026
@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test test_srt_endpoint.py

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/core/test_srt_endpoint.py

@hnyls2002
hnyls2002 merged commit 4df3895 into main Oct 31, 2026
120 of 151 checks passed
@hnyls2002
hnyls2002 deleted the lsyin/unify-handoff-token-commit branch October 31, 2026 00:18
Yuxingwang-intel pushed a commit to Yuxingwang-intel/sglang that referenced this pull request Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant