fix: stop-string check misses early matches during speculative decoding - #23802
Merged
hnyls2002 merged 5 commits intoJun 9, 2026
Merged
Conversation
…ve decoding The tail_str() window for stop-string matching was sized only by stop_str_max_len, ignoring how many tokens were accepted in one speculative decoding step. When multiple tokens were accepted at once, an early stop string could fall outside the tail window and never be detected. This fix passes new_accepted_len through to tail_str() so the window always covers all newly accepted tokens plus the stop-string length.
xythink
requested review from
Ying1123,
hnyls2002,
merrymercy and
xiezhq-hermann
as code owners
April 27, 2026 04:43
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
5 tasks
hnyls2002
added a commit
that referenced
this pull request
Jun 9, 2026
Co-Authored-By: xythink <xythink@users.noreply.github.com>
minkail-m
pushed a commit
to modal-projects/sglang
that referenced
this pull request
Jul 17, 2026
…ng (sgl-project#23802) Clean cherry-pick of upstream d145a61 (sgl-project/sglang PR sgl-project#23802) onto our pre-landing DFlash spec branch. No deviations from the upstream diff: the hunks applied cleanly and land in the same code paths — the widened tail window in Req.tail_str / _check_str_based_finish is driven by new_accepted_len from Req.update_finish_state, which our Spec V2 chunked-accept path (batch_result_processor._process_batch_result_decode) already calls with the full accepted length. Spec V1 paths (eagle/ngram/ dflash verify loops) append + check one token at a time, so the default new_accepted_len=1 keeps their behavior identical. Co-authored-by: xythink <xythink@users.noreply.github.com> Co-authored-by: hnyls2002 <lsyincs@gmail.com> Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com> (cherry picked from commit d145a61) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
minkail-m
pushed a commit
to modal-projects/sglang
that referenced
this pull request
Jul 17, 2026
Clean cherry-pick of upstream 2218622 (sgl-project/sglang PR sgl-project#25980), companion to sgl-project#23802: when a stop string/regex matches inside a multi-token accepted chunk, record Req.finished_len at the match position so tokens accepted after the stop are not emitted, and make the detokenizer's trim_matched_stop end the text exactly at the match (keeping the stop itself under no_stop_trim). No deviations from the upstream diff: our fork already has the finished_len machinery this hooks into (Req.output_ids_through_stop, output_streamer truncation at finished_len), so the fix composes without adaptation. Co-authored-by: gss <2783977641@qq.com> Co-authored-by: hnyls2002 <lsyincs@gmail.com> Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com> (cherry picked from commit 2218622) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Chronostasys
pushed a commit
to MindLab-Research/sglang
that referenced
this pull request
Aug 24, 2026
…ng (sgl-project#23802) Co-authored-by: xythink <xythink@users.noreply.github.com> Co-authored-by: hnyls2002 <lsyincs@gmail.com> Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
jakki-amd
pushed a commit
to jakki-amd/sglang
that referenced
this pull request
Sep 9, 2026
…ng (sgl-project#23802) Co-authored-by: xythink <xythink@users.noreply.github.com> Co-authored-by: hnyls2002 <lsyincs@gmail.com> Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
When speculative decoding accepts multiple tokens in one step,
_check_str_based_finish()can miss stop strings that appear early in the accepted batch.Root cause:
tail_str()decodes only the laststop_str_max_len + 1tokens, but speculative decoding may accept far more tokens per step. A stop string near the beginning of the accepted tokens falls outside this window and is never detected.Impact: Affects all speculative decoding backends (EAGLE/MTP/DFlash/NGRAM) when using string-based
stopparameters.stop_token_idsis unaffected because_check_token_based_finish()already iterates over all newly accepted tokens.Modifications
tail_str(new_accepted_len): expand the decode window tomax(stop_str_max_len, new_accepted_len + stop_str_max_len)_check_str_based_finish(new_accepted_len): pass throughcheck_finished(): passnew_accepted_lento string checkThree call sites changed, no new dependencies.
Checklist
test/registered/unit/managers/new_accepted_len=1preserves existing behavior for non-speculative paths)CI States
Latest PR Test (Base): ⏳ Run #27187587256
Latest PR Test (Extra): ❌ Run #27187587213