[Fix] Make DeepSeek-V4 reasoning and tool-call streaming parsing chunk-invariant by hnyls2002 · Pull Request #34458 · sgl-project/sglang

hnyls2002 · 2026-08-11T22:08:25Z

Summary

Hold back a trailing partial </think> / tool marker in BaseReasoningFormatDetector, so streaming output no longer depends on how decode steps batch tokens (supersedes fix(reasoning_parser): hold back think tags split across stream chunks #31009, [Bug] Preserve DeepSeek-V3.2 tool-call markers in reasoning parsing #21179)
Fix five further DSV4 reasoning + DSML tool-call streaming bugs, each covered by a test that fails on main

Background

Speculative decoding and stream_interval > 1 deliver several tokens per step, splitting multi-character markers across chunk boundaries
_parse_streaming_increment_impl cleared _buffer after every emission and destroyed the fragment, so the end token was never recognised and the rest of the turn leaked into reasoning_content
On main, <think>abc</think>normal at chunk size 3 already yields reasoning='abc</think>normal', normal=''
This affects every detector sharing the base implementation, not only DSV4

Changes

Reasoning parser

Hold back the longest trailing slice that is a proper prefix of think_end_token / tool_start_token / think_start_token, and flush it in finish() when the stream ends first
Move _ends_with_partial_token to BaseReasoningFormatDetector, replacing the copy in Apertus2509Detector
DeepSeekV4Detector passes tool_start_token, so a DSML block that follows reasoning without a </think> reaches the tool-call detector instead of staying in reasoning_content (supersedes fix(parsers): tools are leaking in the reasoning content (deepseek) #32539)
Write the stripped buffer back after removing think_start_token, so stream_reasoning=False no longer carries the token into reasoning_content

Tool-call detector

Catch MalformedJSON from _partial_json_loads instead of letting it escape to the outer handler (supersedes Fix DeepSeekV32/V4 streaming detector: catch MalformedJSON, reset buffer on parse errors #32332, [Fix] deepseekv32 detector: tolerate partial non-string param values raising MalformedJSON (stream tear-down) #28684)
Preserve prose emitted before a tool call, for both wrapped and bare invoke forms (supersedes fix: preserve content before DeepSeek streaming tool calls #31786, Preserve streaming preamble before DeepSeek DSML tool calls for all marker forms #33813, fix(function_call): stream text that precedes DSML tool calls #34244)
On a parse error, drop the buffer and re-emit its text instead of re-parsing it on every later chunk; discard collected calls so a named call with no arguments never reaches the client
detect_and_parse parses every tool_calls section, not only the first (supersedes fix(deepseekv32): parse bare invoke blocks without function_calls wrapper #23786)

Known divergences

Streaming cannot see whether a </think> still follows, so two inputs split differently from detect_and_parse: a model quoting the DSML marker inside its reasoning, and (pre-existing on main) text emitted before <think> when reasoning is not forced
Both are pinned by *_is_chunk_dependent tests so a future fix has to update them rather than change behaviour silently

Test plan

test_reasoning_parser.py: TestStreamingChunkSizeInvariance feeds each case at chunk sizes [1, 2, 3, 5, 7, 11, 23, 1000] and compares against detect_and_parse (supersedes [Tests] Add regression tests for reasoning parser edge cases #34157)
test_deepseekv4_detector.py: preamble across delta splits, MalformedJSON fallback, error-path recovery, multi-section parsing
Two existing assertions are intentionally flipped, both because they pinned behaviour this PR fixes:
- test_finish_drops_partial_end_tag_when_streaming_reasoning becomes ..._flushes_.... Its premise was that anything left in _buffer had already been emitted chunk by chunk; with holdback that text has never been emitted, and dropping it loses content whose only fault is resembling the start of a token
- TestGlm45Detector.test_streaming_no_stream_reasoning no longer expects the raw <think> tag inside reasoning_content. Its comment described the leak it was pinning ("stripped from the local current_text variable but NOT from self._buffer"); stream_reasoning=False now matches detect_and_parse
Supersedes fix(parser): DSV4 reasoning + tool call streaming parser bugs #34178, which reported the buffer-clearing bug this PR fixes

CI States

Latest PR Test (Base): 🚫 Run #31546554111
Latest PR Test (Extra): ❌ Run #31546553893

Co-Authored-By: hao-cyber <89575785+hao-cyber@users.noreply.github.com> Co-Authored-By: Enrico Falco <enrico9034@gmail.com> Co-Authored-By: Svyatoslav <85786374+slivanovich@users.noreply.github.com> Co-Authored-By: Andreas Hassellof <andreas@ombori.com> Co-Authored-By: Leoyzen <leoyzen@gmail.com>

hnyls2002 · 2026-08-11T22:18:43Z

/tag-and-rerun-ci

…ormed call on error

…prefix divergence

Co-Authored-By: Chenglun Hu <chenglunhu@gmail.com> Co-Authored-By: robellliu-dev <robell.liu@huawei.com> Co-Authored-By: Gavin.Zhu <gavin.z@gmicloud.ai> Co-Authored-By: Xinyuan Tong <xinyuantong.cs@gmail.com> Co-Authored-By: tancheng33 <garrytancheng@gmail.com> Co-Authored-By: dineshx29 <dinesh.b.offl@gmail.com> Co-Authored-By: Kangyan Zhou <zky314343421@gmail.com> Co-Authored-By: hao-cyber <89575785+hao-cyber@users.noreply.github.com> Co-Authored-By: Enrico Falco <enrico9034@gmail.com> Co-Authored-By: Svyatoslav <85786374+slivanovich@users.noreply.github.com> Co-Authored-By: Andreas Hassellof <andreas@ombori.com> Co-Authored-By: Leoyzen <leoyzen@gmail.com>

hnyls2002 · 2026-08-11T23:21:11Z

/rerun-test test_reasoning_parser.py test_deepseekv4_detector.py test_kimik3_reasoning_parser.py test_reasoning_content_without_parser.py test_reasoner_grammar_backend.py test_function_call_parser.py test_serving_chat.py test_reasoning.py test_openai_function_calling.py test_anthropic_tool_use.py test_e2e_constrained_reasoning.py test_constrained_decoding_spec_reasoning.py
(2 tries)

github-actions · 2026-08-11T23:21:36Z

Results for /rerun-test test_reasoning_parser.py test_deepseekv4_detector.py test_kimik3_reasoning_parser.py test_reasoning_content_without_parser.py test_reasoner_grammar_backend.py test_function_call_parser.py test_serving_chat.py:

🚀 ubuntu-latest (7 tests): ❌ View workflow run

cd test/ && python3 registered/unit/parser/test_reasoning_parser.py
cd test/ && python3 registered/unit/function_call/test_deepseekv4_detector.py
cd test/ && python3 registered/unit/parser/test_kimik3_reasoning_parser.py
cd test/ && python3 registered/unit/parser/test_reasoning_content_without_parser.py
cd test/ && python3 registered/unit/constrained/test_reasoner_grammar_backend.py
cd test/ && python3 registered/unit/function_call/test_function_call_parser.py
cd test/ && python3 registered/unit/entrypoints/openai/test_serving_chat.py

github-actions · 2026-08-11T23:40:22Z

Results for /rerun-test test_reasoning_parser.py test_deepseekv4_detector.py test_kimik3_reasoning_parser.py test_reasoning_content_without_parser.py test_reasoner_grammar_backend.py test_function_call_parser.py test_serving_chat.py:

🚀 ubuntu-latest (7 tests): ✅ View workflow run

cd test/ && python3 registered/unit/parser/test_reasoning_parser.py
cd test/ && python3 registered/unit/function_call/test_deepseekv4_detector.py
cd test/ && python3 registered/unit/parser/test_kimik3_reasoning_parser.py
cd test/ && python3 registered/unit/parser/test_reasoning_content_without_parser.py
cd test/ && python3 registered/unit/constrained/test_reasoner_grammar_backend.py
cd test/ && python3 registered/unit/function_call/test_function_call_parser.py
cd test/ && python3 registered/unit/entrypoints/openai/test_serving_chat.py

github-actions · 2026-08-11T23:45:44Z

Results for /rerun-test test_reasoning_parser.py test_deepseekv4_detector.py test_kimik3_reasoning_parser.py test_reasoning_content_without_parser.py test_reasoner_grammar_backend.py test_function_call_parser.py test_serving_chat.py test_reasoning.py test_openai_function_calling.py test_anthropic_tool_use.py test_e2e_constrained_reasoning.py test_constrained_decoding_spec_reasoning.py:

🚀 ubuntu-latest (7 tests): ✅ View workflow run

cd test/ && python3 registered/unit/parser/test_reasoning_parser.py
cd test/ && python3 registered/unit/function_call/test_deepseekv4_detector.py
cd test/ && python3 registered/unit/parser/test_kimik3_reasoning_parser.py
cd test/ && python3 registered/unit/parser/test_reasoning_content_without_parser.py
cd test/ && python3 registered/unit/constrained/test_reasoner_grammar_backend.py
cd test/ && python3 registered/unit/function_call/test_function_call_parser.py
cd test/ && python3 registered/unit/entrypoints/openai/test_serving_chat.py

🚀 1-gpu-h100 (3 tests): ✅ View workflow run

cd test/ && python3 registered/reasoning/test_reasoning.py
cd test/ && python3 registered/openai_server/function_call/test_openai_function_calling.py
cd test/ && python3 registered/openai_server/function_call/test_anthropic_tool_use.py

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/constrained/test_e2e_constrained_reasoning.py

🚀 2-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/spec/test_constrained_decoding_spec_reasoning.py

…otentially_dsml trap Three bugs caused agent to stop when it should be calling tools: 1. Exception handler cleared self._buffer (""), permanently losing tool call text. Fix: retain buffer, reset transient tool state instead. 2. str.rstrip(token) treated argument as character set, not substring. rstrip("parameter") truncated streaming values ending with chars in {p,a,r,m,e,t} (e.g. "find /tmp" → "find /"). Fix: use removesuffix. 3. Broad potentially_dsml check (any "｜DSML｜" in buffer) trapped all subsequent text in buffer when a DSML sub-tag appeared without an invoke. No finish() method existed to flush at stream end. Fix: narrow potentially_dsml to tool-call tags only; add finish() method; wire up parse_stream_end() in FunctionCallParser and serving_chat. 4. Reasoning parser used broad tool_start_token="<｜DSML｜" which matched any DSML sub-tag in reasoning, causing false positives. Fix: narrow to "<｜DSML｜tool_calls>". Related: sgl-project#34458 (supersedes sgl-project#34178, which reported the buffer-clearing bug) Tests: 8 new tests + 1 updated test, all 26 V32+V4 tests pass.

…p fix, finish() flush, tool_start_token narrowing Four bugs caused tool call loss, value truncation, text trapping, and DSML tag leakage in DeepSeek-V4 streaming parsing. All four are present in the current main (post sgl-project#34458) and are fixed here: Bug 1 - Exception handler cleared self._buffer, permanently losing tool call text. Fix: retain buffer, reset transient tool state instead. Bug 2 - str.rstrip(token) treated the argument as a character set, not a substring. rstrip("parameter") truncated streaming values ending with chars in {p,a,r,m,e,t}. Fix: use removesuffix. Bug 3 - Broad potentially_dsml check trapped all subsequent text when a DSML sub-tag appeared without an invoke. Fix: narrow to tool-call tags only; add finish() override to DeepSeekV32Detector. Bug 4 - Broad tool_start_token matched any DSML sub-tag in reasoning. Fix: narrow to tool_calls; add holdback at think_end and non-reasoning boundaries using _ends_with_partial_token. Tests: 12 new regression tests + 2 existing sgl-project#34458 tests updated. All 39 tests pass. Co-authored-by: Leoyzen <leoyzen@gmail.com>

…rdening Cherry-pick and squash of three PRs: - sgl-project#34262 infra: BaseFormatDetector.finish(), FunctionCallParser.parse_stream_end(), flush param in serving_chat._process_tool_call_stream - sgl-project#34458: chunk-invariance — partial token holdback, _ends_with_partial_token, tool_start_token for DeepSeekV4Detector, MalformedJSON catch, preamble preservation - sgl-project#34600: hardening — buffer retention on error, rstrip→removesuffix, narrow potentially_dsml to tool-call tags, finish() override on DeepSeekV32Detector, narrow tool_start_token to <｜DSML｜tool_calls, holdback at think_end boundary Co-authored-by: hnyls2002 <hnyls2002@gmail.com> Co-authored-by: Leoyzen <leoyzen@gmail.com>

…k-invariant (sgl-project#34458) Co-authored-by: hao-cyber <89575785+hao-cyber@users.noreply.github.com> Co-authored-by: Enrico Falco <enrico9034@gmail.com> Co-authored-by: Svyatoslav <85786374+slivanovich@users.noreply.github.com> Co-authored-by: Andreas Hassellof <andreas@ombori.com> Co-authored-by: Leoyzen <leoyzen@gmail.com> Co-authored-by: Chenglun Hu <chenglunhu@gmail.com> Co-authored-by: robellliu-dev <robell.liu@huawei.com> Co-authored-by: Gavin.Zhu <gavin.z@gmicloud.ai> Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com> Co-authored-by: tancheng33 <garrytancheng@gmail.com> Co-authored-by: dineshx29 <dinesh.b.offl@gmail.com> Co-authored-by: Kangyan Zhou <zky314343421@gmail.com>

…grapheme streaming Adds regression coverage for malformed nested reasoning tags and Unicode grapheme clusters split across streaming chunks. Chunk-invariance and streaming/full-input consistency tests are omitted because they are already covered by sgl-project#34458.

…rsing onto v0.5.16 Merged upstream fix (2026-08-12) for the streaming tool-call parser dropping text that precedes the DSML tag when it arrives in the same detokenizer increment as the tag (batched detok under speculative decoding). Previously the invoke loop returned normal_text="" once the buffer looked DSML-ish, silently discarding the preamble; non-streaming parsing kept it, so stream and non-stream disagreed. - deepseekv32_detector.py: recover preamble for the first tool call (back to the bot_token, same trailing-newline trim as detect_and_parse); multi-section non-streaming parse via findall; MalformedJSON tolerance; exception path re-emits verbatim instead of swallowing the turn. - Known upstream limitation (tracked by open sgl-project#34518/sgl-project#34600): prose between or after multiple tool-call sections stays buffered. - reasoning_parser.py hunks of sgl-project#34458 NOT ported: 0.5.16 maps deepseek-v4 to _DeepSeekV3Detector (no dsv4 subclass); different base structure.

hnyls2002 requested a review from JustinTong0323 as a code owner August 11, 2026 22:08

github-actions Bot added the deepseek label Aug 11, 2026

hnyls2002 and others added 2 commits August 11, 2026 15:16

trim comments; merge duplicate error-path test

d5a62e3

Merge branch 'main' into lsyin/dsv4-parser-fix

972ab28

github-actions Bot added the run-ci label Aug 11, 2026

hnyls2002 and others added 6 commits August 11, 2026 15:29

keep finish() dropping unfinished end token; cover held-back start token

63e6d19

revert multi-think-block handling; complete holdback with think_start

aeff452

flush held-back suffix on finish; longest-match holdback; drop half-f…

8d45e25

…ormed call on error

flush held-back suffix outside reasoning too; pin pre-existing think-…

bafdb7f

…prefix divergence

trim comments and pin docstrings; drop two redundant tests

6463452

glm45: flushed reasoning no longer carries the opening think tag

7da76e2

hnyls2002 merged commit 5899674 into main Aug 12, 2026
160 of 187 checks passed

hnyls2002 deleted the lsyin/dsv4-parser-fix branch August 12, 2026 03:03

This was referenced Aug 12, 2026

[Tests] Add regression tests for reasoning parser edge cases #34157

Closed

[Fix] deepseekv32 detector: tolerate partial non-string param values raising MalformedJSON (stream tear-down) #28684

Open

tancheng33 mentioned this pull request Aug 12, 2026

[Fix] Stream prose between and after DSML tool calls #34518

Open

Leoyzen mentioned this pull request Aug 12, 2026

[Fix] Harden DeepSeek-V4 tool-call streaming #34600

Open

5 tasks

This was referenced Aug 17, 2026

[Bug] Reasoning parsers: text before the opening think token is routed four different ways (chunk-dependent, and one-shot keeps the literal marker) #35083

Open

[Fix] Route text before the opening think token by position, not by index 0 #35085

Open

dinesh20073 mentioned this pull request Aug 17, 2026

[Tests] Add regression tests for malformed nested tags and Unicode grapheme streaming #35109

Open

ActiveSky mentioned this pull request Aug 20, 2026

[Bug] deepseekv32/deepseekv4 tool-call parsers drop calls when the streamed output has two chunks #35563

Closed

5 tasks

cheferrari mentioned this pull request Aug 24, 2026

[Bug] Streaming response truncates content when both content and tool_calls are present (SGLang 0.5.17) #34214

Open

5 tasks

Navigation Menu

Uh oh!

[Fix] Make DeepSeek-V4 reasoning and tool-call streaming parsing chunk-invariant - #34458

[Fix] Make DeepSeek-V4 reasoning and tool-call streaming parsing chunk-invariant#34458
hnyls2002 merged 10 commits into
mainfrom
lsyin/dsv4-parser-fix

hnyls2002 commented Aug 11, 2026 •

edited by github-actions Bot

Loading

Uh oh!

hnyls2002 commented Aug 11, 2026 •

edited

Loading

Uh oh!

hnyls2002 commented Aug 11, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

hnyls2002 commented Aug 11, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Background

Changes

Known divergences

Test plan

CI States

Uh oh!

hnyls2002 commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hnyls2002 commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

hnyls2002 commented Aug 11, 2026 •

edited by github-actions Bot

Loading

hnyls2002 commented Aug 11, 2026 •

edited

Loading

hnyls2002 commented Aug 11, 2026 •

edited

Loading

github-actions Bot commented Aug 11, 2026 •

edited

Loading

github-actions Bot commented Aug 11, 2026 •

edited

Loading

github-actions Bot commented Aug 11, 2026 •

edited

Loading