Skip to content

fix(deepseek_v4): recover tool calls trapped by a dangling </think> (rebased) - #2

Closed
procr1337 wants to merge 59 commits into
mainfrom
dsv4-toolcall-workaround-HEAD
Closed

fix(deepseek_v4): recover tool calls trapped by a dangling </think> (rebased)#2
procr1337 wants to merge 59 commits into
mainfrom
dsv4-toolcall-workaround-HEAD

Conversation

@procr1337

@procr1337 procr1337 commented Jun 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Now upstream: vllm-project#45862

DeepSeek-V4-Flash occasionally omits the closing </think> before emitting a <｜DSML｜tool_calls> block. This is a model bug: the chat template / encoder explicitly instructs it to output its complete reasoning inside <think>... </think> BEFORE any tool calls or final response, and the model intermittently fails to follow that instruction. Reasoning extraction is gated on </think>, so when it is missing the entire tool-call block stays in the reasoning channel, the tool parser never sees it, and the call is silently lost (finish_reason=stop, empty content). This is malformed model output, not a parser/template bug: it reproduces on the upstream DeepSeek API for V4-Flash and is invariant to MTP/cutlass/concurrency. DeepSeek's own encoding README notes the model can emit malformed output and that production needs "additional error handling". See bench/dangling-think/ for the full investigation and repro tooling.

Workaround: a narrowly-scoped, opt-in recovery in DelegatingParser that promotes a tool-call block trapped in reasoning to a real tool call, framed explicitly as malformed-output recovery (NOT making the tool-call token a reasoning terminator).

  • ToolParser gains tool_call_start_token / tool_call_end_token and an opt-in recovers_tool_calls_in_reasoning flag, so the generic composition code keys off the declared interface instead of sniffing subclass internals. DeepSeekV32ToolParser (and thus V4) opts in.
  • Non-streaming: when auto tool-choice parsed zero tool calls and produced no content, and the reasoning tail contains a balanced start/end tool-call block, re-run the tool parser over it and keep the preceding prose as reasoning. The balance requirement avoids promoting half-formed output or a marker the model merely mentioned.
  • Streaming: watch the reasoning stream for the start marker, buffering partial markers across chunks so they never leak as reasoning; on a full marker, end reasoning right before it and route the remainder to the tool-call phase.
  • Tests exercise the composed reasoning+tool DelegatingParser (CPU-only, stub tokenizer) for both the non-streaming and streaming dangling-</think> shapes.

@procr1337 procr1337 changed the title fix(deepseek_v4): recover tool calls trapped by a dangling </think> fix(deepseek_v4): recover tool calls trapped by a dangling </think> (rebased) Jun 14, 2026
@procr1337
procr1337 force-pushed the dsv4-toolcall-workaround-HEAD branch from 0afb83c to d1751ce Compare June 15, 2026 18:54
@procr1337 procr1337 mentioned this pull request Jun 15, 2026
Merged
yewentao256 and others added 2 commits June 15, 2026 19:50
@procr1337
procr1337 force-pushed the dsv4-toolcall-workaround-HEAD branch from d1751ce to 7073690 Compare June 15, 2026 20:07
jeejeelee and others added 24 commits June 15, 2026 13:17
@jeejeelee @njhill @WoosukKwon
[Model Runner V2][Bugfix] Fix MRV2 LoRA warmup (vllm-project#35536)
eacff17 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
@factnn
Fix parallel_tool_calls: null treated as false instead of default true (
25ee659 
vllm-project#44955)

Signed-off-by: factnn <166481866+factnn@users.noreply.github.com>
@bbrowning @sfeng33
[Frontend] Replace legacy Gemma4 parsers with engine-based implementa…
76a373e 
…tion (vllm-project#45588)

Signed-off-by: Ben Browning <bbrownin@redhat.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
@llx-08 @njhill @ZJY0516
[Bugfix] Defer block freeing until in-flight steps finish under async…
d467a2a 
… scheduling + PD KV consumer (vllm-project#45357)

Signed-off-by: llx-08 <2596671364@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
@itayalroy
nixl_ep: Skip post-receive quantization for NVFP4 (vllm-project#45606)
ab8b0fe 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
@itayalroy
[EP] Query NIXL EP top-k index dtype (vllm-project#45298)
16e9117 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
@itayalroy
[EP] Enable DBO with NIXL EP (vllm-project#45275)
3afe659 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
@WoosukKwon
[DSV4][Minor] Fix supported KV cache dtypes (vllm-project#44892)
f4359a7 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
@xx-thomas
[Misc][Model] add io processor for query/document embeddings from Col…
b00e76f 
…BERT (jinaai/jina-colbert-v2) (vllm-project#45210)

Signed-off-by: thomas <thomas.varghese@columbia.edu>
@BugenZhao
[Rust Frontend] Support max_logprobs validation (vllm-project#45674)
3f65e21 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao
[Rust Frontend] Lower out-of-vocab validation to text layer (vllm-p…
f99260d 
…roject#45685)

Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@Isotr0py
[Multimodal] Add Qwen3-VL video loader (vllm-project#44412)
e3cfea2 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@mrn3088
[BugFix] Support async scheduling with prompt embeds for multimodal m…
2addbb9 
…odels (vllm-project#45673)

Signed-off-by: Ruinan Ma <r7ma3088@gmail.com>
@zhenwei-intel
[XPU] Fix Triton attn fp8/bf16 check failing (vllm-project#45758)
b8bd773 
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
@lucianommartins
[Bugfix][Gemma4] Fix offline parser truncation, adjust_request token …
6607a80 
…leak, and chat template sync (vllm-project#45553)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
@BugenZhao
[Rust Frontend] Require ModelConfig.vocab_size to be present (vllm-…
259ff89 
…project#45696)

Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@bbrowning
[Frontend] [Parser] Migrate Nemotron V3 to streaming parser engine (v…
f3858d5 
…llm-project#45755)

Signed-off-by: Ben Browning <bbrownin@redhat.com>
@gitbisector @claude @DarkLight1337
[Core] Use fastsafetensors ParallelLoader for weight loading (vllm-pr…
9d808e2 
…oject#40183)

Signed-off-by: Git Bisector <gitbisector@gmail.com>
Signed-off-by: gitbisector <gitbisector@gmail.com>
Signed-off-by: git bisector <gitbisector@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
@Bortlesboat @codex
Register parsed config classes before tokenizer init (vllm-project#40299
a9a8a32 
)

Signed-off-by: Bortlesboat <bortstheboat@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
@taneem-ibrahim
[Misc] Added validation for Cohere /v2/embed input field exclusivity (v…
81d8f4e 
…llm-project#45640)

Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
@DarkLight1337
[Cleanup] Remove dead env (vllm-project#45777)
9096659 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@thisisjimmyfb
[Bug Fix] Allow pinned memory for WSL2 (vllm-project#41496)
8bf3749 
Signed-off-by: Jimmy Lee <hirejimmylee@gmail.com>
@bigPYJ1151
[CPU] Support Gemma Diffusion (vllm-project#45690)
a7fdfee 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
@JOSH1024 @joshuaAbraham861 @orozery
[Bugfix] Prevent cuMemcpyBatchAsync segfault with MTP and KV offloadi…
7ad894c 
…ng (vllm-project#44784)

Signed-off-by: joshua <joshua.abraham@multicorewareinc.com>
Co-authored-by: joshua <joshua.abraham@multicorewareinc.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
HanHan009527 and others added 26 commits June 16, 2026 04:35
@HanHan009527
[KV Connector][Mooncake] Pipeline-parallel support for PD-disaggregat…
d53f459 
…ed serving with Mooncake connector (vllm-project#44528)

Signed-off-by: hanhan.hank <hanhan.hank@bytedance.com>
Signed-off-by: Hank Han <hanhan7630@outlook.com>
@yewentao256 @mergify
[Refactor] Remove Fp8OnlineLinearMethod as scheduled (vllm-project#…
3f53e21 
…45463)

Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
@amd-lalithnc @tlrmchlsmth @claude
[ZenCPU] Add zencpu Platform Runtime Logging and Docs (vllm-project#4…
405c7cf 
…2726)

Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
@AndreasKaratzas
[ROCm][CI] Gate incompatible HF references on Transformers v5 (vllm-p…
7e179e4 
…roject#41532)

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@mikekg
[Quant] Support modelopt_mixed on Ampere (SM80/SM86) (vllm-project#45306
a8c86ee 
)

Signed-off-by: Mike G <180722391+mikekg@users.noreply.github.com>
@netanel-haber @mergify
[Bugfix][MoE] Restore routed output unpadding before shared expert add (
c5e5c33 
vllm-project#45707)

Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
@fuscof-ibm @claude
[Perf] Add VLLM_TRITON_FORCE_FIRST_CONFIG to skip Triton autotuning (v…
ced32bb 
…llm-project#42425)

Signed-off-by: Francesco Fusco <ffu@zurich.ibm.com>
Co-authored-by: Claude <noreply@anthropic.com>
@MatthewBonanni @claude
[CI] Fix attention benchmark smoke test (vllm-project#45728)
040df8f 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
@tahsintunan
[Rust Frontend] Add CORS support (vllm-project#45753)
cca3365 
Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com>
@AjAnubolu @MatthewBonanni
[Bugfix] Fix FlashMLA sparse accuracy with topk_length and zero-init …
bf5149b 
…padding (vllm-project#36616)

Signed-off-by: AjAnubolu <anuboluajay@gmail.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
@xiaohongchen1991 @gmagogsfm
[Kernel][Helion][1/N] Add Helion kernel for rms_norm_per_block_quant (v…
ce3ef17 
…llm-project#36895)

Signed-off-by: Sean Chen <seachen@redhat.com>
Co-authored-by: Yanan Cao <gmagogsfm@gmail.com>
@carlyou
feat: MLA prefill enable FA4 fp8 output (vllm-project#43050)
eb04c76 
Signed-off-by: Carl You <4531192+carlyou@users.noreply.github.com>
@tuukkjs @DarkLight1337
[ROCm][Cleanup] Remove stale AITER FA hybrid KV-cache TODO (vllm-proj…
3d34f8c 
…ect#44178)

Signed-off-by: Tuukka Sarvi <tuukka.sarvi@amd.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
@abcd1927 @hmellor
[Model] Add HrmTextForCausalLM (Hierarchical Reasoning Model — Text) (v…
a52205b 
…llm-project#43098)

Signed-off-by: Wuyifei <wuyifei@me.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@CienetStingLin
Upgrade tpu-inference to v0.22.1 (vllm-project#45793)
506ec6d
@micah-wil
[ROCm][CI] Patch conftest to resolve occasional OOMs (vllm-project#45722
6f612fb 
)

Signed-off-by: Micah Williamson <micah.williamson@amd.com>
@yewentao256
[Model Runner V2] Enable GraniteMOE for MRv2 by default (vllm-project…
88a9cdd 
…#45461)

Signed-off-by: yewentao256 <zhyanwentao@126.com>
@xianbaoqian @claude
[Model] Remove Dots1ForCausalLM (vllm-project#45637)
89e8645 
Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
@Sunt-ing @Harry-Chen
[Bugfix][Core] Fall back when numactl --membind is blocked in constra…
c45f681 
…ined containers (vllm-project#45438)

Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
@kouroshHakha @claude
[KVConnector][MoRIIO] Allow overriding the advertised host IP (vllm-p…
188c687 
…roject#45488)

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Dao007forever @claude
[KV Connector][Mooncake] Add cache_prefix to namespace store keys (vl…
44b2512 
…lm-project#45767)

Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@chaunceyjiang
[Frontend] Add Streaming Parser Engine and new MinimaxM2 Parser (vllm…
f00e163 
…-project#45701)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
@alexbi29
[Bugfix] Fix Qwen3 prompt tool-call reasoning false positive (vllm-pr…
7d56717 
…oject#45763)

Signed-off-by: Alex Bilichenko <alexbi29@users.noreply.github.com>
Co-authored-by: Alex Bilichenko <alexbi29@users.noreply.github.com>
@jesse996 @njhill
[PERF] Fuse multi-group block table staged writes (vllm-project#44944)
8e27a9c 
Signed-off-by: jesse <szxfml@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
@hongxiayang
[ROCm][Quant] mxfp8 moe/linear gfx950 tuning for MiniMax-M3 (vllm-pro…
f2beaa8 
…ject#45725)

Signed-off-by: Hongxia Yang <hongxia.yang@amd.com>
@sfeng33
[Misc] Update Mergify tool-calling label (vllm-project#45853)
475a6ad 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
@procr1337
procr1337 force-pushed the dsv4-toolcall-workaround-HEAD branch from 7073690 to 8b9cca7 Compare June 16, 2026 19:58
@procr1337
[Bugfix] DeepSeek V4: recover tool calls within reasoning content
f651dba 
DeepSeek-V4-Flash occasionally omits the closing </think> before emitting a
<｜DSML｜tool_calls> block. This is a model bug: the chat template / encoder
explicitly instructs it to output its complete reasoning inside <think>...
</think> BEFORE any tool calls or final response, and the model intermittently
fails to follow that instruction. Reasoning extraction is gated on </think>, so
when it is missing the entire tool-call block stays in the reasoning channel,
the tool parser never sees it, and the call is silently lost
(finish_reason=stop, empty content). DeepSeek's own encoding README
notes the model can emit malformed output and that production needs "additional
error handling".

Workaround: a narrow, opt-in recovery in DelegatingParser that salvages a
tool-call block left in the reasoning stream and promotes it to a real tool
call. This is deliberately scoped as recovery from malformed model output:
</think> remains the only reasoning terminator; the tool-call marker is not
added to the reasoning grammar.

- ToolParser gains tool_call_start_token / tool_call_end_token and an opt-in
  recovers_tool_calls_in_reasoning flag, so the generic composition code keys
  off the declared interface instead of sniffing subclass internals.
  DeepSeekV32ToolParser (and thus V4) opts in.
- Non-streaming: when auto tool-choice parsed zero tool calls and produced no
  content, and the reasoning tail contains a *balanced* start/end tool-call
  block, re-run the tool parser over it and keep the preceding prose as
  reasoning. The balance requirement avoids promoting half-formed output or a
  marker the model merely mentioned.
- Streaming: watch the reasoning stream for the start marker, buffering partial
  markers across chunks so they never leak as reasoning; on a full marker, end
  reasoning right before it and route the remainder to the tool-call phase.
- Tests exercise the composed reasoning+tool DelegatingParser with a stub tokenizer
  for both the non-streaming and streaming missing-</think> shapes.

Co-authored-by: Claude Opus
Signed-off-by: Procr <193802945+procr1337@users.noreply.github.com>
@procr1337
procr1337 force-pushed the dsv4-toolcall-workaround-HEAD branch from 8b9cca7 to f651dba Compare June 16, 2026 20:16
@procr1337 procr1337 closed this Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.