Skip to content

fix(deepseekv32): parse bare invoke blocks without function_calls wrapper - #23786

Closed
Kangyan-Zhou wants to merge 2 commits into
sgl-project:mainfrom
Kangyan-Zhou:fix_dsv32_toolcall
Closed

fix(deepseekv32): parse bare invoke blocks without function_calls wrapper#23786
Kangyan-Zhou wants to merge 2 commits into
sgl-project:mainfrom
Kangyan-Zhou:fix_dsv32_toolcall

Conversation

@Kangyan-Zhou

Copy link
Copy Markdown
Collaborator

Summary

The nightly 8-GPU H200 job fails on test_deepseek_v32_all_variants with tool_call (3/9) across all four V3.2 variants (run #24754179764, job 72423730579). The pattern is:

basic_format: ; required: ; specific: ; strict: list index out of range; multiturn: list index out of range; thinking: list index out of range

streaming, none, and parallel pass.

Root cause

PR #21593 introduced at_least_one=True structural_tag constraints for tool_choice="required"/named so the grammar forces the model to emit at least one match of the detector's structure_info. For DeepSeek-V3 that PR also updated structure_info() to embed the wrapper tokens (<|tool▁calls▁begin|>...<|tool▁calls▁end|>), but DeepSeek-V3.2's structure_info() was left pointing at the inner block only:

StructureInfo(
    begin=f'<|DSML|invoke name="{name}">',
    end="</|DSML|invoke>",
    trigger="<|DSML|invoke",
)

So under tool_choice="required" the constrained model emits a bare <|DSML|invoke>...</|DSML|invoke> with no surrounding <|DSML|function_calls>...</|DSML|function_calls> wrapper.

DeepSeekV32Detector.has_tool_call and parse_streaming_increment already accept the bare shape (matches both bot_token and <|DSML|invoke), but detect_and_parse (the non-streaming path) required a complete wrapper match and silently returned zero calls. That maps cleanly to the test failures:

  • basic_format / required / specificassert msg.tool_calls and len(msg.tool_calls) > 0 fails with no message because msg.tool_calls is None.
  • strict / multiturn / thinkingtool_calls[0] raises IndexError: list index out of range.
  • streaming passes because it goes through the streaming detector, which is already lenient.
  • parallel passes because it uses tool_choice="auto" and the model emits the wrapper naturally.

Change

Make detect_and_parse mirror the streaming path: find the earliest tool-call marker (bot_token or <|DSML|invoke), use wrapper-bounded content when a complete wrapper is present, otherwise scan invoke blocks directly from the remainder.

Test plan

  • Added three unit tests for the bare-invoke shape (XML params, JSON params, multiple invokes); each fails on main and passes with the fix.
  • pytest test/registered/unit/function_call/test_function_call_parser.py — 193/193 pass.
  • pytest test/registered/unit/function_call/test_function_call_parser.py::TestDeepSeekV32Detector — 10/10 pass (7 pre-existing + 3 new).
  • pre-commit run --files clean (black/ruff/isort/codespell).
  • 8-GPU H200 nightly test_deepseek_v32_all_variants (only available on the nightly runner; will verify once the next nightly runs against this branch, or by manually triggering after merge).

🤖 Generated with Claude Code

…pper

When tool_choice="required"/named triggers a structural_tag with
at_least_one=True (PR sgl-project#21593), the grammar forces the model to emit a
single match of the detector's structure_info. For DeepSeek-V3.2 that
match is the inner <|DSML|invoke>...</|DSML|invoke> only, with no
surrounding <|DSML|function_calls>...</|DSML|function_calls> wrapper.

`has_tool_call` and `parse_streaming_increment` already accept that
shape, but the non-streaming `detect_and_parse` required the wrapper
and silently returned zero calls, surfacing in the nightly 8-GPU H200
job as `tool_call (3/9)` failures across all four V3.2 variants
(`basic_format`/`required`/`specific` with empty asserts;
`strict`/`multiturn`/`thinking` with "list index out of range" from
indexing an empty `tool_calls`). `streaming` and `parallel` passed
because they don't go through this path.

Make `detect_and_parse` mirror the streaming path: scan from the
earliest tool-call marker (`bot_token` or `<|DSML|invoke`), prefer
wrapper-bounded content when a complete wrapper is present, otherwise
parse invoke blocks directly from the remainder.

Add three unit tests for the bare-invoke shapes (XML params, JSON
params, multiple invokes) — all three fail without the fix and pass
with it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the DeepSeek V3.2 function call detector to support both wrapped and bare function call formats, which is necessary when tool selection is forced. It also adds comprehensive unit tests for these new scenarios. The review feedback identifies a potential data loss issue where text appearing after a complete function call wrapper is discarded; a code suggestion was provided to correctly partition the text into normal and scan segments regardless of the wrapper's position.

Comment on lines +189 to +199
# Use the earliest tool-call marker as the boundary for normal_text.
marker_indices = [i for i in (bot_idx, invoke_idx) if i != -1]
start_idx = min(marker_indices)
normal_text = text[:start_idx].strip()

# Prefer the wrapper-bounded content if a complete wrapper is present;
# otherwise scan everything from the first marker onward for invoke blocks.
function_calls_match = re.search(self.function_calls_regex, text, re.DOTALL)
scan_text = (
function_calls_match.group(1) if function_calls_match else text[start_idx:]
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation of normal_text construction only captures text preceding the tool call markers. If a complete <|DSML|function_calls> wrapper is present, any text following the closing tag is discarded and lost from the output.

Additionally, the logic can be simplified by checking for the wrapper match first and using its boundaries to correctly partition the text into normal_text (everything outside the wrapper) and scan_text (everything inside).

Suggested change
# Use the earliest tool-call marker as the boundary for normal_text.
marker_indices = [i for i in (bot_idx, invoke_idx) if i != -1]
start_idx = min(marker_indices)
normal_text = text[:start_idx].strip()
# Prefer the wrapper-bounded content if a complete wrapper is present;
# otherwise scan everything from the first marker onward for invoke blocks.
function_calls_match = re.search(self.function_calls_regex, text, re.DOTALL)
scan_text = (
function_calls_match.group(1) if function_calls_match else text[start_idx:]
)
# Prefer the wrapper-bounded content if a complete wrapper is present.
# This allows capturing text both before and after the wrapper in normal_text.
function_calls_match = re.search(self.function_calls_regex, text, re.DOTALL)
if function_calls_match:
normal_text = (text[:function_calls_match.start()] + text[function_calls_match.end():]).strip()
scan_text = function_calls_match.group(1)
else:
# Use the earliest tool-call marker as the boundary for normal_text.
start_idx = min(i for i in (bot_idx, invoke_idx) if i != -1)
normal_text = text[:start_idx].strip()
scan_text = text[start_idx:]

…ing-text test

- Drop two comments that restated the next line, shorten the top comment
  to keep only the load-bearing "why bare exists" context.
- Add a test that puts assistant text after a bare invoke to lock in
  that the trailing text doesn't bleed into invoke content.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@hnyls2002

Copy link
Copy Markdown
Collaborator

Covered by #34458 (5899674). detect_and_parse now also parses every tool_calls section rather than only the first, and the streaming path handles bare invoke blocks.

You are credited as a co-author on that commit.

If I misread what this PR does and part of it is still missing, please rebase on main and reopen — happy to take it.

@hnyls2002 hnyls2002 closed this Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants