Skip to content

chat : tolerate dropped tool-call wrapper in tag-based formats - #26244

Closed
jamincollins wants to merge 1 commit into
ggml-org:masterfrom
jamincollins:fix/tool-call-wrapper-optional
Closed

chat : tolerate dropped tool-call wrapper in tag-based formats#26244
jamincollins wants to merge 1 commit into
ggml-org:masterfrom
jamincollins:fix/tool-call-wrapper-optional

Conversation

@jamincollins

@jamincollins jamincollins commented Jul 28, 2026

Copy link
Copy Markdown

Overview

Several chat templates use a tagged tool-call format where <tool_call> is the wrapper around an inner <function=name>...</function> block (Qwen3-Coder, Nemotron, Seed-OSS, and others). Testing found that some models (see below) sometimes emit the well-formed inner block but skip the outer wrapper. Before this change, that was rejected entirely - the whole response fell back to plain text, dropping the tool call.

What changed (common/chat-auto-parser-generator.cpp):

  • The wrapper's open/close tags are now optional in the grammar rather than mandatory - the inner function tag stays mandatory, so this can't misfire on unrelated text
  • The scan that looks for "where does the tool call start" now also checks the inner function tag as a fallback, not just the wrapper, so a dropped wrapper doesn't cause the whole response to be swallowed as content
  • The same leniency was applied to both of the two sibling code paths that share this exact pattern (tag_tagged and tag_json formats), for consistency

Test coverage (tests/test-chat.cpp): five new cases covering the dropped-wrapper scenario, an asymmetric case (open tag missing, close tag present), and a parallel-calls case where only one of two calls is missing its wrapper - across both affected code paths.

Additional information

  • Model: unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF
  • Setup: served via llama-server with --jinja (native template handling), dual NVIDIA GPU (RTX 3080 + RTX 3060, 12GB each)
  • Failure rate: measured over batches of 20-30 identical requests - roughly 1 in 3 requests (~33%) failed before the fix
  • After the fix: 0 failures across 70 trials of the same request

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - AI-assisted diagnosis and implementation, with my own validation (against my system and the project's test suite)

Assisted-by: Claude Sonnet
Signed-off-by: Jamin W. Collins <jamin.collins@gmail.com>
@jamincollins
jamincollins requested review from a team and pwilkin as code owners July 28, 2026 20:37
@github-actions github-actions Bot added the testing Everything test related label Jul 28, 2026
@ggml-gh-bot

ggml-gh-bot Bot commented Jul 28, 2026

Copy link
Copy Markdown

Hi @jamincollins, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

@aldehir

aldehir commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Handled in #26252

@aldehir aldehir closed this Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Everything test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants