Add specialized tagged thinking tool parser

[bartdeboer]

Overview

This PR adds a specialized parser for tagged thinking/tool-call outputs.

It was motivated by testing Qwen3.5-9B with reasoning enabled and XML-style tool calls in an agent/tool-loop setup.

Two parser issues showed up:

1. A valid <tool_call> emitted while the parser was inside a <think>...</think> reasoning block was surfaced as reasoning_content instead of being parsed as a tool call.
2. Tagged function parameters such as <parameter=old_string> were required to appear in schema/property iteration order, even though the parameter name is already carried in-band by the tag.

Observed Qwen behavior

Qwen3.5 appears to use three practical response modes:

1. Pure reasoning messages

<think>
reasoning...
</think>

2. Tool-call messages, which may include reasoning before the tool call

reasoning before tool call...
<tool_call>
<function=...>
<parameter=...>...</parameter>
</function>
</tool_call>

3. Final-answer messages

Final answer...

The important detail is that Qwen does not always emit a complete <think>...</think> block for every response mode.

For pure reasoning messages, Qwen reliably emits/uses reasoning tags and closes with </think>.

For tool-call messages, Qwen may emit reasoning-like text before <tool_call>, but it may not emit a matching </think> before the tool call.

For final-answer messages, Qwen may emit plain assistant content without closing a template-prefilled <think> block.

The standard Qwen Jinja template helps by prefilling assistant generation with:

<think>

That works well for pure reasoning messages, because Qwen then emits:

reasoning...
</think>

However, for tool-call and final-answer messages, Qwen may effectively ignore that prefilled reasoning state. Some downstream template variants, such as no-prefill template fixes, remove this <think> prefill entirely.

So the parser should not rely only on whether the Jinja template prefilled <think>. It needs to classify the emitted completion itself.

Parser behavior added by this PR

This PR adds a specialized parser for the tagged thinking/tool-call protocol family used by Qwen-style templates:

<think>...</think>
<tool_call>
<function=name>
<parameter=arg>...</parameter>
</function>
</tool_call>

The parser kicks in when this tag family is detected in the template.

It classifies emitted output as follows:

• </think> marks preceding text as reasoning_content.
• <tool_call> marks preceding text as reasoning_content, even if no explicit <think> tag was emitted in the completion.
• <tool_call>...</tool_call> is parsed as a tool call.
• Text after a parsed tool call is parsed as assistant content.
• If no reasoning/tool tags are emitted, the output is parsed as final assistant content.

This makes the parser robust to both default Qwen templates that prefill <think> and no-prefill template variants.

Tagged parameter handling

For tagged tool calls, function argument names are already present in-band:

<parameter=old_string>...</parameter>
<parameter=new_string>...</parameter>

This PR parses tagged parameters by their emitted names rather than requiring schema/property iteration order.

It also allows string parameters to contain raw multiline content/code until the closing </parameter> tag.

This fixes Qwen-style tool calls such as edit/write calls where multiline code appears inside a string parameter.

Additional information

This was tested downstream with Qwen3.5-9B using reasoning enabled and XML-style tool calls.

A patched CUDA image used for downstream validation is available here:

ghcr.io/bartdeboer/llama-cpp:server-cuda-tagged-thinking-tools-168643697

Digest:

sha256:0b43e14146cb715ff8fa6960083ac9c18aae065eed197f87bc88d3c96f7e0240

Validated locally with:

./build/bin/test-chat --suppress-debug --template Qwen3.5
./build/bin/test-chat --suppress-debug

Related issues / prior art:

• Related to Improve tagged tool parsing with reasoning #23773
• Fixes Eval bug: Qwen3.5 9B often prints tool calls in XML and stops when thinking is enabled - tool calls inside thinking block #20837
• Fixes Eval bug: Qwen3.5 and 3.6 tool call emitted in reasoning_content instead of delta.tool_calls (GitHub CoPilot client) #22684
• Related to Eval bug: unsloth/Qwen3.5-35B-A3B-GGUF peg-native chat format parser fails when model outputs text before <tool_call> (thinking model + tool calling) #20260
• Related to Eval bug: Answer in think tags. Qwen 3.6 27B #22398
• Related to vLLM's Qwen3 reasoning parser behavior, where <tool_call> can act as a reasoning boundary.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - AI assistance was used for investigation and prototyping. I have a software engineering background, but I am not primarily a C++/llama.cpp contributor. I reviewed the changes, tested them locally and downstream, and am responsible for the submitted code.

[ggml-gh-bot]

Hi @bartdeboer, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[hugoduncan]

fwiw, this solved the described problem, which I was seeing when using Qwen3.6-27B

[bartdeboer]

Rebased this PR on current master and reran the chat parser tests:

./build/bin/test-chat --suppress-debug --template Qwen3.5
./build/bin/test-chat --suppress-debug

I also pushed a CUDA sm_86 validation image, matching my RTX 3080 Ti test machine:

ghcr.io/bartdeboer/llama-cpp:server-cuda-tagged-thinking-tools-2d8b21894

Digest: sha256:c0a579e65206122dfe455f98b1154f110a3ba8d6387040cb6fc4ffb82134ab6b
Image reports: version: 9821 (2d8b21894)

[Mushoz]

When can this PR be reviewed? It's solving a real issue with Qwen that currently makes it a pain to use in very long tasks, as it will stop when it emits the toolcall in its thinking trace.

[aldehir]

If masking the tool calling token during reasoning is not an adequate solution, which appears to be the case, then I'm onboard with this change (with some adjustments).

However, it's @pwilkin that you need to convince.

[bartdeboer]

Is there anything I can do to help move this forward? I don’t think this needs to come at the cost of the token-masking direction. @pwilkin?

[pwilkin]

Sorry, I've been thinking about this. I'm seriously wondering whether not to just add a flag to the autoparser to allow specifying models that permit tool calling within reasoning instead of a dedicated parser, as I know Kimi tends to exhibit similar behavior as well. However, I can as well do that in a followup PR not to stall this anymore.

[aldehir]

I'll follow up with some adjustments and we can get this merged in.

[vanbukin]

Any updates?

[aldehir]

The reasoning budget sampler needs to support multiple terminating sequences. Right now anyone who uses it would still be met with errors. I will work this in a separate PR and then we can integrate it into this one.

[bartdeboer]

Sounds good. Let me know if there’s anything I can do to assist.

[Slion]

Related to: QwenLM/Qwen3.8#150

[Slion]

Does not fix GHCP in VS Code on its own:

It does something though as the tool XML is now shown as is in the chat, later on it stops, not sure why though.

[aldehir]

@Slion some clients don't return back reasoning content from assistant responses, which is required by Qwen for agentic workflows. The Copilot agent in VS Code is one such client, which causes these errors. The remedy is to use a client that does or disable reasoning (which may also impact performance).

[Slion]

The Copilot agent in VS Code is one such client, which causes these errors.

You make it sound like it can't be fixed from llama.cpp.
I could fix similar issues when running that same model on vLLM using a Python proxy rewriting vLLM output.
See: QwenLM/Qwen3.8#150 (comment)

Now it looks like I fixed most llama.cpp issues with that patch:
master...Slion:llama.cpp:slion-qwen-copilot

With that patch my agent could run for over an hour uninterrupted and actually managed to fix my issues.
Reasoning worked brilliantly. Once a </think> tag slipped through but without consequence.
There are several issues that can lead to interruption and I'm not saying this fixes everything but it sure seems to be making it a great deal better. I reckon I'll be using that branch and keep patching it as needed over the next few weeks.

Here is my router mode models.ini:

version = 1

[*]
device = cuda0
host = 0.0.0.0
port = 8080
log-file = D:/Models/server.log
reasoning = on
;reasoning-format = deepseek-legacy
reasoning-format = deepseek
chat-template-kwargs = {"enable_thinking": false}
jinja = true

[qwen/qwen3.6-27b-Q8_0]
model = D:/Models/LM-Studio/lmstudio-community/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q8_0.gguf
mmproj = D:/Models/LM-Studio/lmstudio-community/Qwen3.6-27B-GGUF/mmproj-Qwen3.6-27B-BF16.gguf
;chat-template-kwargs = {"enable_thinking": false}
;chat-template-file = D:/Dev/HuggingFace/froggeric/Qwen-Fixed-Chat-Templates/chat_template.jinja
;reasoning = off
;c = 196608

[qwen/qwen3.6-27b-BF16]
model = D:/Models/LM-Studio/lmstudio-community/Qwen3.6-27B-GGUF/Qwen3.6-27B-BF16.gguf
mmproj = D:/Models/LM-Studio/lmstudio-community/Qwen3.6-27B-GGUF/mmproj-Qwen3.6-27B-BF16.gguf
;chat-template-kwargs = {"enable_thinking": false}
;chat-template-file = D:/Dev/HuggingFace/froggeric/Qwen-Fixed-Chat-Templates/chat_template.jinja
;reasoning = off
;c = 196608
spec-type = draft-mtp
spec-draft-n-max = 6

Here is the chatLanguageModels.json I used from VS Code Insider:

[
  {
	  "name": "llama.cpp",
	  "vendor": "customendpoint",
	  "apiType": "chat-completions",
	  "apiKey": "local",
	  "models": [
		  {
			  "id": "qwen/qwen3.6-27b-Q8_0",
			  "name": "llama.cpp - Qwen3.6-27B - Q8_0",
			  "url": "http://LINUX-PC:8080/v1",
			  "toolCalling": true,
			  "vision": true,
			  "maxInputTokens": 131072,
			  "maxOutputTokens": 65536,
			  "thinking": true,
			  "streaming": true
		  },
		  {
			  "id": "qwen/qwen3.6-27b-BF16",
			  "name": "llama.cpp - Qwen3.6-27B - BF16",
			  "url": "http://LINUX-PC:8080/v1",
			  "toolCalling": true,
			  "vision": true,
			  "maxInputTokens": 131072,
			  "maxOutputTokens": 65536,
			  "thinking": true,
			  "streaming": true
		  }
	  ]
  }
]

[aldehir]

You make it sound like it can't be fixed from llama.

This is a workaround to handle a prompt construction issue. I have no interest in adding workarounds to handle this unless you can show the issue is internal to llama.cpp. It doesn't change the fact that you are running a lobotomized version of the model.

[Slion]

Here is an AI generated explanations of the patch linked above, strangely I did not have to directly deal with tool call within think element for some reason. Maybe I just got lucky and will still have to fix that.

---

Qwen 3.6 GHCP Interruption Fix Summary

Scope

This note summarizes the interruption issues we hit while using Qwen 3.6 as a coding agent in VS Code GHCP, what symptoms we observed, and what changes were made to resolve them.

Primary commit reviewed:

• 6674ef2
• Title: Fix Qwen 3.6 GHCP VS Code agent interruptions

Files changed in that commit:

• common/reasoning-budget.cpp
• common/reasoning-budget.h
• common/sampling.cpp
• scripts/llama_log_fast_parse.py
• tools/server/server-schema.cpp
• tools/server/server-task.cpp
• tools/server/tests/unit/test_chat_completion.py
• tools/server/tests/utils.py

Interruption Patterns We Faced

1. Streaming protocol interruption at end of response

Observed behavior:

• GHCP stream terminated unexpectedly after a completion.
• Server emitted a trailing chat.completion.chunk with an empty choices array and usage payload.

Impact:

• Some OpenAI-compatible clients treat this as end-of-stream incompatibility and stop consuming deltas.

Fix:

• Keep usage on the terminal finish chunk instead of emitting a separate empty-choices chunk.
• File: tools/server/server-task.cpp

2. Reasoning-only output parsed as empty assistant message

Observed behavior:

• The model generated reasoning text, but parsed assistant content ended up empty and no tool calls were emitted.
• GHCP treated this as interrupted/useless output.

Fix:

• Enable reasoning_in_content compatibility for streamed deepseek mode, not only deepseek-legacy.
• Files: tools/server/server-schema.cpp, tools/server/server-task.cpp
• A follow-up refinement limited reasoning mirroring to final parse only (avoid partial token-by-token line splitting in the UI).

3. One-token immediate EOS interruption

Observed behavior:

• Only two chunks were emitted: assistant role start, then finish stop.
• completion_tokens was 1.
• Parsed message was empty.

Log signature:

• stopped by EOS
• next token was EOG
• parsed message: assistant with empty content

Root cause:

• Reasoning budget and grammar handoff could still allow early EOS while reasoning state was active.

Fixes:

• Replay matched reasoning end sequence into grammar when reasoning transitions to DONE.
• Add an EOS guard in sampler: if EOG is sampled while reasoning is still active, force reasoning budget to FORCING and resample once.
• Files: common/sampling.cpp, common/reasoning-budget.h, common/reasoning-budget.cpp

Post-Commit Local Adjustment

After commit 6674ef2, one local adjustment was added:

• common/reasoning-budget.cpp

Change:

• common_reasoning_budget_force now allows forcing from WAITING_UTF8 in addition to COUNTING.

Reason:

• The sampler EOS guard can trigger while reasoning state is WAITING_UTF8; forcing must remain possible in that state.

Diagnostics and Verification

Diagnostics helper added:

• scripts/llama_log_fast_parse.py

What it was used for:

• Fast extraction of latest completion IDs from large server.log files.
• Quick classification of interruption signatures:
  • one-token EOS stop
  • reasoning-only stream with empty parsed content
  • finish chunk shape and usage placement

Build and validation notes:

• llama-server builds succeeded after the main fixes.
• Some build attempts failed due to environment/runtime issues (for example locked ggml-base.dll or missing Node/OpenSSL prerequisites), not due to C++ compile errors in the interruption fixes.

Net Result

The fix set addresses three independent interruption classes:

• stream terminal chunk compatibility,
• reasoning visibility/content fallback behavior,
• early EOS during active reasoning budget.

This combination is intended to make Qwen 3.6 agent turns stable in GHCP while keeping changes narrow and model-safe.

[Slion]

@aldehir

This is a workaround to handle a prompt construction issue.

I don't have the pretention of thinking this could be merged as it is. I don't have the expertise to come up with some "proper fix". My goal is merely to get Qwen 3.6 agent working with Copilot. I'm happy keeping that patch on my fork and rebasing it as needed.

I have no interest in adding workarounds to handle this unless you can show the issue is internal to llama.cpp.

I'm not asking you to. Just thought it could help field experts understand the issues we are having with this use case and maybe come up with a "proper fix".

It doesn't change the fact that you are running a lobotomized version of the model.

How so? It's pretty stellar if you ask me. Reasoning and calling tools for over an hour at full context on an RTX 6K and actually fixing complex issues, all in full BF16 while keeping my office space warm and cosy 🔥 That was pretty glorious, certainly my best run ever with that setup.

[bartdeboer]

Thanks, this helped narrow the investigation.

I found a few parser edge cases in #24202 that could cause valid tagged tool calls to be missed and emitted as normal output/reasoning instead. I’m looking into whether those should be addressed in this PR.

I’m not sure yet whether these edge cases are the exact cause of the Copilot/VS Code screenshot, though.

@Slion could you share the raw assistant completion that produced the visible <tool_call> output, plus the tool schema for that request? With the exact raw message and schema, we can add a direct regression test and verify whether the parser covers it.

[aldehir]

Let's tone down the AI analysis and, instead, if you could capture the request/response payloads when these issues occur that would greatly help identify the core issue. I know most people don't leave a mitm proxy running, or the data is sensitive, so it's always hard to get good reproductions.

[Slion]

Sorry guys I don't think I kept all the server logs. They reset every time I restart the server.

Here is how I got an AI agent to investigate and fix my issues:

• In one VS Code instance: Work with Qwen agent on a real project until it interrupts
• In llama.cpp VS code instance: Ask an actually working agent to investigate and patch issues by looking at server logs
• Repeat

@aldehir If you tell me how to collect the logs you need I may try find the time and patience to get them for you. Privacy should not be an issue. I can test that on some projects I own, even if they are not open source, it's no big deal. The thing is, there are multiple issues at play here. So we may have to do a lot of back and forth.

The AI generated explanation above is actually quite good I find. Three interruption issues were found and fixed, see the 1., 2. and 3. bullet points above. Even I can make sense of some of that but I guess it is easier when you actually experience those issues and get a chance to look at the server logs too.

If you are interested, let me know and I'll open an issue for us to work on that.
I'll title it like: "Qwen 3.6 27B GHCP VS Code agent interruptions".

To be fair, any llama.cpp developer serious about getting this fixed should first of all have a setup capable of reproducing the issue and would certainly not need me to collect those logs. I know that can be a lot of work and a big ask. Thus the home made patch. For weeks now I've seen issues with the same interruption outcome from various inference engines. It just does not seem to be something AI developers are willing to address. Again I'm assuming GHCP harness is not being used that much. However, for people like myself, transitioning away from AI cloud services, it is the easier path forward.

In my above post, there is almost all you need to reproduce the issue. Only the BF16 model is missing but I just created it from the official Qwen safetensors using the Python script in this repo. The Q8 variant was downloaded from LM Studio. I also forgot to mention this was done in VS Code Insider.

[Slion]

@bartdeboer Sorry for hijacking your PR 😁 I had an initial patch that was based on it but I had a feeling some of that I did not need. So I started again from master.

[bartdeboer]

No problem, @Slion. We first need to figure out which output edge-case caused Qwen’s tool call to be emitted as normal content.

One possibility is that Qwen emitted the closing </think> tag after the tool call, although I’m not sure whether that matches your failure.

I created a more greedy version of my PR:
https://github.com/bartdeboer/llama.cpp/tree/qwen-style-tagged-parser-focused

Test image:
ghcr.io/bartdeboer/llama-cpp:server-cuda-tagged-thinking-tools-focused-862713d0f

Could you test it with your setup? If it fixes the raw tool-call output, we can check with @aldehir whether it makes sense to update this PR with those changes.

I’ve kept the existing PR stable for now, in case he already has work based on it.

[Slion]

We first need to figure out which output edge-case caused Qwen’s tool call to be emitted as normal content.

I appreciate your effort and contributions as it sets me up on the path to roll out my own patch but I won't be testing your new changes any time soon I'm afraid. This is very time consuming and it seems we may have different use cases too.

Mine is really just about getting Qwen agent actually working in GHCP with reasoning.
I've opened a new issue about it: #25887

As mentioned above I don't think you can efficiently work this out by having someone doing a blind fix and someone else testing it for him, that's just a waste of time for all parties:

To be fair, any llama.cpp developer serious about getting this fixed should first of all have a setup capable of reproducing the issue and would certainly not need me to collect those logs. I know that can be a lot of work and a big ask. Thus the home made patch. For weeks now I've seen issues with the same interruption outcome from various inference engines. It just does not seem to be something AI developers are willing to address. Again I'm assuming GHCP harness is not being used that much. However, for people like myself, transitioning away from AI cloud services, it is the easier path forward.

The way forward is to get a llama.cpp developer to buy in that use case, reproduce the issue locally and come up with a proper fix of his own. Meanwhile, I am happy to keep on using my own patch improve it and rebase it as needed.

[Slion]

strangely I did not have to directly deal with tool call within think element for some reason. Maybe I just got lucky and will still have to fix that.

That could be because I added the following to my config:

chat-template-kwargs = {"enable_thinking": false}

[Slion]

We first need to figure out which output edge-case caused Qwen’s tool call to be emitted as normal content.

Maybe it help for you to know that that screenshot above was before I added "enable_thinking": false to my config.

[bramstein]

I'm running into a similar issue. The Qwen3 Coder (Qwen3-Coder-30B-A3B-Instruct-UD-Q6_K_XL) model infrequently skips the opening <tool_call> and outputs only the function definition and closing </tool_call>:

<function=read>
<parameter=path>
README.md
</parameter>
</function>
</tool_call>

From the llama.cpp logs, the lazy grammar never triggers because it is waiting for <tool_call>, so the response is ultimately returned as assistant content instead of a parsed tool call.

This is an issue with the model/training, but it seems consistent enough that this PR could add this as a recoverable failure case. I'm happy to contribute a patch to this branch if desired, but also understand you have to draw the line somewhere.

[aldehir]

Please try out #26252. The implementation is significantly different than this PR, so I opened a new one.