Skip to content

Fix tool call for Qwen3.5 #1300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ikawrakow merged 2 commits into from
Feb 23, 2026
Merged

Fix tool call for Qwen3.5 #1300

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
98bcff3
Fix tool call for Qwen3.5
sayap Feb 22, 2026
eff53f4
Fix the grammar for the subsequent parameters after the first one
sayap Feb 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 20 additions & 7 deletions common/chat.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1225,6 +1225,17 @@ static common_chat_params common_chat_params_init_qwen3_coder_xml(const common_c
data.prompt = apply(tmpl, params);
data.format = COMMON_CHAT_FORMAT_QWEN3_CODER_XML;

// Qwen3.5 and Step-3.5-Flash use the Qwen3 Coder tool calling with thinking
bool supports_reasoning = (tmpl.source().find("<think>") != std::string::npos);

if (supports_reasoning && string_ends_with(data.prompt, "<think>\n")) {
if (!params.enable_thinking) {
data.prompt += "</think>";

@ghost ghost Feb 22, 2026 •
edited by ghost
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still need to test it more but I have not had much success in no think mode by appending
<think>\n</think>. This might also be the fact that I'm using a small quant (IQ2_XS), but I did get repetition issues occasionally.

It's hard to replicate but I think I got better output by using <think>\n\n</think>\n\n as the original Jinja suggests.

https://huggingface.co/Qwen/Qwen3.5-397B-A17B/blob/main/chat_template.jinja

@ghost ghost Feb 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I haven't looked at the backend too much, but isn't this logic supposed to be delt with by the Jinja?

@sayap sayap Feb 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am actually not too sure what's the purpose of the if-block here. I know the else-block is needed to set thinking_forced_open to true, so that thinking goes under reasoning_content. Let me double check.

Also I just pushed a commit that fixes the grammar to handle tool calls with multiple parameters correctly. It slipped through my testing..

@ghost ghost Feb 22, 2026 •
edited by ghost
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Jinja is being used and the context ends with "<think>\n" then it means that thinking should be on, so the code under if (!params.enable_thinking) should never run anyway.

If you're not using Jinja but rather formatting at front end level with text completion, the formatting should be handled by the front end, not llama.cpp.

That code might add a sneaky extra unwanted token to text completion front end programmers, depending on how params.enable_thinking is set. I don't see the point of it, but I'm not deep into backend so idk.

@sayap sayap Feb 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Jinja is being used and the context ends with "\n" then it means that thinking should be on, so the code under if (!params.enable_thinking) should never run anyway.

You are right. So if I understand correctly, the if-block here should be for chat templates like https://huggingface.co/stepfun-ai/Step-3.5-Flash/blob/main/chat_template.jinja, which don't support the enable_thinking flag by itself. The if-block will then add a "</think>", as a best effort to disable reasoning.

@Ph0rk0z Ph0rk0z Feb 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was Stepfun-3.5-Q4_K_L so not that small.

@sayap sayap Feb 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I downloaded the IQ4_XS quant from https://huggingface.co/ubergarm/Step-3.5-Flash-GGUF/, which has the same chat template as https://huggingface.co/stepfun-ai/Step-3.5-Flash/blob/main/chat_template.jinja

Thinking seems to be working fine. And with the change in this PR, I can now set the "enable_thinking": false chat template kwargs, either as a llama-server cli flag or in the request body, to disable thinking.

@Ph0rk0z Ph0rk0z Feb 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should have downloaded ubergarm's quant, heh. Maybe I will just steal the template out of his metadata.

@sayap sayap Feb 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Stepfun-3.5-Q4_K_L quant is from Bartowski right? Looking at https://huggingface.co/bartowski/stepfun-ai_Step-3.5-Flash-GGUF/edit/main/stepfun-ai_Step-3.5-Flash-Q4_K_L/stepfun-ai_Step-3.5-Flash-Q4_K_L-00001-of-00004.gguf, the chat template is also identical with the official one from stepfun-ai, so it should be working fine with this PR branch.

This is how it looks like with a simple Q&A without tool:

$ ./build/bin/llama-server --host 127.0.0.1 --port 8080 \
  -m /tank/models/ubergarm/Step-3.5-Flash-GGUF/IQ4_XS/Step-3.5-Flash-IQ4_XS-00001-of-00004.gguf \
  -ngl 999 -cmoe --no-mmap -c 1024 \
  --jinja --chat-template-file /tank/models/stepfun-ai/Step-3.5-Flash/chat_template.jinja

...

$ curl -s 127.0.0.1:8080/v1/chat/completions -d '{"messages":[{"role": "user", "content": "who are you?"}]}' | jq .
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "reasoning_content": "Hmm, the user is asking a straightforward \"who are you?\" question. This is a common initial query when someone encounters an AI assistant for the first time. \n\nI should provide a clear and concise introduction that establishes my identity, capabilities, and limitations. The user likely wants to understand what they can expect from interacting with me. \n\nI'll start by stating my name and developer, then summarize my core functions in a simple list format for readability. It's important to mention both what I can do and what I cannot do (like accessing real-time data) to set proper expectations. \n\nThe tone should be friendly and welcoming while remaining professional. I'll end with an invitation for the user to ask questions, encouraging further interaction.\n",
        "content": "I'm **Step**, a large language model developed by **StepFun** (also known as 阶跃星辰).  \n\nI’m designed to understand and generate natural language, perform logical and visual reasoning, answer questions, assist with creative writing, coding, math, document analysis, and more. My goal is to be **honest, helpful, respectful of privacy**, and to provide positive and reliable interactions.\n\nI’m **not affiliated** with other companies or models such as OpenAI’s ChatGPT, Anthropic’s Claude, Baidu’s 文心一言, or Alibaba’s 通义千问 — I’m independently developed by StepFun.\n\nHow can I help you today? 😊"
      }
    }
  ],
  "created": 1771809983,
  "model": "Step-3.5-Flash-IQ4_XS-00001-of-00004.gguf",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 296,
    "prompt_tokens": 16,
    "total_tokens": 312
  },
  ...
}

@sayap sayap Feb 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this is how it looks like with thinking disabled in the request body:

$ curl -s 127.0.0.1:8080/v1/chat/completions -d '{"messages":[{"role": "user", "content": "who are you?"}], "chat_template_kwargs": {"enable_thinking": false}}' | jq .
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! I am Step, a large language model developed by StepFun. I'm designed to understand and generate natural language, perform logical and mathematical reasoning, and assist with a wide range of tasks — from answering questions to creative writing, coding, and more. I aim to be helpful, honest, and respectful while maintaining a friendly and positive tone. If you have any questions or need assistance, feel free to ask!"
      }
    }
  ],
  "created": 1771810347,
  "model": "Step-3.5-Flash-IQ4_XS-00001-of-00004.gguf",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 86,
    "prompt_tokens": 17,
    "total_tokens": 103
  },
  ...
}

The prompt tokens went from 16 to 17, as this particular line adds the </think> token to the prompt.

} else {
data.thinking_forced_open = true;
}
}

data.preserved_tokens = {
"<tool_call>",
"</tool_call>",
Expand All @@ -1234,16 +1245,20 @@ static common_chat_params common_chat_params_init_qwen3_coder_xml(const common_c
"</parameter>",
};

if (supports_reasoning) {
data.preserved_tokens.insert(data.preserved_tokens.end(), {"<think>", "</think>"});
}

// build grammar for tool call
static const xml_tool_call_format form {
/* form.scope_start = */ "<tool_call>\n",
/* form.tool_start = */ "<function=",
/* form.scope_start = */ "",
/* form.tool_start = */ "\n<tool_call>\n<function=",
/* form.tool_sep = */ ">\n",
/* form.key_start = */ "<parameter=",
/* form.key_val_sep = */ ">\n",
/* form.val_end = */ "\n</parameter>\n",
/* form.tool_end = */ "</function>\n",
/* form.scope_end = */ "</tool_call>",
/* form.val_end = */ "\n</parameter>",
/* form.tool_end = */ "\n</function>\n</tool_call>",
/* form.scope_end = */ "",
};
build_grammar_xml_tool_call(data, params.tools, form);

Expand Down Expand Up @@ -2064,9 +2079,7 @@ static common_chat_params common_chat_templates_apply_jinja(
// Detect via explicit XML markers unique to Qwen3-Coder to avoid false positives in other templates.
// Require presence of <tool_call>, <function=...>, and <parameter=...> blocks.
if (src.find("<tool_call>") != std::string::npos &&
src.find("<function>") != std::string::npos &&
src.find("<function=") != std::string::npos &&
src.find("<parameters>") != std::string::npos &&
src.find("<parameter=") != std::string::npos) {
return common_chat_params_init_qwen3_coder_xml(tmpl, params);
}
Expand Down