Autoparser - complete refactoring of parser architecture

[firecoperana]

Port the new Autoparser and optional argument reshuffle capability PR from mainline

ggml-org/llama.cpp#18675 and ggml-org/llama.cpp#20171

Continues #1369

@ikawrakow Can you merge the PEG parser PR first and then this one? This is a large PR and I don't want to squash them into one commit.

[firecoperana]

Streaming for tool call is enabled.

[ikawrakow]

@ikawrakow Can you merge the PEG parser PR first and then this one? This is a large PR and I don't want to squash them into one commit.

So is the PEG parser PR. Are we confident that there are no regressions?

[firecoperana]

I see some new issues with auto parser mentioned in mainline. In PEG parser PR, PEG is just added without being used. It's mostly the new jinja template engine changes that matters. It's been added there for a while, so most bugs should have been fixed.

[hksdpc255]

These changes in mainline llama.cpp appear to work well. All models except MiroThinker (which was added by me) have already been tested by upstream developers.

In principle, this should not introduce regressions, unless there are additional unmerged differences between mainline and ik_llama.cpp. I'll test this branch.

[hksdpc255]

This breaks MiroThinker, but seems an upstream bug

log:

error: the supplied chat template is not supported: /var/llm/MiroThinker.jinja
error: invalid parameter for argument: --chat-template-file
common_chat_templates_init: error: Index 1 out of bounds for array of size 0
common_chat_templates_init: failed to initialize chat template
common_chat_templates_init: please consider disabling jinja via --no-jinja, or using another chat template
common_chat_verify_template: failed to apply template: std::exception

[firecoperana]

There are more issues showing up in mainline. I will wait before they are full resolved.

[sayap]

In examples/server/server-common.cpp, can we change parallel_tool_calls to default to true? It is harder than I thought to try setting this as a request parameter in various agentic coding tools 😬

This capability is from late 2024, so all the newer models should already support this.

[firecoperana]

I could add command line arg to enable it.

[hksdpc255]

@firecoperana I'm planning to send PRs to upstream llama.cpp adding support for MiroThinker with the Refactored chat template. I have tested this patch will work. You can merge it.

diff --git a/common/chat.cpp b/common/chat.cpp
index b799912a..7a76f8a9 100644
--- a/common/chat.cpp
+++ b/common/chat.cpp
@@ -1278,6 +1278,116 @@ static common_chat_params common_chat_params_init_kimi_k2(const common_chat_temp
     return data;
 }
 
+// MiroThinker - uses MCP style toolcalling
+static common_chat_params common_chat_params_init_mirothinker(const common_chat_template &    tmpl,
+                                                          const autoparser::templates_params & inputs) {
+    common_chat_params data;
+
+    data.prompt             = common_chat_template_direct_apply(tmpl, inputs);
+    data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
+    data.supports_thinking  = true;
+    data.thinking_start_tag = "<think>";
+    data.thinking_end_tag   = "</think>";
+    data.preserved_tokens  = {
+        "<think>",
+        "</think>",
+    };
+
+    auto has_tools         = inputs.tools.is_array() && !inputs.tools.empty();
+    auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
+    auto include_grammar   = has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE;
+
+    auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
+        // MiroThinker Thinking format:
+        // - Reasoning: <think>{reasoning}</think>
+        // - Content: text after reasoning
+        // - Tool calls section:
+        //   <use_mcp_tool>
+        //   <server_name>{server_name}</server_name>
+        //   <tool_name>{tool_name}</tool_name>
+        //   <arguments>
+        //   {json_args}
+        //   </arguments>
+        //   ...
+        //   </use_mcp_tool>
+
+        auto reasoning = extract_reasoning ? p.optional("<think>" + p.reasoning(p.until("</think>")) + "</think>") : p.eps();
+
+        // Tool call markers
+        const std::string SECTION_BEGIN = "<use_mcp_tool>";
+        const std::string SECTION_END   = "</use_mcp_tool>";
+        const std::string CALL_BEGIN    = "<server_name>";
+        const std::string ARGS_BEGIN    = "<arguments>";
+        const std::string CALL_END      = "</arguments>";
+
+        auto end = p.end();
+
+        // Content only parser (no tools)
+        if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
+            return reasoning + p.content(p.rest()) + end;
+        }
+
+        // Build tool call parsers for each available function
+        // Function name format is: <tool_name>{tool_name}</tool_name>
+        // We need to match: {what_ever}</server_name>{spaces}<tool_name>{tool_name}</tool_name>
+        auto tool_choice = p.choice();
+        foreach_function(inputs.tools, [&](const json & tool) {
+            const auto & function = tool.at("function");
+            std::string  name     = function.at("name");
+            const auto & schema   = function.at("parameters");
+
+            // Match: {what_ever}</server_name>{spaces}<tool_name>{tool_name}</tool_name>
+            auto tool_parser = p.tool(
+                p.tool_open(
+                    p.until("</server_name>") +
+                    p.literal("</server_name>") +
+                    p.space() +
+                    p.literal("<tool_name>") +
+                    p.tool_name(p.literal(name)) +
+                    p.literal(ARGS_BEGIN)
+                ) + p.space() +
+                p.tool_args(p.schema(p.json(), "tool-" + name + "-schema", schema)) +
+                p.space() + p.tool_close(p.literal(CALL_END))
+            );
+
+            tool_choice |= p.rule("tool-" + name, tool_parser);
+        });
+
+        // Tool calls section: <use_mcp_tool> tool_calls </use_mcp_tool>
+        auto min_calls  = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED ? 1 : 0;
+        auto max_calls  = inputs.parallel_tool_calls ? -1 : 1;
+        auto tool_calls = p.trigger_rule("tool-calls",
+            p.literal(SECTION_BEGIN) + p.space() +
+            p.rule("tool-call", p.repeat(CALL_BEGIN + tool_choice, min_calls, max_calls) +
+            p.space() + p.literal(SECTION_END))
+        );
+
+        auto content_before_tools = p.content(p.until(SECTION_BEGIN));
+
+        return reasoning + content_before_tools + tool_calls + end;
+    });
+
+    data.parser = parser.save();
+
+    if (include_grammar) {
+        data.grammar_lazy = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO;
+        data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
+            foreach_function(inputs.tools, [&](const json & tool) {
+                const auto & function = tool.at("function");
+                auto         schema   = function.at("parameters");
+                builder.resolve_refs(schema);
+            });
+            parser.build_grammar(builder, data.grammar_lazy);
+        });
+
+        data.grammar_triggers = {
+            { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, "<use_mcp_tool>" }
+        };
+    }
+
+    return data;
+}
+
 // LFM2 format:
 // - Reasoning: <think>{reasoning}</think> (optional, only if enable_thinking is true)
 // - Content: text after reasoning (optional)
@@ -1517,6 +1627,14 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
         return common_chat_params_init_kimi_k2(tmpl, params);
     }
 
+    // MiroThinker - uses MCP style toolcalling <use_mcp_tool> ... </use_mcp_tool>
+    // Detection: template has "</use_mcp_tool>" and "</server_name>"
+    if (src.find("</use_mcp_tool>") != std::string::npos &&
+        src.find("</server_name>") != std::string::npos) {
+        LOG_DBG("Using specialized template: MiroThinker\n");
+        return common_chat_params_init_mirothinker(tmpl, params);
+    }
+
     // LFM2 - uses <|tool_list_start|>/<|tool_list_end|> markers and <|tool_call_start|>[name(args)]<|tool_call_end|> format
     // Detection: template has "<|tool_list_start|>" and "<|tool_list_end|>" markers
     if (src.find("<|tool_list_start|>") != std::string::npos &&

[firecoperana]

Add support for MiroThinker with the new template engine as well as add --parallel-tool-calls to enable parallel tool calls.