studio: tool calling for DeepSeek (R1/V3/V3.1), GLM 4.x, Kimi K2 on safetensors + MLX

[danielhanchen]

DRAFT - do not merge. Stacked on PR #5620 (Llama-3 / Mistral / Gemma 4 + healing parity). Validate against real models before merging.

[gemini-code-assist]

Code Review

This pull request adds tool call parsing support for DeepSeek (R1, V3, V3.1), GLM (4.5, 4.6, 4.7), and Kimi K2 models. The implementation includes new regex patterns, specialized parsing functions, and updates to the inference routing logic and test suites. Reviewers suggested making the DeepSeek R1 parser more robust by handling leading whitespace before JSON blocks and recommended moving the ast import to the top level to avoid performance overhead during streaming inference.

[gemini-code-assist]

The DeepSeek R1 path should skip any leading whitespace between the Markdown code fence and the start of the JSON object, similar to the V3 and Kimi parsers. This makes the parser more robust against variations in model output formatting.

Suggested change

	json_start = m.end()
	# Walk a balanced ``{`` even if the trailing fence is truncated.
	if json_start >= len(body) or body[json_start] != "{":
	json_start = m.end()
	# Skip any whitespace before the JSON.
	while json_start < len(body) and body[json_start] in " \t\n\r":
	json_start += 1
	# Walk a balanced { even if the trailing fence is truncated.
	if json_start >= len(body) or body[json_start] != "{":

[gemini-code-assist]

Importing ast inside nested loops (while and for) can impact performance, especially since this parser is called frequently during streaming inference. Move the import to the top of the file with the other imports.

[danielhanchen]

Validated this end to end on 4 B200 GPUs, one Studio per model family, running each safetensors model and its unsloth UD-Q4_K_XL GGUF through the same 6 prompts (weather, GitHub issue search, Python math, latest PyTorch version, Hacker News top story, hash plus date math). Identical generation params on both backends (temperature 0.7, top_p 0.8, top_k 20, min_p 0.0), 3 seeds each, so 18 runs per variant. Tool calls counted from the tool_start stream events.

Results, safetensors vs GGUF, tools fired out of 18:

• Llama-3.1-8B: 18/18 vs 18/18, both averaging 3.3 calls. Safetensors passed 5/6 of the deterministic code checks vs 2/6 for the GGUF.
• Qwen3-14B: 14/18 vs 15/18, essentially identical.
• Mistral-Small-3.2-24B: 0/18 vs 0/18. The model refuses to call tools in both backends with the same "I don't have the capability to access real-time information" answer, so this is model behaviour rather than the parser.
• gemma-3-27b-it: 0/18 vs 0/18. gemma-3 has no tool-calling chat template on either side, so neither backend can fire.

So on the families whose chat template actually advertises tools, the safetensors path now matches the GGUF. Where safetensors does not fire, the GGUF does not fire either, so there is no remaining backend gap.

I also cross-checked each family's parsing against llama.cpp common/chat.cpp, vLLM tool_parsers, and SGLang function_call. The fixes layered on top of the base parser:

• Llama-3.2 bare-JSON form {"name": ..., "parameters": ...} now fires even with no XML signal, since the skip-special-token stream drops <|python_tag|>.
• Mistral [CALL_ID] marker skipped and [THINK]...[/THINK] stripped before parsing.
• <function name="..."> attribute form recognised as a tool signal and stripped.
• GLM zero-argument inline calls parsed, and GLM string-value whitespace preserved.
• Kimi K2 calls parsed without the section wrapper, with malformed-JSON recovery continuing to later calls.

All of these pass on the safetensors tool-loop suite.

I confirmed the same behaviour in the Studio UI. On Qwen3-14B the "Search the web for the latest stable PyTorch release, then compute the sum of primes below 1000" prompt shows a "2 tool calls" badge, real web_search source chips, and the correct python result of 76,127. On Llama-3.1-8B the weather plus Fibonacci prompt returns real web_search source chips and runs the python tool.

One follow-up worth noting separately, unrelated to this parser: for Mistral the Studio swaps in the Unsloth mistral chat template, which does not render the tools schema (the model's native template does), so the safetensors path never advertises tools to Mistral. It does not change the comparison here since the GGUF also fires 0, but it is the reason a fix to Mistral tool calling will need the native template, not just the parser.

[danielhanchen]

Added gemma-4-E4B-it to the comparison, and this one was a real case of the GGUF being better: safetensors fired 0/18 while the GGUF fired 18/18. Three gaps, all fixed in 4127d1f:

1. The Studio swaps in the Unsloth gemma-4 chat template, which does not render the tools schema (the model's native template does), so the model never saw the tools. The text path now falls back to the native template when the override renders identically with and without tools. This also covers any other family whose override template drops tools (Mistral has the same template issue).
2. skip_special_tokens strips the <|tool_call> wrapper and the <|"|> string markers, so a streamed call arrives as a bare call:NAME{k:v, ...} with unquoted values. The parser now reads that form, keeping commas and braces inside a code value, normalising quotes, and stripping the leaked markup from the final answer.
3. Without a grammar the small E4B model loops on one call. The agentic loop now collapses exact-duplicate calls in a turn and forces a final answer after a turn that made no new progress. llama-server's lazy grammar prevents this loop on the GGUF side.

After the fix, safetensors vs GGUF over 18 runs each (same prompts, same generation params):

• tool fire rate: 18/18 (was 0/18) vs 18/18
• avg tool calls: 2.3 vs 2.2
• tool types: web_search 33, python 8 vs web_search 32, python 7
• code-answer correct: 0/6 vs 4/6
• avg latency: 81s vs 8s

Tool firing now matches the GGUF in both rate and the count and kind of tools called. Two gaps remain and are structural to running a small model without llama.cpp's grammar: the E4B model relays computed values into its final answer less reliably (0/6 vs 4/6 on the deterministic code checks) and takes more turns, so it is slower. In the Studio UI the model fires web_search against real weather sources and runs python, returning "The 30th Fibonacci number is: 832040".

Parser tests for the bare and stripped Gemma-4 form are included. The safetensors tool-loop suite is at 157 passing.

[danielhanchen]

One more validation, this time on a new MoE: Qwen3.6-35B-A3B safetensors vs its MTP-GGUF (unsloth/Qwen3.6-35B-A3B-MTP-GGUF, UD-Q4_K_XL). No code change needed here, the multi-format parser and the qwen3-thinking template already handle it, so I am just posting the numbers for the record.

Architecture is qwen3_5_moe (35B total, ~3B active, thinking). safetensors is 71.9 GB and the MTP-GGUF is 22.9 GB, both under the 90 GB budget. The MTP (multi-token-prediction) variant loads and runs fine through llama-server. Both the native and the Unsloth qwen3-thinking override template render tools (Hermes <tools> system block plus <tool_call>{json}). Ran at 32k context since the reasoning plus full-page web_search results overflow 8k.

safetensors vs MTP-GGUF, 18 runs each, same prompts and params:

• tool fire rate: 18/18 vs 18/18
• avg tool calls: 2.5 vs 4.2
• tool types: web_search 37, python 8 vs web_search 69, python 6
• code-answer correct: 6/6 vs 3/6
• avg latency: 165s vs 10s (the GGUF gets MTP speculative decoding, around 190 tok/s)

Both fire on every question, so there is no safetensors gap here. If anything the safetensors path is the more accurate one, passing all 6 deterministic code checks vs 3/6 for the GGUF, and it calls fewer tools (the MTP-GGUF over-searches, averaging about 11 web_search calls on the GitHub-issues question). Confirmed in the Studio UI: the MTP-GGUF fires web_search against real PyTorch sources and runs python, returning the correct sum of primes below 1000 (76,127).