feat: Add streaming support for Mistral v11 tool format

[sjuxax]

Follow-up to #19425
Fixes #20028

Purpose

Based on avigny's work in #19425, we substantially improve the Mistral tool calling parser to handle the tool call format in MistralTokenizer v11.

Test Plan

Used avigny's test suite attached to #19425, passes.

Test Result

Tested it in streaming on https://huggingface.co/jeffcookio/Mistral-Small-3.2-24B-Instruct-2506-awq-sym, works. Didn't test non-streaming or other checkpoints, so not sure if they work yet.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Summary of Changes

Hello @sjuxax, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces comprehensive streaming support for Mistral's v11 tool calling format by completely overhauling the existing tool parsing mechanism. The changes focus on robustly handling incremental output, dynamically adapting to the new format, and improving overall parsing efficiency, ensuring seamless tool integration during model generation.

Highlights

• Streaming Tool Call Support: I have significantly improved the Mistral tool calling parser to add robust streaming support for the new Mistral v11 tool format. This includes handling incremental parsing of tool names and arguments.
• Refactored Parsing Logic: The core streaming parsing logic within MistralToolParser has been completely re-architected. This involves introducing a new state machine (StreamingState enum) and dedicated state variables for more precise and efficient parsing of tool calls as they stream in.
• Dynamic Format Detection: The parser now dynamically detects whether the incoming tool call format is the traditional JSON array or the newer Mistral v11 format (e.g., ToolName{arguments}), adapting its parsing strategy accordingly.
• Performance Optimizations: I've implemented optimized regex patterns and caching mechanisms for JSON parsing to enhance the performance of tool call extraction during streaming.
• Comprehensive Testing: A new, extensive test suite has been added to validate both non-streaming and streaming tool call extraction for various scenarios, including single and multiple tool calls, different argument types, and the new v11 format.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces streaming support for the Mistral v11 tool format. While the refactoring to a state machine is a positive step, there are critical issues related to correctness and robustness, particularly in the v11 parsing logic and test coverage. Addressing these is essential for the stability of the new feature.

[PedroMiolaSilva]

@sjuxax hey! I've tested your solution here and it seems to be working, nice job!

For non-streaming, I think we can fix it by replacing

lines 510:535:

            # jsons is difficult
            try:
                if self.fn_name_regex:
                    matches = self.fn_name_regex.findall(tool_content)

                    function_call_arr = []
                    for match in matches:
                        fn_name = match[0]
                        args = match[1]

                        # fn_name is encoded outside serialized json dump
                        # only arguments are serialized
                        function_call_arr.append({
                            "name": fn_name,
                            "arguments": json.loads(args)
                        })
                else:
                    function_call_arr = json.loads(tool_content)
            except json.JSONDecodeError:
                # use a regex to find the part corresponding to the tool call.
                # NOTE: This use case should not happen if the model is trained
                # correctly. It's a easy possible fix so it's included, but
                # can be brittle for very complex / highly nested tool calls
                raw_tool_call = self.tool_call_regex.findall(tool_content)[0]
                function_call_arr = json.loads(raw_tool_call)

with this:

            #First, use the tool call token to split, and we discard the first item, because it is empty
            raw_tool_calls = model_output.split(self.bot_token)[1:] 
            function_call_arr = []
            for raw_tool_call in raw_tool_calls:
                tool_name = raw_tool_call.split("{")[0]
                tool_arguments_begin = raw_tool_call.find("{")
                tool_arguments = raw_tool_call[tool_arguments_begin:]
                function_call_arr.append({
                                        "name": tool_name,
                                        "arguments": json.loads(tool_arguments)
                })

I've commented this also on #19425, but mentioning here also!

[avigny]

@sjuxax
I've updated the tests for streaming extraction using the new format, in #19425
Feel free to cherrypick them if you want ;)

[aarnphm]

Can you fix these ruff error?

[sjuxax]

Addressed Gemini's comments with Sonnet/Opus. I've been using these changes on my Mistral3.1-rebase branch with success for the last week or so.

@avigny, will take a look at your tests and probably pull them in in place of the Opus-autobuilt ones tomorrow.

@PedroMiolaSilva, thanks for persistently posting that snippet. I'll test and pull it in tomorrow too.

[hibukipanim]

Here is a snippet which reproduces some errors with Mistral Small 3.2 with commit b521f50

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["location", "unit"]
        }
    }
}]

is_stream = False # <--- try also with True
out = client.chat.completions.create(
    model=client.models.list().data[0].id,
    messages=[{"role": "user", "content": "Where is colder tomorrow San Francisco or New York?"}],
    tools=tools,
    stream=is_stream,
    temperature=0,
)

if is_stream:
    for chunk in out:
        print(chunk.choices[0])
else:
    print(out.choices[0])

when I run it with is_stream=False than the tool-parser throws a JSONDecodeError.
when run with is_stream=True than it doesn't error but returns only the first tool (there are 2 here with temperature=0).
This snippet does work well with #19425 though

[PedroMiolaSilva]

hey @sjuxax ! I've followed your instructions from hugging face and tried to run with v0.9.2 but with your modified files:

#!/bin/bash
source .env
export MODEL_ID=jeffcookio/Mistral-Small-3.2-24B-Instruct-2506-awq-sym
export MODEL_ID_PORT=8000
export MODEL_ID_GPU=0

docker run \
--runtime nvidia \
-e VLLM_USE_V1=1 \
--gpus all \
--ipc=host \
-p "${MODEL_ID_PORT}:8000" \
--env "HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}" \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v "$(pwd):/app" \
-v "./mistral_tool_parser_stream.py:/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/tool_parsers/mistral_tool_parser.py" \
-v "./mistral3.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mistral3.py" \
-v "./registry.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py" \
-v "./pixtral.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/pixtral.py" \
-v ~/.cache/vllm/torch_compile_cache:/root/.cache/vllm/torch_compile_cache \
vllm/vllm-openai:v0.9.2 \
--model ${MODEL_ID} \
--tool-call-parser mistral \
--tokenizer_mode mistral \
--enable-auto-tool-choice \
--chat-template /app/template.jinja \
--max-model-len 32000 \
--gpu-memory-utilization 0.8

I've tried with and without the chat template and the same error shows up.

In general, it works almost always, but I've found an example that breaks something on serving_chat.py, and I'm trying to figure it out why.

This is the request example (this is a tool from Hubspot MCP):

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "jeffcookio/Mistral-Small-3.2-24B-Instruct-2506-awq-sym", "temperature": 0.1, "stream": true, "tools": [{"type": "function", "function": {"name": "hubspot-get-schemas", "description": "    🎯 Purpose:      1. Retrieves all custom object schemas defined in the HubSpot account.    🧭 Usage Guidance:      1. Before working with custom objects to understand available object types,         their properties, and associations.    📦 Returns:      1. Provides the objectTypeId and objectType for each schema.      2. These attributes should be used for this object type instead of custom in subsequent requests.  ", "parameters": {"type": "object", "properties": {}, "additionalProperties": false, "$schema": "http://json-schema.org/draft-07/schema#"}}}], "stream_options": {"include_usage": true}, "messages": [{"role": "system", "content": "You are a helpful assistant, that has access to tools. Use your tools when the asks for it."}, {"role": "user", "content": "use your tool to get the schemas"}]}'

And then the server goes:

INFO:     172.17.0.1:50018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-22 06:21:53 [async_llm.py:270] Added request chatcmpl-e3a1ca77194545df8897f963b85de9e1.
ERROR 07-22 06:21:53 [serving_chat.py:948] Error in chat completion stream generator.
ERROR 07-22 06:21:53 [serving_chat.py:948] Traceback (most recent call last):
ERROR 07-22 06:21:53 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 864, in chat_completion_stream_generator
ERROR 07-22 06:21:53 [serving_chat.py:948]     tool_parser.prev_tool_call_arr[index].get(
ERROR 07-22 06:21:53 [serving_chat.py:948]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
ERROR 07-22 06:21:53 [serving_chat.py:948] IndexError: list index out of range

Again, most tools work perfectly, so idk why this happens with this particular one. Any thoughts? Am I doing something wrong?

[avigny]

hey @sjuxax ! I've followed your instructions from hugging face and tried to run with v0.9.2 but with your modified files:

#!/bin/bash
source .env
export MODEL_ID=jeffcookio/Mistral-Small-3.2-24B-Instruct-2506-awq-sym
export MODEL_ID_PORT=8000
export MODEL_ID_GPU=0

docker run \
--runtime nvidia \
-e VLLM_USE_V1=1 \
--gpus all \
--ipc=host \
-p "${MODEL_ID_PORT}:8000" \
--env "HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}" \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v "$(pwd):/app" \
-v "./mistral_tool_parser_stream.py:/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/tool_parsers/mistral_tool_parser.py" \
-v "./mistral3.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mistral3.py" \
-v "./registry.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py" \
-v "./pixtral.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/pixtral.py" \
-v ~/.cache/vllm/torch_compile_cache:/root/.cache/vllm/torch_compile_cache \
vllm/vllm-openai:v0.9.2 \
--model ${MODEL_ID} \
--tool-call-parser mistral \
--tokenizer_mode mistral \
--enable-auto-tool-choice \
--chat-template /app/template.jinja \
--max-model-len 32000 \
--gpu-memory-utilization 0.8

I've tried with and without the chat template and the same error shows up.

In general, it works almost always, but I've found an example that breaks something on serving_chat.py, and I'm trying to figure it out why.

This is the request example (this is a tool from Hubspot MCP):

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "jeffcookio/Mistral-Small-3.2-24B-Instruct-2506-awq-sym", "temperature": 0.1, "stream": true, "tools": [{"type": "function", "function": {"name": "hubspot-get-schemas", "description": "    🎯 Purpose:      1. Retrieves all custom object schemas defined in the HubSpot account.    🧭 Usage Guidance:      1. Before working with custom objects to understand available object types,         their properties, and associations.    📦 Returns:      1. Provides the objectTypeId and objectType for each schema.      2. These attributes should be used for this object type instead of custom in subsequent requests.  ", "parameters": {"type": "object", "properties": {}, "additionalProperties": false, "$schema": "http://json-schema.org/draft-07/schema#"}}}], "stream_options": {"include_usage": true}, "messages": [{"role": "system", "content": "You are a helpful assistant, that has access to tools. Use your tools when the asks for it."}, {"role": "user", "content": "use your tool to get the schemas"}]}'

And then the server goes:

INFO:     172.17.0.1:50018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-22 06:21:53 [async_llm.py:270] Added request chatcmpl-e3a1ca77194545df8897f963b85de9e1.
ERROR 07-22 06:21:53 [serving_chat.py:948] Error in chat completion stream generator.
ERROR 07-22 06:21:53 [serving_chat.py:948] Traceback (most recent call last):
ERROR 07-22 06:21:53 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 864, in chat_completion_stream_generator
ERROR 07-22 06:21:53 [serving_chat.py:948]     tool_parser.prev_tool_call_arr[index].get(
ERROR 07-22 06:21:53 [serving_chat.py:948]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
ERROR 07-22 06:21:53 [serving_chat.py:948] IndexError: list index out of range

Again, most tools work perfectly, so idk why this happens with this particular one. Any thoughts? Am I doing something wrong?

here are where serving_chat.py inspects the internal state of tool parsers

vllm/vllm/entrypoints/openai/serving_chat.py

Lines 841 to 845 in ef4d46c

	if tool_parser:
	auto_tools_called = len(
	tool_parser.prev_tool_call_arr) > 0
	index = len(tool_parser.prev_tool_call_arr
	) - 1 if auto_tools_called else 0

vllm/vllm/entrypoints/openai/serving_chat.py

Line 864 in ef4d46c

tool_parser.prev_tool_call_arr[index].get(

The hack done for pythonic_tool_parser or llama4_pythonic_tool_parser should probably be done for mistral tool parser too.

vllm/vllm/entrypoints/openai/tool_parsers/pythonic_tool_parser.py

Lines 167 to 173 in ef4d46c

	# HACK: serving_chat.py inspects the internal state of tool parsers
	# when determining it's final streaming delta, automatically
	# adding autocompleted JSON.
	# These two lines avoid that nonsense while ensuring finish_reason
	# is set to tool_calls when at least one tool is called.
	if tool_deltas and not self.prev_tool_call_arr:
	self.prev_tool_call_arr = [{"arguments": {}}]

[gaby]

@aarnphm @sjuxax Any progress on merging this?

[Blake-Martin-code]

I have not seen any updates on this in some time. Is anyone actively working on this or is the gpt-oss PR #22386 taking priority? @aarnphm @sjuxax

[Rictus]

Hello, what's the status of this work ?

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!

[mergify]

Hi @sjuxax, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @sjuxax.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Khalid1G]

Hello, any update ? Mistral 3

[hmellor]

Thank you for the effort on improving Mistral tool call streaming! This work has been superseded by PRs #19425 and #30332, which were merged and address the Mistral tool parser v11 streaming support. The referenced issue #20028 is also closed. We are closing this PR accordingly. If you believe there are additional improvements not covered by those merged PRs, please feel free to open a new PR. Thanks for the contribution!