[Feature][V1] Support tool_choice: required when using Xgrammar as the StructuredOutputBackend.

[chaunceyjiang]

FIX #16887

Test

vllm serve stelterlab/Mistral-Small-24B-Instruct-2501-AWQ --tool-call-parser mistral   --enable-auto-tool-choice  --tokenizer-mode mistral --guided-decoding-backend xgrammar

from lmformatenforcer.external.jsonschemaobject import JsonSchemaObject

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description":
                        "The city to find the weather for, e.g. 'Vienna'",
                        "default": "Vienna",
                    },
                    "country": {
                        "type":
                        "string",
                        "description":
                        "The country that the city is in, e.g. 'Austria'",
                    },
                    "unit": {
                        "type": "string",
                        "description":
                        "The unit to fetch the temperature in",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["country", "unit"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_forecast",
            "description": "Get the weather forecast for a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description":
                        "The city to get the forecast for, e.g. 'Vienna'",
                        "default": "Vienna",
                    },
                    "country": {
                        "type":
                        "string",
                        "description":
                        "The country that the city is in, e.g. 'Austria'",
                    },
                    "days": {
                        "type":
                        "integer",
                        "description":
                        "Number of days to get the forecast for (1-7)",
                    },
                    "unit": {
                        "type": "string",
                        "description":
                        "The unit to fetch the temperature in",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["country", "days", "unit"],
            },
        },
    },
]

messages = [
    {
        "role": "user",
        "content": "Hi! How are you doing today?"
    },
    {
        "role": "assistant",
        "content": "I'm doing well! How can I help you?"
    },
    {
        "role":
        "user",
        "content":
        "Can you tell me what the current weather is in Berlin and the "\
        "forecast for the next 5 days, in fahrenheit?",
    },
]

# Non-streaming test
chat_completion = client.chat.completions.create(
    messages=messages,
    model='',
    tools=tools,
    tool_choice="required",
    # tool_choice="auto",
    # extra_body=dict(guided_decoding_backend="outlines"),
)
print("Chat completion response:")
print(f"Chat completion: {chat_completion}")
for choice in chat_completion.choices:
    if choice.message.tool_calls:
        print(
            f"Tool calls: {choice.message.tool_calls}")
    else:
        print("No tool calls found.")
assert chat_completion.choices[0].message.tool_calls is not None
assert len(chat_completion.choices[0].message.tool_calls) > 0

# python test.py
Chat completion response:
Chat completion: ChatCompletion(id='chatcmpl-e61a0dac3e9343cebc6243c8549a245b', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='a5Yaj3JNh', function=Function(arguments='{"city": "Berlin", "country": "Germany", "unit": "fahrenheit"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='K54xN6Hm0', function=Function(arguments='{"city": "Berlin", "country": "Germany", "days": 5, "unit": "fahrenheit"}', name='get_forecast'), type='function')], reasoning_content=None), stop_reason=None)], created=1746757583, model='stelterlab/Mistral-Small-24B-Instruct-2501-AWQ', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=71, prompt_tokens=384, total_tokens=455, completion_tokens_details=None, prompt_tokens_details=None), prompt_logprobs=None)
Tool calls: [ChatCompletionMessageToolCall(id='a5Yaj3JNh', function=Function(arguments='{"city": "Berlin", "country": "Germany", "unit": "fahrenheit"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='K54xN6Hm0', function=Function(arguments='{"city": "Berlin", "country": "Germany", "days": 5, "unit": "fahrenheit"}', name='get_forecast'), type='function')]

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[russellb]

Where is the tool calling fix exactly? Is it that a bug fix was needed in xgrammar 0.19?

[chaunceyjiang]

Where is the tool calling fix exactly?

Here. @russellb

vllm/vllm/entrypoints/openai/protocol.py

Lines 560 to 593 in 7d4aeda

	if self.tool_choice == "required":
	# Pydantic schema generation cannot be used since the JSON schema
	# has to be constructed for a specific instantiation of a tool list
	# so that parameters of a function are correctly generated
	# based on the chosen function name
	def get_tool_schema(tool: ChatCompletionToolsParam) -> dict:
	return {
	"properties": {
	"name": {
	"type": "string",
	"enum": [tool.function.name]
	},
	# parameters are always generated as '{}' in the final
	# output if they are missing from the request
	# (i.e. are None or '{}') so the schema is
	# updated to produce an empty object in that case
	"parameters": tool.function.parameters
	if tool.function.parameters else {
	"type": "object",
	"properties": {}
	}
	},
	"required": ["name", "parameters"]
	}
	json_schema = {
	"type": "array",
	"minItems": 1,
	"items": {
	"type": "object",
	"anyOf": [get_tool_schema(tool) for tool in self.tools]
	}
	}
	return json_schema

"tool_choice: required depends on minItems, but xgrammar v0.18 does not support it."

Is it that a bug fix was needed in xgrammar 0.19?

xgrammar 0.19 supports minItems.

[chaunceyjiang]

/cc @russellb @DarkLight1337 PTAL.

[russellb]

thanks!

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @chaunceyjiang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[chaunceyjiang]

Hi, @DarkLight1337 It seems that the failed CI checks are unrelated to my code. Could you help retry the CI?

[chaunceyjiang]

I think this can be merged. The failed CI seems to be environment-related. Other PRs have similar errors — for example, #18047 also failed on buildkite/ci/pr/distributed-tests.
@DarkLight1337