Repository navigation
[Frontend] Support OpenAI Responses API namespace tools - #47024
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
This pull request has merge conflicts that must be resolved before it can be |
chaunceyjiang
left a comment
There was a problem hiding this comment.
Please fix the DCO and resolve the merge conflicts.
df432ee to
a75d108
Compare
chaunceyjiang
left a comment
There was a problem hiding this comment.
@zhongjing123, thanks for the PR.
I tested it locally with both Qwen3 and GLM-5.2. It works correctly with Qwen3, but GLM-5.2 is still not working as expected.
The test results are as follows:
Test Qwen3:
# vllm serve /mnt/data2/models/Qwen/Qwen3.5-35B-A3B --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --port 8000
(APIServer pid=2726521) INFO 06-30 12:08:59 [api_utils.py:339]
(APIServer pid=2726521) INFO 06-30 12:08:59 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2726521) INFO 06-30 12:08:59 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.23.1rc1.dev566+g064030efe.d20260630
(APIServer pid=2726521) INFO 06-30 12:08:59 [api_utils.py:339] █▄█▀ █ █ █ █ model /mnt/data2/models/Qwen/Qwen3.5-35B-A3B
(APIServer pid=2726521) INFO 06-30 12:08:59 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2726521) INFO 06-30 12:08:59 [api_utils.py:339]
...
....
...
(APIServer pid=2726521) INFO 06-30 12:10:56 [parser_manager.py:37] "auto" tool choice has been enabled.
(APIServer pid=2726521) WARNING 06-30 12:10:56 [model.py:1483] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 1.0, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2726521) INFO 06-30 12:10:56 [hf.py:548] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2726521) INFO 06-30 12:11:08 [base.py:236] Multi-modal warmup completed in 11.357s
(APIServer pid=2726521) INFO 06-30 12:11:08 [base.py:236] Readonly multi-modal warmup completed in 0.637s
(APIServer pid=2726521) INFO 06-30 12:11:08 [api_server.py:610] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:37] Available routes are:
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2726521) INFO 06-30 12:11:08 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2726521) INFO: Started server process [2726521]
(APIServer pid=2726521) INFO: Waiting for application startup.
(APIServer pid=2726521) INFO: Application startup completeclient:
curl -sS http://localhost:8000/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer EMPTY" \
-d '{
"model": "",
"input": "Use Computer Use to open Chrome, go to https://www.google.com, and search for \"vllm\".",
"tools": [
{
"type": "namespace",
"name": "mcp__computer_use",
"description": "Computer control tools.",
"tools": [
{
"type": "function",
"name": "get_app_state",
"description": "Get the current state of a desktop application. for example Google Chrome.",
"parameters": {
"type": "object",
"properties": {
"app": {
"type": "string",
"description": "Application name, for example Google Chrome."
}
},
"required": ["app"],
"additionalProperties": false
}
}
]
}
],
"tool_choice": "auto"
}' client response:
{
"id": "resp_b25caa0bc654f0fb",
"created_at": 1782792687,
"incomplete_details": null,
"instructions": null,
"metadata": null,
"model": "/mnt/data2/models/Qwen/Qwen3.5-35B-A3B",
"object": "response",
"output": [
{
"id": "rs_a45ffed629d84281",
"summary": [],
"type": "reasoning",
"content": [
{
"text": "The user wants me to use Computer Use to:\n1. Open Chrome\n2. Go to https://www.google.com\n3. Search for \"vllm\"\n\nI need to use the mcp__computer_use__get_app_state function to check the current state of Chrome. However, looking at the available function, it seems to be for getting the state of an application, not for controlling it (opening, navigating, etc.).\n\nLet me check what this function does - it says \"Get the current state of a desktop application. for example Google Chrome.\" This seems to be for checking status, not for controlling actions.\n\nActually, looking more carefully at the available tools, I only see this one function for getting app state, not for actually controlling the browser. This is unusual - typically Computer Use tools would have functions for opening apps, navigating, typing, clicking, etc.\n\nLet me proceed with what I have and get the current state of Chrome first to see what information is available.\n",
"type": "reasoning_text"
}
],
"encrypted_content": null,
"status": null
},
{
"arguments": "{\"app\": \"Google Chrome\"}",
"call_id": "chatcmpl-tool-b02e2831f486d13d",
"name": "get_app_state",
"type": "function_call",
"id": "fc_90b28a5144841ad4",
"namespace": "mcp__computer_use",
"status": "completed"
}
],
"parallel_tool_calls": true,
"temperature": 1.0,
"tool_choice": "auto",
"tools": [
{
"description": "Computer control tools.",
"name": "mcp__computer_use",
"tools": [
{
"name": "get_app_state",
"type": "function",
"defer_loading": null,
"description": "Get the current state of a desktop application. for example Google Chrome.",
"parameters": {
"type": "object",
"properties": {
"app": {
"type": "string",
"description": "Application name, for example Google Chrome."
}
},
"required": [
"app"
],
"additionalProperties": false
},
"strict": null
}
],
"type": "namespace"
}
],
"top_p": 0.95,
"background": false,
"max_output_tokens": 261804,
"max_tool_calls": null,
"previous_response_id": null,
"prompt": null,
"reasoning": null,
"service_tier": "auto",
"status": "completed",
"text": null,
"top_logprobs": null,
"truncation": "disabled",
"usage": {
"input_tokens": 340,
"input_tokens_details": {
"cached_tokens": 0,
"input_tokens_per_turn": [],
"cached_tokens_per_turn": []
},
"output_tokens": 241,
"output_tokens_details": {
"reasoning_tokens": 0,
"tool_output_tokens": 0,
"output_tokens_per_turn": [],
"tool_output_tokens_per_turn": []
},
"total_tokens": 581
},
"user": null,
"presence_penalty": 0.0,
"frequency_penalty": 0.0,
"kv_transfer_params": null,
"input_messages": null,
"output_messages": null
}
Test GLM5.2
vllm serve /mnt/data2/models/zai-org/GLM-5.2-FP8 -tp 8 --tool-call-parser glm47 --reasoning-parser glm45 --enable-auto-tool-choice --chat-template-content-format=string --speculative_config '{"method": "mtp", "num_speculative_tokens": 3}' --served-model-name zai-org/GLM-5.2-FP8 --max-model-len 272640
client response:
{
"id": "resp_905b7e31411253a1",
"created_at": 1782790793,
"incomplete_details": null,
"instructions": null,
"metadata": null,
"model": "zai-org/GLM-5.2-FP8",
"object": "response",
"output": [
{
"id": "rs_aa71b5f5df31abaa",
"summary": [],
"type": "reasoning",
"content": [
{
"text": "The user wants me to use Computer Use to open Chrome, navigate to Google, and search for \"vllm\". Let me start by checking the current state of Chrome.",
"type": "reasoning_text"
}
],
"encrypted_content": null,
"status": null
},
{
"id": "msg_8855550bf92e3917",
"content": [
{
"annotations": [],
"text": "I'll help you open Chrome, go to Google, and search for \"vllm\". Let me start by checking the current state of Google Chrome.<tool_call>mcp__computer_use__get_app_state<arg_key>app</arg_key><arg_value>Google Chrome</arg_value></tool_call>",
"type": "output_text",
"logprobs": null
}
],
"role": "assistant",
"status": "completed",
"type": "message",
"phase": null
}
],
"parallel_tool_calls": true,
"temperature": 1.0,
"tool_choice": "auto",
"tools": [
{
"description": "Computer control tools.",
"name": "mcp__computer_use",
"tools": [
{
"name": "get_app_state",
"type": "function",
"defer_loading": null,
"description": "Get the current state of a desktop application. for example Google Chrome.",
"parameters": {
"type": "object",
"properties": {
"app": {
"type": "string",
"description": "Application name, for example Google Chrome."
}
},
"required": [
"app"
],
"additionalProperties": false
},
"strict": null
}
],
"type": "namespace"
}
],
"top_p": 0.95,
"background": false,
"max_output_tokens": 272424,
"max_tool_calls": null,
"previous_response_id": null,
"prompt": null,
"reasoning": null,
"service_tier": "auto",
"status": "completed",
"text": null,
"top_logprobs": null,
"truncation": "disabled",
"usage": {
"input_tokens": 216,
"input_tokens_details": {
"cached_tokens": 0,
"input_tokens_per_turn": [],
"cached_tokens_per_turn": []
},
"output_tokens": 86,
"output_tokens_details": {
"reasoning_tokens": 0,
"tool_output_tokens": 0,
"output_tokens_per_turn": [],
"tool_output_tokens_per_turn": []
},
"total_tokens": 302
},
"user": null,
"presence_penalty": 0.0,
"frequency_penalty": 0.0,
"kv_transfer_params": null,
"input_messages": null,
"output_messages": null
}
The reason is that find_tool_properties and find_tool_name in vllm/tool_parsers/utils.py do not search for tools nested inside a namespace. You'll need to update them accordingly to support namespace tools as well.
| @@ -39,12 +39,89 @@ | |||
|
|
|||
| logger = init_logger(__name__) | |||
|
|
|||
| _NAMESPACE_TOOL_NAME_SEPARATOR = "__" | |||
There was a problem hiding this comment.
I think the changes in this file should be moved to vllm/tool_parsers/utils.py.
In addition, find_tool_properties and find_tool_name in vllm/tool_parsers/utils.py need to support namespace tools.
I think these two functions can be updated as follows:
- In
find_tool_name(), detect whentool.type == "namespace"and recursively inspect the nested functions. - Construct the internal function name using the existing Responses convention:
{namespace}__{nested_tool_name}. find_tool_properties()should also support this internal name and return the nested function'sparameters.properties.- It would be good to add a regression test as well: use a Responses namespace tool with the GLM-4.7 parser, and have the model output
<tool_call>mcp__computer_use__get_app_state.... Assert that it produces:
FunctionCall(name="mcp__computer_use__get_app_state", arguments='{"app":"Google Chrome"}')
and then verify thatbuild_response_output_items()maps it back to:
name="get_app_state"andnamespace="mcp__computer_use".
|
Hi @zhongjing123, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
1cdc63a to
a5118a2
Compare
|
Hi @zhongjing123, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
|
This pull request has merge conflicts that must be resolved before it can be |
|
Hi @zhongjing123, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
chaunceyjiang
left a comment
There was a problem hiding this comment.
Thanks!
I tested this locally with Qwen3.5 and GLM-5.2, and both are working correctly.
Could you add a corresponding E2E test in tests/entrypoints/openai/responses/test_function_call.py? You can submit a new PR to do this.
There was a problem hiding this comment.
| if isinstance(item, ResponseFunctionToolCall): | |
| tool_name = item.name | |
| if item.namespace: | |
| tool_name = flat_namespace_tool_name(item.namespace, item.name) | |
| tool_call = ChatCompletionMessageToolCallParam( | |
| id=item.call_id, | |
| function=FunctionCallTool( | |
| name=tool_name, | |
| arguments=item.arguments, | |
| ), | |
| type="function", | |
| ) |
| ] | ||
|
|
||
|
|
||
| def iter_response_function_tool_dicts( |
There was a problem hiding this comment.
diff --git a/vllm/tool_parsers/utils.py b/vllm/tool_parsers/utils.py
index 9752baf6f..95769bafd 100644
--- a/vllm/tool_parsers/utils.py
+++ b/vllm/tool_parsers/utils.py
@@ -205,9 +205,7 @@ def iter_response_function_tool_dicts(
) -> list[dict[str, Any]]:
function_tools: list[dict[str, Any]] = []
for tool in tools:
- if isinstance(tool, FunctionTool):
- function_tools.append(tool.model_dump())
- elif isinstance(tool, NamespaceTool):
+ if isinstance(tool, NamespaceTool):
namespace = tool.name
for namespaced_tool in tool.tools:
if namespaced_tool.type != "function":
@@ -217,6 +215,8 @@ def iter_response_function_tool_dicts(
namespace, namespaced_tool.name
)
function_tools.append(tool_dict)
+ else:
+ function_tools.append(tool.model_dump())
return function_toolSigned-off-by: zhongjing123 <jimzhong5193@gmail.com>
Signed-off-by: zhongjing123 <jimzhong5193@gmail.com>
Signed-off-by: zhongjing123 <jimzhong5193@gmail.com>
Signed-off-by: zhongjing123 <jimzhong5193@gmail.com>
Purpose
Fixes #46737.
Namespace tools are expanded to an internal
namespace__toolrepresentation for prompting and parsing, but the Responses API should expose the original tool name expected by clients.This PR introduces a consistent mapping between the external tool name and the internal namespace-qualified representation throughout the Responses pipeline. The mapping is applied when:
function_callhistory;This ensures compatibility with clients (such as Codex) while preserving the existing internal representation.
Test Plan
Run the Responses unit tests:
The added regression tests cover:
function_callinputs;Test Result
Completed:
python -m compileallpassed.git diff --checkpassed.