fix: Responses API streaming tool call support for non-harmony models

[giulio-leone]

Purpose

Fix Responses API streaming tool call support for non-harmony models.

Fixes #36435

Problem

When streaming responses from models that use XML-based tool calling (e.g., Qwen3.5 with <think> tags), the Responses API does not emit proper response.function_call_arguments.delta / response.function_call_arguments.done events. Instead, raw tool call XML leaks into response.output_text.delta events, breaking any client that relies on structured function call streaming. This affects all non-harmony models that combine reasoning and tool calling.

Root Causes

1. Mutually exclusive parser chain: reasoning_parser and tool_parser are in an if/elif chain in _process_simple_streaming_events. When a model has both (e.g., Qwen3.5 with <think> tags), the elif tool_parser: branch is never reached, so tool calls are silently dropped.

2. Missing event conversion: No code existed to convert DeltaMessage.tool_calls into Responses API function call events. The original code had a # todo(kebe7jun) tool call support comment placeholder.

Changes

• Handle both reasoning and tool calls together — reasoning is processed first, then tool calls are processed after is_reasoning_end(), removing the mutual exclusion
• Handle tool-call-only models (no reasoning parser) as a separate path
• Emit ResponseFunctionCallArgumentsDeltaEvent / ResponseFunctionCallArgumentsDoneEvent via existing helpers
• Close message output items before function call events when content precedes tool calls
• Sync tool_streaming_state.current_output_index with the main output index to keep event ordering consistent
• Suppress ResponseTextDeltaEvent once tool calls are detected, preventing XML leakage into text events

Test Plan

Added two new unit tests in test_serving_responses.py:

• test_tool_only_stream_emits_function_call_events — verifies correct event sequence for models with only tool calling
• test_reasoning_then_tool_call_stream — verifies correct event sequence when reasoning precedes tool calls

All 18 existing tests in test_serving_responses.py continue to pass, confirming no regressions.

[gemini-code-assist]

Code Review

This pull request introduces a significant fix for streaming tool calls in the Responses API for non-harmony models. The changes correctly address the root causes described, such as handling reasoning and tool parsers in sequence and converting tool call deltas into the proper API events. The addition of new unit tests is also a great step towards ensuring correctness.

However, I've identified a critical issue with the implementation for emitting tool call done events, which appears to be fragile and is not covered by the new tests. Please see my detailed comment.

[gemini-code-assist]

The logic for emitting ResponseFunctionCallArgumentsDoneEvent is fragile and not adequately tested. It relies on the condition args_delta == "}" to detect the end of a tool call's arguments, which has several potential problems:

1. Brittleness: The arguments delta may contain more than just the closing brace (e.g., whitespace or be part of a larger string), causing the strict equality check to fail. This makes the logic highly dependent on a very specific and potentially unreliable output format from the parser.
2. Incomplete State Handling: If a new tool call starts before the previous one's arguments are closed with a }, the previous tool call item will be left in an in_progress state without a done event. The logic at lines 1492-1496 advances to a new item but doesn't finalize the previous one.
3. Insufficient Testing: The new tests in test_serving_responses.py do not cover this done event emission logic. The mocked argument deltas don't trigger this condition, and the mock parser's prev_tool_call_arr is not populated, which would also prevent the done event from being created.

This can lead to clients never receiving a done event for a tool call, which is a critical bug in a streaming API.

A more robust approach should be considered. For example, the parser could explicitly signal the completion of a tool call. A potential improvement could be to have the extract_tool_calls_streaming method return not just DeltaMessage, but a tuple (DeltaMessage, bool) where the boolean indicates if a tool call is complete. This would decouple the streaming logic from the internal parsing details of argument strings.

[mergify]

Hi @giulio-leone, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[chaunceyjiang]

see #29947

[giulio-leone]

Thanks @chaunceyjiang — I see #29947 addresses tool calling in the Responses API more comprehensively. Happy to close this in favor of that PR if the maintainers prefer. My fix is narrower (streaming-only text→function_call detection), so it could serve as a quick stopgap or be superseded by #29947.

[mergify]

Hi @giulio-leone, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[giulio-leone]

Friendly ping — CI is green, tests pass, rebased on latest. Ready for review whenever convenient. Happy to address any feedback. 🙏

[giulio-leone]

@chaunceyjiang Thanks for the pointer. I see #29947 adds Responses API streaming tool call support as well. My PR focuses specifically on the non-harmonized tool parser path (e.g. Hermes, Llama) where function.name and function.arguments are missing from delta chunks — the existing code only handled the harmonized path. Happy to close this if #29947 covers that case too, or we could coordinate to avoid duplication.

[mergify]

Hi @giulio-leone, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

Hi @giulio-leone, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[giulio-leone]

Refreshed the branch as a single signed commit to clear the DCO blocker and keep the latest fixes together. This push preserves the non-harmony Responses API tool-call streaming fix, the more robust function-call done-event emission, and the ruff/pre-commit cleanup in serving.py.

[giulio-leone]

Follow-up on the remaining docs status: the only failing check left is docs/readthedocs.org:vllm, and the raw RTD log shows it is being cancelled by docs/maybe_skip_pr_build.sh because the PR lacks the documentation or ready label. The refreshed branch itself now has pre-commit, DCO, Summary, and Meta Internal-Only Changes Check green on commit 41f1587725283ff1f9e4f4d5a988159d422d7124.

[giulio-leone]

Follow-up for the bot review: I tightened the new streaming regressions so they now populate prev_tool_call_arr and explicitly assert ResponseFunctionCallArgumentsDoneEvent emission in both paths:

• tool-only streaming
• reasoning-then-tool-call streaming

That directly covers the previously untested done event path and avoids relying only on delta assertions.

Local validation completed here:

• python -m py_compile tests/entrypoints/openai/test_serving_responses.py (Python 3.12)
• git diff --check

I also attempted targeted pytest execution, but this local macOS environment does not have the full vLLM test runtime provisioned and collection pulls torch/compilation dependencies outside the current setup. CI on the PR should now validate the strengthened regression coverage on the real project environment.

[qandrew]

thanks for putting this together! could you add an E2E test in test_simple.py also?

cc @chaunceyjiang

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @giulio-leone.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[giulio-leone]

Closing this PR as the feature has been superseded by #29947 (merged in commit 9fe404e), which implements tool/function call streaming support in the Responses API.

The upstream implementation covers the same core fixes:

• Removes the mutual exclusion between reasoning_parser and tool_parser
• Handles the reasoning-then-tool-call transition
• Emits proper ResponseFunctionCallArgumentsDeltaEvent / ResponseFunctionCallArgumentsDoneEvent

Thanks @chaunceyjiang for landing this! 🎉