fix: tool_choice="required" falls back to tool_parser for non-JSON formats (streaming + non-streaming)

[voipmonitor]

Summary

tool_choice="required" fails when the model produces non-JSON tool calls (e.g. Qwen3 XML format), even though a compatible --tool-call-parser (e.g. qwen3_coder) is configured. This affects both the non-streaming and streaming code paths.

Root cause

Both tool_choice="required" code paths bypass the configured tool_parser and use JSON-only parsing, while the tool_choice="auto" branches correctly use the tool parser.

Non-streaming path (engine/serving.py)

In _parse_tool_calls_from_content(), the code uses pydantic TypeAdapter JSON validation only. When the model produces XML-formatted tool calls (e.g. <tool_call>{"name": ...}</tool_call>), parsing fails with ValidationError. Note: #36841 recently changed this block to silently suppress ValidationError (fixing the crash), but this means non-JSON tool calls are now silently dropped instead of being routed to the tool parser.

Streaming path (chat_completion/serving.py)

Three issues in the streaming required branch:

1. tool_parsers not initialized — tool_parsers array is only created when tool_choice_auto is true. When tool_choice="required", it stays empty, so even if a parser is configured, it is never used.

2. if/else prevents tool parsing when reasoning ends — With MTP (multi-token prediction), the model can send the </think> reasoning-end token and the beginning of tool call content in the same chunk. The original if/else structure (reasoning processing vs. tool call processing) meant that the iteration that detected reasoning-end would skip tool call extraction entirely.

3. No tool_parser fallback for non-JSON formats — The required branch only has extract_tool_call_required_streaming() which expects JSON. Models using XML-style tool calls (like Qwen3 with qwen3_coder parser) fail silently.

Fix

Non-streaming (engine/serving.py)

Replace the contextlib.suppress(ValidationError) from #36841 with a try/except that preserves the crash-safety (content = content or "") while adding a fallback: when JSON parsing fails with ValidationError or JSONDecodeError, fall back to the configured tool_parser.extract_tool_calls() (when enable_auto_tools is true and a parser is available). This way both the crash fix from #36841 and the non-JSON fallback coexist.

Streaming (chat_completion/serving.py)

1. Initialize tool_parsers for required too — Create parser instances when tool_choice is "auto" OR "required".

2. Separate if statements — Change the if reasoning / else tool_parsing to two independent if blocks so both can execute in the same iteration when reasoning ends.

3. Dual parser approach — When a tool_parser is available, try it first (for XML/custom formats). If it has not detected tool calls yet, also try the JSON extract_tool_call_required_streaming() as fallback. This handles the non-deterministic output from MTP, where the same model may produce JSON in one request and XML in another.

Note: content alongside tool_calls

When using the tool_parser fallback for required, the parser may emit partial text as delta.content before it detects the tool call pattern (e.g., qwen3_coder looks for XML <tool_call> tags but the model outputs JSON [{"name":...). This means some SSE chunks may contain both content and tool_calls. Clients consuming tool_choice="required" responses should prioritize tool_calls over content when both are present in the stream — the content in that case is leaked parser buffer text, not a real text response.

Discussion point: Dual parser approach

Fix #3 is somewhat opinionated. We use it because with MTP (speculative decoding), the same model (Qwen3) non-deterministically produces either JSON or XML tool calls across different requests. The dual approach handles both formats reliably. However, if the vLLM team prefers a different strategy (e.g., always routing through the tool_parser for required), we are happy to adjust.

Testing

• Unit tests added in tests/tool_use/test_tool_choice_required_fallback.py
• Tested with Qwen3-Coder 32B + MTP (--tool-call-parser qwen3_coder --enable-auto-tool-choice):
  • 30/30 streaming requests with tool_choice="required" succeed
  • 10/10 agentic RAG pipeline queries (multi-turn tool calling) succeed
• Zero impact on JSON-based models (GLM, Llama, Hermes) — JSON path succeeds immediately, fallback never triggers

Files changed

• vllm/entrypoints/openai/engine/serving.py — Non-streaming fallback (replaces silent suppress from [Bugfix] Fix crash when tool_choice=required exceeds max_tokens #36841 with try/except + tool_parser fallback)
• vllm/entrypoints/openai/chat_completion/serving.py — Streaming fixes (3 changes)
• tests/tool_use/test_tool_choice_required_fallback.py — Unit tests (new)

[gemini-code-assist]

Code Review

This pull request effectively addresses a bug where tool_choice="required" would fail for non-JSON tool call formats. The fix, which involves adding a try...except block to fall back to the configured tool_parser, is logical and mirrors the behavior of tool_choice="auto". This is a good, low-risk solution. I have one suggestion to make the exception handling more specific and robust.

[gemini-code-assist]

Using except Exception: is too broad and can mask unexpected errors (e.g., KeyboardInterrupt). It's a best practice to catch more specific exceptions. In this case, TypeAdapter(...).validate_json() raises pydantic.ValidationError on failure. Catching this specific exception makes the code safer and the intent clearer. You will need to add from pydantic import ValidationError at the top of the file.

Suggested change

	except Exception:
	except ValidationError:

[mergify]

Hi @voipmonitor, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @voipmonitor, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @voipmonitor, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @voipmonitor, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10