Skip to content

[Bugfix] Clear conflicting structured outputs in strict tool calling#44134

Draft
alexeldeib wants to merge 1 commit into
vllm-project:mainfrom
alexeldeib:alex/fix-strict-tool-calling-structured-outputs
Draft

[Bugfix] Clear conflicting structured outputs in strict tool calling#44134
alexeldeib wants to merge 1 commit into
vllm-project:mainfrom
alexeldeib:alex/fix-strict-tool-calling-structured-outputs

Conversation

@alexeldeib
Copy link
Copy Markdown
Contributor

Purpose

ToolParser.adjust_request's strict structural-tag path (added in #40894, gated by VLLM_ENFORCE_STRICT_TOOL_CALLING) installs structural_tag on a pre-existing StructuredOutputsParams via in-place attribute assignment, and returns without nulling response_format. The in-place set bypasses StructuredOutputsParams.__post_init__, so the params retain a prior mutually-exclusive constraint (json/regex/choice/grammar/json_object, or one lowered from response_format) alongside the new structural_tag. On the next re-validation this trips the one-constraint invariant:

ValueError: You can only use one kind of structured outputs constraint but multiple are specified

So a strict-mode request that also carries a structured-output constraint or a response_format returns HTTP 400 when it should succeed. This affects any parser that installs a structural tag — currently DeepSeek-V4 and Qwen3-Coder via get_structural_tag. The env var is off by default; requests with no pre-existing constraint are unaffected.

Fix: rebuild structured_outputs with only the structural tag (preserving the whitespace / additional-properties knobs) and null response_format, mirroring Step 2 of the same method. The same "tool constraint wins, response_format dropped" resolution already exists in Step 2 and the DeepSeek-V3.2 override (#41178), and is the intent of the open auto-path fix #39969.

Test Plan

pytest tests/tool_use/test_strict_tool_calling_adjust_request.py

Added CPU-only tests (no model required): enable strict mode, give a parser a structural tag, and send tools together with a response_format or a structured_outputs.json constraint, for both tool_choice="auto" and tool_choice="required".

Test Result

  • Before: adjust_request leaves two constraints; to_sampling_params raises the ValueError above. The conflict tests fail.
  • After: structured_outputs holds only the structural tag, response_format is None, the caller's whitespace knobs are preserved. All tests pass; the no-pre-existing-constraint case passes either way.

Equivalently over HTTP, with strict mode on: a tool_choice="auto" request that also sets response_format returns 400 before this change and a normal tool call after; a required-tool request is unaffected (that path already rebuilds).


Essential Elements of an Effective PR Description Checklist

ToolParser.adjust_request's strict structural-tag path (added in vllm-project#40894, gated by
VLLM_ENFORCE_STRICT_TOOL_CALLING) installs structural_tag on a pre-existing
StructuredOutputsParams via in-place attribute assignment and returns without
nulling response_format. The in-place set bypasses
StructuredOutputsParams.__post_init__, so the params keep a prior
mutually-exclusive constraint (json/regex/choice/grammar/json_object, or one
lowered from response_format) next to the new structural_tag. On the next
re-validation this trips the one-constraint invariant, so a strict-mode request
that also carries a structured-output constraint or a response_format fails with:

    ValueError: You can only use one kind of structured outputs constraint
    but multiple are specified

This affects any parser that installs a structural tag -- currently DeepSeek-V4
and Qwen3-Coder via get_structural_tag. The env var is off by default, and a
request with no pre-existing constraint is unaffected.

Fix: rebuild structured_outputs with only the structural tag (preserving the
whitespace / additional-properties knobs) and null response_format, mirroring
Step 2 of the same method. This "tool constraint wins, response_format dropped"
resolution already exists in Step 2 and the DeepSeek-V3.2 override (vllm-project#41178), and
is the intent of the open auto-path fix vllm-project#39969; the in-place-vs-rebuild trade-off
was discussed on vllm-project#40894 and vllm-project#43155 (whose Kimi path already rebuilds).

Repro / regression test (CPU, no model required):

    pytest tests/tool_use/test_strict_tool_calling_adjust_request.py

The added tests enable strict mode, give a parser a structural tag, and send
tools together with a response_format or a structured_outputs.json constraint
(tool_choice auto and required). On the pre-fix code adjust_request leaves two
constraints, and to_sampling_params raises the ValueError above; with this change
structured_outputs holds only the structural tag, response_format is None, and
the user's whitespace knobs are preserved. The conflict tests fail without this
patch and pass with it; the no-pre-existing-constraint case passes either way.

Equivalently over HTTP: with strict mode on, a tool_choice="auto" request that
also sets response_format returns HTTP 400 (the error above) before this change
and a normal tool call after; a required-tool request is unaffected because that
path already rebuilds.

Signed-off-by: Ace Eldeib <aeldeib@coreweave.com>
@mergify mergify Bot added tool-calling bug Something isn't working labels May 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working tool-calling

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant