[Bugfix] Clear conflicting structured outputs in strict tool calling#44134
Draft
alexeldeib wants to merge 1 commit into
Draft
[Bugfix] Clear conflicting structured outputs in strict tool calling#44134alexeldeib wants to merge 1 commit into
alexeldeib wants to merge 1 commit into
Conversation
ToolParser.adjust_request's strict structural-tag path (added in vllm-project#40894, gated by VLLM_ENFORCE_STRICT_TOOL_CALLING) installs structural_tag on a pre-existing StructuredOutputsParams via in-place attribute assignment and returns without nulling response_format. The in-place set bypasses StructuredOutputsParams.__post_init__, so the params keep a prior mutually-exclusive constraint (json/regex/choice/grammar/json_object, or one lowered from response_format) next to the new structural_tag. On the next re-validation this trips the one-constraint invariant, so a strict-mode request that also carries a structured-output constraint or a response_format fails with: ValueError: You can only use one kind of structured outputs constraint but multiple are specified This affects any parser that installs a structural tag -- currently DeepSeek-V4 and Qwen3-Coder via get_structural_tag. The env var is off by default, and a request with no pre-existing constraint is unaffected. Fix: rebuild structured_outputs with only the structural tag (preserving the whitespace / additional-properties knobs) and null response_format, mirroring Step 2 of the same method. This "tool constraint wins, response_format dropped" resolution already exists in Step 2 and the DeepSeek-V3.2 override (vllm-project#41178), and is the intent of the open auto-path fix vllm-project#39969; the in-place-vs-rebuild trade-off was discussed on vllm-project#40894 and vllm-project#43155 (whose Kimi path already rebuilds). Repro / regression test (CPU, no model required): pytest tests/tool_use/test_strict_tool_calling_adjust_request.py The added tests enable strict mode, give a parser a structural tag, and send tools together with a response_format or a structured_outputs.json constraint (tool_choice auto and required). On the pre-fix code adjust_request leaves two constraints, and to_sampling_params raises the ValueError above; with this change structured_outputs holds only the structural tag, response_format is None, and the user's whitespace knobs are preserved. The conflict tests fail without this patch and pass with it; the no-pre-existing-constraint case passes either way. Equivalently over HTTP: with strict mode on, a tool_choice="auto" request that also sets response_format returns HTTP 400 (the error above) before this change and a normal tool call after; a required-tool request is unaffected because that path already rebuilds. Signed-off-by: Ace Eldeib <aeldeib@coreweave.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
ToolParser.adjust_request's strict structural-tag path (added in #40894, gated byVLLM_ENFORCE_STRICT_TOOL_CALLING) installsstructural_tagon a pre-existingStructuredOutputsParamsvia in-place attribute assignment, and returns without nullingresponse_format. The in-place set bypassesStructuredOutputsParams.__post_init__, so the params retain a prior mutually-exclusive constraint (json/regex/choice/grammar/json_object, or one lowered fromresponse_format) alongside the newstructural_tag. On the next re-validation this trips the one-constraint invariant:So a strict-mode request that also carries a structured-output constraint or a
response_formatreturns HTTP 400 when it should succeed. This affects any parser that installs a structural tag — currently DeepSeek-V4 and Qwen3-Coder viaget_structural_tag. The env var is off by default; requests with no pre-existing constraint are unaffected.Fix: rebuild
structured_outputswith only the structural tag (preserving the whitespace / additional-properties knobs) and nullresponse_format, mirroring Step 2 of the same method. The same "tool constraint wins, response_format dropped" resolution already exists in Step 2 and the DeepSeek-V3.2 override (#41178), and is the intent of the open auto-path fix #39969.Test Plan
Added CPU-only tests (no model required): enable strict mode, give a parser a structural tag, and send tools together with a
response_formator astructured_outputs.jsonconstraint, for bothtool_choice="auto"andtool_choice="required".Test Result
adjust_requestleaves two constraints;to_sampling_paramsraises theValueErrorabove. The conflict tests fail.structured_outputsholds only the structural tag,response_formatisNone, the caller's whitespace knobs are preserved. All tests pass; the no-pre-existing-constraint case passes either way.Equivalently over HTTP, with strict mode on: a
tool_choice="auto"request that also setsresponse_formatreturns 400 before this change and a normal tool call after; a required-tool request is unaffected (that path already rebuilds).Essential Elements of an Effective PR Description Checklist