Skip to content

[Bugfix][Structured Output] Preserve reasoning adapters for shared parser engines - #52830

Merged
sfeng33 merged 2 commits into
vllm-project:mainfrom
sfeng33:parser-fix
Aug 26, 2026
Merged

sfeng33 merged 2 commits into
vllm-project:mainfrom
sfeng33:parser-fix

Conversation

@sfeng33

@sfeng33 sfeng33 commented Aug 18, 2026 •

Copy link
Copy Markdown
Member

Purpose

Fix structured-output reasoning configuration being silently disabled when the reasoning and tool parsers use the same parser engine.

ParserManager.get_parser() returned the raw shared ParserEngine, whose reasoning_parser attribute is None. The Chat Completions and Responses serving paths interpreted this as reasoning being disabled and skipped forwarding
request-specific parser kwargs such as enable_thinking=False.

The structured-output backend could then use the parser's default thinking mode and wait for a reasoning boundary that the model was instructed not to emit, leaving schema generation unconstrained or running until max_tokens.

Test Plan

Case STRICT main (before) PR (after)
required, non-stream 1 no tool_calls ✅ get_weather {"city": "San Francisco"}
required, stream 1 no tool_calls ✅ get_weather {"city": "San Francisco"}
named, non-stream 1 no tool_calls ✅ get_weather {"city": "San Francisco"}
named, stream 1 no tool_calls ✅ get_weather {"city": "San Francisco"}
required, non-stream 0 no tool_calls ✅ get_weather {"city": "San Francisco"}
named, non-stream 0 no tool_calls ✅ get_weather {"city": "San Francisco"}
named, stream 0 no tool_calls ✅ get_weather { "city": "San Francisco" }
auto, non-stream 0,1 content: "" (24 tokens dropped) ✅ content: "<tool_call>\n{"name": "get_weather", ...}\n</tool_call>"
auto, non-stream, no-think 0,1 content: "" ✅ same content restored
auto, non-stream, include_reasoning:false 0 content: "" ✅ same content restored

Signed-off-by: sfeng33 <4florafeng@gmail.com>
@mergify mergify Bot added tool-calling bug Something isn't working labels Aug 18, 2026
@sfeng33
sfeng33 marked this pull request as ready for review August 20, 2026 22:16

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@sfeng33

sfeng33 commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #85559 for commit 553201d58097.

@yewentao256 yewentao256 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the work!

@sfeng33
sfeng33 merged commit 4663885 into vllm-project:main Aug 26, 2026
62 checks passed
@sfeng33
sfeng33 deleted the parser-fix branch August 26, 2026 04:37
am-cohere pushed a commit to am-cohere/vllm that referenced this pull request Sep 1, 2026
…rser engines (vllm-project#52830)

Signed-off-by: sfeng33 <4florafeng@gmail.com>
mikeshawcode pushed a commit to mikeshawcode/vllm that referenced this pull request Sep 1, 2026
…rser engines (vllm-project#52830)

Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: mikeshawcode <michaelwshaw2@gmail.com>
mylibrar pushed a commit to tanyuqian/vllm that referenced this pull request Sep 3, 2026
…rser engines (vllm-project#52830)

Signed-off-by: sfeng33 <4florafeng@gmail.com>
0xjunhao added a commit to 0xjunhao/vllm that referenced this pull request Sep 3, 2026
Signed-off-by: Junhao Li <junhao@ubicloud.com>
wtdcode added a commit to wtdcode/vllm that referenced this pull request Sep 10, 2026
…quest

_apply_structural_tag hardcodes reasoning=False. The two grammars xgrammar
builds are not interchangeable -- for deepseek_v4 with tools and
tool_choice="auto":

  reasoning=False
    TriggeredTagsFormat(triggers=['<|DSML|tool_calls>'], tags=[...],
                        excludes=['<think>', '</think>'])

  reasoning=True
    SequenceFormat(elements=[
        TagFormat(begin='', content=AnyTextFormat(excludes=[]), end='</think>'),
        TriggeredTagsFormat(...)])

reasoning=False puts </think> in the free-text excludes, so a model whose
prompt ends inside an open thinking block can never close it. reasoning=True
opens with an any-text span terminated by </think>, so a non-thinking request
waits for a marker its prompt already consumed and EOS stays masked. Verified
with an xgrammar GrammarMatcher on a DeepSeek-V4-Flash-0731 tokenizer:

                          reasoning=False   reasoning=True
  "...</think>answer"     rejected          accepted
  "...</think>" + call    rejected          accepted
  runaway invoke name     rejected          rejected

This was latent while ParserManager collapsed matching parsers into a bare
ParserEngine, since _apply_structural_tag never ran for them. vllm-project#52830 removed
that shortcut, so the path is now live for every engine-backed parser
(deepseek_v3_2/v4, qwen3, glm47_moe, kimi_k2, minimax_m2, ...) whenever a
structural tag is applied -- today that is tool_choice=required/named, or
auto with a strict tool.

Add ReasoningParser.emits_reasoning_span, defaulting to False so existing
parsers keep their current behaviour, and override it in ParserEngine from
parser_engine_config.initial_state, which is derived per request from
chat_template_kwargs.

The same root cause was reported in vllm-project#46149 (Qwen3.6-27B, tool_choice="auto"
with per-tool strict=true) and closed without a fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: lazymio <mio@lazym.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working tool-calling

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants