[Bugfix][Structured Output] Feed the token that implicitly ends reasoning to the grammar

[simon-veitner-redhat]

Overview

Structured output skipped a reasoning-end token that is also content, such as GLM's <tool_call> with no </think> before it. The grammar then forced a second <tool_call>, and required/named tool calls were lost. This PR feeds that token to the grammar, which matches what extract_content_ids already does. Fixes #59608.

Claims

• For ParserEngine-based parsers, the grammar now consumes a reasoning-end token that extract_content_ids keeps as content, if the grammar accepts it. Close tags such as </think> are still skipped. ReasoningParser and legacy parsers do not change.
• On GLM-5.3-Flash, the doubled-<tool_call> signature went from 7/40 (required) and 8/40 (named) to 0/40. With MTP speculative decoding it went from 9/40 and 11/40 to 0/40.
• Under speculative decoding, a drafted <tool_call> cannot force the end of reasoning, and when it is the last draft the bonus row is masked from the grammar state after it.
• The same failure reproduces on Qwen3 (qwen3 + hermes), and the fix removes it there too. Correct-call rates on the GLM tool-result prompt did not go up, because of a separate argument bug (see Limitations).

Validation

Unit tests

pytest tests/v1/structured_output tests/v1/spec_decode/test_mtp_structured_output.py tests/parser \
  tests/reasoning tests/tool_parsers/test_glm47_moe_tool_parser.py tests/tool_parsers/test_glm4_moe_tool_parser.py \
  --ignore=tests/parser/cohere

Result with 1 GPU visible: 6402 passed, 1 xfailed. pre-commit run --files <changed files> and the manual mypy-3.12 hook pass.

New tests are test_inclusive_reasoning_end_token and test_end_token_committed_alone in tests/v1/structured_output/test_structured_output_manager.py (xgrammar and guidance), and test_implicit_reasoning_end_of_parsers (real glm47_moe config) in tests/parser/engine/test_parser_engine.py. With grammar start forced to the constraint start, 8 of the 13 new cases fail: inclusive, inclusive_first, inclusive_last and no_drafts, on both backends.

CI runs these tests through misc.yaml. tests/parser/cohere is left out because it needs the optional cohere_melody package.

A CPU script that drives the scheduler flow (validate drafts, bitmask, accept, next bitmask) with <tool_call> as the last draft, the only draft, a mid-window draft and the recovered token gives a second <tool_call> in all 8 stock cases and none with the fix.

Live runs

The live runs used the first version of this PR (5805e42cfa). The current commit only restructures that code, and the unit tests and the CPU check give the same results on both versions.

GLM-5.3-Flash (RedHatAI/GLM-5.3-Flash-NVFP4), TP4 on H100, xgrammar, the tool-result repro from the issue, 40 seeds at temperature 0.7:

vllm serve RedHatAI/GLM-5.3-Flash-NVFP4 --tensor-parallel-size 4 --max-model-len 32768 \
  --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm45 \
  [--speculative-config '{"method":"mtp","num_speculative_tokens":3}']

Doubled-<tool_call> signatures, 40 seeds	Before	After
required	7	0
named	8	0
required, MTP	9	0
named, MTP	11	0

MTP draft acceptance was 59.1% before and 58.5% after.

Qwen3 (Qwen/Qwen3-0.6B, --tool-call-parser hermes --reasoning-parser qwen3), logit_bias -100 on </think> and +5 on <tool_call>, temperature 0, 3 repeats:

vllm serve Qwen/Qwen3-0.6B --max-model-len 8192 --gpu-memory-utilization 0.5 \
  --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3

Mode	Before	After
required	0/3 calls, 3/3 signatures	3/3 calls
named	0/3 calls, 3/3 signatures	3/3 calls
auto (control)	3/3 calls	3/3 calls

No standard tool-calling eval (such as BFCL) was run.

Details

Root cause. _get_constraint_start returned the index after the reasoning-end token. That is right for </think>, which is not content. A tool-call start that ends reasoning is content, so the grammar started one token after the content did. This came in with #55223, and #48200 kept it.

Change.

• ParserEngine.find_reasoning_end_offset becomes find_reasoning_end, which returns ReasoningEnd(offset, implicit). implicit is True when the end token is not the close tag, the same rule extract_content_ids uses.
• StructuredOutputManager._get_constraint_start becomes _get_constraint_bounds, which returns ConstraintBounds(grammar_start, constraint_start). For an implicit end that the grammar accepts, the grammar starts at the end token and masking starts one token later. Otherwise both equal the old value.
• validate_tokens, the spec-decode loop of grammar_bitmask and accept_tokens feed the grammar from grammar_start. Masked rows and early returns still use constraint_start, so a drafted <tool_call> cannot force the target to end reasoning.
• The "grammar accepts it" check keeps the old skip when the grammar cannot start with that token, such as a json_schema response format with tools present.

Affected parsers. Every in-tree ParserEngine config whose tool-call start ends reasoning now feeds that token to the grammar: glm47_moe (and ling3), qwen3 (and mimo), nemotron_v3, granite_thinking, seed_oss, gemma4, kimi_k2, minimax_m2, deepseek_v4, deepseek_v41, plamo3, and mistral with reasoning_encoding="special_token". A token counts only if the tokenizer has a single id for it. On the real DeepSeek-V4 tokenizer the DSML opener is several tokens, so deepseek_v4 does not change in practice. For plamo3 the token is the end-of-turn tag, which is normally a stop token; I expect that to be harmless but did not test it.

Tradeoffs.

• No measurable cost. After reasoning ends, _get_constraint_bounds returns before any reasoner call. Only a step with an implicit end token runs one extra single-token validate_tokens on the CPU. There is no CPU/GPU sync.

Limitations.

• A separate bug, similar to [Bug]: GLM required tool-call grammar accepts split marker tokens that the streaming parser treats as content #58315, remains. The GLM tool grammar allows the special <tool_call> token inside <arg_value>. When the model quotes tool-result text that contains it, arguments break (Write({}), missing arguments). Many responses this PR recovers now hit that bug. It will be reported separately.
• Legacy (non-ParserEngine) parsers such as k2_horizon keep the old behavior until they move to ParserEngine.
• Under spec decode, the target can still reject a draft and then sample an implicit end token that the grammar rejects. This predates the PR. @yzong-rh plans to file an issue for it.
• Also unchanged: a prompt that already ends with an implicit end token (continue_final_message) still starts the grammar after it, and multi-token text markers cannot be expressed as a token-id set.

Related. No open PR changes where the grammar starts after reasoning; #36138, #56403, #57718 and #57754 touch nearby paths.

This PR was investigated and drafted with an AI assistant (Claude). The tests and live runs listed here were run on real hardware, and a human reviewed the PR before submission.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[yzong-rh]

Great catch! Thanks for the PR.

I attempted a fix that makes _get_constraint_start return reasoning_end_offset instead reasoning_end_offset + 1 for inclusive_reasoning_end_token_ids. However, it changed the output distribution under spec decode and was incorrect.

The correct fix, as implemented here, is to distinguish between grammar_start and constraint_start.

• validate_tokens validate from grammar_start
• grammar_bitmask will consume tokens from grammar_start but only create bitmasks for tokens from constraint_start
• accept_tokens advance grammar from grammar_start

Note that failures due to implicit reasoning end can still occur during rejection sampling when the target rejects a draft and samples an grammar-invalid reasoning end marker (e.g. <tool_call> is sampled when grammar disallows tool calls). However, this is a pre-existing issue and not introduced by this PR. I'll create a bug report for more discussion on this.

NIT:
Current implementation adds more coupling between StructuredOuptutManager and ReasoningParser. The former now reaches into reasoner.inclusive_reasoning_end_token_ids.

A cleaner impl could be to

• rename and rewrite ParserEngine.find_reasoning_end_offset to also indicate if a reasoning end is implicit or explicit.
• rename and rewrite StructuredOutputManager._get_constraint_start to return both grammar start and constraint start (or constraint start and a grammar start offset)
• trim abstract ReasoningParser changes and limit the fix to engine-based parsers

[simon-veitner-redhat]

@yzong-rh Thanks for the review and the cleaner design. I amended the commit as you suggested:

• ParserEngine.find_reasoning_end_offset is now find_reasoning_end. It returns ReasoningEnd(offset, implicit), where implicit says whether the end token is also content.
• _get_constraint_start is now _get_constraint_bounds. It returns ConstraintBounds(grammar_start, constraint_start). validate_tokens, grammar_bitmask and accept_tokens feed the grammar from grammar_start. Masking and the early returns still use constraint_start.
• ReasoningParser is unchanged. The fix applies only to engine-based parsers, through ParserEngineReasoningAdapter.find_reasoning_end, so the manager no longer reads inclusive_reasoning_end_token_ids.

The flow tests pass with no change to their expected rows, and the CPU bonus-row check gives the same output as before. I added a test that a reasoner that is not engine-based keeps the old grammar start.

Thanks for the rejection-sampling note. I added it to Limitations and will link your issue once it is filed.

Disclaimer: this reply and the changes were drafted with the help of an AI assistant (Claude) and reviewed by me.

[mgoin]

/ci run

[github-actions]

✅ Triggered Buildkite CI #93616 for commit 831bcad5dd18.

[yzong-rh]

I think it might be cleaner to let constraint_start only control where grammar masking begins in grammar_bitmask. validate_tokens would validate from grammar_start, and accept_tokens would feed the grammar from grammar_start.

This lets _get_constraint_bounds always set grammar_start to the implicit end token and removes its grammar.validate_token probe. The grammar can then reject invalid implicit markers through validate_tokens, while accepted markers are passed to the grammar normally.

Diff implementing this: https://gist.github.com/yzong-rh/4b8a967a6d19671ede197483191d9eb7

I also consolidated the implicit-end coverage into FlowCase.

Tested my diff with your repro, tool call remains correct.

The file says: "GLM-5.3 wraps every function call in <tool_call>Write<arg_key>content</arg_key><arg_value>...

[simon-veitner-redhat]

I think it might be cleaner to let constraint_start should only control where grammar masking begins in grammar_bitmask. validate_tokens would validate from grammar_start, and accept_tokens would feed the grammar from grammar_start.

This lets _get_constraint_bounds always set grammar_start to the implicit end token and removes its grammar.validate_token probe. The grammar can then reject invalid implicit markers through validate_tokens, while accepted markers are passed to the grammar normally.

Diff implementing this: https://gist.github.com/yzong-rh/4b8a967a6d19671ede197483191d9eb7

I also consolidated the implicit-end coverage into FlowCase.

Tested my diff with your repro, tool call remains correct.

The file says: "GLM-5.3 wraps every function call in <tool_call>Write<arg_key>content</arg_key><arg_value>...

applied after local validation

[mgoin]

/ci run

[github-actions]

✅ Triggered Buildkite CI #93667 for commit fb87b56a656e.