[Bugfix][Parser] Scope reasoning-end detection to the current turn via turn-boundary tokens

[Xarbirus]

Purpose

ParserEngine.is_reasoning_end() walks the token sequence backwards and returns True on the first </think> it sees. It is called on the full prompt (the entire rendered conversation) from two places:

1. DelegatingParser.parse_delta (vllm/parser/abstract_parser.py) - seeds reasoning_ended at stream start. A stale </think> in history routes the new turn's entire <think>…</think> block into content instead of reasoning (and the answer is lost entirely under include_reasoning=false).
2. StructuredOutputManager.should_fill_bitmask (vllm/v1/structured_output/__init__.py) - seeds the scheduler's reasoning-end gate. A stale </think> applies the grammar bitmask from the first generated token, constraining the model's thinking.

Fix

Add turn_boundary_tokens to ParserEngineConfig; the backward walk now stops at the first turn boundary (<|im_start|>/<|im_end|> for the ChatML family), so markers from earlier turns are no longer visible. Hitting a
boundary falls back to initial_state != REASONING (the same semantics as an empty prompt) which keeps the "template closed the think block in the generation prompt" case working (covered by a test). This generalizes into the
engine what #44551 (Cohere) and poolside_v1 (here) already do per-parser, and follows the model-specific backward-scan direction maintainers requested when closing the generic serving-layer attempts (#46663 review).

Backward compatible: the default boundary set is empty, and boundary tokens missing from a tokenizer's vocab resolve to nothing, so parsers on non-ChatML vocabs (e.g. SeedOssParser) keep the previous behaviour. Both cases are covered by tests.

Why this is not a duplicate

• fix: Correct reasoning-end detection for prompt history #44551 fixed this bug class for Cohere only
• [Bugfix][Reasoning] Mistral: scope streaming reasoning-end detection to current turn #47775 does the same for Mistral
• fix(reasoning): force-open poolside_v1 on bare </think> #49728 / [New Model] Laguna XS.2 implementation #41129 relate to poolside models
• [Bug]: MiniMax-M3 reasoning tokens still leak into content on v0.26.0 and current main #46042 / [Bugfix] Fix MiniMax M3 prompt reasoning initialization #50594 cover MiniMax-M3
• [Bugfix] Improve GLM4 MoE Reasoning Parser's is_reasoning_end Condition #25355 fixed GLM4
• [Bugfix][Frontend] Fix reasoning-end detection to check prompt tail only (streaming + non-streaming) #45891, [Bugfix][Frontend] Fix reasoning-end detection to check prompt tail only #46663 and [Bugfix] Check only prompt tail for reasoning end in DelegatingParser #50152 attempted a generic serving-layer "prompt tail" check and were closed; review feedback asked for a model-specific backward scan inside the parser
• [Bugfix][Parser] qwen3: start in CONTENT when the prompt closed the think block #53302 complementary: it starts qwen3 in CONTENT when the generation prompt legitimately closes the think block, and relies on is_reasoning_end(prompt), this PR makes that check robust against stale history markers
• [Perf][Reasoning] O(delta) reasoning-end check for engine parsers #51238 is an orthogonal perf change to the decode-path is_reasoning_end_streaming

Test Plan

pytest tests/parser/engine/test_qwen3_reasoning.py -v
pytest tests/parser -q
pre-commit run --files <changed files>
pre-commit run mypy-3.12 --all-files --hook-stage manual

E2E: serve Qwen/Qwen3-1.7B with --reasoning-parser qwen3, send streaming requests whose prompt contains a stale </think> (literal tag in user text), plus regression controls, at temperature=0; compare field routing before and after this change.

Test Result

Tested on H100

• tests/parser/engine/test_qwen3_reasoning.py: 57 passed (incl. 10 new turn-boundary tests)

• full tests/parser suite: 4086 passed, 0 failures

• pre-commit on changed files: all hooks passed

• E2E (Qwen/Qwen3-1.7B, --reasoning-parser qwen3, streaming, temperature=0):

  case	main	this PR
  control single turn	reasoning 1227 ch / content 77 ch	identical
  replayed reasoning in history	reasoning 875 ch / content 16 ch	identical
  literal </think> in user text	reasoning 0 ch, whole response incl. <think> tag leaked into content (602 ch)	reasoning 571 ch / content 16 ch, no leak

Control cases are byte-identical before/after.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[bbrowning]

/ci run

[github-actions]

✅ Triggered Buildkite CI #85873 for commit 24a3b24415a1.

[bbrowning]

Note that the SeedOSS models inherit from the Qwen3 parser here. That happens to be harmless in this case, even though the boundary turn tokens are now wrong for SeedOSS, because the code gracefully degrades when those tokens are not in the model vocab.

To get this fixed for SeedOSS models as well, need to override SeedOssParser.TURN_BOUNDARIES with <seed:bos> and <seed:eos> instead of the inherited ChatML tokens (and confirm those are the right turn boundary tokens for the target model and in that model's vocab) and add a multi-turn test containing a stale </seed:think> before the latest Seed turn boundary.

Given that this is harmless for SeedOSS models today, approving this as is. Thanks for the fix!