[Bugfix] Check only prompt tail for reasoning end in DelegatingParser - #50152
pynterest83 wants to merge 1 commit into
Conversation
DelegatingParser.parse_delta scanned the entire prompt with is_reasoning_end(), so any reasoning marker inside the chat template or an earlier turn marked reasoning as already finished and the whole generation was emitted as content. Check only the last tokens of the prompt instead, sized to the parser's end marker. Signed-off-by: quangch <quangdtm2004@gmail.com>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
The proper |
DelegatingParser.parse_delta scanned the entire prompt with is_reasoning_end(), so any reasoning marker inside the chat template or an earlier turn marked reasoning as already finished and the whole generation was emitted as content.
Check only the last tokens of the prompt instead, sized to the parser's end marker.
Purpose
DelegatingParser.parse_deltacallsis_reasoning_end(prompt_token_ids)over theentire prompt. Any reasoning marker present in the prompt -- from chat-template
instructions, few-shot examples, or an earlier assistant turn -- makes it conclude
reasoning already ended, so
state.reasoning_endedis set before the firstgenerated token and the whole response is emitted as
content.Reproduced with MiniMax-M3 (
--reasoning-parser minimax_m3) on v0.25.1 and stillpresent on main. The rendered prompt contains three markers from the model's own
<thinking_instructions>block, at offsets 394 (<mm:think>), 404 and 509(
</mm:think>).rfind(end) > rfind(start)-> True -> reasoning phase skipped.Related: #46042 (same symptom). #46663 fixes the equivalent check in
OpenAIServingChat, but that fix alone is not sufficient:parse_deltarecomputes the value independently. Verified on a deployment already carrying
#46663 -- the bug persisted until this second site was fixed.
Test Plan
Added
tests/parser/engine/test_prompt_reasoning_end.py:End-to-end: MiniMax-M3-NVFP4, 4x H200, TP=4, streaming chat completions.
Test Result
Unit tests: 5 passed with the fix; 4 fail without it.
End-to-end before the fix: every streamed response emitted a literal
<mm:think>delta inside
contentand noreasoning_contentfield was present.After the fix: reasoning is correctly separated into
reasoning_content.Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.