[Bugfix] Detect MTP truncation at reasoning-to-tool-call boundary

[ToastyTheBot]

Summary

With MTP speculative decoding (num_speculative_tokens >= 1), the rejection sampler can produce EOS as the target model's argmax at one position in the burst. The scheduler's per-token check_stop() sees EOS and terminates generation, discarding remaining tokens. This happens at the reasoning→tool-call boundary where the model's output transitions from reasoning content to tool-call XML.

The client receives finish_reason: "stop" with only reasoning_content — no content or tool_calls. This is a silent truncation that the client cannot distinguish from a legitimate stop.

Observed hit rate: ~0.25% under concurrent load with num_speculative_tokens=3, 0% without MTP. The bug is timing-dependent and requires concurrent requests to stress MTP scheduling.

Context

We run Qwen3.6-27B-FP8 with MTP=3 in production and observed a suite of tool-calling issues with speculative decoding. After applying several open PRs together, tool call reliability improved dramatically:

• [Bugfix] Fix Qwen3 reasoning parser: raw text tags, transition loss, end detection, token counting, withhold recovery #40783 — fragmented <think/> tag handling in reasoning parser
• [Bugfix][ToolParser] Fix Qwen3 XML and Coder streaming tool call parser regressions #40861 — structural parsing fixes in Qwen3Coder tool parser
• Fix Qwen3 reasoning tool calls embedded inside think #39055 — embedded tool call recovery from reasoning
• [Bugfix] Grammar was ignored when reasoning ended within speculated tokens #36138 — grammar ignored when reasoning ends within speculated tokens
• [BugFix] Fix streaming tool call empty fields with MTP: Pydantic null serialization + qwen3coder early return #39598 — Pydantic null serialization in streaming tool call deltas

We also opened two other sibling draft PRs in an attempt to perfectly fix the issue:

• [Bugfix] Fix Qwen3Coder prev_tool_call_arr double-emission on parse failure #41466 — prev_tool_call_arr double-emission fallback
• fix(spec decode): suppress EOS at draft positions in rejection sampler #41493 — Suppress EOS at draft positions in rejection sampler

This PR addresses the remaining ~0.25% truncation case that the above fixes don't cover.

Root Cause Analysis

The rejection sampler's greedy kernel stores the target model's argmax as the output token unconditionally. When the argmax is EOS at the boundary, check_stop() triggers and trims remaining tokens in the burst.

In _update_request_with_output, the scheduler iterates MTP burst tokens one-by-one via check_stop(). When EOS is encountered at any position, the scheduler immediately sets FINISHED_STOPPED and discards all remaining tokens in the burst — including the bonus token that would have continued generation.

This is an engine-level issue — this PR provides a serving-layer safety net that converts the silent truncation into an explicit retryable error.

Changes

• Detects the truncation pattern in the streaming generator at the point where finish_reason is determined:
  • finish_reason == "stop"
  • request.tools is non-empty (tools were configured)
  • No tool calls were streamed
  • No auto tools were called
  • A reasoning parser is active
  • The delta message has no content and no tool_calls
• When detected, raises GenerationError with a descriptive message
• The exception handler emits an SSE error event, signaling the client to retry

Why This Is Safe

• Only triggers when ALL detection criteria are met — a very specific pattern
• Legitimate finish_reason="stop" without tools configured is unaffected
• Legitimate finish_reason="stop" with content or tool_calls already emitted is unaffected
• The GenerationError maps to a standard SSE error event, not a server crash
• Clients that don't handle the error receive the same behavior as before (the stream ends)

Reproduction

Using Qwen3.6-27B-FP8 with MTP=3, send tool-calling requests with reasoning enabled under concurrent load. The truncation occurs at ~0.25% hit rate. Stress testing with --concurrent 4 or higher increases the hit rate.

Test Plan

• Verify normal streaming tool calls still work (finish_reason="tool_calls" with tool call data)
• Verify legitimate finish_reason="stop" without tools is not affected
• Verify that when MTP truncation occurs, the client receives an SSE error event instead of a silent stop
• Run pytest tests/entrypoints/openai/chat_completion/test_serving_chat.py

Note: Reproducing the truncation requires concurrent load with MTP enabled. A unit test would need to mock finish_reason and delta_message to simulate the condition.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

[gemini-code-assist]

Code Review

This pull request introduces a detection mechanism for MTP speculative decoding truncation in the chat completion stream generator, raising a retryable error when tool calls are expected but not generated. I have reviewed the implementation and suggest simplifying the redundant boolean check in the condition for tool calls to improve code readability.