[Bugfix] Fix Qwen3Coder tool call streaming with speculative decoding

[voipmonitor]

Summary

Fixes broken tool call JSON when using Qwen3CoderToolParser (--tool-call-parser qwen3_coder) with speculative decoding (--num-speculative-tokens N where N >= 2).

PR #35347 partially addressed this for num_speculative_tokens: 2 by adding Qwen35CoderToolParser, but the fix still fails for num_speculative_tokens: 3, 4, 5 because the root cause is in the serving layer and in missing streamed_args_for_tool tracking in the base Qwen3CoderToolParser. This PR fixes the actual root causes directly in Qwen3CoderToolParser, so no separate parser class is needed.

Root Cause Analysis

Three bugs interact to produce malformed tool call JSON:

Bug 1: Double-serialization in serving.py (the primary cause)

prev_tool_call_arr[index]["arguments"] is already a JSON string (e.g. '{"city": "Prague"}'), but the serving layer calls json.dumps() on it again:

# BEFORE (broken):
expected_call = json.dumps(
    tool_parser.prev_tool_call_arr[index].get("arguments", {}),
    ensure_ascii=False,
)
# Result: '"{\\"city\\": \\"Prague\\"}"' (double-serialized!)

This causes expected_call.replace(actual_call, "", 1) to fail (the format doesn't match), so remaining_call equals the entire double-serialized string, which gets appended as a spurious final delta. The client then assembles:

{"city": "Prague""{\"city\": \"Prague\"}"

Bug 2: Missing streamed_args_for_tool tracking in Qwen3CoderToolParser

The parser never populates self.streamed_args_for_tool, but the serving layer reads it at stream end (streamed_args_for_tool[index]), causing IndexError: list index out of range.

Bug 3: Conditional { sending can be skipped

The original condition if not self.json_started and self.parameter_prefix not in delta_text skips sending { when the delta contains parameter data (common with speculative decoding where multiple tokens arrive at once). But json_started is then set to True anyway, desyncing the tracked state from what was actually streamed.

Changes

• serving.py: Check if arguments is already a string before calling json.dumps(), preventing double-serialization.
• qwen3coder_tool_parser.py:
  • Add streamed_args_for_tool tracking at all argument emission points (header, {, parameters, }).
  • Always send { first regardless of delta content, removing the parameter_prefix not in delta_text condition.
  • Remove dead code (if not self.json_started: self.json_started = True was unreachable after the fix).

How to Reproduce

# Start vLLM with speculative decoding and Qwen3Coder tool parser
vllm serve <model> \
  --tool-call-parser qwen3_coder \
  --speculative-model <draft_model> \
  --num-speculative-tokens 5

# Send a multi-turn tool call conversation. On the second turn
# (when the assistant's tool_call response is sent back), the
# server crashes with either:
#   - json.JSONDecodeError: Expecting ',' delimiter
#   - IndexError: list index out of range (streamed_args_for_tool)

With num_speculative_tokens: 1 it works. With :2 it may work (depending on token boundaries). With :3 and above it consistently fails.

Test Plan

• Tested with num_speculative_tokens: 1, 2, 3, 4, 5 — all work after the fix
• Tested with multi-parameter tool calls (arrays, strings, integers)
• Verified no regression for non-speculative-decoding tool calls
• Unit tests for streamed_args_for_tool population

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[gemini-code-assist]

Code Review

This pull request addresses several bugs related to tool call streaming with speculative decoding in the Qwen3CoderToolParser. The changes prevent double-serialization of arguments in the serving layer and correctly track streamed arguments in the parser to avoid IndexError and state desynchronization. The fixes appear correct and well-targeted. I've added a few comments with high severity regarding potential silent failures in exception handling and state management that could lead to incorrect JSON output, which would undermine the goal of this bugfix.

[mergify]

Hi @voipmonitor, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[jmfirth-arkane]

Thanks, this patch is working great at MTP=3. Test Dockerfile:

FROM vllm/vllm-openai:qwen3_5

# Grab the PR diff and apply it directly to the installed package
RUN apt-get update && apt-get install -y patch curl && \
    curl -L https://github.com/vllm-project/vllm/pull/35421.diff -o /tmp/pr.diff && \
    # Find where vllm is installed
    VLLM_PATH=$(python3 -c "import vllm; import os; print(os.path.dirname(vllm.__file__))") && \
    # Apply the patch - strip the leading 'vllm/' prefix to match installed paths
    cd "$VLLM_PATH" && \
    patch -p2 --forward < /tmp/pr.diff || true && \
    rm /tmp/pr.diff

[chaunceyjiang]

Thanks~

[chaunceyjiang]

you need to DCO.

[voipmonitor]

you need to DCO.

fixed

[voipmonitor]

all those fixes are still not correct and fails at some scenarios with speculative decoding >3 - I'm closing this PR and creating new one: #35615 @chaunceyjiang please check