[Feat] Support thinking_token_budget in Model Runner V2

[chaunceyjiang]

In some quantized models, such as GLM-5.2 or Qwen quantized models, the model may generate long reasoning traces.

Purpose

Support thinking_token_budget in Model Runner V2

Test Plan

see e2e

Test Result

vllm serve /mnt/data4/models/Qwen/Qwen3.5-27B-FP8 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3

curl -sS http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "",
    "messages": [
      { "role": "user", "content": "9.11 and 9.8, which is greater?" }
    ],
    "thinking_token_budget": 10
  }' |jq
{
  "id": "chatcmpl-bc955ba3f0743a51",
  "object": "chat.completion",
  "created": 1782394356,
  "model": "/mnt/data4/models/Qwen/Qwen3.5-27B-FP8",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "\n\n**9.11 is greater than 9.8.**\n\nHere is the breakdown:\n\n*   **9.8** can be written as **9.80**.\n*   **9.11** is **9.11**.\n\nWhen comparing the decimal places:\n*   The whole number part is the same (9).\n*   In the tenths place, **8** is greater than **1**.\n\nTherefore, **9.8 > 9.11**.",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "reasoning": "Thinking Process:\n\n1.  **An"
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null,
      "token_ids": null,
      "routed_experts": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": "vllm-0.23.1rc1.dev418+g6e3a983cf-e4992f95",
  "usage": {
    "prompt_tokens": 24,
    "total_tokens": 141,
    "completion_tokens": 117,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "prompt_text": null,
  "kv_transfer_params": null
}

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[njhill]

Thanks @chaunceyjiang, at first glance this looks pretty good, I'll try to take a closer look soon!

[NickLucche]

thanks @chaunceyjiang I think this is looking clean

[njhill]

Thanks @chaunceyjiang!

I think the other thing we should aim to improve is the scan of whole context each time where we only need to consider new tokens.

[njhill]

I think the other thing we should aim to improve is the scan of whole context each time where we only need to consider new tokens.

I pushed a draft change for this here, not sure about it yet though: njhill@f2d3627

[rishitdholakia13]

From what I understand, during prefill, if we're processing a long context (e.g., ~128k tokens), we currently perform a forward scan over the entire prompt to locate the <START_THINKING> and <END_THINKING> tokens. In the common case where the <START_THINKING> token appears near the end of the prompt template, this results in an unnecessary O(128k) forward scan.

One potential optimization would be to search for the thinking boundary tokens in reverse instead. Most reasoning prompt templates place the <START_THINKING> and <END_THINKING> instructions towards the end of the prompt, so a reverse scan would likely find them much sooner and reduce the amount of work performed during prefill. This would probably help lower the TTFT overhead at large context.
cc: @njhill

[chaunceyjiang]

Done.

[dafeliton]

Hi all,

This PR is causing issues with the thinking token phrase leaking into the chat. V1 previously worked around this but it looks like after X tokens even if we aren't in a reasoning block the phrase will leak out into the main response.

[chaunceyjiang]

@dafeliton, could you provide the steps to reproduce this issue?

[dafeliton]

@chaunceyjiang Sure. With gemma 31b, add these args:

- --enable-auto-tool-choice
  - --reasoning-parser
  - gemma4
  # Enables per-request reasoning caps via extra_body.thinking_token_budget.
  - --reasoning-config
  - '{"reasoning_start_str":"<|channel>","reasoning_end_str":"I have to give the solution based on the reasoning directly now.<channel|>"}'
  - --tool-call-parser
  - gemma4

Then set the reasoning budget to 1500 and tell the model to output something long such as a 50 paragraph essay. The reasoning end str appears in the main response. There were previous PRs for V1 that alleviated this issue.

Sample:

Prompt::
Calculate the enthalpy of neutralization when 500 mL 0.2 M HCl, 300 mL 0.3 M H2SO4 and 200 mL 0.5 M Ba(OH)2 are mixed.
A. -2.72 kcal
B. -11.42 kcal
C. -3.80 kcal
D. -16.0 kJ

Response::
<Reasoning>
The user previously asked to calculate the enthalpy of neutralization for a mixture of $\text{HCl}$, $\text{H}_2\text{SO}_4$, and $\text{Ba(OH)}_2$.
$\text{HCl}$: $500 \text{ mL} \times 0.2 \text{ M} = 0.5 \text{ L} \times 0.2 \text{ mol/L} = 0.1 \text{ mol } \text{H}^+$
$\text{H}_2\text{SO}_4$: $300 \text{ mL} \times 0.3 \text{ M} = 0.3 \text{ L} \times 0.3 \text{ mol/L} = 0.09 \text{ mol } \text{H}_2\text{SO}_4$
Since $\text{H}_2\text{SO}_4$ is diprotic: $0.09 \times 2 = 0.18 \text{ mol } \text{H}^+$
Total $\text{H}^+ = 0.1 + 0.18 = 0.28 \text{ mol } \text{H}^+$
$\text{Ba(OH)}_2$: $200 \text{ mL} \times 0.5 \text{ M} = 0.2 \text{ L} \times 0.5 \text{ mol/L} = 0.1 \text{ mol } \text{Ba(OH)}_2$
Since $\text{Ba(OH)}_2$ is dibasic: $0.1 \times 2 = 0.2 \text{ mol } \text{OH}^-$
$\text{H}^+ + \text{OH}^- \rightarrow \text{H}_2\text{O}$
$\text{H}^+$ available: $0.28 \text{ mol}$
$\text{OH}^-$ available: $0.20 \text{ mol}$
Limiting reactant: $\text{OH}^-$ ($0.20 \text{ mol}$)
Amount of water formed: $0.20 \text{ mol}$
The standard enthalpy of neutralization for a strong acid and strong base is approximately $\Delta H_{neut} \approx -57.3 \text{ kJ/mol}$ (or $\approx -13.7 \text{ kcal/mol}$).
Calculation in kcal: $0.20 \text{ mol} \times (-13.7 \text{ kcal/mol}) = -2.74 \text{ kcal}$.
Calculation in kJ: $0.20 \text{ mol} \times (-57.3 \text{ kJ/mol}) = -11.46 \text{ kJ}$.
A. $-2.72 \text{ kcal}$
B. $-11.42 \text{ kcal}$
C. $-3.80 \text{ kcal}$
D. $-16.0 \text{ kJ}$
Comparison: My calculation $-2.74 \text{ kcal}$ is very close to Option A ($-2.72 \text{ kcal}$). Small differences often arise from the exact value of $\Delta H_{neut}$ used (some texts use $-55.8 \text{ kJ/mol}$ or $-13.3 \text{ kcal/mol}$). Using $-13.6 \text{ kcal/mol}$: $0.2 \times -13.6 = -2.72 \text{ kcal}$.
Calculate moles of $\text{H}^+$ and $\text{OH}^-$.
Identify the limiting reactant.
Multiply moles of reaction by the standard enthalpy of neutralization.
Concise.
No em dashes.
Proper LaTeX.
Professional/Friendly.
<End Reasoning>

To find the enthalpy of neutralization, we first determine the total amount of $\text{H}^+$ and $\text{OH}^-$ ions present in the mixture.

1. Calculate moles of $\text{H}^+$ ions

• From $\text{HCl}$: ( 0.500\text{ L} \times 0.2\text{ M} = 0.1\text{ mol } \text{H}^+ )
• From $\text{H}_2\text{SO}_4$: ( 0.300\text{ L} \times 0.3\text{ M} \times 2\text{ (since it is diprotic)} = 0.18\text{ mol } \text{H}^+ )
• Total $\text{H}^+$ = ( 0.1 + 0.18 = 0.28\text{ mol} )

2. Calculate moles of $\text{OH}^-$ ions

• From $\text{Ba(OH)}_2$: ( 0.200\text{ L} \times 0.5\text{ M} \times 2\text{ (since it is dibasic)} = 0.20\text{ mol } \text{OH}^- )

3. Identify the limiting reactant
The neutralization reaction is:
$$\text{H}^+ + \text{OH}^- \rightarrow \text{H}_2\text{O}$$
Since we have $0.28\text{ mol}$ of $\text{H}^+$ and only $0.20\text{ mol}$ of $\text{OH}^-$, the $\text{OH}^-$ is the limiting reactant. Therefore, $0.20\text{ mol}$ of water will be formed.

4. Calculate the total enthalpy
The standard enthalpy of neutralization for a strong acid and strong base is approximately (-13.6\text{ kcal/mol}) (or (-57.3\text{ kJ/mol})).

$$\text{Total Enthalpy} = \text{moles of reaction} \times \Delta H_{neut}$$ $$\text{Total Enthalpy} = 0.20\text{ mol} \times (-13.6\text{ kcal/mol}) = -2.72\text{ kcal}$$

Correct Option:
**A.I have to give the solution based on the reasoning directly now.A. -2.72 kcal

[njhill]

Thanks @chaunceyjiang! I pushed one more commit with some additional fixes and optimizations.

[mergify]

Hi @chaunceyjiang, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[mergify]

Hi @chaunceyjiang, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[chaunceyjiang]

/ci run

[github-actions]

✅ Triggered Buildkite CI #82766 for commit b813d3c6760b.