archived internal review draft (b)

[valeriyischenko]

Review-only PR. This targets the muse-glimmer-grammar-gate branch inside
my own fork, not vllm-project/vllm. Nothing has been proposed upstream. Basing
it on PR 1's branch means the diff below is exactly this PR's two commits —
which is also how it would be submitted upstream ("stacked on #").

A standalone version of these two commits, applied straight on upstream
41f179b57, is on branch muse-glimmer-thinking-budget. It is not for
submission: without PR 1 the budget works but the schema stays unenforced.

Purpose

thinking_token_budget is unusable with Muse-Glimmer: the server rejects any
request that sets it. The chain, all on current main:

1. MuseGlimmerReasoningParser overrides neither reasoning_start_str nor
   reasoning_end_str, so both inherit the base None
   (abs_reasoning_parsers.py).

2. ReasoningConfig.initialize_token_ids therefore reaches
   if not reasoning_start_str or not reasoning_end_str: return and leaves
   _enabled = False (vllm/config/reasoning.py).

3. InputProcessor then raises for every request carrying a budget
   (vllm/v1/engine/input_processor.py):

   thinking_token_budget is set but reasoning_config is not configured. Please set --reasoning-parser and/or --reasoning-config to use thinking_token_budget.

   which surfaces as HTTP 400 — and the advice in the message is already
   satisfied: the server was started with --reasoning-parser muse_glimmer.

Confirmed on the serving path, not just by reading: with the stock parser a
thinking_token_budget request returns 400, while a qwen3 server on the same
build accepts it. count_reasoning_tokens() also returns 0, so usage accounting
reports no reasoning tokens even when no budget is requested.

Declaring the strings is not enough, because Muse needs two different end
strings:

• natural — <|eom|> closes the reasoning message; short, so it matches
  reliably;
• forced — <|eom|> alone leaves the model free to open another to=self
  block, so exhausting the budget must also open the answer channel:
  <|eom|><|start|>assistant to=user<|message|>.

ReasoningConfig already distinguishes forced (reasoning_end_token_ids) from
natural (natural_reasoning_end_token_ids), but a parser can only declare one
string; the forced one is a config-only override. This PR adds the missing hook.

Why this is not a duplicate

• [Bugfix][Reasoning] Properly detect reasoning end when using thinking_token_budget vllm-project/vllm#43210 makes the parser's end string available to the budget enforcer (the
  natural/forced split now in main). This PR is the other half: it lets the
  parser declare a forced string that differs from its natural one, which no open
  PR does.
• [Reasoning] Break repeating reasoning loops by forcing the reasoning end sequence vllm-project/vllm#52677 forces reasoning_end_str to break repetition loops. It consumes the same
  field and benefits from this change; it does not provide it.
• [Bugfix][Core] Restore incremental thinking-budget marker scans vllm-project/vllm#52106 / fix(security): restore incremental marker search cursors in thinking budget state vllm-project/vllm#51133 are incremental-scan performance fixes in
  thinking_budget_state.py; disjoint.
• [Feat] Support thinking_token_budget in Model Runner V2 vllm-project/vllm#46727 added the natural ids on the MRv2 GPU path; this PR feeds them.

Design

Two commits: the generic property first, then the Muse-Glimmer change that uses it.

1. ReasoningParser.forced_reasoning_end_str — new property, defaults to None,
   documented as "override only when closing reasoning does not commit the model to
   answering". The default keeps every existing parser unchanged.
2. ReasoningConfig.initialize_token_ids prefers
   parser.forced_reasoning_end_str or parser.reasoning_end_str for the forced ids,
   and keeps parser.reasoning_end_str as the natural ids.
3. Muse declares reasoning_start_str = " to=self<|message|>",
   reasoning_end_str = "<|eom|>",
   forced_reasoning_end_str = "<|eom|><|start|>assistant to=user<|message|>".
   The leading space is load-bearing: the chat template ends the generation
   prompt after <|start|>assistant, so the model emits to, a different token
   from to, and the budget matches ids by exact slice — the wrong spelling fails
   silently.
4. count_reasoning_tokens counts tokens inside to=self messages by finding
   <|message|> ids and decoding a short window backwards for the recipient
   (channel framing has no start token to key on), returning 0 if the framing
   markers are not single tokens rather than guessing. Channel bodies are never
   decoded.
5. Docs: notes in docs/features/reasoning_outputs.md on the id-matched-spelling
   trap and on channel-framed models needing a separate forced string.

Files: vllm/reasoning/abs_reasoning_parsers.py, vllm/config/reasoning.py,
vllm/reasoning/muse_glimmer_reasoning_parser.py,
tests/reasoning/test_muse_glimmer_reasoning_parser.py,
docs/features/reasoning_outputs.md.

Where the strings come from

chat_template.jinja in the checkpoint, read directly:

• the generation prompt is '<|start|>assistant' with no trailing space, and a
  channel is rendered as '<|start|>assistant' + ' to=' + recipient + '<|message|>', so the first thing the model generates is to=self<|message|> —
  the leading space is the template's, not a guess;
• a reasoning message is '<|start|>assistant to=self<|message|>' + reasoning + '<|eom|>', and the answer that follows uses
  recipient = message.get('recipient') or 'user', i.e. always a recipient. So
  the transition the model was trained to emit is exactly
  <|eom|><|start|>assistant to=user<|message|>, the forced string here;
• <|start|>, <|message|>, <|eom|>, <|eot|> are all declared special tokens,
  so each is a single id and count_reasoning_tokens can key on
  vocab["<|message|>"].

Tests

pytest tests/reasoning/test_muse_glimmer_reasoning_parser.py -v
pytest tests/v1/worker/test_gpu_thinking_budget.py -v

Seven new tests over a stub tokenizer: the three boundary strings; ReasoningConfig
ending up enabled with forced ≠ natural ids; and count_reasoning_tokens over a
single span, several spans, an unterminated span, a tool channel, and a vocabulary
without the framing tokens.

$ pytest tests/reasoning/test_muse_glimmer_reasoning_parser.py -q
7 passed in 3.41s
$ pytest tests/v1/worker/test_gpu_thinking_budget.py \
      tests/v1/structured_output/test_reasoning_structured_output.py -q
23 passed, 14 warnings in 12.50s

End-to-end enforcement is measured: with the gate open, the chat endpoint returns
exactly 64 and exactly 256 reasoning tokens for those budgets, with an
answer present and no overshoot, against HTTP 400 in the stock state. Combined with
PR 1, the forced answer channel is also schema-valid and ends on <|eot|> at
budgets 256 and 512 — which is the strongest argument for this PR: on a
reasoning-heavy prompt the schema is unusable without a budget, because the model
reasons past max_tokens (4,093 reasoning tokens, finish_reason: length) and
returns no answer at all.

Known limitations, stated rather than inherited silently

The legacy vllm/v1/sample/thinking_budget_state.py path uses
reasoning_end_token_ids (the forced string) for both forcing and natural-end
detection; natural_reasoning_end_token_ids is consumed only by the MRv2 GPU path
(vllm/v1/worker/gpu/sample/thinking_budget.py, added in vllm-project#46727). For a normal
Muse answer turn that still works, because the model emits the whole forced string
verbatim — detection just lags by the header tokens. It breaks for a tool turn,
where the model leaves reasoning with
<|eom|><|start|>assistant to=<tool>.<fn><|message|>: that is not the forced
string, so the legacy path sees no natural end and can force the answer header into
the middle of a tool call. This PR supplies the natural ids that fix it on MRv2;
wiring the legacy path to them is a separate change and pre-existing behaviour.

That failure mode is not hypothetical: open issue vllm-project#44676 reports exactly it for
Qwen3.5+ — the tool parser treats <tool_call> as an implicit reasoning end,
ThinkingBudgetStateHolder does not, so the budget expires mid-JSON and the forced
end marker lands inside the tool-call arguments. It is the reason a parser must be
able to declare a natural end distinct from the string it wants forced; the Muse
channel protocol makes that distinction structural rather than model-specific.

Second: enabling the budget for Muse also opts these requests into the marker-scan
cost that open PR vllm-project#52106 fixes. ThinkingBudgetState's scan_offset only advances
when a thinking section exits, so a single long reasoning span rescans the whole
growing output every decode step — 1.2 ms/step at 16k tokens in pure Python, ~20 s
cumulative per request. Orthogonal to this PR and pre-existing for every budget
user, but fair warning for anyone enabling a budget on long reasoning.

AI assistance

Written with AI assistance (Claude Code), reviewed by submitter.

[valeriyischenko]

Closing — superseded by the 2026-08-30 submission plan. This PR shrinks before upstream submission: keep the two string declarations (budget enablement/enforcement), drop count_reasoning_tokens (ceded to vllm-project#54238) and the forced_reasoning_end_str base property (motivating example removed by the tool-routing A/B; Muse keeps forced=None). Branch is kept for reference.