[Bugfix][Reasoning] muse_glimmer: enable thinking_token_budget - #54462
valeriyischenko wants to merge 1 commit into
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
Documentation preview: https://vllm--54462.org.readthedocs.build/en/54462/ |
mergify
Bot
added
documentation
…ng_token_budget works ReasoningConfig.initialize_token_ids returns early unless it can resolve both boundary strings, so reasoning_config.enabled stayed False for MuseGlimmer and any request carrying a thinking_token_budget was rejected with HTTP 400 and "reasoning_config is not configured. Please set --reasoning-parser", which the user had already done. The parser declared no boundary strings because MuseGlimmer frames reasoning as a channel rather than delimiting it with a <think>/</think> pair. Declare the two strings that framing implies: " to=self<|message|>" opens a reasoning message and "<|eom|>" closes it. The start string keeps its leading space deliberately. The chat template ends the generation prompt after "<|start|>assistant", so the space belongs to the first generated token, and the budget matches token ids by exact slice, so the other spelling would silently never fire. The end string is deliberately not the full transition "<|eom|><|start|>assistant to=user<|message|>". That string does not just end reasoning, it decides the next recipient. Measured with one tool offered and a small budget, forcing it made all 12 samples answer the user first, in 10 of the 12 the tool call never happened and the answers fabricated the data the tool would have returned, while forcing the bare marker left tool calls intact. The budget therefore bounds each reasoning message, and the recipient of the next message stays with the model. Signed-off-by: Valerii Ishchenko <valeriy@ischenko.me> Co-authored-by: Claude
ecbc4a9 to
153bb1f
Compare
chaunceyjiang
left a comment
chaunceyjiang
left a comment
There was a problem hiding this comment.
Could you paste the actual outputs before and after the change?
Could you provide an actual before/after test comparison for this change?
For example, you could test it with something like:
```bash
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer EMPTY" \
-d '{
"model": "",
"messages": [
{
"role": "user",
"content": "What's the weather like in Beijing?"
}
],
"thinking_token_budget": 100,
"tools": [
{
"type": "function",
"strict": true,
"function": {
"name": "get_weather",
"description": "Get current weather for a city",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string"
}
},
"required": ["city"]
}
}
}
]
}'
|
Before the patch: curl -s localhost:11003/v1/chat/completions -H 'Content-Type: application/json' -d @- <<'JSON' | jq .
{"model":"muse","messages":[{"role":"user","content":"What's the weather like in Beijing?"}],"thinking_token_budget":8,"tools":[{"type":"function","strict":true,"function":{"name":"get_weather","description":"Get current weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]}
JSON
{
"error": {
"message": "thinking_token_budget is set but reasoning_config is not configured. Please set --reasoning-parser and/or --reasoning-config to use thinking_token_budget.",
"type": "BadRequestError",
"param": null,
"code": 400
}
}After the patch: curl -s localhost:11003/v1/chat/completions -H 'Content-Type: application/json' -d @- <<'JSON' | jq .
{"model":"muse","messages":[{"role":"user","content":"What's the weather like in Beijing?"}],"thinking_token_budget":8,"tools":[{"type":"function","strict":true,"function":{"name":"get_weather","description":"Get current weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]}
JSON
{
"id": "chatcmpl-97209a65b4443271",
"object": "chat.completion",
"created": 1788289160,
"model": "muse",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"refusal": null,
"annotations": null,
"audio": null,
"function_call": null,
"tool_calls": [
{
"id": "chatcmpl-tool-b6f3d4cb3d2b6264",
"type": "function",
"function": {
"name": "get_weather",
"arguments": "{\"city\": \"Beijing\"}"
}
}
],
"reasoning": "What's the weather like in Beijing? Use"
},
"logprobs": null,
"finish_reason": "tool_calls",
"stop_reason": 200008,
"token_ids": null,
"routed_experts": null
}
],
"service_tier": null,
"system_fingerprint": "vllm-0.27.2rc1.dev116+gd4801990a-77d8df22",
"usage": {
"prompt_tokens": 402,
"total_tokens": 460,
"completion_tokens": 58,
"prompt_tokens_details": null,
"completion_tokens_details": {
"reasoning_tokens": 0
}
},
"prompt_logprobs": null,
"prompt_token_ids": null,
"prompt_text": null,
"kv_transfer_params": null,
"ec_transfer_params": null,
"metrics": null
}
|
This branch has not been deployed
2 participants
Purpose
Setting
thinking_token_budgeton any request served with--reasoning-parser muse_glimmerreturns HTTP 400 with "reasoning_config is not configured. Please set --reasoning-parser and/or --reasoning-config", advice the user has already followed. The budget feature is unusable for this model.The cause is that the parser declares no reasoning boundary strings. MuseGlimmer frames reasoning as a channel (a message addressed
to=selfand closed by<|eom|>) rather than delimiting it with a<think>/</think>pair, so the base-class defaults of None apply,ReasoningConfig.initialize_token_idsreturns early,reasoning_config.enabledstays False, and the input processor rejects every budgeted request.This PR declares the two strings that the channel framing implies:
to=self<|message|>opens a reasoning message and<|eom|>closes it. With them declared, budgets are accepted and enforced. On our serving setup a request withthinking_token_budget: 64produced exactly 64 reasoning tokens followed by a normal answer, and 256 produced exactly 256.The start string keeps its leading space deliberately. The chat template ends the generation prompt after
<|start|>assistant, so the space belongs to the first generated token, andtois a different token id fromto. The budget machinery matches these ids against the output by exact slice, so the other spelling would silently never fire. A docs note about this trap is included.Why the end string is the bare marker
The full transition the model writes to start answering is
<|eom|><|start|>assistant to=user<|message|>, and it looks like the natural forced-end string. It is not, because it does not just end reasoning, it decides the next recipient. Measured with one tool offered and a small budget, forcing the full transition made all 12 samples answer the user first. In 10 of the 12 the tool call never happened, and those answers fabricated the data the tool would have returned. The other 2 opened the tool channel after answering. A control without a budget chose the tool in 6 of 6 samples. Forcing the bare<|eom|>leaves the recipient with the model, and on this model the transition is then produced voluntarily (#44676 describes the corresponding failure class for<think>-family models, where the forced end lands inside tool-call arguments).One semantic point to state explicitly: MuseGlimmer may open several reasoning messages in one turn, and the budget bounds each message, not the turn. That is the existing meaning of
thinking_token_budgetfor every parser. Turn-level accounting is possible future work and is independent of this fix.Related work
count_reasoning_tokensfor MuseGlimmer. That is the usage-accounting side and is independent of this change, which is the enforcement side. They compose and can merge in either order.<think>-family models. The end-string choice above avoids the MuseGlimmer variant of that failure.Duplicate check, run before submission:
The first returns only this PR and #54467. The second returns #54238, #54091, #54186, and #53404, which change reasoning-token accounting, the
reasoning_strengthtemplate kwarg, and the structured-output gate. None of them touches budget enablement or the boundary strings.Test Plan
Three checkpoint-free tests over a stub tokenizer: the two declared strings are exactly what the model generates (leading space pinned),
ReasoningConfigends up enabled with forced == natural ==<|eom|>, and a user-provided--reasoning-configend string still overrides the parser's declaration while the parser's marker remains the natural end.Test Result
Against the real MuseGlimmer tokenizer the declared strings resolve to id slices
[328, 19669, 200023]and[200007], matching the segmentation the model emits (verified by sampling raw turns). End-to-end on our serving setup (Muse-Glimmer-30B, budgets 64/256): HTTP 400 before this change, exact-to-the-token enforcement with a normal answer after it. A full production run (6 seeds x 1874 requests, budget 3072 of max_tokens 4096) completed with zero requests at the token cap and zero parse failures.AI assistance (Claude Code) was used in developing this change. All code was reviewed and validated by the submitter, and the measurements are from our own runs.