Repository navigation
Conversation
An engine-based reasoning parser ends reasoning implicitly on a token that is also content, such as `<tool_call>`, and vllm-project#59627 feeds that token to the grammar. Bitmask rows stay unconstrained while the request reasons, so the model can sample such a token when the grammar cannot start with it (a JSON schema). accept_tokens then fails and the scheduler finishes the request with an error. While a request reasons, mask the implicit reasoning-end tokens its grammar rejects in its initial state. The explicit end and implicit ends the grammar accepts (required and named tool calls) stay allowed. The grammar does not advance before reasoning ends, so the set is computed once per request. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Maroon Ayoub <mayoub@redhat.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
mergify
Bot
added
structured-output
speculative-decoding
tool-calling
bug
github-project-automation
Bot
moved this to Unsorted queue
in Structured Output (arpera)
|
I verified this PR on H200 against the production failure we saw on GLM-5.3 ( Why it fixes the bug. Since #59627, an implicit reasoning end ( Setup. GLM-5.3-Flash-NVFP4, TP2, Without spec decode
With MTP spec decode (
The server log on main has one Notes:
This verification was done with AI assistance (Claude). |
|
cc @yzong-rh |
|
@simon-veitner-redhat your H200 run covers what the CPU validation here could not reach: Model Runner V2, async scheduling and MTP. The CPU end to end ran the V1 model runner, so between the two both runners are covered. I moved your results into the description. Agree the cut content is separate. The parser matches |
|
/ci run |
|
❌ @vMaroon, A reviewer with write access must run |
robertgshaw2-redhat
added
the
ready
|
/ci run |
|
✅ @vMaroon, CI is now available for this PR.
|
|
✅ Triggered Buildkite CI #94748 for commit |
|
@chaunceyjiang see below for E2E test. Reproducing the GLM-5.3 structured-output 500 (PR #61574)What failsGLM-5.3 can end reasoning implicitly by emitting PR #61574 masks implicit reasoning-end tokens in reasoning rows when the grammar cannot start with them. With SetupTwo builds of the same tree: main at git clone https://github.com/vllm-project/vllm && cd vllm
git checkout a2fb4827 # "main"
# PR build: git fetch origin pull/61574/head && git checkout FETCH_HEAD
VLLM_USE_PRECOMPILED=1 pip install -e . # Python-only change, precompiled kernels are fineServe on two GPUs. I used H200s. CUDA_VISIBLE_DEVICES=0,1 vllm serve RedHatAI/GLM-5.3-Flash-NVFP4 --served-model-name glm \
--tensor-parallel-size 2 --max-model-len 32768 --port 8000 \
--enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47MTP variant with 3 speculative tokens. CUDA_VISIBLE_DEVICES=0,1 vllm serve RedHatAI/GLM-5.3-Flash-NVFP4 --served-model-name glm \
--tensor-parallel-size 2 --max-model-len 32768 --port 8000 --max-num-seqs 256 \
--enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 \
--structured-outputs-config '{"backend":"xgrammar"}' \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'The failing requestThe conversation is a
repro_glm_so_500.py# Reproduces the GLM-5.3 "Failed to advance FSM ... for tokens 154843" HTTP 500
# (vllm-project/vllm#61574). Stdlib only. usage: python repro_glm_so_500.py [port] [n]
import json
import sys
import urllib.error
import urllib.request
PORT = sys.argv[1] if len(sys.argv) > 1 else "8000"
N = int(sys.argv[2]) if len(sys.argv) > 2 else 8
THINK_END, TOOL_CALL = 154842, 154843 # </think>, <tool_call>
# The tool result quotes GLM's own tool-call tags; that is what makes the model
# emit <tool_call> while still reasoning.
TAGGED = (
"# Parser notes\n\nGLM-5.3 wraps every function call in <tool_call>...</tool_call>"
" tags; each argument is <arg_key>k</arg_key><arg_value>v</arg_value>. Example:"
" <tool_call>Read<arg_key>file_path</arg_key><arg_value>a.txt</arg_value>"
"</tool_call>\n"
)
PLAIN = (
"# Parser notes\n\nGLM-5.3 wraps every function call in tool-call tags; each "
"argument is a key tag followed by a value tag. Example: a Read call with a "
"file_path argument.\n"
)
TOOLS = [
{"type": "function", "function": {"name": "Read", "description": "Read a file.",
"parameters": {"type": "object", "required": ["file_path"],
"properties": {"file_path": {"type": "string"}}}}},
{"type": "function", "function": {"name": "Write", "description": "Write a file.",
"parameters": {"type": "object", "required": ["file_path", "content"],
"properties": {"file_path": {"type": "string"},
"content": {"type": "string"}}}}},
]
SCHEMA = {
"type": "object",
"properties": {"summary": {"type": "string"}, "mentions_tags": {"type": "boolean"}},
"required": ["summary", "mentions_tags"],
"additionalProperties": False,
}
RF = {"type": "json_schema", "json_schema": {"name": "summary", "schema": SCHEMA}}
def messages(tool_result):
return [
{"role": "user", "content": "Read docs/parser.md, then give me a one-sentence "
"summary of it and whether it mentions tags."},
{"role": "assistant", "content": "", "tool_calls": [{"id": "call_1",
"type": "function", "function": {"name": "Read",
"arguments": "{\"file_path\": \"docs/parser.md\"}"}}]},
{"role": "tool", "tool_call_id": "call_1", "content": tool_result},
]
ROWS = {
"fail2: tagged + json_schema, no tools field": dict(
messages=messages(TAGGED), response_format=RF),
"fail3: tagged + json_schema, tool_choice none": dict(
messages=messages(TAGGED), tools=TOOLS, response_format=RF, tool_choice="none"),
"ctl: tagged + json_schema, tool_choice required": dict(
messages=messages(TAGGED), tools=TOOLS, response_format=RF,
tool_choice="required"),
"ctl2: untagged + json_schema, no tools field": dict(
messages=messages(PLAIN), response_format=RF),
# Deterministic: forces <tool_call> during reasoning. Spec decode rejects
# logit_bias with a 400, so this row only applies without MTP.
"det: logit_bias <tool_call>, json_schema": dict(
messages=[{"role": "user", "content": "What is the capital of France? Reply "
"with a summary and mentions_tags=false."}],
response_format=RF, temperature=0,
logit_bias={str(THINK_END): -100, str(TOOL_CALL): 10}),
}
def post(body):
req = urllib.request.Request(
f"http://localhost:{PORT}/v1/chat/completions", data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"})
try:
with urllib.request.urlopen(req, timeout=600) as r:
return r.status, json.loads(r.read())
except urllib.error.HTTPError as e:
return e.code, None
def json_valid(resp):
try:
obj = json.loads(resp["choices"][0]["message"].get("content") or "")
return isinstance(obj, dict) and set(obj) == {"summary", "mentions_tags"}
except (ValueError, KeyError, TypeError):
return False
for name, extra in ROWS.items():
codes, valid = {}, 0
for seed in range(N):
code, resp = post({"model": "glm", "max_tokens": 4000, "seed": seed, **extra})
codes[code] = codes.get(code, 0) + 1
valid += code == 200 and json_valid(resp)
print(f"{name:50} codes={codes} 500s={codes.get(500, 0)}/{N} json_valid={valid}/{N}",
flush=True)python repro_glm_so_500.py 8000 8 # port, seeds per rowThe rows:
Expected resultsMeasured with N=8 on 2xH200, TP2, 2026-10-14. Cells are HTTP 500 counts out of 8, with valid JSON counts in parentheses.
The Each 500 on main has one matching engine log line, and the PR logs none: grep "Failed to advance FSM" server.log
# ERROR ... [backend_xgrammar.py:269] Failed to advance FSM for request chatcmpl-... for tokens 154843. Please file an issue.The client sees Without a GPUThe PR's unit tests cover the masking logic on CPU: pytest tests/v1/structured_output/test_structured_output_manager.py \
-k "implicit_end or grammar_invalid_implicit_end"This reproduction and verification were done with AI assistance (Claude). |
|
@chaunceyjiang Added them to the description under Validation:
|
|
Thanks Maroon and Simon! This seems to fix the remaining issue from #60722. More specifically, this refines the approach where we read "a model's implicit reasoning end marker and disallow them during sampling" if said reasoning end marker isn't grammar-valid. The changes look reasonable to me. Only concern is that we are now constraining reasoning tokens despite
Would that have resolved this issue? IIUC, when we have a <tool_call>, we can either leak it (leading to #59608) or fail the request (as we do on main). Only way to resolve this is to prevent <tool_call> from being sampled in the first place by constraining reasoning, which is what this PR implements. |
|
/ci retry |
|
✅ Queued 1 failed job(s) for retry in Buildkite CI #94748. |
|
@simon-veitner-redhat Follow-up on the content cut you noted: it is tracked in #55080, with the fix in #55089, and it is separate from this PR. Checked on CPU against main |
|
@yzong-rh On the flag: I think the mask has to be the default, so a flag would only fit as an opt-out. Without the mask, a request whose grammar rejects the implicit end can still sample it and end in a 500, so an opt-in flag keeps that failure as the default. The constraint stays narrow: it removes only the implicit ends the request's own grammar rejects as its first token, On the first version of #59627: agreed. |
This branch has not been deployed
5 participants
Overview
With an engine-based reasoning parser, a structured-output request fails with "grammar rejected tokens" and HTTP 500 when the model samples an implicit reasoning end, such as GLM's
<tool_call>, that the request's grammar cannot start with. Reasoning rows of the bitmask are unconstrained, so the token can be sampled, and since #59627 it is fed to the grammar. This masks those tokens while the request reasons. Implicit ends the grammar accepts stay allowed.Related: #60722 (RFC; this is the "remaining issue" it describes, using its first option restricted to grammar-rejected tokens), #59627 (feeds the implicit end to the grammar), #59608 (
required/named tool calls, which must keep<tool_call>allowed), #61456 (open; also edits_fill_bitmasks).Claims
tool_choice="none"withresponse_format) can no longer sample that token while reasoning, so it no longer ends in "grammar rejected tokens".tool_choicerequired, named, andautowith tools (with or withoutresponse_format).</think>) and every other token stay allowed while reasoning. Draft and bonus rows under speculative decoding are masked the same way.Validation
Unit, CPU.
VLLM_TARGET_DEVICE=cpu, xgrammar and guidance:The 8 that fail on main:
test_real_flow[implicit_end_rejected-*](patternRRUUU; main givesUUUUU),test_reasoning_rows_mask_grammar_invalid_implicit_end[json-*],test_parallel_fill_masks_grammar_invalid_implicit_end[*], and the twoTestReasoningEndTokenIdscases for the new property.test_reasoning_rows_mask_grammar_invalid_implicit_end[regex_starts_with_marker-*]passes on both: a grammar that starts with the marker keeps it allowed and consumes it.Wider run on this branch:
tests/v1/structured_output,tests/parser(withouttests/parser/cohere, which needscohere_melody),tests/reasoning/test_glm4_moe_reasoning_parser.py,tests/v1/worker/test_grammar_invalid_drafts.py,tests/v1/spec_decode/test_mtp_structured_output.py: 6636 passed, 1 xfailed.tests/v1/core/test_async_scheduler.py: 19 passed on both.tests/v1/core/test_scheduler.py: the same 48 multimodal tests fail on main and on this branch, because this kernel-less install cannot inspect the Llava architecture.Real components, no model. GLM-5.3 tokenizer,
--reasoning-parser glm45(Glm47MoeParserReasoningAdapter), xgrammar, prompt ending in<think>. Bitmask rows while reasoning, thenaccept_tokens([<tool_call>])as the scheduler would call it:<tool_call>allowed / masked bitsaccept_tokens<tool_call>only;</think>allowed)requiredstructural tagEnd to end, CPU server. arm64 CPU image of main
206e45d47, whose four changed files are identical toa2fb48273, with the structured-output backends froma2fb48273mounted in both arms and this branch's four files in the second. A 2-layer Llama with the GLM-5.3 tokenizer and vocabulary,--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice, xgrammar, greedy, 3 runs per case.logit_biasmakes<tool_call>(154843) the top token.json_schema, bias<tool_call><tool_call>in reasoningjson_schema+ tools +tool_choice="none", same biasjson_schema, bias<tool_call>100,</think>50, answer tokens 30{"answer": "yes"}, finishstopjson_schema+ tools +tool_choice="auto", same biastool_choice="required"/ named /autowithoutresponse_formatjson_schema, no biasEvery request that returns 200 on main returns the same response on this branch. The engine log on main has one
Unexpected: grammar rejected tokens [154843]per 500 (12 of 12); the branch logs have none.Raw request and responses (third row)
{"model": "glm-tiny", "max_tokens": 96, "temperature": 0, "messages": [{"role": "user", "content": "Is the sky blue? Reply as JSON."}], "response_format": {"type": "json_schema", "json_schema": {"name": "a", "schema": { "type": "object", "properties": {"answer": {"type": "string", "enum": ["yes", "no"]}}, "required": ["answer"], "additionalProperties": false}}}, "logit_bias": {"154843": 100, "154842": 50, "4913": 30, "9214": 30, "788": 30, "330": 30, "9689": 30, "9204": 30, "154820": 30}}main, 3 of 3:
this branch, 3 of 3 (fields from the response):
Speculative decoding, end to end.
logit_biasis rejected with speculative decoding, so these runs use fixed weights instead: layer outputs are zero and the LM head prefers<tool_call>, thenthe, at every position. ngram, 3 speculative tokens,json_schema, 96 tokens:json_schemathejson_schema+ tools +tool_choice="none"json_schemaonly)required,auto+ toolsEach accepted draft position used a masked row; an unmasked draft row would have sampled
<tool_call>and failed the request.CPU harness note
On main the CPU stand-in
_compute_slot_mapping_kernel_impl(vllm/utils/cpu_triton_utils.py) takes its arguments in an order that no longer matchesComputeSlotMappingKernel.kernel, so the dispatcher passesslot_mappingasblock_sizeand the first forward fails. These runs mount a copy of that one function, with the kernel's parameter names, in both arms. Separate from this change.GPU, end to end. @simon-veitner-redhat ran main
a2fb482732against this branch6d144d6478on H200 (results): GLM-5.3-Flash-NVFP4, TP2,glm47tool and reasoning parsers, xgrammar, 8 seeds per row. "Tagged" means a tool result in the conversation quotes GLM's own tags (<tool_call>,<arg_key>,<arg_value>), followed by ajson_schemarequest.json_schema, no toolsjson_schema,tool_choice: "none"logit_biasforcing<tool_call>, blocking</think>json_schematool_choice: "required"<tool_call><tool_call>json_schema, no toolsjson_schema,tool_choice: "none"json_schematool_choice: "required"<tool_call><tool_call>Main logs one
Failed to advance FSMper 500; this branch logs none. The remaining invalid JSON on this branch is a separate frontend issue: the token stream is valid JSON, but the model quotes<tool_call>inside a JSON string andcontentis cut there. The CPU runs above use the V1 model runner, so both runners are covered.Motivation. A GLM-5.3 deployment (MTP with 3 tokens, Model Runner V2, xgrammar,
glm45) logged 649 "grammar rejected tokens" terminations in 8 hours; 625 of them ended on<tool_call>(154843). A non-streaming request terminated this way returns HTTP 500.Details
Path of the failure,
enable_in_reasoning=False:_get_constraint_boundsputsconstraint_startpast every scheduled row, sogrammar_bitmaskfills each row with the full mask.<tool_call>, whichGlm47MoeParsertreats as an implicit reasoning end: it ends reasoning and starts a tool call.accept_tokensfinds that end and, since [Bugfix][Structured Output] Feed the token that implicitly ends reasoning to the grammar #59627, setsgrammar_startto its offset, so<tool_call>is fed to the grammar. A JSON grammar rejects it andaccept_tokensreturns False.FINISHED_ERROR.The change, in
StructuredOutputManager:ParserEngine.implicit_reasoning_end_token_ids: reasoning-end tokens other than the explicit end, the same splitfind_reasoning_enduses to report an implicit end. Exposed throughParserEngineReasoningAdapter. This only adds a read-only property, so no parser changes behavior._masked_reasoning_ends: the implicit ends the grammar rejects as its first token (validate_tokens([t]), non-mutating in every backend). The grammar does not advance before reasoning ends, so the set is computed once per request and cached onStructuredOutputRequest._fill_bitmasks: a row the grammar does not constrain allows every token except those ids. Only rows beforeconstraint_start(including the bonus row while reasoning) get them, in both the serial and the parallel fill. Constrained rows and rows after padding are unchanged.Tradeoffs and limits:
deepseek_v4,inkling) are not covered.validate_tokenscall per implicit end per request, and a bit clear per masked id on reasoning rows. No device work and no host sync._fill_bitmaskswith a batch fill. Whichever lands second has to keep the reasoning-row mask.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
AI assistance (Claude) was used to investigate the failure, write the change and tests, and draft this description. I reviewed every changed line and ran the tests above.