[Bug]: qwen3.5 when enable response_format json_schema outputs garbled spaces

[Yyong25]

Your current environment

Environment
Use vllm docker images :
vllm/vllm-openai:nightly ,the image id is db683b1d4dce and the create time is 2026-03-27
Service startup command
vllm serve --model /data/deploy/Qwen3.5-9B/Qwen3.5-9B --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --served-model-name qwen3.5-9b-test --tensor-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 32768 --limit-mm-per-prompt {"image":12}

🐛 Describe the bug

Prompt
curl --location 'http://127.0.0.1:8000/v1/chat/completions'
--header 'Content-Type: application/json'
--data '{
"messages": [
{
"role": "system",
"content": "## 场景与角色\n\n这是一个AI监控场景。首先摄像头需要绑定CV算法（亦称“能力”）以进行事件检测，然后当事件发生时则会产生相应的告警。摄像头有名称、属地或属主，例如“xx摄像头”、“xx监控”、“银泰城的监控”。根据检测的内容，能力有相关的命名，如：人员入侵检测、烟火烟雾检测、占道经营检测等。事件发生时所产生的告警随之也有相应的类型名称，如：人员入侵、烟火烟雾等，同时事件还有时间、区域、所属摄像头、严重程度等属性。围绕AI监控场景，用户可能需要为摄像头绑定某种能力以进行事件检测，也可能需要查看某摄像头画面以了解当前情况，或者需要查看一些告警的详情，又或者需要对告警做统计分析。你负责结合对话上下文将当前用户输入分类到一个恰当的类别，并对该输入做必要改写使之能够独立完整明确地表达用户意图，以辅助下一步处理。具体地，分类如下：\n\n- 1代表事件统计分析，当用户目的是统计分析告警事件的数量、分布、趋势等情况时，属于此类。例如：“今天发生多少起xx事件”、“查询本周告警按地区（或类型）的分布情况”、“本月xx类事件的趋势”。\n- 2代表事件详情查询，当用户目的是查看告警事件的详情时，属于此类。例如：“查询今日的人员入侵”、“本周有哪些告警”、“显示xx摄像头的事件”。\n- 3代表调取监控，当用户目的是调取某摄像头的实时监控画面时，属于此类。例如：“查看大门口摄像头的画面”、“打开xx摄像头”、“调取xx摄像头”、“看下xx监控画面”。\n- 4代表创建与绑定能力，当用户目的是为摄像头创建和绑定能力时，属于此类。例如：“给门口的摄像头绑定人员入侵算法”、“xx摄像头启动对小女孩的检测”、“识别到蓝色轿车时，xx摄像头需要告警”，上述示例其实质是为某个/某些摄像头绑定某种能力，以开启相应事件的检测。\n- 5代表其它。当前用户输入可能不是一个合理有效的句子（结构混乱、语义不清、没有提供有效的信息（如“啊”“哈”）且结合上下文无法理解），或者虽然含义清楚，但不属于上述已定义的1~4有效类别，均应判为此类。\n对于类别1、2，请仔细判断，若用户意图未提及数量、分布、趋势等含义或者在两者间不易区分时，应理解为查询事件详情，即判定为类别2。\n摄像头的名称是开放的，通常会用属地或属主命名，也可能使用所绑定的能力进行命名。\n\n## 输出规则\n\n严格输出JSON对象，不要输出任何解释文字，不使用 markdown，不要加代码块等标记，只输出单纯的JSON对象，确保能够被JSON库解析。\n输出样例：\n{\n"questionType": "1", //1代表事件统计分析，2代表事件详情查询，3代表调取监控，4代表创建与绑定能力，5代表其它。\n"rewritingQuestion": "改写结果"\n}\n改写的注意事项：\n\n1. 改写结果应该语义完整明确，不依赖上下文就能够独立表达清楚用户当前的意图，并且不丢失相关信息，比如用户要绑定能力，那么为哪个/哪些摄像头绑定怎样的能力必须描述清楚。\n2. 改写结果应该仍是以用户口吻发出的。\n3. 当questionType取5时，rewritingQuestion字段不需要存在，即此时无需改写。"
},
{
"content": [
{
"type": "text",
"text": "465564867"
}
],
"role": "user"
}
],
"model": "qwen3.5-9b-test",
"max_tokens": 1000,
"stream": false,
"temperature": 0.1,
"top_p": 0.95,
"presence_penalty": 1.05,
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "rewritingQuestion",
"schema": {
"additionalProperties": false,
"type": "object",
"title": "QuestionResult",
"required": [
"questionType"
],
"properties": {
"questionType": {
"type": "string",
"description": "问题类型"
},
"rewritingQuestion": {
"type": "string",
"description": "改写后的问题"
}
}
},
"strict": true
}
},
"chat_template_kwargs": {
"enable_thinking": false
}
}'
Problem description
When I modify the user prompt word arbitrarily and call it several times(usually ten or more times), the model returns blank characters and stops outputting when the set length is reached

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Alex-ai-future]

Hi @Yyong25, I'd like to help investigate this issue. Could you please assign it to me?

To help reproduce and debug this problem, could you provide more details:

1. Error logs: Are there any error messages or warnings in the vLLM server logs when this happens?

2. Reproduction rate: You mentioned "usually ten or more times" - roughly what's the failure rate? (e.g., 3 out of 10 calls, or 1 out of 20?)

3. Output analysis: When the model returns blank spaces, is it:

   • All space characters ( )?
   • Empty response?
   • Partial JSON followed by spaces?

4. Version info: What's the exact vLLM version in the nightly image? (can check with vllm --version)

5. Without json_schema: Does the same prompt work correctly if you remove the response_format: json_schema setting?

I'll start by trying to reproduce this locally. Thanks!

[skiby-f]

Hello,
I’m experiencing the same issue on my side.

1. No relevant or specific logs are generated.
2. The issue is reproducible 100% of the time using the same prompt and the same image. If I modify the prompt, it works with that image. I’ve observed the same behavior with some other images as well.
3. The model returns "{\n" and then continues outputting only whitespace until the max_tokens limit is reached.
4. The problem occurs in both versions 0.17 and 0.18.1.
5. The request works correctly when json_schema is not used.

[Alex-ai-future]

This might appear to be a contradictory scenario: for "structure output" and "tool," only one mode should be selected.

However, I still believe this program might require a defensive design.

You could try avoiding the use of the tool.

And the reasoning feature might also make the error , you should not use it

[stltqhs]

I have encountered this issue as well; the output includes various whitespace characters—specifically spaces, newlines, and tabs. Testing with Qwen3.5-27B revealed no such problem, whereas both Qwen3.5-35B-A3B and Qwen3.5-397B-A17B exhibit this behavior. It appears that Dense models are unaffected, while MoE models suffer from this issue. One potential workaround is to suppress the output of whitespace characters by adding the following parameters: --structured-outputs-config.backend xgrammar --structured-outputs-config.disable_any_whitespace 1.

[Sunt-ing]

I can reproduce this on current main, on the reporter's exact Qwen3.5-9B, and it is not fixed since the reporter's nightly. With the issue's system prompt and an ambiguous input, roughly 1 in 3 plain response_format requests loop into whitespace until max_tokens ({\n"questionType": followed by only whitespace, finish_reason: length) — no image, tools, reasoning, or presence_penalty required.

My environment and exact reproduction

• vLLM: reproduced on 0.19.x, 0.20.x, and current main (built around commit 2588ec4f0).
• Model: Qwen/Qwen3.5-9B (the reporter's model; Qwen3_5ForConditionalGeneration, dense + vision).
• GPU: a single NVIDIA RTX PRO 6000 (sm_120). I ran with --enforce-eager --attention-backend TRITON_ATTN (plus VLLM_USE_FLASHINFER_SAMPLER=0); these are just sm_120 workarounds and are unrelated to the bug.
• Minimal server (no tools / reasoning / mm needed):

  vllm serve Qwen3.5-9B --structured-outputs-config.backend xgrammar
  

• Request: the monitoring system prompt from the issue, user message "465564867", the issue's response_format json_schema, temperature: 0.1, top_p: 0.95, max_tokens: 1000.
• Result: ~1/3 of requests return {\n"questionType": then whitespace to max_tokens; the rest return well-formed JSON.

Measured runaway rates (same prompt/schema/sampling)

version	config	runaway rate
0.19.x (≈ reporter's nightly)	default, +tool +reasoning, image+text	36 / 480 ("啊"+image ≈ 100%)
current main	default, +tool +reasoning	11 / 96 (text "465564867" ≈ 42%)
current main	default, plain response_format only	8 / 96 (text "465564867" ≈ 33%)
current main	disable_any_whitespace: true	0 / 64
0.19.x	disable_any_whitespace: true	0 / 64

Ablations (what does and doesn't matter)

• Not the tool/reasoning parser — plain response_format reproduces it; those flags only nudge the rate up slightly. (So "structured output + tool both on" isn't the cause.)
• Not presence_penalty — with presence_penalty: 0 it still loops (if anything slightly more).
• Checkpoint-specific, not a clean dense-vs-MoE split: the dense Qwen3.5-9B reproduces, while the same-architecture dense Qwen3.5-4B did not (0/128). I could not trigger it on the FP8 Qwen3.5-35B-A3B weights I had, but @stltqhs reports the 35B-A3B and 397B-A17B MoE models do reproduce — likely different weights/quantization. It comes down to a checkpoint's propensity to put mass on whitespace at the value position for ambiguous inputs (a clearly-classifiable input never loops).
• Present on 0.19.x, 0.20.x, and current main alike — it didn't regress into, or get fixed in, a particular release.

Mechanism. vLLM's xgrammar backend compiles JSON schemas with any_whitespace=True by default (compile_json_schema(..., any_whitespace=not disable_any_whitespace)) and never passes max_whitespace_cnt, so whitespace between JSON tokens is unbounded: feeding whitespace tokens at a value position, the matcher accepts an arbitrary number (400+ in a row) and never forces a non-whitespace token. When a checkpoint puts enough mass on whitespace there for an ambiguous input, nothing in the grammar forces it out and it runs to max_tokens.

Workaround that works today (matches @stltqhs) — on current main this is only accepted with an explicit xgrammar/guidance backend:

--structured-outputs-config.backend xgrammar --structured-outputs-config.disable_any_whitespace true

A possible direction for maintainers (bounded whitespace)

disable_any_whitespace removes whitespace entirely. For callers who want readable-but-bounded JSON, xgrammar's compile_json_schema already accepts max_whitespace_cnt, which vLLM never passes (so it defaults to unbounded). In my runs, a small bound (max_whitespace_cnt=1 or 2) gave 0 runaways while still allowing some whitespace; a loose value like 8 only let the model relocate the whitespace into the string value. Given this triggers under default settings via the standard response_format API, would the structured-outputs maintainers (cc @russellb @aarnphm) consider exposing that (and possibly defaulting it to a small bound)? Related in-flight work: disable_any_whitespace with backend=auto (#42414, earlier #42174); whitespace_pattern on V1 (#26362 / #34790, currently outlines-only); same root cause reported for Gemma (#42110). Happy to share the full repro scripts.

[estrella159]

Adding measured data here, because I hit this from a different direction (#34650) and spent a day quantifying it before realising it was already your issue. Three things that may be useful: @stltqhs's dense-vs-MoE observation reproduces on the Qwen3.6 generation and I can put numbers on it, speculative decoding turns out to be a large amplifier, and the workaround suggested in this thread has a trap in it.

Setup: NVIDIA RTX PRO 6000 Blackwell (sm120), vLLM 0.24.0, fp8 KV, xgrammar backend, strict response_format: json_schema, temperature=0.0, seed=12345, max_tokens=2048. 24 distinct prompts (simple arithmetic word problems, nothing exotic), 8 fresh server launches per arm, arms interleaved round-robin. Failure = the exact behaviour in this issue: valid JSON up to a key, then whitespace until max_tokens.

{ "answer": 240 , "reasoning_summary": "...240 km." , "confidence":   \n\n\n\n\n... (to 2048 tokens)

1. @stltqhs's dense-vs-MoE split reproduces, with a correction

Numbers below are with #44297 and #44993 applied. That matters: those patches fix a different bug (reasoning-boundary detection under spec-decode, #34650) whose failures also end at max_tokens, so on stock 0.24.0 the two are mixed together and the whitespace rate cannot be read cleanly. With the boundary bug removed, what is left is this issue in isolation:

loops per launch (out of 24 prompts), patched build	Qwen3.6-27B-FP8 (dense)	Qwen3.6-35B-A3B-FP8 (MoE)
spec-decode (MTP) off	0.75	0.25
spec-decode (MTP) on	0.62	3.75

The two failure modes are easy to tell apart in raw output: the boundary bug yields unconstrained prose or markdown fences, this issue yields schema-legal JSON padded with whitespace.

The MoE-is-worse pattern holds on this generation. But "dense models are unaffected" is not quite right: the dense 27B loops at a low but real rate (6 of 8 launches showed at least one), it just does not care whether spec-decode is on. Worth knowing if anyone is using dense-model results as a control.

(Caveat: my dense model is the hybrid-mamba Qwen3.6-27B-FP8, not the Qwen3.5-27B @stltqhs tested, so this is a related but not identical comparison.)

2. Speculative decoding is a large amplifier on MoE

This is the part I have not seen mentioned here. On the MoE, turning MTP on takes the loop rate from 0.25 to 3.75 per launch, roughly 15x. On the dense model it does nothing (0.75 vs 0.62). Paired within launch, patched-MTP-on vs MTP-off on the MoE is -3.50/24, p=0.0078, negative in all 8 reps; the same contrast on dense is +0.12, p=1.0.

So if you are reproducing this and getting inconsistent rates, spec-decode may be a hidden variable. It does not cause the loop (it happens with MTP fully off), it makes it far more likely on MoE.

3. The disable_any_whitespace workaround only works server-side

@stltqhs's suggested workaround is right, but there is a trap: setting disable_any_whitespace per request is silently ignored. It returns HTTP 200 and has no effect, because XgrammarBackend reads the flag from the engine config once at construction and never looks at StructuredOutputsParams. I filed that separately as #48765. Only the server-level form works:

--structured-outputs-config '{"backend": "xgrammar", "disable_any_whitespace": true}'

I mention it because it is the natural way to A/B this hypothesis, and it will quietly give you a false null. It cost me an hour; I only noticed because the "disabled" output still had spaces after the colons.

4. Why the rates in this thread vary so much

Reports here range from "1 in 3 requests" to "only with certain prompts". Some of that is likely launch-to-launch bistability: with an identical image, identical flags, temperature=0.0 and a fixed seed, I get materially different rates from consecutive container launches. Within a launch it is deterministic (a given prompt loops every time, or never, with byte-identical completion_tokens); across launches, which prompts loop changes and how many changes.

Practical consequence, learned the hard way: a single server launch cannot tell you whether a change helped. I published an underpowered A/B on #34650 and had to retract it. If anyone tests a fix for this, please replicate across launches (k>=8) and use many distinct prompts rather than repeating one, since repeats within a launch carry no extra information. Harness (stdlib only, public domain) is in this #34650 comment if useful.

On #44619

@Alex-ai-future's #44619 (max_whitespace_cnt, default 2) looks like it would address this directly, and it was closed unmerged with a needs-rebase. From where I sit the bug is real, current on 0.24.0, hits at least two model families, and has an open issue since April, so it seems worth reviving. Happy to test a rebased version at k=8 per arm on Blackwell across both the dense and MoE models and report per-launch distributions; just say.

The underlying hazard, as far as I can tell, is that the compiled JSON grammar permits an unbounded whitespace path between tokens, so the output stays grammar-legal forever. Every failing response I captured passed the bitmask at every token. That is why it presents as "valid JSON that never ends" rather than as a constraint violation.