[bugfix] Fix online serving crash when text type response_format is received

[cjackal]

Purpose

Fix #26639.

Especially, this PR adds a proper input validation to StructuredOutpusParams not to generate unschedulable structured outputs params, and adjust sampling parameter generation logic in ChatCompletionRequest.to_sampling_parameters() and OpenAIServingResponses.create_responses() to reflect the change and be more robust.

co-authored by @j0shuajun who first reported the issue and minimal reproducible examples on our side.

Test Plan

Pass a chat completion request with "response_format": {"type": "text"}:

curl -XPOST http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"openai/gpt-oss-120b","messages":[{"role":"user","content":"hello"}],"max_tokens":2048,"stream":false,"response_format":{"type":"text"}}'

Plus, pass a chat completion request with "response_format": {"type": "json_object"} to check for regressions.

curl -XPOST http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"openai/gpt-oss-120b","messages":[{"role":"user","content":"hello"}],"max_tokens":2048,"stream":false,"response_format":{"type":"json_object"}}'

Test Result

Return a valid response in both cases.

# response_format text - respond with normal text
{"id":"chatcmpl-b6a22dfca265460ab316d7ea490d974b","object":"chat.completion","created":1760457543,"model":"openai/gpt-oss-120b","choices":[{"index":0,"messages":{"role":"assistant","content":"Hello! How can I assist you today?","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":"We need to respond: greeting. No instructions."},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":70,"total_tokens":99,"completion_tokens":29,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_tokens_ids":null,"kv_transfer_params":null}

# response_format json_object - respond with JSON object
{"id":"chatcmpl-9643189ba2674cc2b471bd9a63472e
69","object":"chat.completion","created":1760457566,"model":"openai/gpt-oss-120b","choices":[{"index":0,"messages":{"role":"assistant","content":"{\"response\":\"Hello! How can I assist you today?\"}","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":"The user says \"hello\". We just respond politely."},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":70,"total_tokens":104,"completion_tokens":34,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_tokens_ids":null,"kv_transfer_params":null}

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results

[gemini-code-assist]

Code Review

This pull request addresses a crash in the online serving endpoint when a chat completion request includes response_format: {"type": "text"}. The fix correctly prevents the creation of an invalid, empty StructuredOutputsParams object by adding stricter validation and refining the parameter handling logic. The changes are well-implemented. However, I've identified a critical regression where the new validation can cause crashes for requests using certain deprecated parameters. A fix is suggested in the detailed comment.

[gemini-code-assist]

This new validation is a great improvement for ensuring StructuredOutputsParams is always in a valid state. However, it introduces a potential regression for deprecated parameters.

Specifically, the logic for handling deprecated guided_* parameters in vllm/entrypoints/openai/protocol.py (lines 794-806) can now raise this ValueError. If a user provides only guided_whitespace_pattern (which maps to whitespace_pattern), the code will attempt to create StructuredOutputsParams with only a non-constraint parameter. This will cause count to be 0 here, triggering this error.

While the problematic code is not in this diff, this change makes it faulty. To prevent this regression, the logic for handling deprecated parameters should be updated to only construct StructuredOutputsParams if at least one constraint parameter (e.g., guided_json, guided_regex) is provided.

For example, in ChatCompletionRequest.to_sampling_params in vllm/entrypoints/openai/protocol.py, the logic could be adjusted:

# ... inside to_sampling_params, after collecting kwargs from deprecated params
            kwargs = {k: v for k, v in kwargs.items() if v is not None}
            constraint_keys = {'json', 'regex', 'choice', 'grammar', 'structural_tag'}
            if any(k in constraint_keys for k in kwargs):
                self.structured_outputs = StructuredOutputsParams(**kwargs)

This would ensure backward compatibility for the deprecated parameters while upholding the new, stricter validation.

[cjackal]

AFAIK guided_whitespace_pattern must be given with one of the structural constraints, otherwise the same no structured parameter error is raised (so it is not a newly introduced regression but pre-existing bug).

Though I'd welcome a better way to validate the structured outputs params.

[cjackal]

I think we can do much better response_format validation after #26519; will rework if this PR does not get merged before #26519

[cjackal]

@chaunceyjiang Would you mind having a look at this PR? As more clients are using response_format, this bug is increasingly disruptive in terms of server stability. I think StructuredOutputsParams without a compilable grammar, which crashes the server at the grammar compilation stage, should not be allowed to be created in the first place.

[zifeitong]

This if clause is redundant

[cjackal]

Indeed, recent code change in upstream makes this if clause funny 😄. Thank you for pointing it.

[zifeitong]

There is no need to introduce kwargs_changes?

How about just replace self.structured_outputs.json_object = True with self.structured_outputs = StructuredOutputsParams(json_object=True)

The same applies to all the other cases.

[cjackal]

We'd like to inherit from the original StructuredOutputsParams due to the other options like whitespace_pattern, so I updated over the existing parameters, not newly create it.

Would we like to ignore these options for the response_format codepath?

[cjackal]

I'd like to mention that the whole logic around kwargs_changes and dataclasses.replace is there just to validate StructuredOutputParams. We can achieve the same effect by validating on assignment; Pydantic dataclasses already support this by adding ConfigDict(validate_assignment=True) and replacing __post_init__ to @pydantic.model_validator decorator.

cc @hmellor Is it the way to go from the point of pydantic validation refactoring? Most models on vllm.sampling_params are using msgpack and currently StructuredOutputParams is the only exception with a small comment on "maybe make msgpack". If we can keep the StructuredOutputParams pydantic, all the mess around the current state of StructuredOutputParams not validated during the creation/modification can be nicely gone.

[chaunceyjiang]

@cjackal I’m really sorry—I missed this PR. Could you rebase main onto your branch so we can move forward?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @cjackal.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[cjackal]

@cjackal I’m really sorry—I missed this PR. Could you rebase main onto your branch so we can move forward?

No worries, I will rebase tonight. Or I have granted push permission to maintainers, feel free to rebase by yourself if urgent.

[chaunceyjiang]

After #32127 is merged, vLLM will no longer crash. However, I still think this PR is a nice improvement.

[cjackal]

After #32127 is merged, vLLM will no longer crash. However, I still think this PR is a nice improvement.

Indeed, this PR looks more like a general code quality improvement + unit test addition for now 😄 Thanks for the review!

[chaunceyjiang]

thanks~