[Mistral] Use get_grammar_for_chat_request in MistralParser and require mistral-common 1.13.0 by juliendenize · Pull Request #64169 · vllm-project/vllm

juliendenize · 2026-10-28T00:36:31Z

Overview

Switch MistralParser.adjust_request to mistral-common's single-pass grammar entry point GrammarFactory.get_grammar_for_chat_request (new in mistral-common 1.13.0) and require mistral_common >= 1.13.0.

Claims

vLLM no longer picks the grammar mode for Mistral tokenizers. It builds a mistral-common ChatCompletionRequest with from_openai(**request.model_dump(include={messages, tools, tool_choice, parallel_tool_calls}, exclude_none=True), response_format=...) and mistral-common renders the grammar. This replaces the manual get_lark_from_jinja / get_lark_for_json_schema routing.
response_format is passed to mistral-common unchanged and follows its defaults: a json_schema is enforced by the grammar only when strict is true; otherwise any JSON object or array is accepted. structured_outputs.json has no strict field and stays enforced. See the open question for maintainers on _mistral_response_format.
check_json_nesting and the errors for unsupported structured_outputs / response_format types stay in vLLM.
mistral-common validation errors (pydantic ValidationError, e.g. an invalid JSON schema, and InvalidRequestException, e.g. a JSON response format with tools and tool_choice="required") surface as VLLMValidationError (400) instead of 500. See the open question for maintainers on the except line.
Bump mistral_common to >= 1.13.0 in requirements/common.txt and the test requirements; the compiled test pins move to 1.13.0. Its dependency floors are already met by the existing pins, so no other pin changes.

Behaviour changes for Mistral grammar-capable tokenizers

Text + no tools + tool_choice auto/None: no grammar is attached and _grammar_from_parser stays False (previously a text grammar was attached).
tool_choice="required" or a named tool: on the base template the model can no longer write text before the tool call; a named choice pins the tool name. Think templates are unchanged.
JSON response format + tools + tool_choice="auto": the grammar allows only the JSON answer or tool calls; plain text is rejected.
json_schema without strict: true: the grammar constrains output to any JSON object or array instead of the schema (previously always enforced).
json_schema with no schema: 400 (previously fell back to json_object).
parallel_tool_calls: null: treated as allowed, as in OpenAI (previously False).

Validation

macOS CPU, VLLM_TARGET_DEVICE=cpu HF_HUB_OFFLINE=1, mistral-common 1.13.0, xgrammar 0.2.8, huggingface_hub 1.31.0.

pytest tests/parser/mistral/ tests/tokenizers_/test_mistral.py tests/renderers/test_mistral.py \
  tests/v1/structured_output/test_backend_xgrammar_lark.py \
  tests/parser/engine/test_replay.py tests/parser/engine/test_delegating_replay.py tests/parser/test_beam_parser.py

Code	mistral-common	Result
this PR	1.13.0	5304 passed, 0 failed
`main`	1.13.0	26 failed: `get_lark_from_jinja` now requires `force_tool_names`, and `reasoning_effort="medium"` is now accepted

New tests check the grammar text adjust_request returns for required, a named tool, and JSON schema + tools + auto. They fail when the grammar is changed back to allow free text before the tool call.
New tests check that strict reaches mistral-common unchanged, that the schema appears in the grammar only when strict is true, and that an invalid schema raises VLLMValidationError.
pre-commit --hook-stage manual passes, including the four pip-compile hooks. In the compiled test requirements only mistral-common moves to 1.13.0, and typing-extensions drops it from its "via" list (mistral-common 1.13.0 no longer depends on it).
Not run: GPU serving, end-to-end requests, model evals.

Details

tests/parser/mistral/test_tool_calls.py: existing adjust_request tests now patch get_grammar_for_chat_request and assert the request it receives. Added grammar-content, strict, invalid-schema and no-grammar tests. Removed the null_tool_choice case of the parametrized grammar test, since that request now correctly gets no grammar.
tests/v1/structured_output/test_backend_xgrammar_lark.py: pass force_tool_names=False, now required by get_lark_from_jinja. This is the only direct get_lark_from_jinja call left in vLLM.
tests/tokenizers_/test_mistral.py: mistral-common 1.13.0 accepts all OpenAI reasoning_effort levels, so the test asserting "medium" is rejected is replaced by a check that every level is accepted.
Removed the now-unused _DEFAULT_JSON_SCHEMA and the manual tool_choice / message / tool conversion.
The non-grammar branch (pre-v11 tokenizers, unsupported structured outputs) is unchanged.
Not a duplicate: no open PR touches get_grammar_for_chat_request or the mistral-common 1.13.0 bump.

Pull Request Checklist

I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Replace the manual get_lark_from_jinja / get_lark_for_json_schema routing in MistralParser.adjust_request with the single-pass mistral-common entry point GrammarFactory.get_grammar_for_chat_request, available from mistral-common 1.13.0. - Build the mistral-common ChatCompletionRequest with from_openai from the request's messages, tools, tool_choice and parallel_tool_calls, and let mistral-common pick the grammar mode. - Pass the client's response_format through, so the schema is enforced according to mistral-common's defaults (only when `strict` is true). structured_outputs.json keeps enforcing its schema. - Convert mistral-common validation errors to VLLMValidationError. - Set no grammar, and leave _grammar_from_parser unset, for text requests without tools and with auto tool choice. - Require mistral-common >= 1.13.0. It accepts every reasoning_effort level, so the test that rejected "medium" is replaced. Committed with AI assistance. Signed-off-by: Julien Denize <40604584+juliendenize@users.noreply.github.com>

juliendenize · 2026-10-28T08:03:10Z

+            "MistralParser only accepts `text`, `json_object` or "
+            f"`json_schema`, got {request.response_format=}"
+        )
+    return request.response_format.model_dump(by_alias=True, exclude_none=True)


Question for maintainers: this passes the client's response_format to mistral-common as-is, so the schema follows mistral-common's defaults. It is enforced by the grammar only when json_schema.strict is true; with strict unset or false the grammar accepts any JSON object or array. That matches OpenAI's meaning of strict and the prompt rendering in vllm/renderers/mistral.py, which already passes the same field.

The trade-off: structured_outputs_from_response_format enforces the schema for other models whatever strict says, and most clients do not send strict. Before this PR, Mistral models also always enforced it. structured_outputs.json stays enforced here because it has no strict field.

Should Mistral follow strict (this PR), or keep enforcing the schema regardless, like other models?

juliendenize · 2026-10-28T08:03:11Z

+            lark_grammar = grammar_factory.get_grammar_for_chat_request(
+                request=mistral_request, template=template
+            )
+        except (ValidationError, InvalidRequestException) as e:


Question for maintainers: mistral-common validates the request here. Its pydantic.ValidationError (e.g. a schema that is not valid Draft 7 JSON Schema) and InvalidRequestException (e.g. a JSON response format with tools and tool_choice="required") both become VLLMValidationError, so the client gets a 400 with mistral-common's message instead of a 500.

Is surfacing mistral-common's messages directly OK, or would you rather validate these cases in vLLM's request validators so the messages and parameter fields match the other models?

We should pass parameter field if you could narrow it down. The error message itself could be forwarded as long as it doesn't leak the user's message (since errors may be stored in logs)

As discussed offline, we could follow up on this later

NickLucche · 2026-10-28T13:29:35Z

/ci run

github-actions · 2026-10-28T13:29:47Z

✅ @juliendenize, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

github-actions · 2026-10-28T13:31:11Z

✅ Triggered Buildkite CI #97962 for commit 0c8d81906f0b.

ℹ️ This PR is 46 commits behind upstream main. Merged upstream main into this branch as 0c8d81906f0b before starting CI. Pull the branch before pushing more changes.

mergify · 2026-10-28T22:34:45Z

Hi @juliendenize, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

DarkLight1337 · 2026-10-29T04:52:11Z

/ci run

github-actions · 2026-10-29T04:52:32Z

✅ Triggered Buildkite CI #98205 for commit 0bd7c99676c1.

NickLucche · 2026-10-29T09:15:59Z

/ci retry

github-actions · 2026-10-29T09:16:22Z

✅ No failed, timed-out, or expired jobs need retrying: https://buildkite.com/vllm/ci/builds/98205

NickLucche · 2026-10-29T09:16:40Z

/ci run

github-actions · 2026-10-29T09:17:59Z

✅ CI is already running for this commit: https://buildkite.com/vllm/ci/builds/98205

NickLucche · 2026-10-29T11:17:26Z

/ci retry

github-actions · 2026-10-29T11:18:01Z

✅ No failed, timed-out, or expired jobs need retrying: https://buildkite.com/vllm/ci/builds/98205

mergify Bot added ci/build mistral Related to Mistral models cpu Related to CPU backends structured-output labels Oct 28, 2026

github-project-automation Bot added this to Structured Output (arpera) Oct 28, 2026

mergify Bot added the tool-calling label Oct 28, 2026

github-project-automation Bot added this to Structured Output, Tool Calling (arpera) and Tool Calling Oct 28, 2026

github-project-automation Bot moved this to Unsorted queue in Structured Output (arpera) Oct 28, 2026

github-project-automation Bot moved this to Unsorted queue in Tool Calling (arpera) Oct 28, 2026

github-project-automation Bot moved this to Unsorted queue in Tool Calling Oct 28, 2026

juliendenize marked this pull request as ready for review October 28, 2026 00:37

juliendenize requested review from aarnphm and mgoin as code owners October 28, 2026 00:37

claude Bot reviewed Oct 28, 2026

View reviewed changes

juliendenize requested review from DarkLight1337, arpera, bbrowning, chaunceyjiang, njhill and sfeng33 as code owners October 28, 2026 00:37

joa-stdn reviewed Oct 28, 2026

View reviewed changes

Comment thread vllm/parser/mistral.py Resolved

joa-stdn reviewed Oct 28, 2026

View reviewed changes

Comment thread vllm/parser/mistral.py Resolved

joa-stdn reviewed Oct 28, 2026

View reviewed changes

Comment thread vllm/parser/mistral.py Resolved

juliendenize force-pushed the mistral-grammar-chat-request branch from 6b527c6 to 6c2d7ee Compare October 28, 2026 01:03

joa-stdn reviewed Oct 28, 2026

View reviewed changes

Comment thread vllm/parser/mistral.py Resolved

juliendenize force-pushed the mistral-grammar-chat-request branch from 6c2d7ee to 8ee9916 Compare October 28, 2026 08:00

juliendenize commented Oct 28, 2026

View reviewed changes

NickLucche added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 28, 2026

Merge branch 'main' into mistral-grammar-chat-request

0c8d819

Merge branch 'main' into mistral-grammar-chat-request

22de22e

juliendenize and others added 2 commits October 29, 2026 00:26

Merge branch 'main' into mistral-grammar-chat-request

e304604

Merge branch 'main' into mistral-grammar-chat-request

0bd7c99

DarkLight1337 approved these changes Oct 29, 2026

View reviewed changes

DarkLight1337 enabled auto-merge (squash) October 29, 2026 05:07

DarkLight1337 merged commit e5571b9 into vllm-project:main Oct 29, 2026
445 checks passed

github-project-automation Bot moved this to Done in Structured Output Oct 29, 2026

vMaroon mentioned this pull request Oct 29, 2026

[Bugfix][Parser] Parse closed GLM-4.7 tool arguments once while streaming #64535

Merged

6 tasks done

Shawnsuun mentioned this pull request Nov 2, 2026

[Bugfix] Require mistral-common 1.13.1 for developer role support #64861

Open

6 tasks done

Conversation

juliendenize commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Claims

Behaviour changes for Mistral grammar-capable tokenizers

Validation

Details

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

juliendenize Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

juliendenize Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

NickLucche commented Oct 28, 2026

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

mergify Bot commented Oct 28, 2026

Uh oh!

DarkLight1337 commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

NickLucche commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

NickLucche commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

NickLucche commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

juliendenize commented Oct 28, 2026 •

edited

Loading