[Rust Frontend] Replay roundtrip output grammars through XGrammar

[BugenZhao]

Purpose

Rust-built output grammars are currently checked only as structural-tag JSON. Nothing compiles them with the real XGrammar or verifies that the matcher accepts a real generation from token zero, which is the meaning of reasoning_ended = Some(true) since #57340.

Step 2.5 of the parser-owned output grammar rollout, following the merged #55269 and #57340. Stacked on #59148, which switches Rust MiMo to the MiMo structural-tag builder in xgrammar-structural-tag 0.4.1; only the last five commits belong to this PR. It adds test coverage only; production behavior is unchanged. The open-PR search found no other XGrammar replay harness for the Rust frontend.

• The chat roundtrip builds the output grammar from the same initialized processor that parses the completion. It runs with --tool-strict-level parameter so every builder-backed parser emits a grammar; the strict level affects grammar inputs only, not parsing.
• tool_call_mix runs once per tool-choice variant for every model:
  • auto, with content before the tool calls;
  • required, with tool calls only, the only shape builder grammars admit under required;
  • named get_weather, with a single call and no content.
• Each model's grammars are snapshotted into one rust/src/chat/tests/grammar_replay/<model>.json via expect_file!. The file records the model, model vocab size, and lowered stop set once. Each variant case records the BuiltOutputGrammar, the completion text, and the generation token IDs ending with a stop token. A stale file for a parser that no longer builds a grammar fails the roundtrip.
• grammar_replay.py is a PEP 723 uv script pinned to the xgrammar version in requirements/common.txt, with CPU torch. It builds TokenizerInfo and compiles each case the same way as backend_xgrammar.py. For from_token_zero cases it replays every token, checking both the bitmask and accept_token, and requires the stop token to terminate the matcher with override_stop_tokens. final_output_only cases are compiled only. It also requires the HF tokenizer to encode the completion to the recorded IDs, so tokenizer drift fails loudly.
• grammar_replay.rs wraps the script as a nextest test, so the existing Rust Frontend Cargo CI step runs it. That step already installs uv.
• BuiltOutputGrammar and GrammarCoverage derive Serialize. serde_json's raw_value is enabled for chat dev-dependencies only, to keep token ID arrays on one line.

Kimi K3 renders request-scoped control messages, such as the tool_choice=required instruction, after the whole history, as the official encoding_k3.py does. The generation prompt is then A + C + G while the closed history is A + G + B + C, so the prompt is no longer a prefix of the full render. The roundtrip now locates the completion B as the unique split of C and G. It falls back to the prefix cut when C is empty, which is the case for every other template. K3's grammar remains final_output_only here; it becomes a token-zero replay case with its native builder in Step 3.

The replay found the MiMo V2.6 grammar mismatch that #59148 fixes. MiMo renders compact <tool_call><function=...> calls, but on main the Rust MiMo parser reuses the Qwen3-Coder builder, whose tags are newline-delimited. Under required the replay rejected the call. Under auto the trigger never fired, so the grammar left the calls unconstrained. With #59148, MiMo runs all three variants, and its auto trigger and tags are compact. #58019 is the Python counterpart.

Test Plan

cd rust
cargo nextest run -p vllm-chat --test roundtrip --test grammar_replay
UPDATE_EXPECT=1 cargo nextest run -p vllm-chat --test roundtrip  # regenerate cases
cargo nextest run -p vllm-parser -p vllm-chat
cargo clippy --workspace --all-targets --all-features -- -D warnings

Negative controls: I copied the cases, mutated them, and ran the script on the copies.

Test Result

• 12 models, 36 cases, xgrammar-structural-tag 0.4.1 output replayed with XGrammar 0.2.7, the version pinned in requirements/common.txt. XGrammar 0.2.7 ignores the new excludes field. The runner pin moves together with requirements/common.txt.
  • auto, required, and named, replayed from token zero and terminated: Qwen3, Qwen3.5, MiMo V2.6, MiniMax M2.5, DeepSeek V3.2/V4/V4.1, GLM-4.7/5.2, Hy3, and Kimi K2.5.
  • Kimi K3: auto, required, and named, final_output_only, compiled.
• All five negative controls fail at the expected token:
  • K3 relabeled from_token_zero: rejects think after <|close|>.
  • Qwen3 without the reasoning wrapper: rejects <think> at token 0.
  • DeepSeek V4 with a stop set lacking its terminator: does not terminate.
  • Unclosed final tool call: rejects <|im_end|>.
  • Completion text drift: reports a tokenizer mismatch.
• The named grammar replayed against the two-call required generation rejects the second call right after the first closes, for Qwen3, DeepSeek V4, and GLM-4.7.
• A modified case and a stale case each fail the roundtrip.
• vllm-parser + vllm-chat: 860 tests passed, including a unit test for the completion split. Workspace Clippy, cargo sort, and pre-commit passed.
• The replay takes about 28 s with warm uv and HF caches, dominated by loading 12 tokenizers. A cold uv environment added about 50 s locally.
• trust_remote_code=True is needed only for the Kimi K3 tokenizer. It is kept for all cases because production deployments of these models load their tokenizers with it.

AI assistance: implementation and validation assisted by Claude Code.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[BugenZhao]

/ci run --allow-stale

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @BugenZhao.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @BugenZhao.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #92000 for commit 58ae5aa8bfa3.

[mergify]

Hi @BugenZhao, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #92066 for commit e6ce31dce2ec.