[Frontend] Upgrade XGrammar to 0.2.7 and Rust structural tags to 0.3.0

[BugenZhao]

Purpose

Pin the shared Python frontend/engine dependency to xgrammar==0.2.7 and regenerate the CPU, CUDA, ROCm, and XPU test lockfiles. The CPU lockfile also updates apache-tvm-ffi to satisfy XGrammar's minimum version.

Use the upstream deepseek_v4_1 structural-tag builder through the standard builtin route, removing the local wrapper and older-version fallback. This makes V4.1 parameter-schema enforcement consistently available. The CLI parser name remains deepseek_v41; serving continues to pass reasoning=False because the engine owns the reasoning boundary. Existing tests cover required/named/strict-auto choices, invalid parameters, and non-strict parallel calls.

Upgrade Rust xgrammar-structural-tag to 0.3.0, which tracks XGrammar 0.2.7. Adapt the local Kimi K3 builder to ReasoningMode, preserving its enabled/disabled behavior and handling the new auto mode. Refresh Kimi K3 and HY serialization snapshots for the updated structural-tag format; token and character budgets remain unset.

Replace Rust's local DeepSeek V4.1 builder with Model::DeepSeekV41.builder(), enabling parameter-schema enforcement through deepseek_v4_1_xml. Kimi K3 keeps its handwritten typed-argument grammar, including type/value correlation and marker-safe enum handling, alongside its existing channel framing and raw-JSON argument branch.

Related to #54497's dependency-upgrade proposal. Its max_whitespace_cnt configuration plumbing is separate work. A search of open XGrammar upgrade PRs, 0.2.7 references, and #54497 references found no overlapping implementation. #56408 already added conditional use of the upstream V4.1 builder; this change makes that dependency available consistently and removes the compatibility path.

Test Plan

.venv/bin/python -m pytest -q \
  tests/tool_parsers/test_structural_tag_registry.py \
  tests/parser/engine/test_deepseek_v41.py

.venv/bin/python -m pre_commit run --files \
  requirements/common.txt \
  requirements/test/cpu.txt requirements/test/cuda.txt \
  requirements/test/rocm.txt requirements/test/xpu.txt \
  vllm/tool_parsers/structural_tag_registry.py \
  vllm/tool_parsers/deepseekv41_engine_tool_parser.py \
  tests/tool_parsers/test_structural_tag_registry.py

.venv/bin/python -m pre_commit run --files \
  rust/src/parser/src/tool/deepseek_dsml/deepseek_v41.rs \
  rust/src/parser/src/tool/deepseek_dsml/deepseek_v41/tests.rs \
  rust/src/parser/src/unified/kimi_k3/structural_tag.rs

cargo fmt --manifest-path rust/Cargo.toml --all --check
cargo clippy --manifest-path rust/Cargo.toml \
  --workspace --all-targets --all-features --locked -- -D warnings
cargo nextest run --manifest-path rust/Cargo.toml \
  -p vllm-parser --all-features --locked

Test Result

• Final builder-routing and DeepSeek V4.1 parser suites: 109 passed with XGrammar 0.2.7.
• Final Rust parser suite: 457 passed, 0 skipped. Workspace Clippy with all targets/features and warnings denied passed during the upgrade; final formatting checks passed.
• A temporary local Rust exporter and Python XGrammar 0.2.7 matcher harness verified 40 DeepSeek V4.1 cases, covering auto/required/named choices, strict opt-out, any_order, missing/duplicate/invalid parameters, enums, nested references, and integer/null unions. This supplemental harness is outside the repository.
• Additional validation during the upgrade passed for structured-output request validation, stop-token masking, regex compilation timeouts, structured-output manager behavior, and Cohere structural tags. The manager suite retained one existing Guidance xfail.
• Pre-commit passed, including requirement-lock regeneration, Ruff, and type checking; git diff --check passed.
• Validation used Python 3.12 on Linux aarch64 with existing precompiled vLLM extensions, Torch 2.11.0, and Transformers 5.11.0. CPU grammar/parser tests were run; GPU model generation and accuracy evaluations remain pending.

AI assistance: Codex implemented the changes, ran local validation, and prepared the PR for review.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[abmfy]

LGTM! Two non-blocking edge-case notes inline on the K3 KimiK3Xml migration (union-typed argument correlation; the dropped unsafe-enum guard). The upgrade otherwise looks correct.

[abmfy]

cc @Ubospica @ywang96

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #89585 for commit 891bb8ef3aae.

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #89776 for commit 47641c047814.