[Rust Frontend] Split argument grammars into schema resolution and per-model rendering

[BugenZhao]

Purpose

Kimi K3 day-zero support (#50104) shipped with a handwritten argument grammar, because XGrammar had no K3 style. XGrammar 0.2.7 later added kimi_k3_xml, but it still disagrees with the K3 parser: it pins the type attribute only for single-type schemas, so under a union it accepts a label that does not match the value, for example type="number" followed by abc, while the parser decodes each value by its label. The XGrammar 0.2.7 upgrade (#57272) therefore kept the handwritten grammar.

The underlying problem is that depending on a backend style ties every new or fixed argument format to an XGrammar change, release, and upgrade. For keyed-parameter protocols, the frontend can build the argument grammar itself from structural-tag primitives, except for a few relaxations listed below. That work splits into two independent parts: resolving the JSON schema into parameters and their value options, which is generic, and rendering one parameter, which is model-specific. This PR makes that split and moves K3 onto it.

Stacked on #59393; only the last three commits belong to this PR. The open-PR search found no other Rust work on schema-derived argument grammars.

• output_grammar::arguments::arguments(schema, syntax, options) owns the parameter list and value options:
  • parameters follow the declared order, required ones are mandatory, and additional parameters come last; any_order allows any sequence instead;
  • each parameter's value options come from the schema, with local $refs resolved and anyOf/oneOf, type arrays, and enums split per option, each with its JSON type;
  • each reference target expands at most once per value, so a definition shared by several alternatives cannot blow up exponentially, and a schema whose references only cycle accepts any value. Twelve levels of four-way unions over references would otherwise yield 4^12 options, over 30 s from a few kilobytes of tool schema.
• A parser implements ArgumentSyntax to render one parameter from its value options, using building blocks rather than configuration: ValueOption::json() for a JSON region with the root definitions, ValueOption::raw_string(terminators) for raw text, and small helpers (one_of, group_by, ParameterKey::tag). Values nested below a parameter stay XGrammar JSON schema regions.
• Kimi K3's XtmlArguments replaces the handwritten typed arguments. The call tag and the raw JSON alternative are unchanged. Behavior changes:
  • parameters follow the declared order with required ones mandatory, instead of any order with at least one parameter;
  • a string enum value containing a close marker is dropped by itself, instead of turning the whole enum into free text;
  • $refs resolve before the type label is chosen;
  • properties: {} allows no parameters.
• Parsers whose XGrammar style already matches them keep it. This is for protocols that XGrammar does not cover, or covers in a way that does not match the parser.

Deliberate differences from XGrammar's XML styles, documented in the module: additional keys are not required to differ from declared ones, string format and length constraints and minProperties/maxProperties are not enforced, and allOf with several schemas accepts any value. Recursive non-JSON nesting is out of scope.

Test Plan

cd rust
cargo nextest run -p vllm-parser -p vllm-chat
cargo clippy --workspace --all-targets --all-features -- -D warnings

Test Result

• vllm-parser + vllm-chat: 874 tests passed, including all 36 XGrammar replay cases. K3 is still final_output_only on this base, so its three cases are compiled only; [Rust Frontend] Build the Kimi K3 full-output grammar from its parser mode #59190 replays them from token zero.
• Workspace Clippy and pre-commit passed.
• Out-of-tree differential check (not in this PR). The same K3 rendering, implemented through the public trait in a separate crate, was compiled with XGrammar 0.2.8 next to the kimi_k3_xml style. Both grammars judged 44 handwritten argument strings over 9 schemas, and they agreed on all but 6. In each of those 6, kimi_k3_xml accepted a type label that does not match the value, and this grammar rejected it. The samples come from a handwritten renderer, not the chat template, so this is supporting evidence only.

AI assistance: implementation and validation assisted by Claude Code.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[BugenZhao]

/ci run --allow-stale

[github-actions]

✅ Triggered Buildkite CI #92339 for commit 2eb9de550936.

⚠️ This PR is 75 commits behind upstream bz/grammar-outline. Running CI at your own risk because --allow-stale was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest bz/grammar-outline, then rerun /ci run on the latest PR commit.

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #92346 for commit cc8bd9872d20.