[Seven-Streams]

feat: support string exclusions in JSON schema structural tags

Extracted from #901 by @izhuhaoran, fourth piece after #906, #909 and #910. The branch keeps the original commit and its revert at the bottom for attribution; the net diff against main is the feat: support string exclusions in JSON schema structural tags commit plus feat: drop string length constraints when exclusions are present, a refactor of the exclusion helpers, and feat: apply exclusions only to strings without pattern or format. The Kimi-K3 builtin wiring that turns this on by default is the last piece and comes separately.

Motivation

Kimi-K3 tool arguments are raw text. The model can emit control markers such as <|close|>response<|sep|> inside an argument that is still open, then start another tools/call block. The schema grammar accepts all of that as string content, while the tool-call parser reads it as structure. There is no way today to tell the converter that certain substrings must never appear inside a string value.

API

JSONSchemaFormat gets an optional excludes: list[str] (C++, Python, TypeScript, docs):

JSONSchemaFormat(json_schema=..., style="kimi_k3_xml", excludes=["<|open|>", "<|close|>", "<|sep|>"])

• Forbids each substring in string values and property names, including nested JSON strings. It applies to the strings without pattern or format: a pattern or format is the schema's own contract for the string and is matched as is. Filtering those would intersect an arbitrary regex with the exclusion automaton, which reproduces the length blowup below for any pattern with a wide repetition (^[\s\S]{0,256}$ gives 4592 rules).
• minLength/maxLength are dropped when exclusions are set, with a warning naming the string and the bounds. Bounding a filtered string's length would multiply the grammar by the bound: one rule per position and exclusion state, about 18 rules per character for the three Kimi-K3 markers, which is 18 s and 111 MiB for maxLength: 256 and 110 s for ten maxLength: 128 properties. Without the bounds those schemas compile in 96 ms and 71 ms, as the unbounded strings do.
• Matches the emitted text before JSON/XML unescaping, may span tokens, and never spans separate strings. JSON delimiter quotes, numbers, punctuation and XML parameter wrappers are outside the scope, so <|close|> can still begin a valid <|close|>argument<|sep|> terminator.
• const/enum alternatives and fixed property names that contain an excluded substring are dropped; a required one makes the schema unsatisfiable rather than silently allowed.
• An omitted or empty list changes nothing: every exclusion code path is behind !excludes_.empty().
• Errors instead of silent bypass: XML styles reject exclusions that start or end with formatting whitespace, and cohere_xml and minimax_m3_xml reject non-empty exclusions.

Implementation

ExcludingString builds the FSM of the converter's own string body (the JSON string content with escapes, or the XML identifier), intersects it with the complement automaton of the excluded substrings (Intersect with the budgets from #906), prunes states that can no longer reach an accepting state so no dead-end prefix is ever offered in a mask, and re-emits the result as EBNF rules with codepoint transitions. UTF-8 byte paths are collapsed back to codepoint classes and never turned into a second transition for an ASCII character, which relies on the canonical encodings from #909. The closing JSON quote is appended after the intersection because it is syntax, not content. Property names go through the same intersection for additionalProperties keys and through a substring check for fixed keys; keys constrained by patternProperties/propertyNames patterns are matched as is like patterned values. A length-bounded string with exclusions uses the shared string rule (json) or the raw TagDispatch body (XML styles) instead of an intersected length bound.

Validation

• tests/python/test_json_schema_excludes.py: 91 cases covering every style, nested JSON, property names and literals, Unicode and overlapping exclusions, emitted-text semantics, patterned and formatted strings left unfiltered, dropped length bounds and their warning, whitespace-boundary and unsupported-style errors, compiler cache isolation, and token-mask replays that reject the original Kimi-K3 bad output at response while accepting the repaired output and a valid argument end.
• xgrammar_test 82/82. Full tests/python with HF_TOKEN: 4947 passed, 3 skipped; the two CUDA kernel tests that JIT-build a torch extension fail identically on main in this environment (CUDA 11.1 toolkit).
• Rebased onto main after Bugfix: DeepSeek XML parameter string attribute binding #908: the DeepSeek key exclusion check now applies to both deepseek_xml and deepseek_v4_1_xml, which share the raw-key path.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.