Repository navigation
fix: preserve Unicode character class ranges - #909
Merged
Seven-Streams merged 5 commits intoSep 20, 2026
Merged
Conversation
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
This reverts commit 13d3432. The change is re-applied in separate commits so that the FSM state-limit fix, the Unicode character class fix, the regular-grammar flattener, the excludes feature, and the Kimi-K3 wiring can be reviewed independently.
Negated character classes truncated their codepoint endpoints to bytes and complemented over a 128-entry ASCII bitset, then unconditionally accepted every multibyte sequence. [^\x00-\U0010ffff] therefore accepted every non-ASCII character, and [^你] rejected '`' instead of 你. Positive ranges that cross UTF-8 widths were encoded with the packed-byte helper, whose minimum byte sequences include overlong encodings such as C0 80, and the three-byte upper-tail path used a two-byte lower bound. Complement negated classes in codepoint space, split every range by UTF-8 width so only canonical encodings are accepted, and fix the three-byte tail bound. FSM snapshots and token-mask goldens change because the leading bytes C0, C1, and F5-F7 are now rejected. Extracted from mlc-ai#901. Signed-off-by: yuchuan <yuchuan.7streams@gmail.com>
Extracted from mlc-ai#901 by @izhuhaoran. This is the first piece of that PR split into independently reviewable units. The branch keeps the original commit and a revert of it at the bottom for attribution, so the net diff against `main` is the last commit only (`cpp/fsm.cc`, `tests/cpp/test_fsm.cc`). `FSMWithStartEnd::ToDFA(max_num_states)` only rejected an input that already had more than `max_num_states` states. The subset construction itself was never bounded, so a small NFA could still expand into an arbitrarily large DFA and the budget was effectively a no-op. `Intersect(lhs, rhs, max_result_num_states)` had two related gaps: it determinized both operands with `ToDFA()`, i.e. the default limit of 1000 states rather than the budget it was given, and it never bounded the product automaton either. ## Changes - `ToDFA`: check the DFA state count before every new closure is inserted (character-range, rule-ref, repeat-ref, token and exclude-token edges) and return an error once it reaches `max_num_states`. - `Intersect`: pass `max_result_num_states` through to both `ToDFA` calls, and return an error once the product automaton reaches the limit. For callers this means a budget is now actually enforced (an error instead of unbounded construction), and intersecting automata with more than 1000 states works when the caller allows it. `Not` and `MinimizeDFA` go through `ToDFA` and pick up the same bound. Nothing on `main` outside the tests reaches these functions today; the string-exclusion feature in mlc-ai#901 is the first production caller and needs the larger budget for strings with `maxLength` of roughly 1000 or more. ## Validation - Three new tests in `tests/cpp/test_fsm.cc`: - `a{1001}` ∩ `a*` succeeds with a budget of 4096 and accepts exactly 1001 `a`s; the same call with a budget of 1000 fails. - An NFA whose determinization needs 64 subset states fails under a budget of 16 and succeeds under 64 with the expected language. - A 2-cycle ∩ 3-cycle product fails under a budget of 4 and succeeds under 6. - `xgrammar_test`: 77/77 pass, including all 15 `XGrammarFSMTest` cases. --------- Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com> Co-authored-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
Signed-off-by: yuchuan <yuchuan.7streams@gmail.com>
Seven-Streams
requested review from
DarkSharpness and
Ubospica
as code owners
September 20, 2026 17:09
Contributor
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Reintroduce validation for reversed ranges, which can currently compile silently as an empty language.
Review effort: Lite
Findings: None
What changed in this PR
Fixes Unicode character-class FSM handling by preserving codepoint ranges, rejecting overlong UTF-8, and correcting three-byte range boundaries.
Changes:
- Complement negated classes in Unicode codepoint space.
- Split ranges at UTF-8 width boundaries.
- Update FSM snapshots, token masks, and regression coverage.
| File | Description |
|---|---|
tests/python/test_serialization.py |
Updates serialized FSM goldens. |
tests/python/test_grammar_matcher_json.py |
Updates JSON token-mask expectations. |
tests/python/test_grammar_matcher_json_schema.py |
Updates schema token-mask expectations. |
tests/python/test_grammar_matcher_ebnf.py |
Updates EBNF Unicode expectations. |
tests/python/test_grammar_matcher_basic.py |
Updates matcher expectations. |
tests/python/test_grammar_fsm_builder.py |
Adds Unicode regression coverage. |
tests/cpp/test_fsm_builder.cc |
Updates FSM construction snapshots. |
cpp/grammar_functor.cc |
Implements Unicode-safe range construction. |
cpp/fsm_builder.cc |
Corrects three-byte UTF-8 range bounds. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Seven-Streams
added a commit
that referenced
this pull request
Sep 20, 2026
…910) Extracted from #901 by @izhuhaoran, third piece after #906 and #909. The branch keeps the original commit and its revert at the bottom for attribution; the net diff against `main` is the `feat(fsm): add GrammarFSMBuilder::FromRegularGrammar` commit, additions only. `Grammar::FromRegex` sends some patterns through `RegexToEBNF` and produces a small CFG with sub-rules instead of a single FSM. `JSONSchemaFormat.excludes` (#901) needs an FSM for every string constraint so it can intersect it with the exclusion automaton, so it needs a way to flatten that CFG back. `GrammarFSMBuilder::FromRegularGrammar(grammar, max_num_states)` lowers the regular subset `RegexToEBNF` emits without re-parsing its terminals: byte strings and character classes go through the existing FSM builders. Each finite rule call is inlined with private entry and exit states, so one call cannot return into another call's continuation, and a direct tail self-reference loops back to that call's own entry. Bounded repetitions are unrolled, unbounded ones become a loop. Mutual or non-tail recursion, non-byte terminals such as `Token(...)`, annotated rules (lookahead, `max_tokens`, capture, lazy, temperature, suffix stop) and exceeding `max_num_states` return an error instead of a wrong automaton. Nothing on `main` calls it yet; the excludes feature is the first consumer. Validation: four new tests in `tests/cpp/test_regular_grammar_fsm.cc`. Two cover the call-return isolation (`part ::= "a" part | "b"` used twice, and a recursive callee next to an optional exit in the caller). One is differential: six regex patterns, including nested quantifiers, `[é你]+`, `[^你]` and an email pattern, agree with `GrammarMatcher` on 24 samples. One checks the error cases and the state budget. `xgrammar_test` 81/81; no Python changes. --------- Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com> Signed-off-by: yuchuan <yuchuan.7streams@gmail.com> Co-authored-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
Seven-Streams
added a commit
that referenced
this pull request
Sep 21, 2026
feat: support string exclusions in JSON schema structural tags Extracted from #901 by @izhuhaoran, fourth piece after #906, #909 and #910. The branch keeps the original commit and its revert at the bottom for attribution; the net diff against `main` is the `feat: support string exclusions in JSON schema structural tags` commit plus `feat: drop string length constraints when exclusions are present`, a refactor of the exclusion helpers, and `feat: apply exclusions only to strings without pattern or format`. The Kimi-K3 builtin wiring that turns this on by default is the last piece and comes separately. ## Motivation Kimi-K3 tool arguments are raw text. The model can emit control markers such as `<|close|>response<|sep|>` inside an argument that is still open, then start another tools/call block. The schema grammar accepts all of that as string content, while the tool-call parser reads it as structure. There is no way today to tell the converter that certain substrings must never appear inside a string value. ## API `JSONSchemaFormat` gets an optional `excludes: list[str]` (C++, Python, TypeScript, docs): ```python JSONSchemaFormat(json_schema=..., style="kimi_k3_xml", excludes=["<|open|>", "<|close|>", "<|sep|>"]) ``` - Forbids each substring in string values and property names, including nested JSON strings. It applies to the strings without `pattern` or `format`: a `pattern` or `format` is the schema's own contract for the string and is matched as is. Filtering those would intersect an arbitrary regex with the exclusion automaton, which reproduces the length blowup below for any pattern with a wide repetition (`^[\s\S]{0,256}$` gives 4592 rules). - `minLength`/`maxLength` are dropped when exclusions are set, with a warning naming the string and the bounds. Bounding a filtered string's length would multiply the grammar by the bound: one rule per position and exclusion state, about 18 rules per character for the three Kimi-K3 markers, which is 18 s and 111 MiB for `maxLength: 256` and 110 s for ten `maxLength: 128` properties. Without the bounds those schemas compile in 96 ms and 71 ms, as the unbounded strings do. - Matches the emitted text before JSON/XML unescaping, may span tokens, and never spans separate strings. JSON delimiter quotes, numbers, punctuation and XML parameter wrappers are outside the scope, so `<|close|>` can still begin a valid `<|close|>argument<|sep|>` terminator. - `const`/`enum` alternatives and fixed property names that contain an excluded substring are dropped; a required one makes the schema unsatisfiable rather than silently allowed. - An omitted or empty list changes nothing: every exclusion code path is behind `!excludes_.empty()`. - Errors instead of silent bypass: XML styles reject exclusions that start or end with formatting whitespace, and `cohere_xml` and `minimax_m3_xml` reject non-empty exclusions. ## Implementation `ExcludingString` builds the FSM of the converter's own string body (the JSON string content with escapes, or the XML identifier), intersects it with the complement automaton of the excluded substrings (`Intersect` with the budgets from #906), prunes states that can no longer reach an accepting state so no dead-end prefix is ever offered in a mask, and re-emits the result as EBNF rules with codepoint transitions. UTF-8 byte paths are collapsed back to codepoint classes and never turned into a second transition for an ASCII character, which relies on the canonical encodings from #909. The closing JSON quote is appended after the intersection because it is syntax, not content. Property names go through the same intersection for `additionalProperties` keys and through a substring check for fixed keys; keys constrained by `patternProperties`/`propertyNames` patterns are matched as is like patterned values. A length-bounded string with exclusions uses the shared string rule (json) or the raw `TagDispatch` body (XML styles) instead of an intersected length bound. ## Validation - `tests/python/test_json_schema_excludes.py`: 91 cases covering every style, nested JSON, property names and literals, Unicode and overlapping exclusions, emitted-text semantics, patterned and formatted strings left unfiltered, dropped length bounds and their warning, whitespace-boundary and unsupported-style errors, compiler cache isolation, and token-mask replays that reject the original Kimi-K3 bad output at `response` while accepting the repaired output and a valid argument end. - `xgrammar_test` 82/82. Full `tests/python` with `HF_TOKEN`: 4947 passed, 3 skipped; the two CUDA kernel tests that JIT-build a torch extension fail identically on `main` in this environment (CUDA 11.1 toolkit). - Rebased onto `main` after #908: the DeepSeek key exclusion check now applies to both `deepseek_xml` and `deepseek_v4_1_xml`, which share the raw-key path. --------- Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com> Co-authored-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
Seven-Streams
added a commit
that referenced
this pull request
Sep 21, 2026
Extracted from #901 by @izhuhaoran, last piece after #906, #909, #910 and #912. The branch keeps the original commit and its revert at the bottom for attribution; the net diff against `main` is the `feat(kimi_k3): exclude control markers from tool-argument strings` commit plus a docs commit that states the exclusion rules of #912 as a list. ## What changes The Kimi-K3 builtin now passes `excludes=["<|open|>", "<|cloase|>", "<|sep|>"]` to the `JSONSchemaFormat` of every tool-argument schema, in `auto`, `required` and forced tool modes. A control marker inside a raw string argument is rejected at the mask instead of being swallowed as text, which is the original #901 bad case: the model emitted `<|close|>response<|sep|>` inside an open `response` argument and started another tools/call block. The existing `exclude_special_tokens=False` option disables this together with the free-text exclusions; the free-text reasoning/response scope is unchanged. Per #912, the exclusions apply to argument strings without `pattern`/`format` and to property names, nested JSON strings included; `minLength`/`maxLength` on such strings are dropped with a warning. Argument and call wrappers are outside the scope, so `<|close|>argument<|sep|>` still terminates a value. `web/src/structural_tag.ts` gains the `kimi_k3_xml` member of the `JSONSchemaFormat.style` union, missed when the style was introduced in #806. ## Validation - `tests/python/test_builtin_structural_tag.py`: the instance that accepted a complete `<|close|>response<|sep|>` inside a string argument now expects rejection (`malformed-argument-end-in-string`); `test_kimi_k3_exclude_special_tokens` checks the three exclusions on every tool-argument schema across the three tool-choice modes and the opt-out; `test_kimi_k3_tool_argument_exclusions` covers raw and nested JSON string values with and without the option, including complete valid argument and call endings. - `test_builtin_structural_tag`, `test_json_schema_excludes`, `test_structural_tag_converter`: 1733 passed. The C++ tree is identical to `main`. Full `tests/python` with `HF_TOKEN`: 4946 passed, 3 skipped; the only failures are the CUDA kernel tests that JIT-build a torch extension, which fail identically on `main` in this environment (CUDA 11.1 toolkit). --------- Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com> Co-authored-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extracted from #901 by @izhuhaoran, second piece after #906. The branch keeps the original commit and its revert at the bottom for attribution; the net diff against
mainis thefix(fsm): preserve Unicode character class rangescommit. Same change as the closed #904, rebased.Character-class FSMs mishandled Unicode in three places:
[^\u0000-\U0010FFFF]accepted all non-ASCII characters and non-ASCII endpoints were dropped with a "clamping to 127" warning (Negated character class over non-ASCII ranges silently fails to enforce (clamped to 127) #848).C0 80.0x80instead of0x8080, so[一-龥]rejected every character with lead byteE9(道,部,金, ...).Fix: complement negated classes in codepoint space, split every range at the UTF-8 width boundaries so only canonical encodings are accepted, and fix the tail bound.
FSM snapshots grow because canonical UTF-8 needs more states. The
hf_token_requiredtoken-mask goldens change in two ways, verified by diffing per-position masks againstmain: JSON string positions reject exactly 5 more single-byte tokens (C0 C1 F5 F6 F7), and the[a-zа-я一-龥]+test accepts 78 more CJK tokens that the tail bug had rejected. Nothing else changes.Overlaps #849, which fixes only the negated-class complement and still admits overlong encodings. Closes #848.
Validation: 11 new regression tests that all fail on
main;xgrammar_test77/77; fulltests/pythonwithHF_TOKEN4764 passed. Performance was measured in #904 on this same diff: a few percent more compile time and mask latency on JSON, +12.6% mask latency on a 1024-char bounded Kimi XML string, from the extra states. This is a correctness trade-off.