[adimalkar]

Fixes #1001

What

A JSON-schema pattern that the regex engine cannot parse made the whole schema fail to compile, so one such pattern in one tool took down a request with many tools. On main (614ab01):

construct	JSON converter	XML tool-calling converter (qwen_xml)
(?=…), (?!…) lookahead	compiles; lookahead ignored with a warning (since #765)	fails: Lookahead is not supported yet.
(?<=…), (?<!…) lookbehind	fails	fails
\1 backreference	fails	fails
\b word boundary	fails	fails
\p{L}	fails	fails
(?s), (?>…)	fails	fails
a{,3}	fails	fails

The issue's own example, ^(?!\.\.?(?:\/|$))[^\s]+$ from the Claude Code tool set, still breaks Qwen-style XML tool calling on main.

With this PR, each of these compiles and logs

pattern "(?<=a)b" is not supported (Regex parsing error at position 3: Lookbehind is not supported yet.); ignoring pattern, minLength/maxLength still apply

and the string is generated as if the pattern were absent, still bounded by minLength/maxLength. This is request 2 of the issue. It's the same policy the FSM engine already applies to lookaheads, and xgrammar already applies to an unsupported multipleOf (and, with #969, an unknown format): a relaxed grammar only accepts more strings, and the caller can validate after decoding. Details and the full table are in #1001 (comment).

How

JSONSchemaConverter::PatternExpression wraps RegexExpression for a string schema's pattern, in both JSONSchemaConverter::GenerateString and XMLToolCallingConverter::GenerateString. If the parser reports a syntax it cannot handle, it returns std::nullopt after logging, and the caller falls through to the plain or length-bounded branch.

Why at the point of use and not in ParseString: the two converters compile patterns through different engines. The JSON path tries the FSM engine first, which skips lookaheads; the XML path forces the CFG parser, which rejects them. So "unsupported" depends on the converter. Every syntax the parser can't handle comes out of RegexConverter::RaiseError ("Regex parsing error …"). The FSM engine returns an error instead, and RegexExpression then falls back to that parser. So catching exactly that error means "the engine on this path can't parse it", and nothing has been added to the grammar when it throws. Other failures still propagate (a pattern containing a null character still raises, and is tested).

This does not overlap with #986. That PR rejects patterns that parse but whose JSON-safe language is empty; those errors don't come from RaiseError, so they still fail as intended. Key regexes in patternProperties and the EBNF Regex(...) / Lark paths are unchanged.

One thing to flag: when a pattern with minLength/maxLength is ignored, the existing #896 warning from ParseString ("pattern combined with minLength/maxLength is not supported; ignoring minLength/maxLength") still fires, although the bounds are then enforced. ParseString can't know which engine will compile the pattern, so I left #896's warning alone and made the new warning say that the bounds still apply. Happy to rework that if you prefer.

Tests

In tests/python/test_json_schema_converter.py, next to the #896 tests and using the same _compile_and_capture helper:

• seven unsupported constructs × json / qwen_xml: the schema compiles and warns
• the issue's lookahead example compiles for qwen_xml
• an ignored pattern accepts exactly what a plain string accepts, and still honours minLength/maxLength
• a supported pattern on a sibling property is still enforced
• a pattern with a null character still raises

24 of the 25 new tests fail on main and pass with the change; the null-character one guards existing behaviour.

Locally, tests/python gives 4661 passed. The only 26 failures are in test_builtin_structural_tag_alignment.py and come from the optional cohere_melody package, which isn't installed here. pre-commit and ruff check pass. The C++ tests aren't built by default (XGRAMMAR_BUILD_CXX_TESTS=OFF), and none of tests/cpp/test_json*.cc covers pattern errors.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.