Skip to content

fix: never fall back to an unescaped CFG for JSON string patterns #986

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mahmoudhas wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: never fall back to an unescaped CFG for JSON string patterns #986

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 12 additions & 13 deletions cpp/json_schema_converter.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2208,21 +2208,16 @@ int32_t JSONSchemaConverter::GenerateFromSpec(
* json_string=true so the pattern is compiled into a single automaton by GrammarFSMBuilder;
* json_string=true excludes the characters that must be escaped in a JSON string ('"', '\\'
* and the control characters) from every character match, so classes like \S cannot emit an
* unescaped quote. Fall back to the CFG expansion when the FSM regex engine does not support
* the pattern, or when the exclusion makes the pattern unmatchable (e.g. a pattern requiring
* a literal '"').
* unescaped quote. JSON-string patterns must never fall back to the raw regex CFG: that
* path does not escape quotes or control characters. If the safe canonical language is
* empty or unsupported, reject the schema rather than generate malformed JSON.
*/
int32_t JSONSchemaConverter::RegexExpression(
const std::string& regex, bool json_string, bool force_cfg_expansion
) {
bool can_use_fsm = !force_cfg_expansion;
if (json_string) {
can_use_fsm =
can_use_fsm && std::all_of(regex.begin(), regex.end(), [](unsigned char character) {
return character >= 0x20 && character <= 0x7e;
});
}
if (can_use_fsm) {
// The FSM handles UTF-8 patterns too; routing those through the raw CFG loses
// JSON-string exclusions (for example, in a Unicode pattern containing \\S).
if (!force_cfg_expansion) {
auto fsm_result = GrammarFSMBuilder::Regex(regex, json_string);
Comment on lines +2220 to 2221
if (fsm_result.IsOk()) {
auto fsm = std::move(fsm_result).Unwrap();
Expand All @@ -2238,8 +2233,12 @@ int32_t JSONSchemaConverter::RegexExpression(
}
}

// Keep regex conversion independent. Only the uncommon fallback path converts its existing
// EBNF result to a subgrammar; the JSON Schema rule graph itself is still built directly.
XGRAMMAR_CHECK(!json_string)
<< "JSON string pattern has no JSON-safe canonical representation or uses unsupported regex "
"syntax; patterns requiring escaped quotes, backslashes or control characters are not "
"supported";
Comment on lines +2236 to +2239

// Raw regex/numeric conversion is independent and may still use its CFG fallback.
return AddSubGrammar(Grammar::FromEBNF(RegexToEBNF(regex)));
}

Expand Down
30 changes: 30 additions & 0 deletions tests/cpp/test_json.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,5 +52,35 @@ TEST(JSONTest, JSONSchemaCarriageReturnWhitespaceModes) {
EXPECT_FALSE(MatchesEntireString(capped, "{\r\n\r\"value\": 3}"));
}

TEST(JSONTest, UnicodePatternsRetainJSONStringExclusions) {
GrammarCompiler compiler(TokenizerInfo(std::vector<std::string>{}), 1, false);
auto grammar = compiler.CompileJSONSchema(
R"({"type":"string","pattern":"^[é\\s\\S]{1,64}$"})"
);
EXPECT_TRUE(MatchesEntireString(grammar, "\"hello é 🙂 world\""));
EXPECT_FALSE(MatchesEntireString(grammar, "\"hello\"world\""));
for (char control = 0; control < 0x20; ++control) {
EXPECT_FALSE(MatchesEntireString(grammar, std::string("\"hello") + control + "world\""));
}
auto literal = compiler.CompileJSONSchema(R"({"type":"string","pattern":"^é🙂+$"})");
EXPECT_TRUE(MatchesEntireString(literal, "\"é🙂🙂\""));
EXPECT_FALSE(MatchesEntireString(literal, "\"éé\""));
}

TEST(JSONTest, EscapedOnlyPatternsFailBeforeGenerationInsteadOfUsingRawCFG) {
GrammarCompiler compiler(TokenizerInfo(std::vector<std::string>{}), 1, false);
for (const auto* schema : {
R"({"type":"string","pattern":"^\\\"$"})",
R"({"type":"string","pattern":"^\\n$"})",
R"({"type":"string","pattern":"^\\\\$"})",
R"({"type":"object","properties":{"text":{"type":"string","pattern":"^\\\"$"}}})",
}) {
EXPECT_ANY_THROW(compiler.CompileJSONSchema(schema));
}
// The independent raw-regex API is not JSON and must keep accepting these bytes.
auto raw = compiler.CompileRegex("^\"$");
EXPECT_TRUE(MatchesEntireString(raw, "\""));
}

} // namespace
} // namespace xgrammar