Build JSON schema grammars directly as ASTs

[Ubospica]

Summary

• build Grammar::FromJSONSchema output directly in the grammar AST instead of generating and reparsing one large EBNF string
• extend GrammarBuilder with nestable expression specs for strings, regexes, character classes, references, sequences, choices, repeats, dispatches, and token expressions
• keep JSONSchemaToEBNF available for testing, debugging, and format-specific compatibility paths
• add focused builder tests

Implementation

The schema parser and SchemaSpec model stay shared with the existing converter. The new JSONSchemaASTConverter mirrors the established JSON behavior for whitespace, arrays, object-property cardinality, any_order, refs/cycles, unions, constants/enums, numeric ranges, string patterns, and additional/pattern properties, then normalizes the completed grammar.

print_converted_ebnf=True prints the generated normalized grammar directly, so it no longer invokes the legacy converter for debugging output. Regex constructs whose FSM representation does not yet preserve the existing converter semantics use the mature regex frontend for that fragment only; the generated JSON grammar is no longer reparsed as a whole.

Benchmark

Compared origin/main (ff923ddf) with this branch using RelWithDebInfo builds in separate processes. Results are medians of 5 runs after 1 warmup. legacy parser is the EBNF parser portion of the old conversion path. Real schemas come from JSONSchemaBench; synthetic cases were generated by an out-of-tree benchmark harness. Benchmark-only artifacts are not part of this PR.

Schema	Bytes	Legacy total (ms)	Legacy parser (ms)	Direct AST (ms)	Speedup
JSONSchemaStore/cloudformation	6,786,385	1160.690	676.850	692.422	1.68x
JSONSchemaStore/config	1,329,080	114.493	35.692	94.061	1.22x
Kubernetes/kb_1161_Normalized	1,280,078	77.447	39.444	52.300	1.48x
JSONSchemaStore/service-schema	1,005,269	130.194	48.933	92.688	1.40x
synthetic/wide_object_5000	230,074	43.468	27.926	26.811	1.62x
synthetic/ref_object_2000	362,084	151.820	26.484	38.308	3.96x
synthetic/wide_object_10000	460,074	122.443	82.187	65.382	1.87x
synthetic/ref_object_5000	905,084	1112.451	115.806	155.498	7.15x

Testing

• pre-commit run
• ruff check python/xgrammar/grammar.py
• ctest --test-dir build-cxx --output-on-failure -j 4 (65 passed)
• pytest tests/python/test_json_schema_converter.py tests/python/test_grammar_regex_macro.py (592 passed)
• full tests/python with -m 'not hf_token_required' (2788 passed, 1 skipped, 622 remote-token tests deselected)
• structural-tag compatibility subset (1331 passed, 290 remote-token tests deselected)

[Ubospica]

#726