Reduce first-time grammar compilation cost

[Ubospica]

改动

本请求降低首次语法编译在结构化标签分派、JSON Schema（用于描述 JSON 数据结构的规格）转换、语法优化和有限状态机构造中的开销。有限状态机是用状态和转换表示允许路径的数据结构。

• 用完整的 Aho-Corasick 多模式字符串匹配自动机替代重复的标签子串扫描，包括失败跳转和终结结果传播。
• 把每个分派状态拒绝的词元集合存为稀疏差异，并遍历稀疏候选，避免重建或扫描密集表。
• 把生成的 JSON 语法按规则解析到共享语法树，避免构造一份巨大的扩展巴科斯范式（Extended Backus-Naur Form，EBNF）文本和中间词元流。
• 跳过不会改变结果的字节串合并和规则内联，以线性时间收集已使用规则，流式连接状态机序列，并直接追加简单序列元素。
• 新增字符串匹配、解析器、JSON Schema 和结构化标签的回归测试。

原因

大型规格和大型结构化标签集会同时触发多种首次启动开销：重复子串匹配、状态与词元之间的密集计算、大型语法文本生成、无效优化重写和状态机复制。本改动分别去除这些开销，不改变原有语法与匹配行为。

性能

• 4,000 个密集标签的字符串匹配输入：实际耗时从 5,243 毫秒降到 1,115 毫秒，快 4.70 倍；处理器时间从 29,433 毫秒降到 6,058 毫秒，快 4.86 倍。
• 包含 50,000 个并列属性的 JSON Schema：在 JSON 转换、状态机构造和优化组合后，实际耗时从 4,856 毫秒降到 3,948 毫秒，降低 18.7%；处理器时间从 11,456 毫秒降到 10,142 毫秒，降低 11.5%。
• 6 个专项压力输入全部通过，包括密集、唯一和共享前缀标签，以及并列属性和循环引用规格。

验证

• 已重放到当时最新的主分支提交 d6fc0420。
• 干净重建后，C++ 语言测试 54 项全部通过。
• 专项压力测试 6 项全部通过。
• JSON Schema 与结构化标签的 Python 语言专项测试分别通过 202 项和 149 项。
• Python 语言完整测试通过 3,092 项，跳过 1 项。另外 51 项需要当时不可用的远程词表或缓存文件；一项编译器集成测试单独重跑时通过。

[gemini-code-assist]

Code Review

This pull request introduces significant performance and memory optimizations to the grammar compiler and JSON schema converter. Key changes include parsing EBNF rules directly into a GrammarBuilder to avoid materializing a combined script, optimizing TagDispatch rules using a new byte-oriented Aho-Corasick matcher with sparse candidate matching, and reducing memory allocations during FSM sequence construction. A high-severity issue was identified in the Aho-Corasick trie builder, where holding a reference to a vector element while calling emplace_back can lead to a dangling reference and undefined behavior if reallocation occurs.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Holding a reference to nodes_[state].next[byte] (via next_state) while calling nodes_.emplace_back() is unsafe. If emplace_back() triggers a reallocation of the nodes_ vector, the reference next_state becomes dangling, leading to undefined behavior when it is subsequently read or written to. Although reserve() is called beforehand, relying on it to prevent dangling references is error-prone and discouraged. It is safer to store the state index by value and perform the lookup/assignment after emplace_back().

      int32_t next_state = nodes_[state].next[byte];
      if (next_state == -1) {
        next_state = static_cast<int32_t>(nodes_.size());
        nodes_.emplace_back();
        nodes_[state].next[byte] = next_state;
      }
      state = next_state;

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR reduces cold-start grammar compilation overhead by avoiding repeated substring scans, avoiding dense per-token work during structural-tag dispatch, and parsing generated JSON-schema EBNF per-rule into a shared GrammarBuilder to reduce intermediate materialization.

Changes:

• Introduces a byte-oriented Aho–Corasick automaton and uses it for TagDispatch token preclassification.
• Adds a per-rule EBNF parsing path (ParseEBNFRules) and a JSON Schema → Grammar conversion path that bypasses the combined EBNF script.
• Optimizes FSM sequence concatenation / optimizer no-ops and adds focused C++ and Python regression tests.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/python/test_json_schema_converter.py	Adds a regression ensuring direct JSON Schema→Grammar matches the text-EBNF path.
tests/python/test_grammar_matcher_structural_tag.py	Adds coverage ensuring sparse TagDispatch candidate masks match direct parsing behavior.
tests/cpp/test_parser.cc	Adds a regression comparing per-rule parsing vs parsing a combined EBNF script.
tests/cpp/test_aho_corasick.cc	Adds unit tests for failure links, terminal propagation, offsets, and exhaustive equivalence.
cpp/support/aho_corasick.h	Adds the Aho–Corasick matcher interface.
cpp/support/aho_corasick.cc	Implements the Aho–Corasick matcher with resolved transitions and terminal propagation.
cpp/json_schema_converter.h	Exposes ConvertToGrammar and JSONSchemaToGrammar APIs.
cpp/json_schema_converter.cc	Implements per-rule generation + direct Grammar conversion from JSON schema.
cpp/grammar_parser.h	Exposes ParseEBNFRules API.
cpp/grammar_parser.cc	Implements per-rule parsing into a shared builder to avoid combined script/token stream.
cpp/grammar_functor.cc	Optimizes used-rule discovery, streams FSM sequence concatenation, and skips no-op optimizer passes.
cpp/grammar_compiler.cc	Adds TagDispatch token optimization with Aho–Corasick + sparse candidate traversal.
cpp/grammar.cc	Switches default JSON Schema frontend to the direct Grammar conversion path when not printing EBNF.
cpp/ebnf_script_creator.h	Exposes generated rules directly to avoid materializing the combined EBNF script.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Ubospica]

这个组合请求中的改动现已拆分或由更完整的实现替代：#726 已合并直接构造 JSON 语法树，#727 已合并共享状态机构造，#729 已合并原地语法优化；剩余的结构化标签分派优化已单独提交为 #794。为避免重复改动和当前主分支冲突，关闭这个旧组合请求。