perf(json): optimize lexer and parser hot paths

[ya7010]

Summary

• replace regex-based JSON number lexing with an allocation-free byte scanner while preserving the previous accepted language
• scan long ordinary ASCII strings with AArch64 NEON and retain the existing scalar/memchr fallback elsewhere
• carry escape metadata from the lexer into the parser and avoid rescanning unescaped strings
• use the map entry API to avoid hashing object keys twice
• cover number compatibility and 15/16/31/32/33-byte string scanner boundaries with the existing declarative lexer test macros

Performance

Parse-only release AB/BA benchmarks against 70b90cf9ff7b7437b3eab51a86ffa7fc5d8afe5a on Apple M2 Max used 31 paired samples per case, alternating order, with about 30 ms per side. Repeated forward/reverse measurements showed stable improvements for the primary synthetic workloads:

• tiny JSON: 18.6–21.9% faster
• long strings: 83.5–84.0% faster
• numeric-heavy: 72.8–75.1% faster
• object-heavy: 36.3–38.8% faster

Short strings, Unicode, escaped strings, and the representative Cargo/Tombi/pyproject schemas showed order-sensitive results in the final exact-tree reruns, so this PR makes no improvement claim for those workloads. Non-AArch64 fallback performance was not measured; its build paths were checked separately.

Validation

• cargo fmt --all --check
• cargo test -p tombi-json-lexer -p tombi-json --locked (107 tests)
• cargo test -p tombi-schema-store --locked (75 tests; pre-existing dead-code warnings)
• cargo clippy --workspace --all-targets --locked -- -D warnings
• cargo check --workspace --all-targets --locked
• cargo check -p tombi-json --target wasm32-unknown-unknown --locked
• cargo check -p tombi-json --target x86_64-pc-windows-msvc --locked
• cargo check -p tombi-json --target x86_64-unknown-illumos --locked
• git diff --cached --check

Review

• independent read-only pre-PR review: no actionable findings
• reviewed fingerprint: 70b90cf9ff7b7437b3eab51a86ffa7fc5d8afe5a:a8e8d7882b580db5cf7089c2355f567904b5281eb9acd38333ad3dafa4bc6c1e
• residual risk: non-AArch64 runtime performance remains unmeasured

[Copilot]

Pull request overview

tombi の JSON lexer / parser のホットパスを最適化し、特に数値トークン判定・長い ASCII 文字列スキャン・オブジェクトキー処理のオーバーヘッドを削減する PR です。lexer 側で「文字列に escape が含まれるか」をメタデータとして保持し、parser 側の再スキャンを避けることで、全体としてパース処理のスループット改善を狙っています。

Changes:

• JSON 数値 lexing を regex から allocation-free なバイトスキャナに置き換え、従来互換の受理範囲をテストで固定
• 長い通常 ASCII 文字列の prefix スキャンを AArch64 NEON で高速化し、非 AArch64 は memchr + scalar にフォールバック
• lexer の contains_escape を parser へ伝搬し、未エスケープ文字列の unescape 処理を回避＋オブジェクトキー挿入を entry API で 1 回ハッシュに

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
crates/tombi-json/src/parser.rs	contains_escape による unescape 回避と、object properties の entry 挿入でキー重複検出＋二重ハッシュ回避
crates/tombi-json/Cargo.toml	memchr 依存を削除（parser 側で不要化）
crates/tombi-json-lexer/tests/test_lexer.rs	contains_escape を考慮した期待トークン生成へ更新し、数値互換・文字列境界のテストを追加
crates/tombi-json-lexer/src/token.rs	Token に contains_escape を追加し、new_string / accessor を提供
crates/tombi-json-lexer/src/scanner.rs	通常 ASCII 文字列 prefix の SIMD/非 SIMD 実装を ordinary_ascii_prefix に集約し、AArch64 で NEON を使用
crates/tombi-json-lexer/src/lib.rs	regex ベース数値判定を json_number_len に置換し、文字列トークンで contains_escape を設定
crates/tombi-json-lexer/src/cursor.rs	regex 用の peek_with_current_while を削除（不要な String 生成を排除）
crates/tombi-json-lexer/Cargo.toml	tombi-regex 依存を削除
Cargo.lock	依存関係の追従更新（memchr/tombi-regex の削除反映）

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.