fix(holographic): add LIKE fallback for CJK full-text search - #42268
fix(holographic): add LIKE fallback for CJK full-text search#42268maurovideosmr-wq wants to merge 1 commit into
Conversation
|
Thanks for isolating the CJK recall gap. The current schema still uses a default FTS5 table ( Problems
Suggested changes
This is an automated hermes-sweeper review. |
FTS5's default unicode61 tokenizer does not segment CJK text, so Chinese characters without whitespace or punctuation boundaries are indexed as single tokens. For example, "香港腾讯云" is indexed as one token, and searching for "香港" returns no results because FTS5 does exact token matching. This fix: 1. Catches FTS5 MATCH exceptions (e.g. hyphens in queries like "sing-box" are parsed as column references by FTS5). 2. Always runs a LIKE "%query%" fallback alongside FTS5 MATCH, scanning both content and tags columns to match the FTS5 index surface. 3. Escapes LIKE metacharacters (% and _) in user input with an ESCAPE clause so the fallback is a literal sub-string search, not wildcard. 4. Merges results from both sources, deduplicated by fact_id — FTS5 results keep their ranking, LIKE additions get a neutral rank. The LIKE scan is O(n) but the fact store is typically small (<1000 entries), so the performance impact is negligible. Built on top of the existing _sanitize_fts_query() sanitizer (commit cb6d6d4) — the FTS5 path retains its behavior, LIKE only adds recall for sub-string matches the tokenizer cannot segment. Testing: - Searching "香港" now returns all matching facts (was: 0) - Searching "腾讯" matches via tags column (was: 0) - Searching "sing-box" finds facts with hyphens (was: FTS5 crash) - Searching "43.132" matches numeric IP fragments - Literal % and _ in queries are escaped, not treated as wildcards - Existing FTS5-exact searches (English/ASCII) remain unaffected - LIKE results merge with FTS5 results without duplicates
b9ce562 to
eac4296
Compare
|
Updated per @teknium1's review:
All 17 tests pass (10 existing + 7 new, 0.48s). |
FTS5s default unicode61 tokenizer does not segment CJK text, so Chinese characters without whitespace or punctuation boundaries are indexed as single tokens. For example, "香港腾讯云" is indexed as one token, and searching for "香港" returns no results because FTS5 does exact token matching.
This fix:
The LIKE scan is O(n) but the fact store is typically small (<1000 entries), so the performance impact is negligible.
Testing: