Skip to content

fix(holographic): add LIKE fallback for CJK full-text search - #42268

Open
maurovideosmr-wq wants to merge 1 commit into
NousResearch:mainfrom
maurovideosmr-wq:fix/cjk-search
Open

fix(holographic): add LIKE fallback for CJK full-text search#42268
maurovideosmr-wq wants to merge 1 commit into
NousResearch:mainfrom
maurovideosmr-wq:fix/cjk-search

Conversation

@maurovideosmr-wq

Copy link
Copy Markdown

FTS5s default unicode61 tokenizer does not segment CJK text, so Chinese characters without whitespace or punctuation boundaries are indexed as single tokens. For example, "香港腾讯云" is indexed as one token, and searching for "香港" returns no results because FTS5 does exact token matching.

This fix:

  1. Catches FTS5 MATCH exceptions (e.g. hyphens in queries like "sing-box" are parsed as column references by FTS5).
  2. Always runs a LIKE "%query%" fallback alongside FTS5 MATCH.
  3. Merges results from both sources, deduplicated by fact_id — FTS5 results keep their ranking, LIKE additions get a neutral rank.

The LIKE scan is O(n) but the fact store is typically small (<1000 entries), so the performance impact is negligible.

Testing:

  • Searching "香港" now returns all 3 matching facts (was: 0 or partial)
  • Searching "sing-box" finds facts with hyphens (was: FTS5 crash)
  • Searching "43.132" matches numeric IP fragments
  • Existing FTS5-exact searches (English/ASCII) remain unaffected

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have tool/memory Memory tool and memory providers comp/plugins Plugin system and bundled plugins labels Jun 8, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for isolating the CJK recall gap. The current schema still uses a default FTS5 table (plugins/memory/holographic/store.py:48-49), and current query preparation only splits on whitespace before emitting quoted FTS phrases (plugins/memory/holographic/retrieval.py:605-619), so the fallback idea remains relevant.

Problems

  • The proposed LIKE "%{query}%" uses raw % and _ from user input as SQL wildcards. Escape literal LIKE metacharacters and add an ESCAPE clause so fallback results remain a literal substring search.
  • The fallback only scans f.content, although FTS indexes both content and tags (plugins/memory/holographic/store.py:48-49). Include tags so the fallback preserves the searchable surface.
  • The patch predates current main's sanitizer/OR recall path at plugins/memory/holographic/retrieval.py:499-503 and 585-619, introduced in cb6d6d46ab6b20b173c8215a1f066b53847e9ee5.

Suggested changes

  • Rework the fallback around the current sanitizer, retain its existing behavior, and add integration coverage for CJK content/tags plus literal % and _ queries.

This is an automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 14, 2026
FTS5's default unicode61 tokenizer does not segment CJK text, so Chinese
characters without whitespace or punctuation boundaries are indexed as
single tokens. For example, "香港腾讯云" is indexed as one token, and
searching for "香港" returns no results because FTS5 does exact token
matching.

This fix:
1. Catches FTS5 MATCH exceptions (e.g. hyphens in queries like "sing-box"
   are parsed as column references by FTS5).
2. Always runs a LIKE "%query%" fallback alongside FTS5 MATCH, scanning
   both content and tags columns to match the FTS5 index surface.
3. Escapes LIKE metacharacters (% and _) in user input with an ESCAPE
   clause so the fallback is a literal sub-string search, not wildcard.
4. Merges results from both sources, deduplicated by fact_id — FTS5
   results keep their ranking, LIKE additions get a neutral rank.

The LIKE scan is O(n) but the fact store is typically small (<1000
entries), so the performance impact is negligible.

Built on top of the existing _sanitize_fts_query() sanitizer (commit
cb6d6d4) — the FTS5 path retains its behavior, LIKE only adds recall
for sub-string matches the tokenizer cannot segment.

Testing:
- Searching "香港" now returns all matching facts (was: 0)
- Searching "腾讯" matches via tags column (was: 0)
- Searching "sing-box" finds facts with hyphens (was: FTS5 crash)
- Searching "43.132" matches numeric IP fragments
- Literal % and _ in queries are escaped, not treated as wildcards
- Existing FTS5-exact searches (English/ASCII) remain unaffected
- LIKE results merge with FTS5 results without duplicates
@maurovideosmr-wq

Copy link
Copy Markdown
Author

Updated per @teknium1's review:

  1. LIKE metacharacters escaped — added _escape_like_pattern() that escapes %, _, and \ in user input, with ESCAPE '\' clause in the SQL. Literal % and _ in queries are now treated as characters, not wildcards.

  2. Tags included — LIKE fallback scans both content and tags (f.content LIKE ? OR f.tags LIKE ?), matching the FTS5 index surface.

  3. Built on current sanitizer — the FTS5 path still uses _sanitize_fts_query() (cb6d6d4) unchanged. LIKE only adds recall for sub-string matches the tokenizer cannot segment (CJK, numeric fragments). No duplicate logic.

  4. Integration tests added — 7 new tests covering:

    • CJK sub-string in content ("香港" → matches "香港腾讯云...")
    • CJK sub-string in tags ("腾讯" → matches via tags column)
    • FTS5 + LIKE merge dedup (no duplicate fact_id)
    • Literal % escaped (searching "95%" does not wildcard-match everything)
    • Literal _ escaped ("DATABASE_URL" matches literally)
    • Numeric IP fragments ("43.132")
    • Existing English search unaffected

All 17 tests pass (10 existing + 7 new, 0.48s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/memory Memory tool and memory providers type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants