fix: holographic memory CJK (Chinese/Japanese/Korean) support

[edward1229]

Problem

Facts stored in the holographic memory plugin are completely unsearchable for CJK (Chinese/Japanese/Korean) users:

• search("科技公司") returns empty (the FTS index swallows whole sentences into single tokens)
• probe("张伟") returns empty (entity extraction recognizes no Chinese names)
• reason([a, b]) returns irrelevant facts (HRR vectors behave as near-random noise for Chinese)

Root causes (6 layers, all verified in source)

1. FTS tokenization: store.py creates FTS tables without a tokenizer → default unicode61 merges consecutive CJK characters into one giant token (verified via fts5vocab)
2. Query routing: retrieval.py::_sanitize_fts_query splits on whitespace + English stopwords, no CJK handling
3. Trigram limitation: the trigram tokenizer (community fix for Terminal tool #1) still cannot match CJK runs shorter than 3 chars — Chinese given names are typically 2 chars
4. Entity extraction: store.py entity regexes are English-only (capitalized words / English quotes / "aka") → zero extraction for Chinese
5. HRR vectors: holographic.py::encode_text splits on whitespace → whole Chinese sentences hash into a single bucket
6. Jaccard rerank: retrieval.py::_tokenize whitespace splitting → CJK query/fact token intersection is always 0

Fix

holographic.py

• add _cjk_tokenize(): CJK runs split into overlapping bigrams, runs ≥3 chars also keep the full run (exact phrase match); non-CJK split on whitespace
• encode_text now uses _cjk_tokenize

retrieval.py

• _CJK_RE class attribute (CJK Unicode block regex)
• _is_short_cjk_query(): detects queries consisting only of <3-char CJK runs
• _cjk_fts_query(): CJK runs ≥3 chars → phrase literal (trigram-matchable); non-CJK words keep stopword+OR logic
• _sanitize_fts_query delegates to the CJK path first; pure-English logic unchanged
• _get_fts_candidates dual-arm execution: MATCH arm (English / ≥3-char CJK) + LIKE arm (<3-char CJK runs, per-run LIKE), merged & deduped in Python; LIKE arm ordered by trust_score/updated_at; each arm fails independently

Verification (real DB: 5 facts / 19 entities, all Chinese)

Query	Before	After
search("张伟") 2-char name	0	2 (correct facts)
search("李娜") / search("王芳")	0	1 (LIKE arm hit)
search("科技公司 改革")	0	MATCH rank 1.0 top
search("张伟 年度规划") mixed	0	dual-arm merged [1,5]
Jaccard CJK intersection	always 0	{科技, 张伟, 负责}
English queries	working	unchanged

Design notes

• SQLite FTS5 has no built-in CJK tokenizer; trigram is the best option without extensions, but requires ≥3-char substrings — the LIKE arm covers that gap
• Chinese names carry no morphological markers, so regex extraction cannot find them by principle; long-term this needs LLM-assisted extraction or manual curation
• Zero behavior change for pure-English workloads

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

fix: holographic memory CJK (Chinese/Japanese/Korean) support

The bigram-tokenization + FTS/LIKE routing split is a sensible approach for unspaced CJK. Observations:

1. plugins/memory/holographic/retrieval.py — the short-CJK LIKE arm builds %run% patterns with leading wildcards ((f.content LIKE ? OR f.tags LIKE ?)), which cannot use an index and forces a full scan of facts per query. For a large fact store this is a real cost on every short-CJK query. Consider bounding it (e.g. a LIKE against a limited subquery, or deferring to the reranker over a smaller candidate pool) or documenting the expected table size.
2. _is_short_cjk_query() is defined as a classmethod but the new _fts_candidates routing computes short/long runs inline (cjk_runs = self._CJK_RE.findall(query)); _is_short_cjk_query appears to be dead code now. If it is no longer called anywhere, remove it or update _fts_candidates to use it so the two CJK detection paths can't drift.
3. Merged ordering across the two arms is inconsistent: MATCH rows are ordered by FTS rank, LIKE rows by trust_score DESC, updated_at DESC, then concatenated with dedupe. Since results feed a reranker later this is probably fine, but the rank merge means the final LIMIT applies per-arm, so the total can be up to 2× limit rows; make sure downstream reranking handles that gracefully.
4. _cjk_fts_query skips CJK runs < 3 chars and falls back to the raw query when no tokens remain — for a pure short-CJK query the MATCH arm never runs (routing skips it), so this fallback is only reachable in mixed queries. The if not tokens: return query fallback returning the raw query into FTS5 MATCH could produce a malformed-query exception, which is caught and degrades to the LIKE arm — acceptable, but a comment noting that path would help.