fix(search): keep natural-language queries on the FTS5 path instead of crashing to the LIKE fallback

[Milgauss]

Problem

Any conversational search query containing an apostrophe, comma, question mark, or most other punctuation (for example don't) is a syntax error to FTS5. store.search catches the error and silently degrades to the LIKE fallback, so in practice nearly every natural-language query runs on the slow, unranked fallback path instead of the FTS index.

Mechanism

sanitize_fts5_query (search_query.py:47) only replaces the fixed operator set _FTS5_SPECIAL_CHARS (search_query.py:36), leaving apostrophes, commas, and question marks in the unquoted query text. store.search (store.py:931) passes the result to messages_fts MATCH, FTS5 raises fts5: syntax error near "'", and the handler at store.py:997 logs "FTS message search failed, falling back to LIKE" and degrades. dag.py:427 has the same path for summary search.

Fix

Outside quoted phrases, keep only bareword-safe characters (alphanumerics, underscore, whitespace) and replace everything else with spaces. That is exactly the token alphabet the unicode61 tokenizer indexed, so the sanitized query matches the index instead of erroring; don't becomes don t, which matches rows containing don't. Quoted phrases pass through verbatim, as before.

The LIKE fallback must not use that strict sanitization: it matches raw stored content, so apostrophes, emoji, and CJK punctuation have to survive for the fallback to find them (an emoji-only query would otherwise produce zero terms and return nothing). The helper is therefore split in two: sanitize_fts5_query (strict, used by the FTS MATCH paths) and sanitize_like_terms_query (the previous operator-stripping behavior, used by _search_like in store.py and dag.py).

Verification

New tests cover: don't returning its row on the FTS path with no fallback warning, a full conversational sentence (what did I tell you, don't you remember?) searching without any OperationalError or fallback, quoted phrases with trailing conversational punctuation still matching as phrases, and sanitizer unit cases. On unpatched main the new tests fail with the exact production error (fts5: syntax error near ","). The full tests/test_lcm_core.py suite passes (288 tests), including the existing emoji and LIKE-fallback regression tests.

Incident evidence

One live store logged the FTS-to-LIKE fallback warning over 300 times, meaning nearly every conversational search had been silently running on the degraded fallback path.

[Milgauss]

Closing as superseded. Current main carries this fix in a more thorough shape than this PR: _fts5_safe_char turns every non-alphanumeric outside a quoted phrase into a separator (keeping combining marks, with NFC composition first), sanitize_like_query keeps the LIKE path on the weaker cleanup, and bare AND/OR/NOT/NEAR are neutralized. That landed with b2f228c ("recall: reduce a raw question to FTS5 terms before MATCH", #168) and its review-finding follow-ups, via #436.

Verified rather than assumed: this PR's three store.search regression tests (apostrophe query, full conversational question, quoted phrase plus conversational punctuation) pass unmodified against main at 10cbb78, and sanitize_fts5_query on main returns exactly the strings this PR's unit tests asserted (don't -> don t, the quoted-phrase case verbatim). Nothing left here to rebase. Thanks for taking the underlying problem on.