fix(search): keep natural-language queries on the FTS5 path instead of crashing to the LIKE fallback

[100yenadmin]

Important

This executable LCM-X PR was recreated by the migration operator from the exact upstream commit head. GitHub did not transfer the original PR actor, dates, review objects, or approval state.

Source and attribution

• Original pull request: fix(search): keep natural-language queries on the FTS5 path instead of crashing to the LIKE fallback stephenschoettler/hermes-lcm#354
• Originally authored by @Milgauss
• Upstream author profile: Milgauss
• Preservation record: [Upstream PR #354] fix(search): keep natural-language queries on the FTS5 path instead of crashing to the LIKE fallback #96
• Exact upstream head: 529407f73ebe84f8fe1b49963af41f2ae017fda7
• Original head repository/ref: Milgauss/hermes-lcm:fix/fts5-natural-language-queries
• Original base branch: main
• Destination branch: upstream/pr-354
• Original upstream draft state: False
• Migration containment state: draft (a maintainer can mark it ready when current-head review is wanted)

@coderabbitai ignore

Original commit authorship and history remain in the commits. Historical discussion and review text are imported below as attributed ordinary comments; they are not new approvals or change requests.

Original upstream PR description

Problem

Any conversational search query containing an apostrophe, comma, question mark, or most other punctuation (for example don't) is a syntax error to FTS5. store.search catches the error and silently degrades to the LIKE fallback, so in practice nearly every natural-language query runs on the slow, unranked fallback path instead of the FTS index.

Mechanism

sanitize_fts5_query (search_query.py:47) only replaces the fixed operator set _FTS5_SPECIAL_CHARS (search_query.py:36), leaving apostrophes, commas, and question marks in the unquoted query text. store.search (store.py:931) passes the result to messages_fts MATCH, FTS5 raises fts5: syntax error near "'", and the handler at store.py:997 logs "FTS message search failed, falling back to LIKE" and degrades. dag.py:427 has the same path for summary search.

Fix

Outside quoted phrases, keep only bareword-safe characters (alphanumerics, underscore, whitespace) and replace everything else with spaces. That is exactly the token alphabet the unicode61 tokenizer indexed, so the sanitized query matches the index instead of erroring; don't becomes don t, which matches rows containing don't. Quoted phrases pass through verbatim, as before.

The LIKE fallback must not use that strict sanitization: it matches raw stored content, so apostrophes, emoji, and CJK punctuation have to survive for the fallback to find them (an emoji-only query would otherwise produce zero terms and return nothing). The helper is therefore split in two: sanitize_fts5_query (strict, used by the FTS MATCH paths) and sanitize_like_terms_query (the previous operator-stripping behavior, used by _search_like in store.py and dag.py).

Verification

New tests cover: don't returning its row on the FTS path with no fallback warning, a full conversational sentence (what did I tell you, don't you remember?) searching without any OperationalError or fallback, quoted phrases with trailing conversational punctuation still matching as phrases, and sanitizer unit cases. On unpatched main the new tests fail with the exact production error (fts5: syntax error near ","). The full tests/test_lcm_core.py suite passes (288 tests), including the existing emoji and LIKE-fallback regression tests.

Incident evidence

One live store logged the FTS-to-LIKE fallback warning over 300 times, meaning nearly every conversational search had been silently running on the degraded fallback path.

[100yenadmin]

Triage: your red CI is stale, not broken — and here is the actual blocker.

This PR shows a failing check, but the failure is a CodeQL infrastructure error from 2026-08-16, not a problem with your code:

##[error]Encountered a fatal error ... CodeQL could not process any code written
in JavaScript/TypeScript. Exit code was 32

This is a Python repo, so that extractor found nothing to analyse and aborted. The job has passed on every PR landed since, so the configuration is already fixed and this PR is simply carrying an old red. It needs a re-run, not repair. Twelve of the sixteen "failing" PRs in this repo are in the same position — detail and recommendation in #241.

The real blocker is the rebase. This branch is ~40 commits behind and conflicts against current main. That is what needs doing before it can be evaluated on its merits.

Recorded disposition: NEEDS-REBASE. Not closed, not abandoned — it is a feature branch whose capability I verified is still absent from main. Priority order in this lane is external contributions, then correctness fixes, then reproducible issues, then feature drafts, so this sits behind the correctness work rather than being forgotten.

If you want it prioritised sooner, say so on the PR and I will pick it up next.

[100yenadmin]

Closing as salvaged-and-superseded — the useful half of this PR is now on main via #254 (60a110d3).

What was salvaged (with credit to the original author, milgauss, in the commit): 4 regression tests, including three FTS-path fences verified non-vacuous — each asserts requires_like_fallback(query) is False in-test, so they provably exercise the FTS MATCH path rather than passing via the silent LIKE fallback.

What was deliberately dropped, and why: the source hunks to dag.py/search_query.py/store.py. Main has since implemented punctuation handling differently and better — applying this branch's sanitizer would regress main ("Portland, OR hotel" would become a real FTS5 boolean OR; decomposed naïve would split to nai ve). One test was dropped as substantively redundant with main's test_sanitize_fts5_query_reduces_a_question_to_terms.

Full salvage record: session-notes/2026-08-20/lcmx-clearance/artifacts/salv159-*.txt + the #254 thread. The upstream original (hermes-lcm#354) remains open upstream; nothing here forecloses their own fix.