perf(state): CJK bigram FTS index — replace trigram+LIKE routing for session search

[Soju06]

Stacked on #65541 (read-path split) — review the last three commits.

Problem

search_messages routes queries three ways: unicode61 FTS5, the trigram table, or a LIKE fallback. unicode61 indexes a CJK run as ONE token, so a 2-char Korean term (일본, 구글, 우리, …) can never match it; the trigram tokenizer needs ≥3 chars per term. Result: any query containing a 1-2 char CJK token full-scans messages with LIKE across three columns — measured 3-6.4s CPU per query on a 6.8GB production DB, and the dominant base cost behind a 12.4s in-gateway session_search average (Korean/Japanese/Chinese conversation workloads hit this constantly — 2-char words are everywhere in Korean).

The v1 layout also triple-stores text (standalone FTS content copies in both tables; the trigram index alone was larger than the messages data) and its unconditional AFTER UPDATE triggers re-tokenize whole messages on flag-only updates.

Design

1. cjk_unicode61 tokenizer (native/fts5_cjk/fts5_cjk.c, ~250 lines, no deps): wraps unicode61; CJK runs inside its tokens are re-emitted as overlapping character bigrams (Lucene CJKAnalyzer semantics), everything else passes through. FTS5 turns consecutive tokens from one query term into a phrase, so a bare term like 캘린더 gets exact substring semantics down to 2 chars at index speed. Built with native/fts5_cjk/build.sh → ~/.hermes/lib/libfts5_cjk.so (override: HERMES_FTS5_CJK_SO).
2. messages_fts_v2: one standalone 3-column FTS5 table whose single MATCH path replaces the 3-way routing. Triggers are scoped AFTER UPDATE OF content, tool_name, tool_calls. Lone 1-char CJK terms keep the LIKE path (a bigram index only holds unigrams for isolated chars).
3. Zero-regression fallback: no .so → nothing changes (legacy tables and routing untouched). A DB that carries v2 triggers but can't load the tokenizer self-heals by dropping the triggers (writes never fail; index goes stale until re-migrated).
4. Online migration (scripts/fts_v2_migrate.py): create table+triggers → idempotent DELETE+INSERT batched backfill (resume state in state_meta) → integrity-check → sets a fts_v2_ready marker. Reads are gated on that marker so a partially-backfilled index is never served. Production run: 385k messages in 91s with live writers.
5. Config surface (per the env-var-for-config policy): agent.fts_v2_read (default on once the index is ready) and agent.search_slow_ms are config.yaml-authoritative, bridged to env at both the startup export block and the per-turn reload path. A slow-search log line names the routing path (fts_v2/fts5/trigram/like_scan) so future routing regressions are a journalctl grep.
6. v1 retirement (scripts/fts_v1_drop.py, operator-run): preflight (tokenizer loadable, v2 objects present, ready marker, integrity-check, rowcount parity) then drops the six v1 triggers + two tables. Freed ~5.9GB on our production DB. Fresh DBs with a loadable tokenizer are v2-native and never create the v1 tables.

Production results (fork, 2026-07-16)

query	before (in-gateway)	after
나쵸 minpeter	58.1s	50ms
"shared default" AND "웅기"	72.5s	86ms
구글 캘린더 일정 없음 브리핑 …	45.4s	27ms

Recall parity held on a replayed real-query benchmark (10/13 confirmed-relevant targets found, identical to legacy), and v2's FTS5 semantics fix a latent bug: the LIKE fallback matched multi-token queries as OR instead of AND.

Tests

tests/test_fts_v2_cjk.py (tokenizer built on the fly; skips without a C toolchain), tests/test_search_slow_query_log.py, tests/gateway/test_fts_v2_config_bridge.py — bigram matching incl. no-false-positive-across-words, trigger mirroring, flag-only-update scoping, self-heal, partial-backfill refusal, fresh-DB v2-native, config bridges.

[teknium1]

Thanks for targeting a verified performance gap: current main still sends 1–2-character CJK terms to the LIKE route (hermes_state.py:5353-5380).

Problems

• hermes_state.py:1293-1295 drops v2 triggers when one process cannot load the extension, but leaves the ready marker intact. A later extension-capable open recreates triggers at hermes_state.py:2051 and accepts that marker at :1300-1310; writes made in between are absent from an index now considered ready.
• This branch predates current main's FTS recovery work (9e1b1d753). repair_state_db_schema() probes/rebuilds only v1 tables (hermes_state.py:561-668), while maintenance enumerates only v1 indexes (:7311). A v1-retired v2 DB needs these paths updated together.
• scripts/fts_v2_migrate.py:15-17 still instructs users to put HERMES_FTS_V2_READ in .env; the advertised config defaults are not added to hermes_cli/config.py.

Suggested changes

• Make trigger loss invalidate v2 durably; require a complete verified backfill before reads resume.
• Integrate v2/tokenizer loading into repair, rebuild, and optimize paths, with end-to-end recovery coverage.
• Add config defaults and remove the user-facing .env cutover path.

Automated hermes-sweeper review.

[teknium1]

Dropping triggers leaves fts_v2_ready=1. A later extension-capable SessionDB recreates these triggers in _init_schema() and _probe_fts_v2() accepts the old marker, so rows written during this gap are silently missing from a supposedly ready v2 index. Persist an invalid state here and require a verified full backfill before re-enabling reads.

[teknium1]

After dropping v1, current main maintenance still enumerates only messages_fts and messages_fts_trigram in SessionDB._FTS_TABLES (hermes_state.py:7311), so optimize/rebuild no longer covers the serving v2 index. Please integrate v2 with those paths before making v1 retirement available.

[teknium1]

This is a user-facing non-secret behavior setting, so it must be configured through agent.fts_v2_read in config.yaml rather than .env. Add the documented defaults to hermes_cli/config.py and update this cutover instruction.

[Soju06]

Thanks for the review — all three points addressed.

Trigger-drop invalidation: dropping the v2 triggers now durably invalidates the index by flipping the fts_v2_ready marker to needs_backfill before the drop (same state_meta mechanism; marker-first ordering so a crash between the two statements is only conservative). An extension-capable open recreates the triggers but keeps reads on the v1/LIKE route until fts_v2_migrate.py completes a verified backfill — the script discards stale snapshot/progress on a needs_backfill marker and only restores the ready marker after an FTS integrity-check plus rowcount parity with messages, checked in the same transaction. The same invalidation applies when _init_schema recreates the table on a populated DB (post-repair). A regression test covers the full lifecycle including rows written during the gap.

Repair/maintenance integration: messages_fts_v2 joins _FTS_TABLES, so optimize/rebuild enumerate it whenever it exists (existence-probed like the trigram table, so tokenizer-less processes skip it). repair_state_db_schema() and the _db_opens_cleanly write probe load the tokenizer best-effort, the in-place rebuild pass covers v2, and a missing FTS table with dangling triggers is now reported unhealthy instead of passing the no-such-table carve-out. Also gated the v2-only schema path on the ready marker so a repaired-but-unmigrated DB gets its v1 index rebuilt as the serving fallback, keeping the drop_fts_rebuild recovery contract intact on tokenizer-capable hosts. End-to-end tests cover both the corrupt-index (in-place rebuild, no marker loss) and absent-table (repair → legacy serves → verified migrate → v2 serves) paths.

Config surface: agent.fts_v2_read and agent.search_slow_ms now have documented defaults in DEFAULT_CONFIG, and the migrate script and tokenizer README point at config.yaml only — no user-facing .env instruction remains; the env var is purely the internal gateway bridge carrier.

Also restacked onto the updated #65541.

[Soju06]

@teknium1 Gentle ping — all points from the review here have been addressed (summary in the comment above), the branch is rebased on current main, and CI is green. Ready for another look whenever convenient.

[teknium1]

Merged via #69423 (rebase-merge — your tokenizer and slow-query-log commits are on main under your authorship: f13f845..).

The cjk_unicode61 tokenizer landed verbatim — it's excellent work, and the bigram design proved out exactly as your benchmarks promised (2-char Korean terms: LIKE full scan → ~1ms indexed MATCH in our E2E).

The schema integration was rebuilt rather than cherry-picked, because #65798 (schema v23, external-content FTS) merged after your branch and replaced the storage model this PR integrated against:

• messages_fts_v2 (standalone, inline copies) became messages_fts_cjk — external-content over a tool-row-excluding view, consistent with the v23 discipline (zero inline text copies; the storage bloat your PR also fixed is now handled there).
• fts_v2_migrate.py / fts_v1_drop.py were folded into the existing hermes sessions optimize-storage engine (throttled, resumable, disk-preflighted) — one command owns every index transition.
• Your ready-marker/needs_backfill machinery became the v23-style marker-gated triggers + stale breadcrumb. One behavioral difference from your last revision: after a tokenizer-less process drops the triggers, we never reinstall them over the gap (an external-content 'delete' on an unindexed rowid corrupts FTS5 shadow tables) — the stale index is rebuilt from scratch on the next optimize run instead.
• Config moved to sessions.cjk_fts / sessions.search_slow_ms alongside the other session-store knobs.

Also kept: your trigger-drop invalidation ordering (marker first, crash-conservative), the lone-1-char-run LIKE routing, and the routing-path slow-log attribution (now fts_cjk/fts5/trigram/like_scan).

Thanks for the thorough work across three review rounds — the responsiveness on the sweeper findings made this salvage straightforward. #65541 (read-path split) is untouched by this merge and remains open for separate review.