feat(state): honor HERMES_DISABLE_FTS_TRIGRAM to skip the trigram FTS index - #45916
feat(state): honor HERMES_DISABLE_FTS_TRIGRAM to skip the trigram FTS index#45916DayanaLorza wants to merge 1 commit into
Conversation
… index The trigram FTS5 table mirrors every message into a second full-text index for CJK substring search, roughly doubling FTS storage. It was already documented as toggleable via HERMES_DISABLE_FTS_TRIGRAM (see optimize_fts), and the search/optimize paths already probe for the table — but the flag was never read, so the index was always built and rebuilt on every startup. Wire the flag for real: - _fts_trigram_enabled() reads HERMES_DISABLE_FTS_TRIGRAM via env_bool. - On open, when disabled, skip creating the trigram table and drop it (plus its triggers) if a previous run or the v10/v11 migrations created it, so the storage is reclaimed on the next VACUUM. The expected-trigger set used for repair detection is narrowed to the main triggers so no spurious rebuild fires. - Re-enabling rebuilds and backfills the index from existing rows. Also fix a latent gap this exposed: a 3+ CJK-character query used only the trigram MATCH path and returned empty when the table was absent. Extract the existing LIKE block into _cjk_like_search() and reuse it as the fallback so CJK search degrades gracefully whenever the trigram index is missing (disabled, mid-backfill, or failed creation). English search is unaffected. Adds TestDisableFtsTrigram (default-on, disabled-fresh, CJK LIKE fallback, drop-on-open, re-enable-rebuild) and documents the variable. 276 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Thanks for addressing a real state.db storage concern. This automated hermes-sweeper review is closing the PR under the standing configuration policy.
Please feel free to re-submit a focused config.yaml-based opt-out. The existing CJK fallback at Closed as not-planned per standing maintainer policy ( Closed as not-planned per standing maintainer policy ( |
What & why
The trigram FTS5 table (
messages_fts_trigram) mirrors every message into asecond full-text index to support CJK substring search. It roughly
doubles FTS storage — on a real install it was ~70 MB, about half of a
137 MB
state.db.The codebase already documents a
HERMES_DISABLE_FTS_TRIGRAMswitch (seeoptimize_fts's docstring) and the search/optimize paths already probe for thetable's existence — but the flag was never actually read. The trigram
table was created and rebuilt on every startup regardless, so the documented
opt-out did nothing.
This PR wires the flag for real, for deployments that never search CJK text and
want the storage back.
Changes
_fts_trigram_enabled()readsHERMES_DISABLE_FTS_TRIGRAMvia the sharedenv_boolhelper.its triggers) if a previous run or the v10/v11 migrations created it, so the
storage is reclaimed on the next
VACUUM. The expected-trigger set used forrepair detection is narrowed to the main triggers so no spurious rebuild fires.
the trigram
MATCHpath and returned empty when the table was absent. Theexisting LIKE block is extracted into
_cjk_like_search()and reused as thefallback, so CJK search degrades to LIKE whenever the trigram index is missing
(disabled, mid-backfill, or failed creation) instead of silently returning
nothing. English/keyword search is unaffected throughout.
reference/environment-variables.md.How to test
Automated:
TestDisableFtsTrigramcovers default-on, disabled-fresh, CJK LIKEfallback, drop-on-open, and re-enable-rebuild.
Test results / platforms
pytest tests/test_hermes_state.py— 276 passed (271 existing + 5 new), no regressionsruff check hermes_state.py— clean