feat(state): add config.yaml flag to disable trigram FTS index - #68089
Closed
soe219 wants to merge 1 commit into
Closed
feat(state): add config.yaml flag to disable trigram FTS index#68089soe219 wants to merge 1 commit into
soe219 wants to merge 1 commit into
Conversation
Closes #55233. The messages_fts_trigram FTS5 index over-indexes every 3-character window and becomes the largest object in state.db (multiple GB on long-lived gateways) purely to serve CJK/substring search many deployments never use. Adds an opt-in config.yaml switch — `state.disable_fts_trigram` (default false, so existing behavior is unchanged). Per AGENTS.md, the user-facing knob lives in config.yaml, not a HERMES_* env var (the reason #27770 and #42190 were closed); HERMES_DISABLE_FTS_TRIGRAM is retained only as an internal bridge for tests and managed deploys. When enabled: - startup and migration skip creating messages_fts_trigram and its triggers; - an already-present trigram index + triggers are dropped so a bloated state.db reclaims the space (pages return to the OS on the next VACUUM); the index is fully rebuildable from messages if re-enabled; - _fts_trigger_count()/the boot repair check expect the base-only trigger set when trigram is disabled, so a disabled index no longer looks like "triggers missing" and force a full FTS rebuild on every boot (the regression the reviewer flagged on the closed #57761); - standard word/token search is unaffected (messages_fts); only CJK/substring queries fall back to LIKE. Also removes the misleading optimize_fts() docstring that referenced an unimplemented HERMES_DISABLE_FTS_TRIGRAM switch. Tests: adds tests/test_hermes_state_trigram_disable.py (6 cases: fresh-DB skip, no boot-rebuild loop, default keeps trigram, existing index dropped on opt-out, config.yaml read path, env bridge). Existing tests/test_hermes_state.py: 382 passed, no regressions.
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #55233. Adds an opt-in
config.yamlswitch —state.disable_fts_trigram(defaultfalse) — to skip themessages_fts_trigramFTS5 index. That index over-indexes every 3-character window and becomes the largest object instate.db(multiple GB on long-lived gateways) purely to serve CJK/substring search many deployments never use.Default is unchanged behavior; existing users see no difference unless they opt in.
Addressing the prior reviews
This supersedes the closed #27770 / #42190 and folds in the reviewer feedback from the closed #57761:
config.yaml: state.disable_fts_trigram, read via the samefast_safe_load+ managed-scope-overlay pattern as_read_logging_config().HERMES_DISABLE_FTS_TRIGRAMis retained only as an internal bridge (tests/managed deploys), not a documented switch — so.envstays credentials-only._fts_trigger_count()/ the startup repair check now expect the base-only trigger set when trigram is disabled. Previously a disabled index looked like "triggers missing" and forced a full FTS rebuild on every boot (the regression flagged on feat(state): opt-out flag for the trigram FTS index (HERMES_FTS_TRIGRAM=0) #57761). Covered bytest_disabled_no_boot_rebuild_loop.Behavior when enabled
messages_fts_trigramand its triggers._drop_fts_trigram), so a bloatedstate.dbreclaims the space — pages return to the OS on the nextVACUUM. Fully rebuildable frommessagesif re-enabled.messages_fts); only CJK/substring queries fall back toLIKE.optimize_fts()docstring that referenced an unimplementedHERMES_DISABLE_FTS_TRIGRAMswitch.Tests
New
tests/test_hermes_state_trigram_disable.py(6 cases): fresh-DB skip, no boot-rebuild loop, default keeps trigram, existing index dropped on opt-out,config.yamlread path (true/false/absent), env-bridge override. Existingtests/test_hermes_state.py: 382 passed, no regressions.Reclaiming space on an already-bloated database
Restart the gateway (drops the index), then
sqlite3 state.db "VACUUM;"to return the freed pages to the OS.