Skip to content

feat(state): add config.yaml flag to disable trigram FTS index - #68089

Closed
soe219 wants to merge 1 commit into
NousResearch:mainfrom
soe219:feat/config-disable-fts-trigram
Closed

feat(state): add config.yaml flag to disable trigram FTS index#68089
soe219 wants to merge 1 commit into
NousResearch:mainfrom
soe219:feat/config-disable-fts-trigram

Conversation

@soe219

@soe219 soe219 commented Jul 20, 2026

Copy link
Copy Markdown

Summary

Closes #55233. Adds an opt-in config.yaml switch — state.disable_fts_trigram (default false) — to skip the messages_fts_trigram FTS5 index. That index over-indexes every 3-character window and becomes the largest object in state.db (multiple GB on long-lived gateways) purely to serve CJK/substring search many deployments never use.

Default is unchanged behavior; existing users see no difference unless they opt in.

Addressing the prior reviews

This supersedes the closed #27770 / #42190 and folds in the reviewer feedback from the closed #57761:

  • Config, not env (AGENTS.md). The user-facing knob is config.yaml: state.disable_fts_trigram, read via the same fast_safe_load + managed-scope-overlay pattern as _read_logging_config(). HERMES_DISABLE_FTS_TRIGRAM is retained only as an internal bridge (tests/managed deploys), not a documented switch — so .env stays credentials-only.
  • No boot rebuild loop. _fts_trigger_count() / the startup repair check now expect the base-only trigger set when trigram is disabled. Previously a disabled index looked like "triggers missing" and forced a full FTS rebuild on every boot (the regression flagged on feat(state): opt-out flag for the trigram FTS index (HERMES_FTS_TRIGRAM=0) #57761). Covered by test_disabled_no_boot_rebuild_loop.

Behavior when enabled

  • Startup and the legacy migration path skip creating messages_fts_trigram and its triggers.
  • An already-present trigram index + triggers are dropped (_drop_fts_trigram), so a bloated state.db reclaims the space — pages return to the OS on the next VACUUM. Fully rebuildable from messages if re-enabled.
  • Standard word/token search is unaffected (messages_fts); only CJK/substring queries fall back to LIKE.
  • Also removes the misleading optimize_fts() docstring that referenced an unimplemented HERMES_DISABLE_FTS_TRIGRAM switch.

Tests

New tests/test_hermes_state_trigram_disable.py (6 cases): fresh-DB skip, no boot-rebuild loop, default keeps trigram, existing index dropped on opt-out, config.yaml read path (true/false/absent), env-bridge override. Existing tests/test_hermes_state.py: 382 passed, no regressions.

Reclaiming space on an already-bloated database

# config.yaml
state:
  disable_fts_trigram: true

Restart the gateway (drops the index), then sqlite3 state.db "VACUUM;" to return the freed pages to the OS.

Closes #55233. The messages_fts_trigram FTS5 index over-indexes every
3-character window and becomes the largest object in state.db (multiple GB on
long-lived gateways) purely to serve CJK/substring search many deployments
never use.

Adds an opt-in config.yaml switch — `state.disable_fts_trigram` (default false,
so existing behavior is unchanged). Per AGENTS.md, the user-facing knob lives in
config.yaml, not a HERMES_* env var (the reason #27770 and #42190 were closed);
HERMES_DISABLE_FTS_TRIGRAM is retained only as an internal bridge for tests and
managed deploys.

When enabled:
- startup and migration skip creating messages_fts_trigram and its triggers;
- an already-present trigram index + triggers are dropped so a bloated state.db
  reclaims the space (pages return to the OS on the next VACUUM); the index is
  fully rebuildable from messages if re-enabled;
- _fts_trigger_count()/the boot repair check expect the base-only trigger set
  when trigram is disabled, so a disabled index no longer looks like "triggers
  missing" and force a full FTS rebuild on every boot (the regression the
  reviewer flagged on the closed #57761);
- standard word/token search is unaffected (messages_fts); only CJK/substring
  queries fall back to LIKE.

Also removes the misleading optimize_fts() docstring that referenced an
unimplemented HERMES_DISABLE_FTS_TRIGRAM switch.

Tests: adds tests/test_hermes_state_trigram_disable.py (6 cases: fresh-DB skip,
no boot-rebuild loop, default keeps trigram, existing index dropped on opt-out,
config.yaml read path, env bridge). Existing tests/test_hermes_state.py: 382
passed, no regressions.
@soe219 soe219 closed this Jul 20, 2026
@soe219
soe219 deleted the feat/config-disable-fts-trigram branch July 20, 2026 14:50
@alt-glitch alt-glitch added type/feature New feature or request comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/config Config system, migrations, profiles P3 Low — cosmetic, nice to have sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add config flag to disable trigram FTS index (prevent state.db bloat)

2 participants