Skip to content

feat(state): honor HERMES_DISABLE_FTS_TRIGRAM to skip the trigram FTS index - #45916

Closed
DayanaLorza wants to merge 1 commit into
NousResearch:mainfrom
DayanaLorza:feat/disable-fts-trigram
Closed

feat(state): honor HERMES_DISABLE_FTS_TRIGRAM to skip the trigram FTS index#45916
DayanaLorza wants to merge 1 commit into
NousResearch:mainfrom
DayanaLorza:feat/disable-fts-trigram

Conversation

@DayanaLorza

Copy link
Copy Markdown

What & why

The trigram FTS5 table (messages_fts_trigram) mirrors every message into a
second full-text index to support CJK substring search. It roughly
doubles FTS storage — on a real install it was ~70 MB, about half of a
137 MB state.db.

The codebase already documents a HERMES_DISABLE_FTS_TRIGRAM switch (see
optimize_fts's docstring) and the search/optimize paths already probe for the
table's existence — but the flag was never actually read. The trigram
table was created and rebuilt on every startup regardless, so the documented
opt-out did nothing.

This PR wires the flag for real, for deployments that never search CJK text and
want the storage back.

Changes

  • _fts_trigram_enabled() reads HERMES_DISABLE_FTS_TRIGRAM via the shared
    env_bool helper.
  • On open, when disabled: skip creating the trigram table and drop it (plus
    its triggers) if a previous run or the v10/v11 migrations created it, so the
    storage is reclaimed on the next VACUUM. The expected-trigger set used for
    repair detection is narrowed to the main triggers so no spurious rebuild fires.
  • Re-enabling (unset the var) rebuilds and backfills the index from existing rows.
  • Graceful-degradation fix: a 3+ CJK-character query previously used only
    the trigram MATCH path and returned empty when the table was absent. The
    existing LIKE block is extracted into _cjk_like_search() and reused as the
    fallback, so CJK search degrades to LIKE whenever the trigram index is missing
    (disabled, mid-backfill, or failed creation) instead of silently returning
    nothing. English/keyword search is unaffected throughout.
  • Document the variable in reference/environment-variables.md.

How to test

# default: trigram present
python -c "from pathlib import Path; from hermes_state import SessionDB; \
  db=SessionDB(db_path=Path('a.db')); print(db._fts_table_exists('messages_fts_trigram'))"  # True

# disabled: trigram absent, English + CJK (via LIKE) still work
HERMES_DISABLE_FTS_TRIGRAM=1 python -c "...same... "  # False

Automated: TestDisableFtsTrigram covers default-on, disabled-fresh, CJK LIKE
fallback, drop-on-open, and re-enable-rebuild.

Test results / platforms

  • pytest tests/test_hermes_state.py276 passed (271 existing + 5 new), no regressions
  • ruff check hermes_state.py — clean
  • Tested on Linux (Python 3.11)

… index

The trigram FTS5 table mirrors every message into a second full-text index
for CJK substring search, roughly doubling FTS storage. It was already
documented as toggleable via HERMES_DISABLE_FTS_TRIGRAM (see optimize_fts),
and the search/optimize paths already probe for the table — but the flag was
never read, so the index was always built and rebuilt on every startup.

Wire the flag for real:
- _fts_trigram_enabled() reads HERMES_DISABLE_FTS_TRIGRAM via env_bool.
- On open, when disabled, skip creating the trigram table and drop it (plus
  its triggers) if a previous run or the v10/v11 migrations created it, so the
  storage is reclaimed on the next VACUUM. The expected-trigger set used for
  repair detection is narrowed to the main triggers so no spurious rebuild fires.
- Re-enabling rebuilds and backfills the index from existing rows.

Also fix a latent gap this exposed: a 3+ CJK-character query used only the
trigram MATCH path and returned empty when the table was absent. Extract the
existing LIKE block into _cjk_like_search() and reuse it as the fallback so
CJK search degrades gracefully whenever the trigram index is missing
(disabled, mid-backfill, or failed creation). English search is unaffected.

Adds TestDisableFtsTrigram (default-on, disabled-fresh, CJK LIKE fallback,
drop-on-open, re-enable-rebuild) and documents the variable. 276 tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/feature New feature or request comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have duplicate This issue or pull request already exists labels Jun 14, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #27770 — same feature: wiring the documented HERMES_DISABLE_FTS_TRIGRAM env var in hermes_state.py to skip the CJK trigram FTS5 index (also overlaps #42190). Addresses #22478.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for addressing a real state.db storage concern. This automated hermes-sweeper review is closing the PR under the standing configuration policy.

  • The PR’s public mechanism is the non-secret behavioral flag HERMES_DISABLE_FTS_TRIGRAM (commit d32698aab615), including documentation in website/docs/reference/environment-variables.md.
  • AGENTS.md:102-107 reserves .env / HERMES_* variables for credentials and requires behavioral settings and feature flags to be exposed through config.yaml.
  • The underlying feature gap remains real: current hermes_state.py:1661-1675 still creates and maintains the trigram index unconditionally. This close is about the configuration mechanism, not the reported storage cost.
  • The duplicate discussion on feat(state): make trigram FTS5 index optional #27770 records the same direction: re-scope this as a config.yaml setting and carry the current migration, trigger-repair, cleanup/re-enable, and CJK fallback behavior forward.

Please feel free to re-submit a focused config.yaml-based opt-out. The existing CJK fallback at hermes_state.py:4899-4956 and FTS maintenance behavior at hermes_state.py:1465-1555 / 1657-1678 are the current paths it should cover.


Closed as not-planned per standing maintainer policy (env-var-for-config). This is an automated hermes-sweeper review.


Closed as not-planned per standing maintainer policy (env-var-for-config). This is a design-direction decision, not a code-quality judgment — see the Contribution Rubric in AGENTS.md for what the project is looking for. If you believe this policy was misapplied to your change, comment here and a maintainer will take a look.

@teknium1 teknium1 closed this Jul 14, 2026
@teknium1 teknium1 added the sweeper:not-planned Sweeper: closed per standing maintainer policy (design direction) label Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists P3 Low — cosmetic, nice to have sweeper:not-planned Sweeper: closed per standing maintainer policy (design direction) type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants