feat(state): add sessions.trigram_fts config flag to disable the trigram FTS index

[fadhlillah2]

What does this PR do?

Adds sessions.trigram_fts (default true) — a config.yaml flag to disable the messages_fts_trigram FTS5 index, the secondary trigram index that powers CJK/substring recall in session_search.

That index carries an INSERT/UPDATE/DELETE trigger on every message and is frequently the single largest contributor to state.db growth (on one production install: ~273 MB of a 549 MB DB — 88% of it FTS, mostly the doubled trigram data/content shadow tables). Operators who don't need CJK or substring search can now opt out; CJK/substring queries fall back to LIKE — the exact graceful path already taken when the SQLite build lacks the trigram tokenizer.

Why this approach (and why the prior attempts were closed)

This is deliberately a config knob, not an env var. Every earlier attempt at this feature used a HERMES_* environment variable and was closed under the AGENTS.md policy that reserves .env/HERMES_* for credentials and requires behavioral flags in config.yaml:

• feat(fts): implement HERMES_DISABLE_FTS_TRIGRAM env gate (docstring parity) #66690 (HERMES_DISABLE_FTS_TRIGRAM) — closed, config-policy
• feat(state): opt-out flag for the trigram FTS index (HERMES_FTS_TRIGRAM=0) #57761 (HERMES_FTS_TRIGRAM=0) — closed, dup + config-policy
• fix(state): honour HERMES_DISABLE_FTS_TRIGRAM to skip CJK index #42190, feat(state): make trigram FTS5 index optional #27770, feat(state): make trigram FTS5 index optional #22710 — same cluster, all closed

This PR is the first to implement the maintainers' stated requirement (a sessions.* config key, matching auto_prune/vacuum_after_prune/write_json_snapshots). It also folds in the two technical notes left on the closed cluster (#57761 → #27770): the trigger-repair count now tolerates the deliberately-absent trigram triggers, and all migration paths are covered — not just steady-state init.

Related Issue

Fixes #55233 (and addresses the size evidence in #22478 / #43690).

Type of Change

• New feature (non-breaking change which adds functionality)

Changes Made

• hermes_cli/config.py — new sessions.trigram_fts: True default (deep-merged at read time, no _config_version bump needed since it's an optional default).
• hermes_state.py
  • _trigram_fts_enabled() — reads the knob via a deferred load_config_readonly() import (keeps the storage layer cycle-free; a single global read keeps every SessionDB open in agreement so the schema can't oscillate across processes).
  • _drop_trigram_fts_triggers() — trigger-only drop (leaves base FTS intact).
  • Gated all three trigram creation paths: v10 migration, v11 migration, steady-state _init_schema. When disabled, creation/backfill is skipped and existing trigram triggers are dropped so writes stop paying the cost immediately.
  • Trigger-repair heuristic now expects only the triggers the active config requires → no full-FTS rebuild loop on every open when trigram is off.
  • optimize_fts() — the deliberate, lock-held teardown point: drops the (possibly large) trigram shadow table when disabled so a following VACUUM reclaims the pages. The heavy DROP is intentionally not in __init__ (avoids contending with a concurrent writer during a plain open).
• tests/test_hermes_state.py — two tests: disabled path (no table, base search works, no rebuild loop, idempotent reopen) and re-enable flip-back (index rebuilt).
• website/docs/user-guide/sessions.md — documents the knob + LIKE fallback + reclaim path.

How to Test

pytest tests/test_hermes_state.py -q -k trigram_fts

1. Default (no config) → trigram index built as before (parity).
2. sessions.trigram_fts: false → messages_fts_trigram never created; db._trigram_available is False; base word search still returns results; substring/CJK degrade to LIKE.
3. Reopen while disabled → idempotent, no rebuild loop, trigram triggers stay absent.
4. Flip back to true → trigram index rebuilt on next open.

Verified end-to-end against real runtime deps (disabled path, idempotent reopen, flip-back) plus the config-read path (trigram_fts: false → False, absent → True).

Checklist

Code

• I have read the Contributing Guide
• Conventional Commits (feat(state): ...)
• Searched existing PRs for duplicates — this supersedes the closed env-var cluster (feat(fts): implement HERMES_DISABLE_FTS_TRIGRAM env gate (docstring parity) #66690/feat(state): opt-out flag for the trigram FTS index (HERMES_FTS_TRIGRAM=0) #57761/fix(state): honour HERMES_DISABLE_FTS_TRIGRAM to skip CJK index #42190/feat(state): make trigram FTS5 index optional #27770/feat(state): make trigram FTS5 index optional #22710) by using config.yaml per AGENTS.md
• PR contains only related changes (one logical change: the trigram config flag)
• Added tests
• Tested on Linux

Documentation & Housekeeping

• Updated docs (website/docs/user-guide/sessions.md) + inline config schema comment
• cli-config.yaml.example — N/A (it does not enumerate the sessions.* keys; peers auto_prune/vacuum_after_prune are not listed there either)
• CONTRIBUTING/AGENTS — N/A (no architecture change)
• Cross-platform — no OS-specific calls added

[teknium1]

Thanks for preserving the config.yaml approach and for documenting the storage rationale. The underlying opt-out is still absent on current main, so this is not redundant.

Problems

• This patch predates the v23 FTS redesign. Current main creates trigram indexes in both schema paths at hermes_state_schema.py:689-725, and optimize_fts_storage() recreates them while migrating legacy databases at hermes_state_search.py:393-438. The implementation needs a port across those current paths rather than a cherry-pick of the old hermes_state.py edits.
• Current main independently enables the CJK-bigram index (hermes_cli/config_defaults.py:2579-2586; initialization at hermes_state_schema.py:726-728). The proposed LIKE-fallback documentation needs an explicit contract for that index.
• The new tests monkeypatch _trigram_fts_enabled, so they do not verify the YAML/config-loader path described by the feature.

Suggested changes

• Rework against the current schema and optimize-storage lifecycle, then add temp-HERMES_HOME YAML E2E coverage for default, disable, and re-enable behavior.

Automated hermes-sweeper review.

[GottZ]

This was generated by AI during triage.

Summary

Eleven PRs address or reference the FTS storage complex: #20239 and merged #65798 replace inline indexes with external-content layouts; #35826, #43701, and #65798 reduce trigram indexing of tool data; and #22710, #27770, #42190, #45916, #57761, #68089, and #68394 add variants of a trigram opt-out. The merged #65798 addresses the principal duplication and tool-row amplification causes, while the open #68394 targets the still-absent operator opt-out but predates the resulting v23 architecture.

Related pull requests

• perf(state): reduce FTS storage with external-content indexes #20239 [closed] partial — (+587/-76) — n/a: Moves both FTS tables to three-column external-content storage and adds migration/lifecycle coverage; closed because the same design landed on main via perf(state): compact v23 FTS layout — external-content indexes, tool-row-free trigram, opt-in hermes sessions optimize-storage #65798 at 9acc4b4.
• feat(state): make trigram FTS5 index optional #22710 [closed] partial — (+305/-23) — n/a: Adds an HERMES_DISABLE_FTS_TRIGRAM opt-out, CJK LIKE fallback, cleanup/VACUUM APIs, tests, and docs; closed and succeeded by the equivalent feat(state): make trigram FTS5 index optional #27770.
• feat(state): make trigram FTS5 index optional #27770 [closed] partial — (+305/-23) — n/a: Implements the same environment-variable opt-out and cleanup path as feat(state): make trigram FTS5 index optional #22710; closed as not planned because AGENTS.md requires behavioral flags in config.yaml, with feat(state): add config.yaml flag to disable trigram FTS index #68089 and feat(state): add sessions.trigram_fts config flag to disable the trigram FTS index #68394 providing config-based successors.
• fix: exclude tool messages from FTS indexes to prevent trigram bloat #35826 [closed] partial — (+12/-8) — n/a: Excludes tool rows and tool metadata from both FTS indexes, but that would remove supported base-FTS tool search; closed after perf(state): compact v23 FTS layout — external-content indexes, tool-row-free trigram, opt-in hermes sessions optimize-storage #65798 implemented the salvageable form by excluding tool rows only from trigram.
• fix(state): honour HERMES_DISABLE_FTS_TRIGRAM to skip CJK index #42190 [closed] partial — (+170/-22) — n/a: Covers the environment-variable gate across migrations, startup repair, rebuilds, trigger counting, and CJK fallback; closed as not planned because the public behavioral switch must use config.yaml.
• fix(db): exclude tool_calls JSON from trigram FTS5 index to reduce bloat #43701 [closed] partial — (+133/-8) — n/a: Removes tool_calls and tool_name from trigram indexing and adds a rebuild migration, directly targeting FTS5 Trigram Index Bloat: 18.3x Expansion on Structured JSON Data (tool_calls) #43690; closed after perf(state): compact v23 FTS layout — external-content indexes, tool-row-free trigram, opt-in hermes sessions optimize-storage #65798 superseded it with tool-row-free trigram plus external-content indexes, while also avoiding its occupied migration version and CJK metadata-search gaps.
• feat(state): honor HERMES_DISABLE_FTS_TRIGRAM to skip the trigram FTS index #45916 [closed] partial — (+230/-67) — n/a: Adds the same environment-variable opt-out as feat(state): make trigram FTS5 index optional #27770 with artifact cleanup, re-enable backfill, repair-aware trigger counting, and CJK LIKE fallback; closed as a duplicate and for the same config-policy conflict.
• feat(state): opt-out flag for the trigram FTS index (HERMES_FTS_TRIGRAM=0) #57761 [closed] partial — (+2022/-10) — n/a: Adds another environment-variable opt-out with useful trigger-repair and re-enable coverage, but bundles unrelated Nextcloud and authorization changes; closed as a duplicate of feat(state): make trigram FTS5 index optional #27770 with those FTS details carried into the cluster discussion.
• perf(state): compact v23 FTS layout — external-content indexes, tool-row-free trigram, opt-in hermes sessions optimize-storage #65798 [merged] best fix — (+152804/-13782) — n/a: Merged reference implementation and recorded best fix for state.db FTS trigram index bloat: 70% of DB size is full-text indexes #22478, FTS5 Trigram Index Bloat: 18.3x Expansion on Structured JSON Data (tool_calls) #43690, and Add config flag to disable trigram FTS index (prevent state.db bloat) #55233, delivering v23 external-content indexes, tool-row-free trigram, and an opt-in optimize-storage lifecycle while preserving base-FTS tool search.
• feat(state): add config.yaml flag to disable trigram FTS index #68089 [closed] duplicate — (+265/-26) — n/a: Implements a config.yaml opt-out under state.disable_fts_trigram with startup/migration cleanup and repair-aware trigger counting; although closed, it remains a relevant config-based predecessor to feat(state): add sessions.trigram_fts config flag to disable the trigram FTS index #68394.
• feat(state): add sessions.trigram_fts config flag to disable the trigram FTS index #68394 fixes — (+191/-20) — n/a: Adds sessions.trigram_fts with trigger suppression, deferred teardown, re-enable behavior, tests, and docs; despite the keep_open review, its old hermes_state.py diff must be ported to the v23 schema and optimize-storage paths, define interaction with CJK bigram indexing, and test the real YAML loader.

Duplicates

#22710 and #27770 are the same patch lineage; #42190 and #45916 substantially duplicate that HERMES_DISABLE_FTS_TRIGRAM approach, while #57761 implements the same opt-out under another environment variable. #68089 and #68394 are competing config.yaml-based successors; #20239, #35826, and #43701 supplied storage-layout or tool-indexing changes later consolidated in #65798.

Suggested consolidation

Author action on #68394: rebase onto current main, or split out the config-based opt-out and port it across hermes_state_schema.py:689-728 and hermes_state_search.py:393-438, including the optimize-storage lifecycle, an explicit CJK-bigram contract, and temp-HERMES_HOME YAML tests for default, disable, and re-enable. This follows the keep_open salvage review rather than bypassing it; the closed environment-variable duplicates #22710, #27770, #42190, #45916, and #57761 remain closed, #20239/#35826/#43701 are superseded by merged #65798, and #68089 can remain closed as the competing config-based predecessor.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I22478(["issue #22478 (closed)"])
    I43690(["issue #43690 (closed)"])
    I55233(["issue #55233 (closed)"])
    subgraph Dup68089 ["PRs duplicating each other"]
        P68089["PR #68089 (closed)"]
        P68394["PR #68394 (open)"]
    end
    P68394 -.->|partial| I22478
    P68394 -.->|partial| I43690
    P68394 -->|fixes| I55233
    class I22478 closed
    class I43690 closed
    class I55233 closed
    class P68089 closed
    class P68394 open
    class P68394 target
    click I22478 "https://github.com/NousResearch/hermes-agent/issues/22478"
    click I43690 "https://github.com/NousResearch/hermes-agent/issues/43690"
    click I55233 "https://github.com/NousResearch/hermes-agent/issues/55233"
    click P68089 "https://github.com/NousResearch/hermes-agent/pull/68089"
    click P68394 "https://github.com/NousResearch/hermes-agent/pull/68394"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 11 pull requests and 3 issues in this complex. Diffs were read for 10 of 11 PRs (rest unavailable); Assessment working set: 249 kB of PR diffs, 53 kB of issue/PR text, 37 kB of discussion (35 comments), 39 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.