perf(state): merge FTS5 segments + add handoff_state index to curb write-lock contention

[kshitijk4poor]

Summary

Salvages two verified, complementary state.db performance fixes into one
reviewable change. On long-lived databases shared by the gateway + cron
processes, write-lock hold times grew until competing writers hit
database is locked.

Two orthogonal fixes, together:

1. FTS5 segment merge (hermes_state.py) — original work by @isair (fix(state): periodically merge FTS5 segments to curb write-lock contention #50124)
   • The messages triggers append one FTS5 segment per insert; the existing
     optimize_fts() helper was never called, so segments accumulated
     unbounded (~34k trigram segments / 27k messages observed), lengthening
     every MATCH scan and every insert's automerge cost — and the WAL
     write-lock hold.
   • Wires optimize_fts() into _execute_write on a coarse 1000-write cadence
     beside the existing checkpoint; _try_optimize_fts is best-effort /
     never-raises, off-lock. optimize is a no-op once merged, so steady-state
     cost is negligible.
2. handoff_state index (hermes_state.py) — original work by @kenyonxu (fix(gateway): add handoff_state index to prevent heartbeat blocking #43504)
   • Adds idx_sessions_handoff_state ON sessions(handoff_state, started_at) so
     the gateway heartbeat's handoff query stops doing a full table scan.

Credit

Cherry-picked preserving original authorship:

• fix(state): periodically merge FTS5 segments to curb write-lock contention — @isair (fix(state): periodically merge FTS5 segments to curb write-lock contention #50124)
• fix(gateway): move handoff_state index to DEFERRED_INDEX_SQL — @kenyonxu (fix(gateway): add handoff_state index to prevent heartbeat blocking #43504, closes [Bug]: Discord gateway heartbeat can be blocked by synchronous handoff SQLite polling #40695)

One trivial conflict resolved: the DEFERRED_INDEX_SQL block gained two indexes
on main since #43504 was authored; all three index definitions are kept.

Verification

• tests/test_hermes_state.py → 295 passed (full suite, incl. fix(state): periodically merge FTS5 segments to curb write-lock contention #50124's two
  new behavior-contract tests: cadence fires + search unaffected; failing
  optimize never breaks the surrounding write).

Note

Deliberately NOT bundled here: #43701 (trigram index redesign, schema migration
v15→v16) changes trigram search behavior (drops tool_name/tool_calls from
the index) and interacts with this FTS path — it deserves its own review.

[tonydwb]

Code Review Summary

Verdict: LGTM

Performance improvement: periodically merges fragmented FTS5 segments (every 1000 writes) to prevent tens-of-thousands of segments from lengthening write-lock hold time and starving competing writers. Adds idx_sessions_handoff_state index.

Looks Good

• Best-effort optimize never fails writes
• Coarse cadence amortizes merge cost far below checkpoint cadence
• optimize is a no-op once merged — idle DB pays almost nothing
• Index uses CREATE INDEX IF NOT EXISTS (safe for existing DBs)
• Tests cover cadence triggering and failure resilience

---

Reviewed by Hermes Agent

[alt-glitch]

This was generated by AI during triage.

Related: salvages two open PRs into one reviewable change — #50124 (FTS5 segment-merge cadence, @isair) and #43504 (handoff_state index, @kenyonxu). All three are open; this is the combined superset. Flagging for a maintainer to pick the canonical PR (this salvage vs. the two originals).

[tonydwb]

Code Review Summary

Verdict: LGTM

Good performance fix to merge FTS5 segments and add a handoff_state index. The implementation is clean with proper error handling and comprehensive tests. The periodic merge cadence is well-documented and the failure handling is robust.

Looks Good

• Clean implementation with proper error handling
• Good test coverage (2 tests)
• Well-documented merge cadence and failure handling
• Adds useful index for handoff_state queries
• Follows existing patterns in the codebase

---

Reviewed by Hermes Agent