perf(state): merge FTS5 segments on VACUUM + add 'hermes sessions optimize'

[kshitijk4poor]

Summary

The FTS5 indexes backing session search (messages_fts, messages_fts_trigram) grow as a series of incremental b-tree segments — one per trigger-driven insert batch. SQLite's background automerge caps at ~16 segments, so a long-lived store ends up scanning many segments per MATCH and never collapses them unless the special 'optimize' command is issued.

Nothing in the codebase ever ran it. SessionDB.vacuum() only fired after a prune that deleted rows, and even then it never merged FTS segments — so the indexes stay fragmented for the life of the database.

This PR wires FTS5 segment-merge into the existing reclamation path and adds an on-demand command. It's a layout-only optimization: search results and snippet() output are unchanged.

Changes

• hermes_state.py
  • SessionDB.optimize_fts() — merges each FTS5 index to a single segment via the INSERT INTO <fts>(<fts>) VALUES('optimize') command. Probes for the trigram table first (it's lazily created and can be disabled via HERMES_DISABLE_FTS_TRIGRAM), so it's safe to call unconditionally. Returns the number of indexes optimized.
  • vacuum() now calls optimize_fts() before VACUUM, so the pages freed by the segment merge are returned to the OS in the same pass.
• hermes_cli/main.py
  • hermes sessions optimize — on-demand FTS merge + VACUUM with before/after size reporting. Previously there was no way to compact the session store without a prune that deleted rows.
• tests/test_hermes_state.py
  • TestOptimizeFts — index count, search + snippet preservation, missing-trigram path, idempotency.

Why this is orthogonal to the existing FTS-size PRs

The big structural size cuts are already in flight: #20239 (external-content FTS, kills the duplicate _content shadow tables) and #27770 (make the trigram index optional). Segment merge is complementary to both — it helps regardless of inline vs external-content storage and regardless of whether the trigram index exists. This PR does not touch the schema or migration path, so it doesn't conflict with either.

Benchmark

8000 messages across 200 sessions, committed in small batches to fragment the indexes (8 segments each):

Metric	Before	After
messages_fts segments	8	1 (−88%)
messages_fts_trigram segments	8	1 (−88%)
porter MATCH latency	0.449 ms/q	0.081 ms/q (5.5× faster)
trigram MATCH latency	0.632 ms/q	0.207 ms/q (3.0× faster)
'needle' matches	8000	8000 — identical row ids

The query-latency win is the consistent, provable benefit. On-disk size reclaim is variable: it's large on heavily fragmented indexes and negligible on an already-merged one (e.g. a real 1.8 GB store whose segments were already merged reclaimed only ~5 MB). The PR does not overclaim size savings — the durable value is sustained query speed plus the on-demand command.

Compatibility / safety

• Pure maintenance operation: no schema change, no migration, no behavior change to search or snippets (verified: identical result set before/after).
• optimize_fts() skips any absent FTS table, so it's a no-op-safe call on databases with the trigram index disabled.
• VACUUM still acquires an exclusive lock; the command is intended to run when the gateway/CLI is idle (same constraint already documented on vacuum()).

Test plan

• python -m pytest tests/test_hermes_state.py — 227 passed.
• hermes sessions optimize smoke-tested on a fresh DB and on a 1.8 GB real-data copy (integrity check ok, message/FTS counts intact, search + snippets working).

[github-actions]

🔎 Lint report: perf/fts-optimize vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 9568 on HEAD, 9566 on base (🆕 +2)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 5034 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

[teknium1]

Merged via PR #34596. Your commit was cherry-picked onto current main with your authorship preserved in git log (3869525). One small follow-up on top: vacuum() now returns the FTS index count so the CLI summary uses the real merged-index count instead of probing private members. Thanks for the clean, well-benchmarked PR.