Skip to content

perf(state): reduce FTS storage with external-content indexes - #20239

Closed
JabberELF wants to merge 1 commit into
NousResearch:mainfrom
JabberELF:feat/fts-v12-external-content
Closed

perf(state): reduce FTS storage with external-content indexes#20239
JabberELF wants to merge 1 commit into
NousResearch:mainfrom
JabberELF:feat/fts-v12-external-content

Conversation

@JabberELF

@JabberELF JabberELF commented May 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Move messages_fts and messages_fts_trigram from inline FTS5 storage to three-column external-content FTS5 on current main (schema v20 → v21). This removes duplicated message/tool text from FTS content shadow tables while keeping messages as the source of truth.

Changes

  • hermes_state.py
    • Bump SCHEMA_VERSION from 20 to 21 while preserving the existing v18 gateway-metadata and v20 usage-attribution migrations.
    • Define both FTS tables with content, tool_name, and tool_calls, backed by content='messages' and content_rowid='id'.
    • Update insert/update/delete triggers to use the external-content FTS5 command pattern.
    • Rebuild derived indexes with FTS5's special rebuild command.
    • Detect an already-current external-content layout and avoid a redundant full rebuild.
    • If an existing inline trigram table cannot be dropped because the tokenizer is unavailable, keep the prior schema version and retry when trigram support returns.
    • Use snippet(..., -1, ...) so matches in any indexed column can produce a useful snippet.
  • tests/test_hermes_state.py
    • Cover v20 inline FTS → v21 external-content migration.
    • Cover insert/update/delete trigger lifecycle and searches across content, tool_name, tool_calls, and CJK trigram text.
    • Cover the no-trigram migration retry path and successful recovery when tokenizer support returns.
    • Verify that an already-current external-content layout does not rebuild again.
  • website/docs/developer-guide/session-storage.md
    • Document the three-column external-content schema, triggers, rebuild behavior, compatibility notes, and schema v21 migration.

Compatibility and behavior

  • messages remains the source of truth and is not mutated by the migration.
  • Normal FTS and CJK trigram searches continue to cover message content, tool names, and serialized tool calls.
  • Because the indexed fields are separate FTS columns, phrase queries no longer span content / tool_name / tool_calls boundaries. Ranking and snippet selection may also differ slightly from the previous concatenated single-column index.
  • Existing inline FTS databases migrate through the version-gated path. If trigram support is temporarily unavailable while an old trigram table still exists, the migration remains incomplete instead of recording a mixed schema as v21.
  • Rebuilding makes pages reusable inside SQLite. Returning space to the filesystem still requires VACUUM or the existing maintenance path; this migration does not run VACUUM automatically.
  • The FTS tables are derived indexes. Users who need rollback should restore a pre-upgrade state.db backup.

Validation

  • python -m pytest -o addopts='' tests/test_hermes_state.py -q --tb=short
    • 356 passed
  • python -m ruff check hermes_state.py tests/test_hermes_state.py
    • All checks passed!
  • python -m compileall -q hermes_state.py tests/test_hermes_state.py
  • git diff --check

@alt-glitch alt-glitch added type/perf Performance improvement or optimization P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels May 5, 2026
@LeonSGP43 LeonSGP43 mentioned this pull request May 6, 2026
Closed
@alt-glitch alt-glitch mentioned this pull request May 18, 2026
Closed
19 tasks
@JabberELF
JabberELF force-pushed the feat/fts-v12-external-content branch from f72b1f6 to ad232f3 Compare May 22, 2026 11:00
@JabberELF JabberELF changed the title feat(state): switch FTS5 to external-content mode (schema v12) perf(state): reduce FTS storage with external-content indexes (schema v13) May 22, 2026
@JabberELF

JabberELF commented May 22, 2026

Copy link
Copy Markdown
Contributor Author

Refreshed this PR to make it easier to review.

What changed:

  • Rebased onto current main.
  • Removed the unrelated fork workflow cleanup from the PR branch.
  • Updated the schema bump to v13, since upstream main already uses v12.
  • Kept this focused as an FTS storage reduction: external-content FTS tables now reference messages instead of duplicating message text.
  • Kept the positioning complementary to feat(state): make trigram FTS5 index optional #27770: if trigram / FTS search remains enabled, this reduces the storage overhead of the enabled indexes.

Verification added/updated:

  • v12 inline FTS → v13 external-content migration coverage.
  • External-content trigger lifecycle coverage for insert/update/delete.
  • Search coverage for content, tool_name, tool_calls, and CJK trigram text.
  • Snippet behavior now uses snippet(..., -1, ...) so non-content column matches can produce useful snippets too.

Local verification:

  • python -m pytest -o addopts='' tests/test_hermes_state.py -q --tb=short
  • Result: 217 passed.
  • git diff --check: clean.

Synthetic SQLite benchmark on 2,500 generated messages:

  • inline FTS DB: 26.80 MiB
  • external-content FTS DB: 17.22 MiB
  • size reduction: 35.7%
  • normal FTS median query time: 0.008 ms vs 0.008 ms
  • trigram CJK median query time: 0.022 ms vs 0.022 ms

Also adjusted the rollback wording: since the FTS tables are derived indexes over messages, rollback should restore a pre-upgrade state.db backup rather than rely on the migration being reversible in-place.

Happy to rebase again or adjust around #27770 if that lands first.

@plcunha plcunha mentioned this pull request May 26, 2026
Closed
@kshitijk4poor kshitijk4poor mentioned this pull request May 28, 2026
Closed
teknium1 pushed a commit that referenced this pull request May 29, 2026
@kshitijk4poor @teknium1
perf(state): merge FTS5 segments on VACUUM + add 'hermes sessions opt…
3869525 
…imize'

The FTS5 indexes (messages_fts, messages_fts_trigram) grow as a series of
incremental b-tree segments — one per trigger-driven insert batch. SQLite's
automerge caps at ~16 segments, so a long-lived store keeps scanning many
segments per MATCH and never collapses them unless the special 'optimize'
command runs. Nothing in the codebase ever ran it: vacuum() only fired after
a prune that deleted rows, and even then never merged FTS segments.

Changes:
- SessionDB.optimize_fts(): merges each FTS5 index to a single segment,
  probing for the (optional/lazy) trigram table first so it is safe to call
  unconditionally. Layout-only — search results and snippet() are unchanged.
- vacuum() now calls optimize_fts() before VACUUM so freed index pages are
  returned to the OS in the same pass.
- 'hermes sessions optimize' CLI subcommand for on-demand reclamation +
  segment compaction (previously there was no way to compact the store
  without a prune deleting rows), with before/after size reporting.

Benchmark (8000 msgs, fragmented to 8 segments/index):
- segments 8 -> 1 on both indexes
- porter MATCH 5.5x faster (0.449 -> 0.081 ms/q)
- trigram MATCH 3.0x faster (0.632 -> 0.207 ms/q)
- 8000 matches before == 8000 after, identical row ids (no functional change)

Orthogonal to the structural FTS-size PRs (#20239 external-content,
#27770 optional trigram) — segment merge helps regardless of those.

Tests: TestOptimizeFts covers index count, search+snippet preservation,
missing-trigram path, and idempotency. Full test_hermes_state.py green (227).
@JabberELF
JabberELF force-pushed the feat/fts-v12-external-content branch from ad232f3 to 6ba4830 Compare May 30, 2026 10:54
@JabberELF

JabberELF commented May 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Refreshed again onto current main.

What changed since the previous refresh:

  • Rebased cleanly on current main.
  • Moved the external-content FTS migration to schema v15 because upstream main is now schema v14.
  • Kept the PR focused on the same storage optimization: messages_fts and messages_fts_trigram now use external-content tables backed by messages instead of duplicating message text in FTS shadow storage.
  • Preserved compatibility with the newer main changes, including the FTS optimize/VACUUM maintenance path.
  • Cleaned stale migration comments after review.

Verification:

  • python -m pytest -o addopts='' tests/test_hermes_state.py -q --tb=short → 231 passed
  • python -m ruff check hermes_state.py tests/test_hermes_state.py → All checks passed
  • python -m compileall -q hermes_state.py tests/test_hermes_state.py
  • git diff --check

I also ran an independent read-only review of the rebased diff. It found no Critical/Must Fix issues; the only suggestions were comment wording cleanups, which are included in this update.

@JabberELF JabberELF changed the title perf(state): reduce FTS storage with external-content indexes (schema v13) perf(state): reduce FTS storage with external-content indexes (schema v15) May 30, 2026
@JabberELF
JabberELF force-pushed the feat/fts-v12-external-content branch from 6ba4830 to 903b6f9 Compare May 31, 2026 07:54
@JabberELF

JabberELF commented May 31, 2026

Copy link
Copy Markdown
Contributor Author

Refreshed this PR onto current main and resolved the hermes_state.py conflict.

What changed in the refresh:

  • Kept the external-content FTS5 schema as schema v15.
  • Preserved the newer upstream FTS5-unavailable migration guards.
  • Updated the shared FTS rebuild helper to use FTS5's external-content rebuild command.
  • Kept the PR scoped to hermes_state.py and tests/test_hermes_state.py.

Local verification:

  • python -m pytest -o addopts='' tests/test_hermes_state.py -q --tb=short → 234 passed
  • python -m ruff check hermes_state.py tests/test_hermes_state.py → All checks passed
  • python -m compileall -q hermes_state.py tests/test_hermes_state.py
  • git diff --check origin/main...HEAD

KKT-OPT pushed a commit to KKT-OPT/hermes-agent that referenced this pull request May 31, 2026
@kshitijk4poor @KKT-OPT
perf(state): merge FTS5 segments on VACUUM + add 'hermes sessions opt…
a19251e 
…imize'

The FTS5 indexes (messages_fts, messages_fts_trigram) grow as a series of
incremental b-tree segments — one per trigger-driven insert batch. SQLite's
automerge caps at ~16 segments, so a long-lived store keeps scanning many
segments per MATCH and never collapses them unless the special 'optimize'
command runs. Nothing in the codebase ever ran it: vacuum() only fired after
a prune that deleted rows, and even then never merged FTS segments.

Changes:
- SessionDB.optimize_fts(): merges each FTS5 index to a single segment,
  probing for the (optional/lazy) trigram table first so it is safe to call
  unconditionally. Layout-only — search results and snippet() are unchanged.
- vacuum() now calls optimize_fts() before VACUUM so freed index pages are
  returned to the OS in the same pass.
- 'hermes sessions optimize' CLI subcommand for on-demand reclamation +
  segment compaction (previously there was no way to compact the store
  without a prune deleting rows), with before/after size reporting.

Benchmark (8000 msgs, fragmented to 8 segments/index):
- segments 8 -> 1 on both indexes
- porter MATCH 5.5x faster (0.449 -> 0.081 ms/q)
- trigram MATCH 3.0x faster (0.632 -> 0.207 ms/q)
- 8000 matches before == 8000 after, identical row ids (no functional change)

Orthogonal to the structural FTS-size PRs (NousResearch#20239 external-content,
NousResearch#27770 optional trigram) — segment merge helps regardless of those.

Tests: TestOptimizeFts covers index count, search+snippet preservation,
missing-trigram path, and idempotency. Full test_hermes_state.py green (227).
alt-glitch pushed a commit that referenced this pull request Jun 14, 2026
@kshitijk4poor @teknium1
perf(state): merge FTS5 segments on VACUUM + add 'hermes sessions opt…
6a30f06 
…imize'

The FTS5 indexes (messages_fts, messages_fts_trigram) grow as a series of
incremental b-tree segments — one per trigger-driven insert batch. SQLite's
automerge caps at ~16 segments, so a long-lived store keeps scanning many
segments per MATCH and never collapses them unless the special 'optimize'
command runs. Nothing in the codebase ever ran it: vacuum() only fired after
a prune that deleted rows, and even then never merged FTS segments.

Changes:
- SessionDB.optimize_fts(): merges each FTS5 index to a single segment,
  probing for the (optional/lazy) trigram table first so it is safe to call
  unconditionally. Layout-only — search results and snippet() are unchanged.
- vacuum() now calls optimize_fts() before VACUUM so freed index pages are
  returned to the OS in the same pass.
- 'hermes sessions optimize' CLI subcommand for on-demand reclamation +
  segment compaction (previously there was no way to compact the store
  without a prune deleting rows), with before/after size reporting.

Benchmark (8000 msgs, fragmented to 8 segments/index):
- segments 8 -> 1 on both indexes
- porter MATCH 5.5x faster (0.449 -> 0.081 ms/q)
- trigram MATCH 3.0x faster (0.632 -> 0.207 ms/q)
- 8000 matches before == 8000 after, identical row ids (no functional change)

Orthogonal to the structural FTS-size PRs (#20239 external-content,
#27770 optional trigram) — segment merge helps regardless of those.

Tests: TestOptimizeFts covers index count, search+snippet preservation,
missing-trigram path, and idempotency. Full test_hermes_state.py green (227).
T02200059 pushed a commit to T02200059/hermes-agent that referenced this pull request Jun 18, 2026
@kshitijk4poor @teknium1
perf(state): merge FTS5 segments on VACUUM + add 'hermes sessions opt…
0fba10e 
…imize'

The FTS5 indexes (messages_fts, messages_fts_trigram) grow as a series of
incremental b-tree segments — one per trigger-driven insert batch. SQLite's
automerge caps at ~16 segments, so a long-lived store keeps scanning many
segments per MATCH and never collapses them unless the special 'optimize'
command runs. Nothing in the codebase ever ran it: vacuum() only fired after
a prune that deleted rows, and even then never merged FTS segments.

Changes:
- SessionDB.optimize_fts(): merges each FTS5 index to a single segment,
  probing for the (optional/lazy) trigram table first so it is safe to call
  unconditionally. Layout-only — search results and snippet() are unchanged.
- vacuum() now calls optimize_fts() before VACUUM so freed index pages are
  returned to the OS in the same pass.
- 'hermes sessions optimize' CLI subcommand for on-demand reclamation +
  segment compaction (previously there was no way to compact the store
  without a prune deleting rows), with before/after size reporting.

Benchmark (8000 msgs, fragmented to 8 segments/index):
- segments 8 -> 1 on both indexes
- porter MATCH 5.5x faster (0.449 -> 0.081 ms/q)
- trigram MATCH 3.0x faster (0.632 -> 0.207 ms/q)
- 8000 matches before == 8000 after, identical row ids (no functional change)

Orthogonal to the structural FTS-size PRs (NousResearch#20239 external-content,
NousResearch#27770 optional trigram) — segment merge helps regardless of those.

Tests: TestOptimizeFts covers index count, search+snippet preservation,
missing-trigram path, and idempotency. Full test_hermes_state.py green (227).
@JabberELF
JabberELF force-pushed the feat/fts-v12-external-content branch 3 times, most recently from 050d038 to 978e44a Compare June 30, 2026 03:29
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
@kshitijk4poor @teknium1
perf(state): merge FTS5 segments on VACUUM + add 'hermes sessions opt…
e709714 
…imize'

The FTS5 indexes (messages_fts, messages_fts_trigram) grow as a series of
incremental b-tree segments — one per trigger-driven insert batch. SQLite's
automerge caps at ~16 segments, so a long-lived store keeps scanning many
segments per MATCH and never collapses them unless the special 'optimize'
command runs. Nothing in the codebase ever ran it: vacuum() only fired after
a prune that deleted rows, and even then never merged FTS segments.

Changes:
- SessionDB.optimize_fts(): merges each FTS5 index to a single segment,
  probing for the (optional/lazy) trigram table first so it is safe to call
  unconditionally. Layout-only — search results and snippet() are unchanged.
- vacuum() now calls optimize_fts() before VACUUM so freed index pages are
  returned to the OS in the same pass.
- 'hermes sessions optimize' CLI subcommand for on-demand reclamation +
  segment compaction (previously there was no way to compact the store
  without a prune deleting rows), with before/after size reporting.

Benchmark (8000 msgs, fragmented to 8 segments/index):
- segments 8 -> 1 on both indexes
- porter MATCH 5.5x faster (0.449 -> 0.081 ms/q)
- trigram MATCH 3.0x faster (0.632 -> 0.207 ms/q)
- 8000 matches before == 8000 after, identical row ids (no functional change)

Orthogonal to the structural FTS-size PRs (NousResearch#20239 external-content,
NousResearch#27770 optional trigram) — segment merge helps regardless of those.

Tests: TestOptimizeFts covers index count, search+snippet preservation,
missing-trigram path, and idempotency. Full test_hermes_state.py green (227).
teknium1
teknium1 reviewed Jul 12, 2026
View reviewed changes

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused FTS storage optimization. The underlying premise is still present on current main: hermes_state.py:815-868 uses inline FTS tables containing a concatenation of content, tool_name, and tool_calls.

Problems

  • The migration version is now occupied. Current main is schema v19 (hermes_state.py:125), and current v18 performs the gateway metadata backfill at hermes_state.py:1543-1555 (landed in 747386ecfa34eb5cbc10e104d326259fb538f565). The PR's SCHEMA_VERSION = 18 and current_version < 18 FTS migration would either suppress the FTS migration for current databases or displace that gateway migration during conflict resolution.

Suggested changes

  • Preserve the v18 migration and carry this as the next schema migration (v20 on current main), with a test beginning from a v19 inline-FTS database.
  • Update the session-storage developer documentation to reflect the external-content, three-column FTS schema.

Automated hermes-sweeper review.

Comment thread hermes_state.py Outdated
@@ -122,7 +122,7 @@ def _delete_delegate_children(conn, parent_ids: List[str]) -> List[str]:

DEFAULT_DB_PATH = get_hermes_home() / "state.db"

SCHEMA_VERSION = 17
SCHEMA_VERSION = 18

@teknium1 teknium1 Jul 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: current main already uses schema v18 for the gateway metadata backfill (hermes_state.py:1543-1555) and is now schema v19. Please retain that migration and move this FTS conversion to the next schema version, with a matching current_version < 20 gate.

@teknium1 teknium1 added the sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state label Jul 12, 2026
@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jul 12, 2026
@JabberELF
JabberELF force-pushed the feat/fts-v12-external-content branch from 978e44a to 461d14c Compare July 13, 2026 02:50
@JabberELF JabberELF changed the title perf(state): reduce FTS storage with external-content indexes (schema v15) perf(state): reduce FTS storage with external-content indexes Jul 13, 2026
@JabberELF

JabberELF commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the sweeper review against current main: preserved the existing v18/v20 migrations, moved the FTS migration to v21, added v20 inline-FTS and trigram-recovery migration coverage, updated the session-storage documentation, and avoided redundant rebuilds for already-external schemas. Local validation: 356 tests passed; ruff and diff checks are clean.

santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
@kshitijk4poor @teknium1
perf(state): merge FTS5 segments on VACUUM + add 'hermes sessions opt…
ac95b28 
…imize'

The FTS5 indexes (messages_fts, messages_fts_trigram) grow as a series of
incremental b-tree segments — one per trigger-driven insert batch. SQLite's
automerge caps at ~16 segments, so a long-lived store keeps scanning many
segments per MATCH and never collapses them unless the special 'optimize'
command runs. Nothing in the codebase ever ran it: vacuum() only fired after
a prune that deleted rows, and even then never merged FTS segments.

Changes:
- SessionDB.optimize_fts(): merges each FTS5 index to a single segment,
  probing for the (optional/lazy) trigram table first so it is safe to call
  unconditionally. Layout-only — search results and snippet() are unchanged.
- vacuum() now calls optimize_fts() before VACUUM so freed index pages are
  returned to the OS in the same pass.
- 'hermes sessions optimize' CLI subcommand for on-demand reclamation +
  segment compaction (previously there was no way to compact the store
  without a prune deleting rows), with before/after size reporting.

Benchmark (8000 msgs, fragmented to 8 segments/index):
- segments 8 -> 1 on both indexes
- porter MATCH 5.5x faster (0.449 -> 0.081 ms/q)
- trigram MATCH 3.0x faster (0.632 -> 0.207 ms/q)
- 8000 matches before == 8000 after, identical row ids (no functional change)

Orthogonal to the structural FTS-size PRs (NousResearch#20239 external-content,
NousResearch#27770 optional trigram) — segment merge helps regardless of those.

Tests: TestOptimizeFts covers index count, search+snippet preservation,
missing-trigram path, and idempotency. Full test_hermes_state.py green (227).
donbowman pushed a commit to donbowman/hermes-agent that referenced this pull request Jul 13, 2026
@kshitijk4poor @donbowman
perf(state): merge FTS5 segments on VACUUM + add 'hermes sessions opt…
6177f3a 
…imize'

The FTS5 indexes (messages_fts, messages_fts_trigram) grow as a series of
incremental b-tree segments — one per trigger-driven insert batch. SQLite's
automerge caps at ~16 segments, so a long-lived store keeps scanning many
segments per MATCH and never collapses them unless the special 'optimize'
command runs. Nothing in the codebase ever ran it: vacuum() only fired after
a prune that deleted rows, and even then never merged FTS segments.

Changes:
- SessionDB.optimize_fts(): merges each FTS5 index to a single segment,
  probing for the (optional/lazy) trigram table first so it is safe to call
  unconditionally. Layout-only — search results and snippet() are unchanged.
- vacuum() now calls optimize_fts() before VACUUM so freed index pages are
  returned to the OS in the same pass.
- 'hermes sessions optimize' CLI subcommand for on-demand reclamation +
  segment compaction (previously there was no way to compact the store
  without a prune deleting rows), with before/after size reporting.

Benchmark (8000 msgs, fragmented to 8 segments/index):
- segments 8 -> 1 on both indexes
- porter MATCH 5.5x faster (0.449 -> 0.081 ms/q)
- trigram MATCH 3.0x faster (0.632 -> 0.207 ms/q)
- 8000 matches before == 8000 after, identical row ids (no functional change)

Orthogonal to the structural FTS-size PRs (NousResearch#20239 external-content,
NousResearch#27770 optional trigram) — segment merge helps regardless of those.

Tests: TestOptimizeFts covers index count, search+snippet preservation,
missing-trigram path, and idempotency. Full test_hermes_state.py green (227).
@alt-glitch alt-glitch mentioned this pull request Jul 17, 2026
Closed
19 tasks
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
@kshitijk4poor @teknium1
perf(state): merge FTS5 segments on VACUUM + add 'hermes sessions opt…
46d851c 
…imize'

The FTS5 indexes (messages_fts, messages_fts_trigram) grow as a series of
incremental b-tree segments — one per trigger-driven insert batch. SQLite's
automerge caps at ~16 segments, so a long-lived store keeps scanning many
segments per MATCH and never collapses them unless the special 'optimize'
command runs. Nothing in the codebase ever ran it: vacuum() only fired after
a prune that deleted rows, and even then never merged FTS segments.

Changes:
- SessionDB.optimize_fts(): merges each FTS5 index to a single segment,
  probing for the (optional/lazy) trigram table first so it is safe to call
  unconditionally. Layout-only — search results and snippet() are unchanged.
- vacuum() now calls optimize_fts() before VACUUM so freed index pages are
  returned to the OS in the same pass.
- 'hermes sessions optimize' CLI subcommand for on-demand reclamation +
  segment compaction (previously there was no way to compact the store
  without a prune deleting rows), with before/after size reporting.

Benchmark (8000 msgs, fragmented to 8 segments/index):
- segments 8 -> 1 on both indexes
- porter MATCH 5.5x faster (0.449 -> 0.081 ms/q)
- trigram MATCH 3.0x faster (0.632 -> 0.207 ms/q)
- 8000 matches before == 8000 after, identical row ids (no functional change)

Orthogonal to the structural FTS-size PRs (NousResearch#20239 external-content,
NousResearch#27770 optional trigram) — segment merge helps regardless of those.

Tests: TestOptimizeFts covers index count, search+snippet preservation,
missing-trigram path, and idempotency. Full test_hermes_state.py green (227).
@kshitijk4poor

kshitijk4poor commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Thanks @JabberELF — this was a well-built and well-tested change, and you correctly identified the FTS shadow-table duplication problem. External-content FTS5 has since landed on main via 9acc4b4 (#65798), which brings the schema to v23 with the same content='messages'/content_rowid='id' design plus incremental-rebuild tracking and tool-column indexing, so this PR is now fully superseded (and hermes_state.py has been split into mixins, so it no longer applies). Closing as implemented-on-main; your analysis of the storage win was spot on. Appreciate the contribution!

@GottZ GottZ mentioned this pull request Aug 3, 2026
Open
11 tasks
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
@kshitijk4poor @teknium1
perf(state): merge FTS5 segments on VACUUM + add 'hermes sessions opt…
d7bb744 
…imize'

The FTS5 indexes (messages_fts, messages_fts_trigram) grow as a series of
incremental b-tree segments — one per trigger-driven insert batch. SQLite's
automerge caps at ~16 segments, so a long-lived store keeps scanning many
segments per MATCH and never collapses them unless the special 'optimize'
command runs. Nothing in the codebase ever ran it: vacuum() only fired after
a prune that deleted rows, and even then never merged FTS segments.

Changes:
- SessionDB.optimize_fts(): merges each FTS5 index to a single segment,
  probing for the (optional/lazy) trigram table first so it is safe to call
  unconditionally. Layout-only — search results and snippet() are unchanged.
- vacuum() now calls optimize_fts() before VACUUM so freed index pages are
  returned to the OS in the same pass.
- 'hermes sessions optimize' CLI subcommand for on-demand reclamation +
  segment compaction (previously there was no way to compact the store
  without a prune deleting rows), with before/after size reporting.

Benchmark (8000 msgs, fragmented to 8 segments/index):
- segments 8 -> 1 on both indexes
- porter MATCH 5.5x faster (0.449 -> 0.081 ms/q)
- trigram MATCH 3.0x faster (0.632 -> 0.207 ms/q)
- 8000 matches before == 8000 after, identical row ids (no functional change)

Orthogonal to the structural FTS-size PRs (NousResearch#20239 external-content,
NousResearch#27770 optional trigram) — segment merge helps regardless of those.

Tests: TestOptimizeFts covers index count, search+snippet preservation,
missing-trigram path, and idempotency. Full test_hermes_state.py green (227).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@teknium1 teknium1 teknium1 left review comments

Assignees

No one assigned

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/perf Performance improvement or optimization

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@JabberELF @kshitijk4poor @teknium1 @alt-glitch