feat: FTS corruption detection, auto-recovery, and repair command

[tuancookiez-hub]

Summary

Adds FTS5 corruption detection and auto-recovery to state.db. When FTS indexes become corrupt (malformed), Hermes now self-heals on startup instead of silently breaking session_search and all FTS-backed features.

Closes #33865.

Background

Related reports: #5563, #30908, #23717, #30445 — all describe SQLite corruption in state.db or kanban.db caused by interrupted WAL checkpoints, concurrent process contention, or force-kills during active transactions.

Changes

hermes_state.py

• _init_schema(): Now catches sqlite3.DatabaseError (corrupt FTS) in addition to sqlite3.OperationalError (missing FTS). On either, drops and recreates FTS tables, then backfills from messages.

• _drop_fts() / _drop_fts_trigram(): Static helpers that cleanly drop FTS virtual tables and their triggers. Shared by _init_schema() and rebuild_fts().

• rebuild_fts(): Public method that drops, recreates, and backfills both FTS indexes. Returns (fts_count, trigram_count). Used by hermes sessions repair and hermes doctor --fix.

• fts_integrity_check(): Returns a dict comparing FTS rowcount against messages table. Detects both corruption (DatabaseError) and index drift (count mismatch).

• integrity_check(): Wraps PRAGMA integrity_check, returns list of issues.

hermes_cli/doctor.py

The state.db check now:

1. Runs PRAGMA integrity_check (catches B-tree corruption, page errors)
2. Validates FTS rowcount matches messages table (catches index drift)
3. Reports specific errors instead of generic "has issues"
4. hermes doctor --fix can auto-rebuild corrupt FTS indexes

hermes_cli/main.py

New hermes sessions repair subcommand:

• Runs integrity check + FTS health validation
• Auto-rebuilds corrupt FTS indexes
• --check-only flag for read-only diagnostics
• Handles both corruption (malformed) and drift (count mismatch)

Testing

python -c "
from hermes_state import SessionDB
import tempfile, os
os.environ['HERMES_HOME'] = tempfile.mkdtemp()
db = SessionDB()
fts_count, tri_count = db.rebuild_fts()
msg_count = db.message_count()
assert fts_count == msg_count
assert tri_count == msg_count
fts = db.fts_integrity_check()
assert fts['fts_ok'] and fts['trigram_ok']
print('All assertions passed')
db.close()
"

Breaking Changes

None. All changes are additive. Existing behavior preserved for healthy databases.

Checklist

• Bug fix (crash/data loss prevention)
• Cross-platform (Windows + Linux + macOS — pure sqlite3, no platform-specific code)
• No new dependencies
• Backward compatible
• Follows existing patterns (v11 migration FTS rebuild, _reconcile_columns() declarative approach)

[alt-glitch]

Addresses #33865. Note competing open PRs: #32589 (doctor FTS detection + repair) and #30514 (MATCH-based FTS corruption detection). This PR is broader — covers startup auto-recovery, hermes sessions repair subcommand, and doctor integration — but overlaps significantly with both.

[teknium1]

Closing as superseded by PR #43149.

This PR correctly identified that the FTS recovery has to happen before normal session search paths can work, but the merged fix goes lower-level: it repairs malformed sqlite_master / duplicate messages_fts schema rows before SessionDB() can open, then lets the derived FTS objects rebuild from canonical messages. PR #43149 includes you as a co-author for this repair direction. Thanks for the detailed write-up and implementation work.

[agifree939-spec]

Real-world reproduction: VPS gateway read-path FTS corruption → Desktop client cascade failure → manual repair

Environment

• VPS: Oracle Cloud Osaka, Ubuntu 24.04, Docker (Dokploy v0.29.2)
• Hermes: v0.19.0 (2026-07-20), commit 07e97d2
• Client: macOS Hermes Desktop App (App Sandbox)

Failure chain

1. VPS state.db (1.3 GB) FTS shadow tables corrupted (PRAGMA integrity_check → database disk image is malformed)
2. Gateway starts — no write occurs, only read paths (session_search, /resume title resolution) hit FTS MATCH queries
3. Read path raises sqlite3.DatabaseError: database disk image is malformed
4. Desktop App receives error → shows 恢复失败: database disk image is malformed + 无法加载此会话
5. Auto-repair NOT triggered — _try_runtime_fts_rebuild only fires on write failure (_execute_write exception handler), not on read-path corruption
6. Manual hermes sessions repair (which calls repair_state_db_schema) fixed it

Gap in PR #33869

• Added fts_integrity_check(), rebuild_fts(), repair_state_db_schema() — great offline tools
• Added _try_runtime_fts_rebuild for write-path self-heal
• Missing: gateway/CLI startup health probe that catches read-path FTS corruption and auto-repairs before serving traffic

Suggested fix
In gateway entrypoint (and SessionDB.__init__ for CLI), add:

from hermes_state import _db_opens_cleanly, repair_state_db_schema, DEFAULT_DB_PATH

reason = _db_opens_cleanly(DEFAULT_DB_PATH)
if reason is not None:
    logger.warning(f"state.db unhealthy at startup: {reason} — attempting auto-repair")
    report = repair_state_db_schema(DEFAULT_DB_PATH, backup=True)
    if not report.get("repaired"):
        raise RuntimeError(f"Auto-repair failed: {report.get('error')}")
    logger.info(f"state.db auto-repaired via {report.get('strategy')}")

This would have prevented the entire cascade (VPS gateway → Desktop client) without any manual intervention.

Workaround deployed: Added Dokploy Health Check that runs the same probe+repair every 30s, so future corruption self-heals before traffic hits.