feat: FTS corruption detection, auto-recovery, and repair command

ISSUE-draft.md

@@ -0,0 +1,34 @@
+# state.db FTS corruption goes undetected — no integrity check, no repair path
+
+## Summary
+
+The `messages_fts` and `messages_fts_trigram` FTS5 indexes in `state.db` can become corrupt ("database disk image is malformed"), silently breaking `session_search`, `/resume`, `/history`, and any feature backed by FTS. There is currently:
+
+1. **No integrity check on startup** — `_init_schema()` creates/reconciles tables but never runs `PRAGMA integrity_check`
+2. **No FTS health validation** — `hermes doctor` only checks `SELECT COUNT(*) FROM sessions`; it doesn't validate FTS indexes match the messages table
+3. **No repair command** — `hermes sessions` has `list`, `prune`, `stats`, `rename`, `export`, `delete`, `browse` — but no `repair`
+4. **No auto-recovery** — When FTS is corrupt, `_init_schema()` catches `sqlite3.OperationalError` (table missing) but not `sqlite3.DatabaseError` (table corrupt/malformed)
+
+## Root Cause
+
+FTS5 virtual tables and their triggers insert into the FTS index as part of the message INSERT transaction. If that transaction is interrupted mid-commit (force-kill, WAL checkpoint failure, power loss), the FTS and messages tables desync. The `_try_wal_checkpoint()` runs every 50 writes but is best-effort with bare `except Exception: pass` — corrupt FTS during checkpoint is silently swallowed.
+
+## Reproduction
+
+1. Run Hermes with heavy session activity (gateway + CLI + worktree agents sharing state.db)
+2. Force-kill the process (`taskkill /F /IM hermes.exe` on Windows, or SIGKILL on Linux)
+3. Restart — `session_search` returns "database disk image is malformed"
+
+## Related Issues
+
+- #5563 — broader state.db corruption report that includes page-level corruption
+- #30908 — same root cause pattern for kanban.db (WAL checkpoint interruption)
+- #23717 — documents the "hot-update death spiral" causing state.db corruption
+- #30445 — multi-gateway concurrent SQLite access causing corruption
+
+## Impact
+
+- `session_search` (the only way for Hermes to recall cross-session context) is completely broken
+- `/resume`, `/title`, `/history`, `/branch` all fail
+- The only recovery path is manual: stop gateway, export JSON, rebuild from scratch
+- Users lose session history if they don't know the manual recovery procedure

PR-draft.md

@@ -0,0 +1,70 @@
+## Summary
+
+Adds FTS5 corruption detection and auto-recovery to `state.db`. When FTS indexes become corrupt (malformed), Hermes now self-heals on startup instead of silently breaking `session_search` and all FTS-backed features.
+
+Closes #33865.
+
+## Background
+
+Related reports: #5563, #30908, #23717, #30445 — all describe SQLite corruption in state.db or kanban.db caused by interrupted WAL checkpoints, concurrent process contention, or force-kills during active transactions.
+
+## Changes
+
+### `hermes_state.py`
+
+- **`_init_schema()`**: Now catches `sqlite3.DatabaseError` (corrupt FTS) in addition to `sqlite3.OperationalError` (missing FTS). On either, drops and recreates FTS tables, then backfills from messages.
+
+- **`_drop_fts()` / `_drop_fts_trigram()`**: Static helpers that cleanly drop FTS virtual tables and their triggers. Shared by `_init_schema()` and `rebuild_fts()`.
+
+- **`rebuild_fts()`**: Public method that drops, recreates, and backfills both FTS indexes. Returns `(fts_count, trigram_count)`. Used by `hermes sessions repair` and `hermes doctor --fix`.
+
+- **`fts_integrity_check()`**: Returns a dict comparing FTS rowcount against messages table. Detects both corruption (`DatabaseError`) and index drift (count mismatch).
+
+- **`integrity_check()`**: Wraps `PRAGMA integrity_check`, returns list of issues.
+
+### `hermes_cli/doctor.py`
+
+The state.db check now:
+1. Runs `PRAGMA integrity_check` (catches B-tree corruption, page errors)
+2. Validates FTS rowcount matches messages table (catches index drift)
+3. Reports specific errors instead of generic "has issues"
+4. `hermes doctor --fix` can auto-rebuild corrupt FTS indexes
+
+### `hermes_cli/main.py`
+
+New `hermes sessions repair` subcommand:
+- Runs integrity check + FTS health validation
+- Auto-rebuilds corrupt FTS indexes
+- `--check-only` flag for read-only diagnostics
+- Handles both corruption (malformed) and drift (count mismatch)
+
+## Testing
+
+```bash
+python -c "
+from hermes_state import SessionDB
+import tempfile, os
+os.environ['HERMES_HOME'] = tempfile.mkdtemp()
+db = SessionDB()
+fts_count, tri_count = db.rebuild_fts()
+msg_count = db.message_count()
+assert fts_count == msg_count
+assert tri_count == msg_count
+fts = db.fts_integrity_check()
+assert fts['fts_ok'] and fts['trigram_ok']
+print('All assertions passed')
+db.close()
+"
+```
+
+## Breaking Changes
+
+None. All changes are additive. Existing behavior preserved for healthy databases.
+
+## Checklist
+
+- [x] Bug fix (crash/data loss prevention)
+- [x] Cross-platform (Windows + Linux + macOS — pure sqlite3, no platform-specific code)
+- [x] No new dependencies
+- [x] Backward compatible
+- [x] Follows existing patterns (v11 migration FTS rebuild, `_reconcile_columns()` declarative approach)

hermes_cli/doctor.py

@@ -1022,16 +1022,47 @@ def run_doctor(args):
     state_db_path = hermes_home / "state.db"
     if state_db_path.exists():
         try:
-            import sqlite3
-            conn = sqlite3.connect(str(state_db_path))
-            cursor = conn.execute("SELECT COUNT(*) FROM sessions")
-            count = cursor.fetchone()[0]
-            conn.close()
-            check_ok(f"{_DHH}/state.db exists ({count} sessions)")
+            from hermes_state import SessionDB
+            db = SessionDB()
+            session_count = db.session_count()
+
+            # PRAGMA integrity_check
+            integrity_issues = db.integrity_check()
+            if integrity_issues:
+                detail = integrity_issues[0] if len(integrity_issues) == 1 else f"{len(integrity_issues)} issues"
+                check_fail(f"state.db integrity check failed: {detail}")
+                issues.append(f"state.db integrity compromised — run 'hermes sessions repair'")
+                if should_fix:
+                    # Cannot auto-fix integrity issues — needs manual rebuild
+                    check_warn("Cannot auto-fix integrity issues", "(rebuild with 'hermes sessions repair')")
+            else:
+                check_ok(f"state.db integrity OK ({session_count} sessions)")
+
+            # FTS index health
+            fts_status = db.fts_integrity_check()
+            if fts_status["error"]:
+                check_fail(f"FTS indexes corrupt: {fts_status['error']}")
+                issues.append("FTS indexes corrupt — run 'hermes sessions repair' to rebuild")
+                if should_fix:
+                    fts_count, tri_count = db.rebuild_fts()
+                    check_ok(f"FTS rebuilt: {fts_count} messages indexed")
+                    fixed_count += 1
+            elif not fts_status["fts_ok"] or not fts_status["trigram_ok"]:
+                msg = f"FTS index mismatch: messages={fts_status['message_count']} fts={fts_status['fts_count']} trigram={fts_status['trigram_count']}"
+                check_warn(msg)
+                issues.append("FTS index out of sync — run 'hermes sessions repair' to rebuild")
+                if should_fix:
+                    fts_count, tri_count = db.rebuild_fts()
+                    check_ok(f"FTS rebuilt: {fts_count} messages indexed")
+                    fixed_count += 1
+            else:
+                check_ok(f"FTS indexes healthy ({fts_status['fts_count']} messages indexed)")
+
+            db.close()
         except Exception as e:
-            check_warn(f"{_DHH}/state.db exists but has issues: {e}")
+            check_warn(f"state.db exists but has issues: {e}")
     else:
-        check_info(f"{_DHH}/state.db not created yet (will be created on first session)")
+        check_info(f"state.db not created yet (will be created on first session)")
 
     # Check WAL file size (unbounded growth indicates missed checkpoints)
     wal_path = hermes_home / "state.db-wal"

hermes_cli/main.py

@@ -13332,7 +13332,7 @@ def cmd_mcp(args):
     # =========================================================================
     sessions_parser = subparsers.add_parser(
         "sessions",
-        help="Manage session history (list, rename, export, prune, delete)",
+        help="Manage session history (list, rename, export, prune, delete, repair)",
         description="View and manage the SQLite session store",
     )
     sessions_subparsers = sessions_parser.add_subparsers(dest="sessions_action")
@@ -13376,6 +13376,16 @@ def cmd_mcp(args):
 
     sessions_subparsers.add_parser("stats", help="Show session store statistics")
 
+    sessions_repair = sessions_subparsers.add_parser(
+        "repair",
+        help="Detect and repair corrupt FTS indexes or DB integrity issues",
+    )
+    sessions_repair.add_argument(
+        "--check-only",
+        action="store_true",
+        help="Check integrity without making changes",
+    )
+
     sessions_rename = sessions_subparsers.add_parser(
         "rename", help="Set or change a session's title"
     )
@@ -13560,6 +13570,52 @@ def cmd_sessions(args):
                 size_mb = os.path.getsize(db_path) / (1024 * 1024)
                 print(f"Database size: {size_mb:.1f} MB")
 
+        elif action == "repair":
+            check_only = getattr(args, "check_only", False)
+
+            # 1. Integrity check
+            print("Checking database integrity...")
+            integrity_issues = db.integrity_check()
+            if integrity_issues:
+                print(f"  ✗ integrity_check found {len(integrity_issues)} issue(s):")
+                for issue in integrity_issues[:5]:
+                    print(f"    {issue}")
+                if len(integrity_issues) > 5:
+                    print(f"    ... and {len(integrity_issues) - 5} more")
+                if not check_only:
+                    print("  ⚠ Integrity issues cannot be auto-repaired.")
+                    print("    Consider: export sessions → fresh DB → reimport.")
+            else:
+                print("  ✓ integrity_check passed")
+
+            # 2. FTS health
+            print("Checking FTS indexes...")
+            fts_status = db.fts_integrity_check()
+            if fts_status["error"]:
+                print(f"  ✗ FTS corrupt: {fts_status['error']}")
+                if not check_only:
+                    print("  Rebuilding FTS indexes...")
+                    fts_count, tri_count = db.rebuild_fts()
+                    print(f"  ✓ FTS rebuilt: {fts_count} messages indexed")
+                    print(f"  ✓ Trigram rebuilt: {tri_count} messages indexed")
+            elif not fts_status["fts_ok"] or not fts_status["trigram_ok"]:
+                print(
+                    f"  ⚠ FTS mismatch: messages={fts_status['message_count']} "
+                    f"fts={fts_status['fts_count']} trigram={fts_status['trigram_count']}"
+                )
+                if not check_only:
+                    print("  Rebuilding FTS indexes...")
+                    fts_count, tri_count = db.rebuild_fts()
+                    print(f"  ✓ FTS rebuilt: {fts_count} messages indexed")
+                    print(f"  ✓ Trigram rebuilt: {tri_count} messages indexed")
+            else:
+                print(f"  ✓ FTS healthy ({fts_status['fts_count']} messages indexed)")
+
+            if check_only:
+                print("\nDone (check-only mode, no changes made).")
+            else:
+                print("\nRepair complete.")
+
         else:
             sessions_parser.print_help()