fix(state): FTS read-path corruption detection, search self-heal, and REINDEX repair strategy - #68653
Merged
Merged
Conversation
…x corruption is detected (#66724) `hermes sessions repair --check-only` opens cleanly on state.db files with partial FTS5 index corruption — base tables read fine, the rolled-back write probe from #50502 succeeds, and `PRAGMA integrity_check` returns "ok". But every session_search / /resume title resolution / feature backed by MATCH / snippet / rank queries errors out with `database disk image is malformed` because internal shadow-table segments are bad. The official repair tool then gives false confidence. Add a representative FTS5 read probe against both `messages_fts` and `messages_fts_trigram` (the latter backs title resolution). Empty MATCH strings are accepted by every FTS5 index without requiring populated content, so the probe is safe on a freshly-init'd DB; missing-table / missing-column errors fall through to the existing "not yet a populated DB" branch, matching the write-probe's behaviour. Any other OperationalError is surfaced as the check reason, which sends `hermes sessions repair` to its existing FTS 'rebuild' path (repair_state_db_schema, line 616). Single-file change in hermes_state.py::_db_opens_cleanly. No public API change. No new imports. Fixes #66724.
The FTS5 read probe in _db_opens_cleanly() only caught sqlite3.OperationalError. But the corruption class #66724 actually wants caught — partial shadow-table damage where MATCH / snippet / rank queries raise DatabaseError("database disk image is malformed") — is a DatabaseError, not OperationalError. Without this catch the probe crashes the caller instead of returning a reason, which is exactly the silent-fail mode the issue describes. Move the try/except inside the for-loop so each FTS table is probed independently (one table corrupted should still surface as a reason), add a separate except clause for DatabaseError that surfaces the same reason format, and use continue instead of pass so the loop still walks both tables when only one is missing on a brand-new DB. Tested by hand: with a corrupted messages_fts_trigram shadow table the function now returns 'fts5 read probe failed on messages_fts_trigram: database disk image is malformed' instead of crashing out. Without this fix it would still crash.
… classifier Two follow-ups on top of f842733 (the FTS5 read probe added in #66906): 1. The original probe query used MATCH '', which FTS5 rejects with 'fts5: syntax error near '. Empty MATCH syntax is not valid FTS5. Switch to MATCH '""' — a quoted empty phrase that parses, scans zero rows, and exercises the same shadow-table read path the search tools use. The probe previously never reached the shadow segments at all on a healthy DB; the read-corruption class was only being detected because the existing write probe happens to fail first on a DatabaseError. 2. The probe's degraded-runtime branch only checked the substrings 'no such table' / 'no such column'. On a SQLite build without the fts5 module, MATCH against a legacy messages_fts table raises 'no such module: fts5' (a different OperationalError class). The substring check would misclassify that as corruption and trigger repair, whose final fallback deletes the messages_fts% schema (#66906 review). Use SessionDB._is_fts5_unavailable_error() — the canonical classifier already used by the degraded-runtime init path — to recognize both 'no such module: fts5' and 'no such tokenizer: trigram' as capability errors. Add tests covering: - Partial shadow-table damage (read-corruption class) - Repair brings reads back online - Healthy degraded DB without fts5 module stays healthy (regression for the misclassification risk) - Healthy degraded DB without trigram tokenizer stays healthy Closes #66906 review feedback Refs #66724
Complements #66296 (self-heal on the write path): search_messages()'s main FTS5 MATCH query caught only sqlite3.OperationalError (a query-syntax error → return empty). A corrupt FTS index raises the malformed / "fts5: corrupt structure record" class, which is a sqlite3.DatabaseError — the parent of OperationalError, so it was NOT caught and propagated straight out of search_messages, crashing session/history search. The write path now rebuilds and retries on that class, but a read-only session (cron/CLI history search, or a search issued before any write) never triggers a write, so its search stayed broken until the next process restart ran the offline repair. Catch the DatabaseError corruption class on the search MATCH read too and route it through the existing one-shot _try_runtime_fts_rebuild(), then retry the query. The catch is moved outside `with self._lock` so rebuild_fts() can re-acquire the lock (mirrors _execute_write). The one-shot guard is shared with the write path, so a single instance never loops on a genuinely unrecoverable index. OperationalError syntax handling is unchanged (caught first). Adds a regression test: with a corrupted messages_fts and no post-corruption write, search_messages() rebuilds in place and returns the match; without the fix it raises DatabaseError.
The trigram MATCH branch in search_messages() had the same OperationalError-only catch that #66420 fixed on the main FTS5 branch: a corrupt messages_fts_trigram shadow table raises the malformed / 'fts5: corrupt structure record' class (sqlite3.DatabaseError, parent of OperationalError), which propagated straight out of search_messages and crashed CJK session/history search for read-only sessions. Route that class through the shared one-shot _try_runtime_fts_rebuild() and retry the trigram query (catch moved outside self._lock so rebuild_fts() can re-acquire it, mirroring the main branch). If the rebuild is refused (guard consumed / FTS disabled / different error) or the retry fails, fall through to the existing LIKE substring fallback — which reads only the canonical messages table — instead of raising, so CJK search degrades gracefully rather than crashing. Adds two regression tests: trigram search self-heals in place after shadow-table corruption (answers from the rebuilt trigram index, not the LIKE fallback), and degrades to LIKE without raising when the one-shot rebuild was already consumed. Follow-up to #66420; refs #66296 #66724
When PRAGMA integrity_check reports 'wrong # of entries in index' for B-tree indexes (e.g. idx_sessions_handoff_state), the existing repair strategies (FTS rebuild, sqlite_master dedup, drop-FTS+VACUUM) don't address the mismatch. Add Strategy 0.5: run REINDEX to rewrite the index b-tree from canonical table rows before escalating to more destructive strategies.
Replace the mocked test for #63398's REINDEX strategy: the original monkeypatched _db_opens_cleanly to return the corruption string, so the REINDEX pass itself was never exercised against actual index corruption — the test would pass even if REINDEX didn't fix anything. New fixture _corrupt_btree_index() builds genuine on-disk staleness with a writable_schema hack: rewrite the index definition to a partial index (WHERE 0), REINDEX so the b-tree is rebuilt empty, then restore the full definition. integrity_check then reports the real 'wrong # of entries in index idx_messages_session' / 'row N missing from index' class from #63386 — no mocks anywhere. The rewritten test asserts end-to-end with real function calls: - the real _db_opens_cleanly detects the stale index, - repair_state_db_schema repairs it with strategy 'reindex_btree', - post-repair the detector and raw PRAGMA integrity_check both report healthy, and a query forced through the rebuilt index (INDEXED BY) sees every row. Adds a second test asserting the REINDEX strategy is non-destructive (all sessions/messages survive, readable via SessionDB). Follow-up to #63398; refs #63386
alt-glitch
added
type/bug
Contributor
૮ >ﻌ< ა ci reviewrunning on a19b7ca looks good to me! |
This was referenced Jul 21, 2026
19 tasks
5 tasks
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hermes sessions repair --check-onlycan no longer report a corrupt state.db as healthy while session_search fails: the integrity probe now exercises a real FTS5 MATCH read, the search path self-heals FTS corruption at runtime (including the CJK/trigram branch), and the repair ladder gains a REINDEX strategy for corrupt B-tree indexes — with a real corrupted-index test fixture, no mocks.Salvages #66906 by @Enough1122, #66420 by @Frowtek, #63398 by @liuhao1024 (authorship preserved).
Changes
hermes_state.py_db_opens_cleanly: FTS5 MATCH read probe + DatabaseError classification (fix(state): probe FTS5 read path in _db_opens_cleanly so partial index corruption is detected (#66724) #66906)search_messages: one-shot FTS rebuild self-heal on the read path (fix(state): self-heal FTS corruption on the SessionDB search path too #66420), extended to the CJK/trigram branch with graceful LIKE fallbackrepair_state_db_schema: REINDEX strategy for 'wrong # of entries in index' (fix(state): add REINDEX strategy to repair stale B-tree indexes #63398), tested against a genuinely corrupted index (writable_schema fixture), end-to-endValidation
Targeted state-db suite: 482 passed, 0 failed. Fixes #66724.
Infographic