fix(gateway): retry failed transcript appends and rebuild corrupted FTS - #66256
Merged
kshitijk4poor merged 2 commits intoJul 17, 2026
Merged
kshitijk4poor merged 2 commits into
kshitijk4poor merged 2 commits into
Conversation
alt-glitch
added
type/bug
tonydwb
reviewed
Jul 17, 2026
tonydwb
left a comment
tonydwb
left a comment
There was a problem hiding this comment.
Code Review Summary
Verdict: Approved (LGTM)
Overview
Retries failed transcript appends and rebuilds corrupted FTS in gateway. +320/0.
Security
- No hardcoded secrets or credentials
Code Quality
- Clean retry and recovery logic
- Proper FTS rebuild mechanism
Looks Good
- Well-scoped resilience fix
Reviewed by Hermes Agent
Queue failed session DB appends so disk order cannot silently lag memory.\nRebuild corrupt FTS indexes once and surface repeated failures as warnings.
Follow-up fixes for salvaged PR NousResearch#65637: 1. Clear _dirty_transcripts in rewrite_transcript + rewind_session — stale pending messages were re-inserted after /retry, /undo, /compress. 2. Narrow _is_fts_corruption_error to specific SQLite error strings — bare 'fts' substring matched 'shifts', 'gifts', etc. 3. Move DB write outside _transcript_retry_lock — holding the lock during writes serialized all sessions' transcript appends and blocked during FTS rebuild. Now the lock guards only the pending queue. 4. Push rebuild_fts() into SessionDB — SessionStore was reaching into _conn/_lock private attrs. SessionDB.rebuild_fts() follows the same pattern as optimize_fts(). 5. Cap pending per session at 200 — prevents unbounded memory growth when DB is persistently broken. Oldest messages dropped with warning. Added 4 new tests: dirty-clear on rewrite/rewind, FTS matcher false positives, pending cap enforcement.
kshitijk4poor
force-pushed
the
salvage/65637-transcript-append-retry
branch
from
July 17, 2026 13:18
f88ea2c to
95b3c7d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Transcript DB append failures (including SQLite FTS corruption) no longer silently drop messages — failed appends are queued in-memory and retried in order on the next append, with FTS rebuild attempted once per store on corruption errors.
Cherry-picked from #65637 by @MaartenDMT, with five hardening fixes on top.
Changes
gateway/session.py:append_to_transcriptqueues failed messages per session and retries them in order on subsequent appends. FTS corruption errors triggerSessionDB.rebuild_fts()once per store. Log level escalated from debug to warning with failure/pending counters.hermes_state.py: AddedSessionDB.rebuild_fts()method (mirrorsoptimize_fts()pattern — acquires_lock, iterates_FTS_TABLES, calls_fts_table_exists, executes FTS5rebuildcommand).tests/gateway/test_session.py: 5 new tests covering retry+rebuild flow, dirty-clear on rewrite/rewind, FTS matcher false positives, pending cap enforcement.Hardening fixes (on top of contributor's commit)
_dirty_transcriptsinrewrite_transcript+rewind_session— stale pending messages were re-inserted after /retry, /undo, /compress replaced the transcript._is_fts_corruption_error— bare"fts"substring matched "shifts", "gifts", etc. Now matches specific SQLite error strings._transcript_retry_lock— holding the lock during writes serialized all sessions' transcript appends and blocked during FTS rebuild. Lock now guards only the pending queue.rebuild_fts()intoSessionDB— SessionStore was reaching into_conn/_lockprivate attrs.SessionDB.rebuild_fts()follows the existingoptimize_fts()pattern.Validation
rebuildattempted once per storeE2E tests: 6/6 passed (real SessionDB, real FK constraint, real FTS rebuild, concurrency verified).
Unit tests: 9/9 passed (5 new + 4 existing in TestGatewaySessionDbRecovery). 3 pre-existing failures on origin/main confirmed unrelated.
Closes #65637