fix(state): harden state.db against corruption — write lock, WAL protections, reconnect self-heal - #82280
Closed
shali10 wants to merge 3 commits into
Closed
fix(state): harden state.db against corruption — write lock, WAL protections, reconnect self-heal#82280shali10 wants to merge 3 commits into
shali10 wants to merge 3 commits into
Conversation
alt-glitch
added
type/bug
7 tasks
added 2 commits
August 10, 2026 09:40
…ections, reconnect self-heal
Production hardening for the canonical session store (state.db), fixing
corruption classes observed on a live v0.20.0 deployment:
- Runtime connection corruption ("file is not a database"): a sibling
process replacing/truncating the backing file breaks the live
connection. Add a bounded one-shot reconnect self-heal on the write
path.
- Mixed journal-mode corruption: journal_mode=DELETE applied while
sibling connections hold WAL open unlinked live -wal/-shm files.
Refuse to downgrade when on-disk mode is unknown; raise
WalUnsupportedError when require_wal is set.
- No write-level serialization: concurrent writers could interleave.
Add state_db_write_lock (process-local write lock).
- NUL-token guard: decoded binary bytes tokenized into bogus script
paths crashed Path.expanduser() when HOME is unset.
- close() checkpoint safety: close() must never issue TRUNCATE/RESTART
checkpoint, only PASSIVE.
- PRAGMA journal_mode reads retry transient disk i/o error; short-lived
ledger connections verify WAL ownership instead of re-running
journal_mode=WAL against the gateway's live WAL/SHM files.
- terminal_tool blocks deleting/moving the live state.db from inside
the gateway process.
…t timeout The local busy_timeout=5000 patch (PRAGMA busy_timeout + 5s connect timeout) made SQLite's built-in busy handler silently wait out lock holds, bypassing _execute_write's jittered patience loop entirely: test_write_lock_patience's 0.2s patience contract (raise OperationalError naming the lock holder) never fired because SQLite waited up to 5s and the write succeeded. Restore upstream's timeout=1.0 and drop the PRAGMA so patience semantics stay intact.
teknium1
pushed a commit
that referenced
this pull request
Aug 13, 2026
…try transient EIO on journal-mode probe Salvaged remainder of PR #82280 (state.db hardening rollup): - Runtime connection corruption: a sibling process replacing/truncating the backing file breaks the live write connection — every subsequent write raises 'file is not a database' and the gateway wedges permanently (messages pile up in memory). Add a bounded one-shot reconnect on the write path: close the broken connection, reopen the DB file (re-running WAL activation + schema reconciliation), retry the failed write once. - _on_disk_journal_mode: retry transient 'disk i/o error' (virtualized block devices) a few times before returning None, so a one-shot EIO doesn't push callers onto the fail-closed unknown-mode branch. The rollup's write-lock machinery, checkpoint-strategy changes, and repair serialization are intentionally NOT included — superseded by PRs #84277 and #69609, or wrong-direction per the POSIX lock-cancellation findings (#71724 lineage).
Author
|
Closing as superseded — upstream has since implemented the core protections itself (reconnect self-heal, WalUnsupportedError, PASSIVE-only checkpoints, write serialization, NUL/zeroed-db guards) across #69603/#45383/#68474 and related commits. Any remaining delta will be proposed as a fresh PR against current main. |
Author
|
Closing as superseded — upstream has since implemented the core protections itself (reconnect self-heal, WalUnsupportedError, PASSIVE-only checkpoints, write serialization, NUL/zeroed-db guards). Remaining delta will be proposed as a fresh PR against current main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Production hardening for the canonical session store (
state.db), fixing the corruption classes observed on a live deployment (v0.20.0):file is not a database): a sibling process (forked curator agent, external repair pass) replacing/truncating the backing file breaks the live connection. Fix: bounded one-shot reconnect self-heal on the write path.journal_mode=DELETEwas being applied while sibling connections held WAL open, unlinking live-wal/-shmfiles. Fix: refuse to downgrade when on-disk mode is unknown (existing is None→ conservative raise; withrequire_wal→ clearWalUnsupportedError).state_db_write_lock(process-local write lock).Path.expanduser()when HOME is unset.close()checkpoint safety:close()must never issue TRUNCATE/RESTART checkpoint (only PASSIVE), matching the WAL-reset vulnerability fix.Also:
PRAGMA journal_mode=WALreads now retry transientdisk i/o error(ZFS/APFS-CoW), andterminal_toolblocks deleting/moving the livestate.dbfrom inside the gateway process.Commits (8)
chore: snapshot local patches before upstream rebase— baseline snapshot of local patches (state.db protection, busy_timeout, qqbot/telegram fixes)fix(tool): precise state.db protection— only block destructive ops on the live file, not read-only mentionsfix(state): refuse journal downgrade when on-disk mode is unknownfix(state): commit pending state.db hardening batch— write lock, NUL-token guard, WAL fallback teststest(state): close() must never issue TRUNCATE/RESTART checkpointfix(state): reconnect self-heal for runtime connection corruptionchore: drop terminal_tool.py.bak-state-db-protect leftoverfix(state): restore WalUnsupportedError for unknown-mode + require_wal— fixes dead code introduced by the upstream merge (mergedexisting is Noneguard shadowed the local branch)Testing
tests/test_hermes_state_wal_fallback.py,test_wal_checkpoint_strategy.py,test_hermes_state.py,test_sqlite_wal_reset_gate.py: 269 passed