fix(state): log WAL checkpoint failures instead of silently swallowing - #44834
Open
liuhao1024 wants to merge 1 commit into
Open
fix(state): log WAL checkpoint failures instead of silently swallowing#44834liuhao1024 wants to merge 1 commit into
liuhao1024 wants to merge 1 commit into
Conversation
The bare `except Exception: pass` in `_try_wal_checkpoint()` silently discards errors from the destructive TRUNCATE checkpoint. If the checkpoint fails mid-operation, the WAL is already truncated to zero bytes but data was not fully written back to the main DB, causing state.db corruption. Changes: - `_try_wal_checkpoint()`: log warning on exception, run PRAGMA quick_check integrity guard after failure, log error if integrity check also fails - `close()`: log debug on checkpoint failure (non-destructive context) - `vacuum()`: log debug on pre-VACUUM checkpoint failure The TRUNCATE checkpoint itself is unchanged — only the error handling improves from silent to observable, enabling operators to detect and respond to corruption before it cascades. Fixes NousResearch#44795
13 tasks
19 tasks
teknium1
reviewed
Jul 14, 2026
teknium1
left a comment
Contributor
There was a problem hiding this comment.
Thanks for making a real silent failure path observable. Current main still has the bare catch in hermes_state.py:1256-1267, so the logging change remains relevant.
Problems
- The proposed guard at
hermes_state.py:883executesPRAGMA quick_check(1)but does not consume its returned row. Current integrity handling reads and validates PRAGMA results athermes_state.py:532-535; therefore a returned corruption status is ignored by this implementation. - The new probe runs after
_try_wal_checkpoint()leaves theself._lockscope. The checkpoint query itself is serialized athermes_state.py:1256-1260, so the follow-up query should use the same mutex.
Suggested changes
- Fetch and validate the quick-check result under
self._lock, logging an error for a missing or non-okvalue. - Add a test that returns a non-
okquick-check row; the current test only verifies invocation or an exception. - Consider keeping routine lock/busy checkpoint outcomes out of the warning-plus-probe path, consistent with
hermes_state.py:1220-1231.
Automated hermes-sweeper review.
| except Exception as exc: | ||
| logger.warning("WAL checkpoint failed (non-fatal): %s", exc) | ||
| try: | ||
| self._conn.execute("PRAGMA quick_check(1)") |
Contributor
There was a problem hiding this comment.
quick_check(1) reports integrity failures as result rows, so this call must fetch and validate the returned value (and run under self._lock). Current main's probe does this at hermes_state.py:532-535; otherwise a readable corrupt database is treated as healthy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Replaces the bare
except Exception: passin_try_wal_checkpoint()and related methods with proper exception logging and an integrity guard. The destructivePRAGMA wal_checkpoint(TRUNCATE)was silently swallowing all errors — when it fails mid-operation, the WAL is already truncated to zero bytes but data was not fully written back, causing state.db corruption that cascades to TUI session store, holographic memory provider, and SessionDB.Related Issue
Fixes #44795
Type of Change
Changes Made
hermes_state.py(_try_wal_checkpoint): Replaceexcept Exception: passwithexcept Exception as exc: logger.warning(...)plusPRAGMA quick_check(1)integrity guard that logs an error if the DB is corrupted after a failed checkpoint.hermes_state.py(close): Replaceexcept Exception: passwithexcept Exception as exc: logger.debug(...)for the final checkpoint.hermes_state.py(vacuum): Replaceexcept Exception: passwithexcept Exception as exc: logger.debug(...)for the pre-VACUUM checkpoint.tests/test_wal_checkpoint_exception.py: New test file with 6 tests covering warning logging, integrity check on failure, error logging on integrity failure, silent success, close resilience, and vacuum resilience.How to Test
python -m pytest tests/test_wal_checkpoint_exception.py -v— all 6 tests should pass_try_wal_checkpointencounters a checkpoint failure, it now logs a warning instead of silently continuing, and runsPRAGMA quick_check(1)to detect corruptiontest_hermes_state.pytests remain unaffected — no behavioral change in the happy pathChecklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests passDocumentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/ACode Intelligence
_try_wal_checkpoint(),close(),vacuum()inhermes_state.py(3 call sites forPRAGMA wal_checkpoint(TRUNCATE))