fix(state): surface failed WAL checkpoints instead of silently swallowing them

[tangtaizong666]

What does this PR do?

SessionDB._try_wal_checkpoint() runs PRAGMA wal_checkpoint(TRUNCATE) — which zeroes the WAL file — and then swallows every failure with a bare except Exception: pass. When a checkpoint fails mid-operation the database can be left inconsistent with no log trace at all; the failure only surfaces later as an opaque disk I/O error on the next connection, at which point the TUI session store, holographic memory provider, and SessionDB all fail simultaneously and the operator has nothing to correlate it with.

This PR keeps the TRUNCATE checkpoint (introduced in #39058 to bound WAL growth) but makes its failures visible and diagnosable:

• Benign lock contention (database is locked / busy) is logged at debug level and skipped — this is normal under multi-process concurrency and the next periodic checkpoint (every 50 writes) retries.
• Any other checkpoint failure is logged at warning level, then a PRAGMA quick_check(1) probe runs immediately: if the DB is damaged, an error log pins the corruption to the checkpoint that produced it instead of leaving a mystery for the next connection attempt.
• close() logs its best-effort checkpoint failure at debug level instead of discarding it.
• The checkpoint path still never raises — _execute_write() callers are unaffected.

Also fixes the stale "PASSIVE" comment on _CHECKPOINT_EVERY_N_WRITES (it has been TRUNCATE since #39058).

Related Issue

Fixes #44795

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• hermes_state.py — _try_wal_checkpoint(): replace silent except Exception: pass with busy-aware logging and a post-failure quick_check integrity probe (new helper _probe_integrity_after_checkpoint_failure()); close(): log failed close-time checkpoint; fix stale PASSIVE comment.
• tests/test_hermes_state.py — new TestWalCheckpointErrorHandling class (7 tests) covering: busy skip stays quiet, I/O error logs warning + probes integrity, healthy DB produces no error log, unusable DB logs error, malformed quick_check result logs error, checkpoint never raises across exception types, close() logs failed checkpoint.

How to Test

1. pytest tests/test_hermes_state.py::TestWalCheckpointErrorHandling -q — 7 passed.
2. Revert the hermes_state.py hunk and re-run: 6 of 7 fail (the seventh, test_checkpoint_never_raises, asserts an invariant that holds before and after).
3. Full suite via the canonical runner: scripts/run_tests.sh (same per-file isolation as CI) — passes; ruff check . passes; ty check hermes_state.py introduces no new diagnostics vs main (88 pre-existing before and after).

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: Ubuntu (WSL2), Python 3.12

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — docstrings updated
• I've updated cli-config.yaml.example if I added/changed config keys — N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — pure SQLite/stdlib, no platform-specific behavior
• I've updated tool descriptions/schemas if I changed tool behavior — N/A

[alt-glitch]

Duplicate of #44834 — same fix for #44795 (WAL checkpoint TRUNCATE swallows exceptions): replace the bare except: pass in _try_wal_checkpoint() with busy-aware logging plus a PRAGMA quick_check(1) integrity probe. #44834 was opened slightly earlier with the identical approach.

[tangtaizong666]

Acknowledged — #44834 was opened first (by ~2 minutes) and takes the same overall approach, so happy to defer to the maintainers on which to keep. Noting three behavioral differences in this PR for whoever consolidates, since they affect whether the guard actually detects corruption:

1. The quick_check result is inspected. PRAGMA quick_check(1) does not raise when it finds corruption — it returns rows like database disk image is malformed. fix(state): log WAL checkpoint failures instead of silently swallowing #44834 executes the pragma without reading the result, so its integrity guard only fires when the DB is so broken the pragma itself throws; a damaged-but-readable DB (the common post-checkpoint-failure state) passes silently. This PR checks row[0] != "ok" and logs an error with the reported status.
2. The probe re-acquires self._lock (with a None guard). All other self._conn access in SessionDB is serialized under the lock; probing without it races concurrent writers on a connection shared across threads.
3. Benign database is locked/busy contention is logged at debug and skipped. TRUNCATE checkpoints need exclusive access, and this DB is documented as heavily multi-process (the jitter-retry block above _execute_write). Warning + a full quick_check probe on every routine busy collision (every 50 writes) would add log noise and wasted I/O on multi-megabyte DBs; the next periodic checkpoint retries anyway.

#44834 additionally covers the pre-VACUUM checkpoint in vacuum(), which this PR doesn't touch — that hunk is worth keeping whichever way this goes. Tests here were verified to fail on the unfixed code (6/7; the seventh asserts the never-raises invariant) and live in the existing tests/test_hermes_state.py suite.

[liuhao1024]

Verification: LGTM ✅

Reviewed the full diff (2 files, +181/-5). Replaces silent except Exception: pass in _try_wal_checkpoint() with proper error stratification and integrity probing.

Key improvements:

1. OperationalError with "locked"/"busy" → debug level (normal concurrency)
2. Other OperationalError → warning + PRAGMA quick_check(1) integrity probe
3. Generic Exception → warning + integrity probe
4. close() checkpoint failure → debug level (non-fatal)
5. _probe_integrity_after_checkpoint_failure() catches corrupted DB before it surfaces as opaque "disk I/O error" on next connection

Concurrency safety: The with self._lock block in _try_wal_checkpoint releases the lock before the except handler calls _probe_integrity_after_checkpoint_failure(), which acquires the lock again — no deadlock risk.

Test coverage: 129-line test class covering: busy checkpoint (debug only), I/O error (warning + probe), unexpected error (warning), unusable DB (error), corrupt quick_check (error), checkpoint never raises, close() logging. Uses _CheckpointFailConn proxy pattern for clean fault injection.

No findings.

[teknium1]

Thanks for the focused diagnostics and for checking the quick_check(1) result rather than only whether the pragma raises. The underlying periodic checkpoint still swallows failures on current main at hermes_state.py:1266-1267, so the premise is live.

Problems

• The busy handling at hermes_state.py:1215 only runs for an exception. The checkpoint result path still inspects only result[1]/result[2] (hermes_state.py:1258-1265 on current main), so it does not classify or log a returned busy status. Add explicit status handling and a returned-busy test.
• SessionDB.vacuum() retains the same silent TRUNCATE checkpoint catch at hermes_state.py:6510-6513. The PR discussion correctly notes that #44834 has that hunk; consolidate it so the stated checkpoint-diagnostics goal covers the sibling path.

Suggested changes

• Inspect the checkpoint status result before declaring success, logging benign busy status at debug without probing or raising.
• Carry over the pre-VACUUM checkpoint logging change and test its non-fatal continuation.

Automated hermes-sweeper review.

[teknium1]

This only recognizes busy when execute() raises. The returned checkpoint tuple is still not status-checked before the success path; handle a returned busy status too, and add a test for that non-exception case so the documented debug behavior is covered.