fix(state): retry 'no more rows available' across all sqlite3.Error classes

[Dannou]

PR Draft — fix(state): retry "no more rows available" across all sqlite3.Error classes

Repo upstream: https://github.com/NousResearch/hermes-agent
Branche locale: fix/session-persistence-retry-catchall (commit fd8a9612c)
Diff: hermes_state.py +48/−4

---

Symptom

On Windows dual-writer setups (a standalone messaging gateway process + a desktop-spawned serve process sharing the same state.db in WAL mode), conversation turns sporadically die with:

Session DB append_message failed: no more rows available
Turn ended: reason=session_persistence_failed

and the user-facing "session storage could not be written" stop message. Always on large appends (17–26 KB tool results, which trigger heavy FTS5 trigram sync work inside the messages INSERT triggers), always under concurrent write load.

Root cause

_execute_write's retry net handles the transient SQLite engine error "no more rows available" (documented in the existing comment as dual gateway/agent WAL contention) only when it surfaces as sqlite3.OperationalError or sqlite3.DatabaseError.

On this SQLite build (uv CPython 3.11, bundled sqlite3.dll), the error surfaces as a class that lives outside DatabaseError (most likely InterfaceError), so it propagates on attempt 0 without any retry.

Timing proof from production logs: the append failed 1.16 s after the last tool result, while 60 retries at the minimum 20 ms jitter take ≥ 1.18 s — the retry loop was provably never entered.

The existing code comment already acknowledges "exact subclass varies with the SQLite build" — the net was just one branch short.

Fix

Add a catch-all except sqlite3.Error branch (after the OperationalError and DatabaseError branches) that applies the same jittered retry when the message contains "no more rows", and re-raises anything else unchanged.

Also includes the local dual-writer hardening the retry constants needed:

• _WRITE_MAX_RETRIES: 15 → 60 (1.3 s tolerance lost races against dual-writer FTS sync on large appends)
• _WRITE_RETRY_MAX_S: 150 ms → 300 ms
• engine timeout: 1.0 s → 2.0 s

Verification

• 3/3 targeted unit tests on _execute_write:
  • InterfaceError("no more rows available") on first attempt → retried, succeeds on attempt 2
  • InterfaceError with a different message → propagates immediately, 1 call only (no behavior change)
  • OperationalError("database is locked") → retried as before (no regression)
• python -m py_compile hermes_state.py clean
• Production soak: zero session_persistence_failed since the patched backend started (previously 3 occurrences in 24 h)

Reproduction notes

• Standalone single-writer: identical SQL always succeeds (verified against a production-copy 2.5 GB state.db, 32 KB payloads, triggers firing).
• Dual-writer stress (2 writers × 26 KB trigram-heavy payloads + a concurrent checkpoint/FTS-optimize loop): massive database is locked contention (all correctly OperationalError); the no more rows race is rarer and build-dependent — hence the catch-all rather than a class-specific branch.

Out of scope

• The dual-gateway topology itself (documented as supported); this PR only makes the write path resilient to it.
• FTS shadow-table corruption salvage (_try_runtime_fts_rebuild) — untouched.

---

Checklist avant publication (reste à faire avec validation utilisateur)

• gh auth status — compte GitHub actif (publication publique sous l'identité de l'utilisateur)
• Fork NousResearch/hermes-agent → Dannou/hermes-agent
• Push branche fix/session-persistence-retry-catchall sur le fork
• gh pr create avec ce corps de description
• Vérifier que la CI upstream passe

[teknium1]

Thanks for tracing the exception hierarchy. Current main still has the relevant gap: _execute_write retries only OperationalError messages containing locked/busy (hermes_state.py:2369-2398), while its following DatabaseError branch is for FTS-corruption recovery (hermes_state.py:2399-2411). An InterfaceError with the reported message would escape both paths.

Problems

• The branch conflicts with current main because commit 8da8a7887d06373e169af6e431ec52ebb439ec7a replaced _WRITE_MAX_RETRIES with deadline-based write patience (hermes_state.py:1757-1763, 2346-2387). The proposed fixed-attempt loops need to be ported to that current mechanism.
• The PR changes only hermes_state.py; no regression test covers the claimed InterfaceError case.

Suggested changes

• Add the narrow retry classification while preserving current patience_s deadlines and fast/slow jitter.
• Add tests for retrying InterfaceError("no more rows available") and immediately propagating unrelated InterfaceError messages.

Automated hermes-sweeper review.

[teknium1]

Please port this catch-all to current main's deadline-based retry policy. Commit 8da8a78 replaced _WRITE_MAX_RETRIES with patience_s and fast/slow jitter, so this fixed-attempt branch cannot be salvaged verbatim without discarding the newer transcript-specific wait budget.

[teknium1]

Merged via salvage PR #75883 (#75883). Your diff predated the deadline-patience rewrite (8da8a78), so we ported the substantive piece — a message-scoped 'no more rows available' retry that catches ALL sqlite3.Error classes including InterfaceError — onto the current patience loop, authored under your name. The constants changes were superseded by the rewrite. Fixes the gap you identified. Thanks for the Windows dual-writer analysis!