fix: prevent silent message loss when state.db persistence fails

[mlinquan]

Problem

When append_message() fails (WAL lock contention, disk full, SQLITE_READONLY, etc.), messages accumulate in messages[] and _last_flushed_db_idx is never advanced. On agent eviction, context compression, or process restart, those unflushed messages are silently lost. The Gateway also blindly trusts agent_persisted — it assumes DB writes succeeded as long as _session_db is not None, never verifying the agent actually committed anything.

Additionally, SQLITE_READONLY (not just LOCKED/BUSY) was not retried in _execute_write, causing transient WAL-readonly states to permanently fail. And background review agents ran without a session_db handle, so their writes went nowhere when the session DB had been recovered from a previous failure.

Changes (12 commits, rebased on main)

Pending Fallback — Direct, No Sticky Flag

run_agent.py — _flush_messages_to_session_db
Per-message try/except. Each successful append_message advances _last_flushed_db_idx. On failure, immediately calls _write_pending_fallback(messages, last_successful_idx) to dump unflushed messages to sessions/<sid>.pending.jsonl — no sticky flag, no deferred write.

run_agent.py — _persist_session
Added a catch: when _session_db is None (SQLite init never succeeded), writes ALL messages to pending fallback via _write_pending_fallback(messages, 0).

run_agent.py — _write_pending_fallback
New method. Wraps each message with its _flush_timestamp as {"_fallback_timestamp": t, "message": m} and appends to <sid>.pending.jsonl, respecting HERMES_PROFILE isolation.

Shared Flush Timestamp for Safe Dedup

Every append_message in _flush_messages_to_session_db now generates a timestamp ONCE and passes it to both the DB write (via the new timestamp= param of SessionDB.append_message) and the message dict as _flush_timestamp. This anchors the dedup comparison: if a DB write succeeded but the fallback was still written (false alarm), both carry the same timestamp and won't be double-inserted during recovery.

Deduplication is based on (role, content, tool_name, tool_calls, tool_call_id, timestamp) — max-overlap suffix-vs-prefix match between existing DB messages and pending entries.

SQLITE_READONLY Retry

hermes_state.py — _execute_write
Added "readonly" and "read-only" to the retry-eligible error string set. Previously only "locked" and "busy" were retried — WAL-mode readonly states were fatal on first attempt.

Background Review Agent Gets session_db

agent/background_review.py
Pass session_db=getattr(agent, "_session_db", None) to review agents so their writes land in the correct session store.

Gateway Startup Recovery

gateway/run.py — _recover_pending_messages
New function, called in GatewayRunner.__init__. Scans <hermes_home>/sessions/*.pending.jsonl, reads wrapped + plain messages, deduplicates against existing DB rows, replays new ones via SessionDB.append_message (with original timestamps, multimodal handling, reasoning fields), and deletes the pending file on success.

Hardening & Cleanup

• Log append_message failure reason before triggering pending fallback (was silently swallowed)
• Clear _session_db_create_fail_count on successful recovery so consecutive-failure logging starts fresh
• Restore accidentally removed comments in _flush_messages_to_session_db that documented the cursor-tracking pattern
• Delete pending files after recovery instead of renaming (simpler, safer)
• Add _session_db_create_fail_count init in agent_init.py

Design Principle

Memory is the most unsafe place. When DB persistence fails, get messages to disk immediately — JSONL fallback is the last line of defense before data loss. Dedup via shared timestamp guarantees safe replay on restart.

Related work

• Gateway replies with stale context when another process appends to the same session (cross-process agent-cache split-brain) #45966 — fix(gateway): invalidate agent cache on cross-process session writes
  Directly related: addresses the same cross-process session DB coordination problem from the Gateway side (agent writes to DB → Gateway cache was stale). This PR complements it by ensuring writes actually land before the Gateway ever has to invalidate.

• Session DB turn-end flush drops assistant after repair_message_sequence compacts list (orphan user → \n\n merge) #44837 — fix(agent): clamp flush cursor after repair_message_sequence compaction
  Related but not identical: manages _last_flushed_db_idx correctness during message sequence repair — the same cursor this PR tracks per-message.

• fix(agent): persist repaired-turn responses #46071 — fix(agent): persist repaired-turn responses
  Related: touches the same _persist_session / _flush_messages_to_session_db path to ensure repaired messages are persisted, not just held in memory.

• fix(state.db): recover from malformed sqlite_master so hidden sessions reappear #43149 — fix(state.db): recover from malformed sqlite_master
  Related: another state.db resilience fix — covers the sqlite_master corruption case while this PR covers the runtime write-failure case.

• fix(agent): clear _session_messages in AIAgent.close() #42123 — fix(agent): clear _session_messages in AIAgent.close()
  Related: session message lifecycle — prevents stale references from surviving agent teardown.

• feat(delegation): async background subagents via delegate_task(background=true) #40946 / fix(delegation): forward background flag so delegate_task(background=true) runs async #46968 — feat/fix(delegation): async background subagents
  Related: the background review agent path that this PR fixes (was missing session_db).

• fix: pass session_db to background review agents #47113 — extracted from this PR: standalone fix that passes session_db to background review agents.

• 8905ee6 — fix(agent): rewind flush cursor exactly when repair compacts before the cursor
  Follow-up to Session DB turn-end flush drops assistant after repair_message_sequence compacts list (orphan user → \n\n merge) #44837, already on main. A min() clamp only fixes cursor overshoot past the new end of the list. When repair_message_sequence drops/merges messages at indexes below the cursor, the clamp leaves the cursor pointing past unflushed rows and the turn-end flush silently skips them. This PR's per-message cursor tracking (last_successful_idx) avoids the entire class of cursor healing bugs.

• 8e4c447 — fix(gateway): prevent duplicate user messages in state.db
  Related: Gateway-side dedup for user messages. When the agent has its own _session_db, _flush_messages_to_session_db() persists user messages to SQLite. Two Gateway fallback paths also wrote the same user message without skip_db=True. Same problem domain — this PR adds dedup on recovery.

• 46b2caf56 — fix(state): use TRUNCATE WAL checkpoint to prevent unbounded WAL growth
  Related: WAL management. Unbounded WAL growth can trigger readonly states in concurrent writer scenarios — exactly the class of SQLITE_READONLY errors that this PR's _execute_write retry now handles.

• fix(compression): disable compression on background-review fork (#38727) #41708 — fix(compression): disable compression on background-review fork to prevent cross-turn stale-parent fork
  Related: background review fork isolation. The review fork's lifecycle edge case (stale parent state across turns) mirrors the missing session_db issue that fix: pass session_db to background review agents #47113 fixes.

[liuhao1024]

I found one issue that looks worth fixing before merge.

run_agent.py:_ensure_db_session — the recovery log always reports "0 failures":

if self._session_db_failed:
    self._session_db_failed = False
    self._session_db_create_fail_count = 0          # ← resets to 0
    logger.info("Session DB creation recovered after %d failures",
                self._session_db_create_fail_count)  # ← always logs 0

The counter is zeroed before the log statement reads it, so the message always says "recovered after 0 failures" regardless of how many consecutive failures actually occurred.

Why it matters: Operators debugging degraded session persistence rely on this log to understand recovery timing. A misleading "0 failures" message hides the real severity — a session that failed 50 times before recovering looks identical to one that never failed.

Suggested fix:

if self._session_db_failed:
    failed_count = self._session_db_create_fail_count
    self._session_db_failed = False
    self._session_db_create_fail_count = 0
    logger.info("Session DB creation recovered after %d failures", failed_count)

[mlinquan]

@alt-glitch Week-old ping — ready to merge?