hermes_state: opt-in in-place self-heal for a lost WAL generation (fixes #109687)

[Sahilvishnaliya]

Fixes #109687

The problem

A short-lived reader (one sqlite3.connect() + close, e.g. hermes sessions stats) unlinks the live gateway writer's state.db-wal/-shm on Linux. The gateway keeps serving while committing into the unlinked inode — session writes are silently discarded — and the shipped deleted-WAL guard then fails every other process closed. The halt is sticky for the life of the writer process: the only remediation is an operator killing all holders.

What this adds

The issue's second explicit ask — "a writer that does lose its generation should be able to recover without operator intervention" — as an opt-in self-heal, database.wal_self_heal: true (default OFF; the fail-closed posture is unchanged for unflagged installs):

1. Capture the retired frames durably via the existing capture_retired_wal_generation machinery (WAL generation split-brain: DeletedWalGenerationError guarded writes, but the graceful-shutdown checkpoint then corrupted state.db (field report) #105670) — nothing that was committed to the orphaned WAL is lost; the artifact stays for inspection.
2. Close this handle's orphaned descriptors with SQLite's close-time checkpoint disabled (SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE); if the checkpoint cannot be suppressed (Python < 3.12), the heal fails closed and the shipped pin path stays.
3. Reopen through refuse_deleted_wal_generation exactly like a fresh process — if any other process still holds an orphaned sidecar, the guard refuses and the canonical DeletedWalGenerationError stands. The heal never mints a second WAL while the box is not clear.
4. Adopt the current generation (identity re-recorded, sticky halt flags cleared) and resume writes.

Read-pool hygiene: pooled read connections minted before the heal hold fds into the orphaned generation; an epoch allowlist set at heal time evicts them at checkout (_fresh_read_conns), so no stale descriptor ever serves reads afterwards.

Design notes / honesty

• The producer is not touched. The unlink is SQLite's own last-connection close semantics; the reporter of [Bug]: A single plain CLI invocation orphans the live gateway's state.db WAL generation on Linux (reproduces on 2026-09-12 main, after the #102589 fix) — gateway keeps serving while silently dropping session writes #109687 explicitly did not pin a line-level producer in Hermes, and several prior fixes (cron/lifecycle_guard raw-opens state.db when a command mentions its path, dropping the gateway's POSIX locks -> WAL split-brain #102589, state.db WAL split-brain inside a SINGLE process: apply_wal_with_fallback runs the set-pragma when the journal-mode probe fails (unlinks sibling connections -wal/-shm) #104596) chased specific producers while the bare 3-line sqlite3.connect() repro still orphans the WAL. This PR makes the victim recoverable instead of racing the next producer.
• Fail-closed everywhere it can fail: capture failure → halt stays sticky; checkpoint-suppression unavailable → no heal attempted; foreign holder remains → canonical refusal. The heal only clears the sticky halt after a successful reopen.
• Developed on Windows; the repro is Linux-only. Cross-platform tests (6) cover the flag default, the unflagged sticky halt, the capture-failure fallback, the guard refusal, the full adopt-and-resume flow (capture stubbed, real guard + reopen + resumed writes), and pool eviction. The real unlink/adopt end-to-end test is skipif(not linux) — same gate as the existing guard tests in test_deleted_wal_generation_guard.py (23 of those skip on this host too). The Linux end-to-end path itself is not executed locally and needs CI's Linux runners to validate.
• Regression-checked: the pre-existing 9 failures in tests/hermes_state on this Windows host are identical on pristine main (inode-replacement/quarantine semantics that need Linux); the guard/capture suites pass; ruff clean; zero new ty diagnostics (diffed against base — only line-number shifts of the 9 pre-existing ones).

Tests

tests/hermes_state/test_wal_generation_self_heal.py — 7 tests:

test	platform	covers
test_flag_defaults_off	all	flag OFF unless configured
test_heal_off_keeps_fail_closed_halt	all	unflagged halt never attempts the heal
test_heal_attempted_then_sticky_when_it_cannot_heal	all	capture failure → sticky refusal stands
test_heal_refuses_when_guard_detects_foreign_holders	all	guard refusal, descriptor closed, sticky halt
test_heal_adopts_current_generation_and_resumes	all	full heal: reopen, adopt, writes resume, read-pool epoch
test_pool_evicts_pre_heal_read_conns	all	pre-heal pooled reader is evicted at checkout
test_heal_recovers_writer_without_operator	Linux (skipif)	real unlink → second generation → heal end-to-end