state.db no longer wedges on a healthy WAL: OpenZFS/close() false DeletedWalGenerationError, unconfirmed-WAL read pool, transient read IOERR, stale lock banner (#107401 #105567 #86515 #100871 #108031, salvage #107411 #105578 #105711 #106958)

[teknium1]

state.db no longer wedges itself on a healthy WAL: the deleted-WAL guard stops misreading OpenZFS dentries and its own close(), the read pool needs a confirmed WAL header and rides out a transient read IOERR, a both-pragmas-rejected filesystem no longer crashes DB init, and the gateway stops warning about a lock that already cleared.

What changed

• [Bug]: deleted-WAL guard misreads OpenZFS (deleted) dentries as unlinked sidecars — permanent DeletedWalGenerationError, JSONL fallback #107401 / salvage fix(state): compare fd identity against the watched path, not st_nlink #107411 (@chelsealong) — iter_deleted_sqlite_sidecar_holders() and the in-process twin SessionDB._wal_generation_was_lost() treated the kernel's " (deleted)" /proc/<pid>/fd suffix as proof of an unlinked sidecar. On OpenZFS a live, still-linked -wal/-shm is reported that way (dentry unhashed, nlink=1), so the guard fired permanently and the gateway fell back to JSONL forever. Both sites now compare the fd's (st_dev, st_ino) against the CURRENT watched path; a mismatch (or unstat-able watched path) still fails closed, so a real orphan kept alive by a hard link is still flagged (the PR's third test).
• [Bug]: clean SessionDB.close() racing append_message can trigger sticky DeletedWalGenerationError (synthetic repro) #105567 / salvage fix(state): serialize replaced/generation probe with close() in _execute_write (#105567) #105578 (@ca-shrimp) — _execute_write() ran the replaced/generation probe before taking self._lock, while close() unlinks the sidecars and clears _db_sidecar_identity under that lock. A writer racing a clean close saw "sidecars gone, identity still recorded" and set the sticky _db_wal_generation_lost flag on its own handle. The probe now runs inside the lock (11 LOC).
• state: WAL-reset fallback returns 'wal' when journal-mode probe is indeterminate, wrongly enabling lock-free read pool #86515 (based on fix(state): disable WAL read pool when journal mode is unknown #87044, @QDung210 co-authored) — apply_wal_with_fallback() returns "wal" as the assumed mode on both indeterminate-probe paths (vulnerable gate and the a7f2a59 probe-unknown path); SessionDB turned that into _wal_active=True and enabled the lock-free mode=ro read pool on a file that may really be DELETE. _open_writer_conn now confirms the header before enabling the pool; unknown → reads queue on the writer lock. Return contract unchanged (the other 14 callers ignore the value).
• [Bug] Repeated SQLITE_IOERR in SessionDB.get_session read path on WSL2 (ext4-on-vhdx), multi-process, starting at 0.21.0 — WAL stable but DELETE fallback collapses read throughput ~30000x #100871 (design from fix(sessions): keep WSL recovery alive through transient read IOERR #100882 @fangliquanflq; diagnosis fix(state): pooled read connections survive transient EIO (WSL2/vhdx, ZFS) #100883 @Sahilvishnaliya) — a warm pooled reader hitting the WAL-transition window propagated disk I/O error out of get_session(). _read_one/_read_all (every SELECT in the SessionDB siblings) now replay the idempotent statement on the SAME connection within the existing read-only IOERR budget (3×50 ms); persistent IOERR still propagates; never quarantined, never close+reopen (would cancel sibling POSIX locks).
• salvage fix(db): survive DELETE fallback failure in WAL setup #106958 (@nikkoxgonzales) — when PRAGMA journal_mode=WAL and the DELETE fallback both raise (APFS external SSD under contention), _enable_wal propagated and crashed every DB-init caller. Now logs once and returns the read-back mode.
• salvage fix(state): distinguish deleted_wal banner wording from replaced #105711 (@gaoanze888) — DeletedWalGenerationError classified as "replaced" (its parent), sending operators after the main file. New deleted_wal cause with guidance pointing at the retired-generation capture.
• [Bug]: gateway broadcasts a state.db lock warning after the lock already cleared #108031 — _send_session_db_warning_notifications() broadcast the startup error verbatim. It now drives one open through the RecoverableHandleCache first (which clears _session_db_init_error on recovery) and only warns if the store is still down.

Live repro (real SQLite, temp HERMES_HOME, origin/main vs this branch)

Issue	Harness	Before (origin/main)	After
#107401	/tmp/statedb_campaign/wal-repro-107401.py — live linked sidecars, readlink reports (deleted) (kernel d_path emulation, detector untouched); control = real unlink	open-guard holders=2, write path DeletedWalGenerationError; control holders=2	holders=0, write ok; control holders=2
#105567	wal-repro-105567-det.py — widen close() teardown window by 0.5 s, concurrent append_message	['DeletedWalGenerationError', 'sticky:DeletedWalGenerationError'], flag=True	[], flag=False
#105567 issue script (100×40)	as filed	0 failed rounds on this host (scheduling-dependent)	0 failed rounds
#86515	test_wal_active_confirmed.py — DELETE file, sibling BEGIN EXCLUSIVE during open	_wal_active is True (pool enabled on a DELETE file)	False, _checkout_read_conn() is None
#100871	test_read_path_transient_ioerr.py — pooled conn raises one disk I/O error on SELECT	get_session raises after 1 attempt	2 attempts, row returned; persistent IOERR raises after budget, no quarantine
#108031	test_session_db_warning_recheck.py — priming open locked, store heals before connect	banner broadcast	not broadcast; still broadcast when still down

Root cause

The 0.21.0 deleted-WAL guard, WAL-mode gate and read pool all trusted a single indirect signal (a /proc suffix, an assumed return value, one EIO, a startup error) as proof of a permanent condition.

Already on main (not re-fixed)

d0653ab + a7f2a59 closed the WAL-branch half of #86515 (no set-pragma on probe-unknown, require_wal honoured); the remaining half — _wal_active going True on the assumed mode — is fixed here. d616424 / d93f72c / d87bf0d / 64fe13a (lost-generation capture + retire) are the recovery side of #107401/#105567 and are untouched; this PR stops the false positives that fed them.

Assessed, not covered

• [Bug]: live gateway session row is transiently marked ended in state.db; #54878 stale-route self-heal fires repeatedly and silently reopens it #98495 / [Bug]: idle agent cache eviction ends session in DB but leaves sessions.json stale → messages silently dropped until restart #54878 (live row transiently marked ended): not a WAL/journal defect — no writer identified; fix(state): audit-log end_session/reopen_session with the calling module #98500's audit-log is diagnostics, left for the sessions lane.
• [Bug]: Desktop/TUI resume can snapshot state.db mid-write from another process, then never revalidates before running a turn — answers as if the conversation never happened #105964 (TUI resume mid-write snapshot): tui_gateway/server.py history revalidation; duplicate of [Bug]: Desktop — replying to a cron session opened mid-run feeds the agent a frozen open-time transcript snapshot #91508 with fix PR fix(tui): refresh resumed history before prompt submit #91523 open, out of this lane.
• doctor: flags a fully-checkpointed WAL and a PATH-resolved hermes launcher as issues; repairs are unnecessary/counter-productive #96976 / fix(doctor): classify WAL health by checkpoint state; respect a working hermes on PATH #97133 / fix(doctor): prevent false positives for WAL and launcher checks #97456 / fix(doctor): flag a configured journal_mode=delete that never took effect #104714 (doctor WAL classification): hermes_cli/doctor*.py, not the state.db engine; fix(doctor): classify WAL health by checkpoint state; respect a working hermes on PATH #97133 is the fuller of the two and should be reviewed on its own.
• fix(state): converge concurrent WAL initialization #101384 (converge concurrent WAL init), fix(state): avoid config-writer waits in journal-policy checks #103214 (config-lock contention): distinct startup races, not in the DeletedWalGeneration/IOERR class; not reproduced here.

Fixes #107401
Fixes #105567
Fixes #86515
Fixes #100871
Fixes #108031
Refs #105670 #104596 #98495 #105964 #96976
Supersedes #107411 #105578 #105711 #106958 #87044 #100882 #100883 (authorship preserved by cherry-pick / Co-authored-by)