fix(state): refuse a fresh read-pool open on a quarantined SessionDB handle - #101303
Open
nftpoetrist wants to merge 2 commits into
Open
nftpoetrist wants to merge 2 commits into
nftpoetrist wants to merge 2 commits into
Conversation
…handle
_get_read_conn() (the WAL read-pool's miss path) opens a brand-new
sqlite3 connection to self.db_path with no awareness of self._db_corrupt
at all — it is a completely separate connection-opening path from
_reopen_after_close_locked, which already refuses exactly this class of
operation on the write side ("reopening would hand a fresh connection...
to a file we already know is structurally damaged").
Opening a fresh read-only connection to a file already known to be
structurally corrupt is the same "reopen a damaged image" the write path
refuses — it just does it through a code path the quarantine work never
wired up. Confirmed empirically: forcing _wal_active=True on a quarantined
handle and calling _get_read_conn() directly opened a new connection
(mocked _connect_tracked_db was called) with zero guard.
Fixed by returning None (the existing "read path unavailable, fall back
to the locked path" contract _get_read_conn() already uses for every
other unavailability reason) when self._db_corrupt is set. The locked
fallback in _read_ctx() then either reuses the already-open self._conn (no
new open — a no-op change to the quarantine contract, which only ever
promised "no write, no reopen, no checkpoint," not "no read of an
already-open handle") or, if self._conn is already None, hits
_reopen_after_close_locked's existing quarantine check.
Scope note: deliberately NOT touching two other "reuse an already-open
connection" paths — _checkout_read_conn()'s pool-hit branch (a connection
opened before corruption was ever observed, sitting idle in the pool) and
_read_ctx()'s locked self._conn yield when self._conn is not None. Both
reuse a handle that's already open rather than opening a new one, so they
fall under the same "no reopen, not no read" carve-out the quarantine
docstring documents — a narrower, more defensible fix than reasoning
about every reuse path at once.
Added test_get_read_conn_refuses_fresh_open_when_quarantined to
tests/hermes_state/test_state_db_corrupt_quarantine.py, reusing this
file's own _quarantined_db() fixture: forces _wal_active=True (this
sandbox's SQLite build falls back to journal_mode=DELETE, so WAL is never
really active here — forcing it exercises the guarded path directly, the
same technique this file already uses elsewhere), mocks
hermes_state._connect_tracked_db, and asserts _get_read_conn() returns
None without ever calling it. Mutation-verified: reverting hermes_state.py
makes the test fail — _get_read_conn() actually opens a connection
(returns the mock) instead of refusing.
Ran tests/state/ + tests/hermes_state/ + the read-conn-pool/read-path-split/
conn-lock-audit suites (653 passed, 29 skipped — pre-existing WAL/py3.12+
gated skips in this sandbox) — no regressions.
…lted states _get_read_conn()'s fresh-open guard (this branch's first commit) only checked self._db_corrupt. Since that commit's base, hermes_state.py grew two more halted states that _reopen_after_close_locked already refuses to reopen for on the write side: self._db_replaced (the path now resolves to a different file generation, NousResearch#89332) and self._db_wal_generation_lost (this handle's WAL/SHM generation was deleted out from under it, salvage of NousResearch#101081/NousResearch#101221). _get_read_conn() was unaware of both — it would still happily open a brand-new sqlite3 connection to the file in either state, the same "reopen a damaged/replaced image" hazard the first commit fixed for the corrupt case. Fixed by returning None (falling back to the locked path, same as the _db_corrupt check) when either flag is set. Deliberately checks only the already-set flags, not the proactive _db_file_was_replaced()/ _wal_generation_was_lost() detection probes _reopen_after_close_locked also runs — that keeps this hot-path check as cheap as the existing _db_corrupt check (a bare flag test), consistent with this function's established style. An out-of-band replace/WAL-loss not yet observed by this handle is still caught the next time a write or an explicit check touches it. Added two tests mirroring the existing test_get_read_conn_refuses_fresh_open_when_quarantined: test_get_read_conn_refuses_fresh_open_when_replaced and test_get_read_conn_refuses_fresh_open_when_wal_generation_lost, both forcing _wal_active=True on a live (not yet closed) handle and asserting _get_read_conn() returns None without ever calling the mocked _connect_tracked_db. Mutation-verified: reverting the hermes_state.py hunk alone makes both new tests fail with the mock connection object returned instead of None; restoring it makes them pass. Ran tests/hermes_state/ (11 passed, 1 pre-existing skip) and tests/state/ + tests/hermes_state/ together (417 passed, 7 skipped, all pre-existing) — no regressions. Fresh competitor check: no open or closed PR covers this specific gap. NousResearch#101042 ("set busy_timeout=5000 on writer and reader connections") also touches _get_read_conn() but only inside the try block after the connection is already opened (adds a busy_timeout PRAGMA) — no line overlap with this commit's checks near the top of the function.
nftpoetrist
force-pushed
the
fix/state-quarantine-read-guard
branch
from
September 3, 2026 10:31
ac1661c to
9876bab
Compare
Contributor
PR 101303 — fix(state): refuse a fresh read-pool open on a quarantined DB. Extends the existing quarantine refusal in Non-blocking observations:
|
14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_get_read_conn()(the WAL read-pool's miss path,hermes_state.py) opens a brand-newsqlite3connection toself.db_pathwith no awareness ofself._db_corruptat all — it is a completely separate connection-opening path from_reopen_after_close_locked(), which already refuses exactly this class of operation on the write side:Opening a fresh read-only connection to a file already known to be structurally corrupt is the same "reopen a damaged image" the write path refuses —
_get_read_conn()just does it through a code path the quarantine work (bcc2e65818, #101095/#101224) never wired up.Empirically confirmed: forced
_wal_active = Trueon a quarantined handle (this sandbox's SQLite build falls back tojournal_mode=DELETE, so WAL is never really active here — forcing it exercises the path directly) and called_get_read_conn()— it opened a new connection (the mocked_connect_tracked_dbwas called) with zero guard.Fix
Return
None— the existing "read path unavailable, fall back to the locked path" contract_get_read_conn()already uses for every other unavailability reason (WAL inactive, read-only handle, pool closed, backoff window, permit exhausted) — whenself._db_corruptis set. The locked fallback in_read_ctx()then either reuses the already-openself._conn(no new open — a no-op change to the existing quarantine contract) or, ifself._connis alreadyNone, hits_reopen_after_close_locked's existing quarantine check (already correctly raisesStateDbCorruptError, proven by the existingtest_reopen_after_close_refused_when_quarantined).Scope note (deliberately narrow, matching the docstring's actual promise): the
StateDbCorruptError/quarantine contract explicitly documents "no further writes, no automatic reopen, no explicit WAL checkpoint at close" — not "no read." I left two other "reuse an already-open connection" paths untouched:_checkout_read_conn()'s pool-hit branch (a connection opened before corruption was ever observed, sitting idle in the pool) — reuse, not reopen._read_ctx()'s lockedself._connyield whenself._conn is not None— same: reuse, not reopen.Both are weaker cases (arguably already covered by "no read" not being part of the stated contract), and fixing only the unambiguous "fresh open == reopen" case keeps this PR narrowly scoped and easier to reason about than touching every reuse path at once.
Tests
Added
test_get_read_conn_refuses_fresh_open_when_quarantinedtotests/hermes_state/test_state_db_corrupt_quarantine.py, reusing this file's own_quarantined_db()fixture and its established "force_wal_activeto exercise the WAL-only path directly" technique. Mockshermes_state._connect_tracked_dband asserts_get_read_conn()returnsNonewithout ever calling it.Mutation-verified: reverting
hermes_state.pymakes the test fail —_get_read_conn()actually opens a connection (returns the mock) instead of refusing.Ran
tests/state/+tests/hermes_state/+ the read-conn-pool/read-path-split/conn-lock-audit suites (653 passed, 29 skipped — pre-existing WAL/py3.12+-gated skips in this sandbox) — no regressions.Competitor check
Searched
gh pr list --state allacross several keyword combinations. Two open PRs touch nearby code but address different concerns, noted transparently:#100882("keep WSL recovery alive through transient read IOERR") adds a_RetryingReadConnectionwrapper at the very end of_get_read_conn()'s try block (after the CJK-extension load, right before return) — textually close but semantically distinct (retrying transient WSL I/O errors on an otherwise-healthy connection vs. refusing to open at all on a known-corrupt handle). My change is a small, isolated insertion near the top of the function; if#100882merges first, this PR should still apply cleanly since neither touches the other's lines.#85255("evict a poisoned pooled read connection instead of requeuing it") touches_read_ctx()but its base predates the entire quarantine feature (confirmed viagit merge-base --is-ancestor— its diff shows no_db_corrupt/reopen-refusal logic existing at all yet) and targets a different corruption class (NOTADB/replaced-file, evicting a connection that starts failing mid-use — not the "never open a fresh connection to an already-quarantined handle" gap this PR fixes).No open or closed PR covers this specific gap.
Update (second commit): Since this PR's base commit,
hermes_state.pygained two more halted states that_reopen_after_close_locked()already refuses to reopen for on the write side:self._db_replaced(the path now resolves to a different file generation, #89332) andself._db_wal_generation_lost(this handle's WAL/SHM generation was deleted out from under it, added by the merged salvage of #101081/#101221 — after this PR's original base)._get_read_conn()'s fresh-open guard only knew aboutself._db_corruptand was unaware of both — same hazard, same fix shape: returnNone(fall back to the locked path) when either flag is set, checking only the already-set flags (not the proactive_db_file_was_replaced()/_wal_generation_was_lost()detection probes) to keep this hot-path check as cheap as the existing_db_corruptcheck.Added
test_get_read_conn_refuses_fresh_open_when_replacedandtest_get_read_conn_refuses_fresh_open_when_wal_generation_lost, mirroring the existingtest_get_read_conn_refuses_fresh_open_when_quarantined. Mutation-verified (patch-based, notgit stash, per this session's worktree-sharing caution): reverting the newhermes_state.pyhunk alone makes both new tests fail (mock connection returned instead ofNone); restoring it makes them pass. Rantests/hermes_state/(11 passed, 1 pre-existing skip) andtests/state/+tests/hermes_state/together (417 passed, 7 skipped, all pre-existing) — no regressions.Fresh competitor check: no open or closed PR covers this specific gap.
#101042("set busy_timeout=5000 on writer and reader connections") also touches_get_read_conn(), but only inside the try block after the connection is already opened (adds abusy_timeoutPRAGMA) — no line overlap with this commit's checks near the top of the function.