perf(state): stop taking the state.db write lock to open a database that needs no writes

[jonpol01]

What does this PR do?

Stops a read-write SessionDB open from taking the state.db write lock when it has nothing to write. On a machine where several Hermes processes share one profile — a gateway, a hermes serve backend, the dashboard, plus every hermes CLI invocation — opening the database currently blocks behind whichever process happens to be writing.

The three writes

A write statement takes the database write lock even when it matches no rows. Three ran unconditionally on every read-write open:

where	statement	why it is almost always a no-op
hermes_state.py, _ensure_db_file_generation	INSERT OR IGNORE INTO state_meta …	the stamp is minted once per file, so every open after the first inserts nothing
hermes_state_schema.py, the NULL-active heal	UPDATE messages SET active = 1 WHERE active IS NULL	matches nothing on a healthy database
hermes_state_schema.py, the FTS layout stamp	INSERT … ON CONFLICT DO UPDATE	re-writes the identical value on every open of an already-optimized database

The connection is opened with timeout=1.0 ("Short timeout — application-level retry with random jitter handles contention"), so each blocked write costs a full busy timeout, and the open path's patience loop can ultimately give up and raise.

Measured

Real 99 MB state.db copy, 239 sessions / 6,470 messages, macOS arm64, Python 3.11 / SQLite 3.53. Every number below is from this branch versus a pristine origin/main export, same database, same box.

A sibling holds the write lock and does not release it (3 opens):

origin/main   2117.3 / 2127.2 / 2136.0 ms      <- two full busy timeouts
this branch      7.1 /    3.6 /    3.6 ms

Same test on an already-optimized database (the population that hits the FTS stamp):

origin/main   sqlite3.OperationalError: database is locked   <- the open FAILS
this branch      6.8 / 3.7 / 4.0 ms

A sibling running 200 ms write transactions in a loop, n=20 opens:

origin/main   min 208.0  p50 894.6  p90 1094.7  max 1098.7 ms
this branch   min   4.8  p50   9.6  p90   13.9  max   14.9 ms      p50 93x, p90 79x

A settled database now issues zero main-database write statements to open (traced via set_trace_callback; temp. statements excluded, they do not touch the main write lock).

The added reads cost nothing

state_meta primary-key seek                       min 2.0 us   p50 2.2 us
messages active-IS-NULL probe, modern schema      min 1.7 us   p50 1.8 us
  plan: unsatisfiable constraint against `active INTEGER NOT NULL DEFAULT 1`, short-circuited
messages probe, legacy default-less column, 300k rows, no NULL rows
                                                  min 0.4 us   p50 0.5 us
  plan: SEARCH messages USING COVERING INDEX idx_messages_active_null (active=?)

That index is created by the DEFERRED_INDEX_SQL executescript twelve lines above the heal, so it is always in place before the probe runs. Uncontended open is unchanged: min 3.65 → 3.94 ms, a delta ~100× larger than the ~4 µs actually added, i.e. box noise on a loaded machine.

Semantics preserved

• INSERT OR IGNORE still resolves the first-opener race inside SQLite, and racers still converge on the winner's token through the re-read.
• The application_id gate and the PASSIVE-only checkpoint (state.db recurring B-tree corruption from WAL TRUNCATE checkpoint on large FTS5 databases #45383) are untouched.
• The heal is still considered on every startup, which is the deliberate decision from fix(state): set active=1 explicitly in message INSERTs + heal NULL rows on startup (#51646) #60108 — the comment explains it was previously gated at current_version < 12 and so never re-ran for v12+ databases. Only the write is now conditional, on a read proving there is something to repair. test_startup_heals_null_active_rows passes unchanged.
• The probe is deliberately not INDEXED BY: I measured that INDEXED BY idx_messages_active_null raises OperationalError: no query solution against the modern NOT NULL column, and the existing except sqlite3.OperationalError: pass would swallow it — silently disabling the repair forever.

Correctness bonus

Read-first is not only faster, it fixes two silent failures under contention:

• The generation block was abandoned by its except sqlite3.Error handler, so a process ended up with no generation token at all even though the value was on disk and a plain read would have returned it. That token feeds the deleted-WAL / replaced-file guards added by merged fix(state): refuse SessionDB open and writes on a deleted WAL generation (salvage #101081) #101221.
• The heal's except sqlite3.OperationalError: pass skipped the repair silently, so the unconditional form did not deliver the unconditional repair it advertised precisely when contention made it slowest.

Related Issue

Related: #93799 ("SessionDB write-lock contention stalls concurrent Hermes processes") — that issue is about the writer side (autocheckpoint, FTS write amplification, busy_timeout); this PR removes three writers from the open path, so it is complementary rather than overlapping.

Merge-adjacent, disclosed for the reviewer, no semantic conflict:

• fix(db): set busy_timeout=5000 on writer and reader connections #101042 (open) adds busy_timeout PRAGMAs in hermes_state.py; different lines, and it confirms the writer stays around 1 s.
• fix(state): drop tool_calls from the trigram index and advertise the v1 storage rebuild (salvage #88217) #101729 (open) bumps FTS_STORAGE_VERSION to 2 and touches hermes_state_schema.py. My gate compares against the constant, not a literal, so under that bump a database stamped 1 correctly fails the probe and gets re-stamped. Textual conflict in that file is possible; the semantics compose.
• fix(state): don't stamp FTS complete while a projection is short #94894 (open) touches _fts_external_index_empty_with_messages, a sibling condition in the same if, but in hermes_state_search.py.

Type of Change

• ⚡ Performance improvement (non-breaking)

Changes Made

• hermes_state.py — _ensure_db_file_generation reads the stamp before inserting
• hermes_state_schema.py — the NULL-active heal is gated on an existence probe; the FTS layout stamp is gated on a value comparison
• tests/test_hermes_state.py — two tests: an open must not block on a sibling's write transaction, and the generation stamp must survive write-lock contention

How to Test

1. pytest tests/test_hermes_state.py tests/hermes_state -q — 548 passed on this branch. One unrelated failure, TestFTS5Search::test_search_projection_skips_context_enrichment_queries, reproduces identically on pristine origin/main (verified by running that single test in a clean git archive export), so it is pre-existing.
2. Both new tests fail on origin/main with sqlite3.OperationalError: database is locked after exhausting the open path's write patience, and pass here — that is the regression proof, not just a timing assertion.
3. To see it end to end: hold the write lock from a second connection (BEGIN IMMEDIATE plus any write) and time SessionDB(path) on a settled database.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched for existing PRs and issues — _ensure_db_file_generation, fts_storage_version, INSERT OR IGNORE state_meta, "state.db write lock open", "heal NULL active startup"; the only hits are the merge-adjacent PRs disclosed above
• My PR contains only changes related to this improvement
• Ran the state suites, ruff check, and scripts/check-windows-footguns.py --all; relying on CI for the full matrix
• I've added tests for my changes
• I've tested on my platform: macOS 26 arm64, Python 3.11, SQLite 3.53

Documentation & Housekeeping

• Documentation — the rationale lives in comments at each of the three sites, next to the existing load-bearing ones
• cli-config.yaml.example — N/A
• CONTRIBUTING.md / AGENTS.md — N/A
• Cross-platform — pure SQLite/Python; no platform APIs; footgun linter clean
• Tool descriptions/schemas — N/A

[Enough1122]

AI code review — automated review for reference; please use your judgment.

Overall: Removes ~2s of spurious write-lock contention from every SessionDB open: the generation-stamp INSERT OR IGNORE, the NULL-active repair UPDATE, and the FTS-layout marker write are now all gated on a read proving work is needed (hermes_state.py:26, hermes_state_schema.py:68,91). The key insight — a no-op write still takes the SQLite write lock — is documented at each site.

Correctness:

• Generation stamp: first opener still wins via INSERT OR IGNORE, racers converge through the re-read (hermes_state.py:35); under contention the stamp is now readable instead of lost (second new test).
• NULL-active repair: probe uses idx_messages_active_null on legacy columns and short-circuits on modern NOT NULL DEFAULT 1 columns; the INDEXED BY pitfall is explicitly avoided with rationale.
• FTS marker: read-before-write on the exact key/value pair, so a settled DB does zero writes on open.

Non-blocking nits:

• The < 0.5s timing assertion in test_open_does_not_take_the_write_lock_when_nothing_needs_writing could flake on a heavily loaded CI box (it measures wall time including interpreter startup of the open path); the 0.5s bound vs 1s timeout gives reasonable headroom, but a retry or looser bound may be needed if flakes appear.
• SELECT→INSERT still has a benign race window (two first-openers both see empty, one INSERT wins, loser re-reads the winner) — correctly handled, just noting the re-read is load-bearing, not defensive.

Verdict: LGTM. Excellent comments explaining why reads must precede writes.

[teknium1]

Landed on main in #108067 (ff59fcd). Your commit was cherry-picked with authorship intact underneath #106751's salvage; thank you @jonpol01.