fix(state): SessionDB open waits out a DELETE-mode lock reported as 'vtable constructor failed' instead of failing

[teknium1]

A SessionDB open, writable or read-only, now waits out another process's lock in DELETE journal mode even when SQLite words the lock as vtable constructor failed: messages_fts. Before this change the open failed after 1 s, persistence was disabled for the run, and the dashboard answered 500.

Live repro: before: SessionDB() raised OperationalError: vtable constructor failed: messages_fts (SQLITE_BUSY) after 1.06 s. The read-only open raised after 1.02 s. The error counted as not transient and its cause as unknown. After: both opens wait for the 2.5 s lock, then succeed (2.66 s / 2.55 s) with fts_enabled=True and the seeded search hit. If a 7 s lock outlasts the bounded read-only budget, the open raises after 4.24 s and the error counts as transient (503) with cause locked.

Root cause

When FTS5's table constructor loses the lock while reading messages_fts_config, SQLite keeps result code SQLITE_BUSY but replaces the message. Every state.db lock classifier looked for the words "locked" or "busy" in the message.

Changes

• hermes_state_errors.py: new is_sqlite_lock_error. It checks for SQLITE_BUSY / SQLITE_LOCKED by result code when SQLite supplies one. Only when there is no code (our own re-raised messages, RPC-wrapped strings) does it fall back to the text. is_transient_sqlite_error (dashboard 503 vs 500) and classify_persistence_error (locked bucket) use the result code too.
• hermes_state.py:
  • The writer open's lock-patience loop and the _execute_write retry now use the helper.
  • _open_read_only now retries a lock inside its existing bounded retry budget, alongside the transient IOERR case. Each retry waits the busy timeout again. Before, a lock on a read-only open was never retried.
• The rest of the class moved to the same helper:
  • the _reconcile_columns re-raise (hermes_state_schema.py)
  • the WAL→DELETE flip (hermes_state_wal.py)
  • the maintenance lock probe (hermes_state_holders.py)
• Docs: developer-guide/session-storage.md now says lock contention is classified by result code and describes open patience for both open kinds.

Open under a DELETE-mode lock (2.5 s hold)	before	after
writable SessionDB()	raises after 1.06 s	opens after 2.66 s
SessionDB(read_only=True)	raises after 1.02 s	opens after 2.55 s
error once the budget is exhausted	not transient (500), cause unknown	transient (503), cause locked

Tests

New tests in tests/hermes_state/test_write_lock_patience.py. They are red on origin/main (3 failed) and green with the fix:

• test_open_waits_out_lock_lost_inside_fts_constructor[writer|read_only]: real DELETE-mode SessionDB. A second connection takes BEGIN EXCLUSIVE for 2.5 s at the moment the open reaches the messages_fts probe. The open must succeed with FTS enabled and the transcript readable.
• test_lock_lost_inside_fts_constructor_classifies_as_busy: a real vtable constructor failed error from SQLite must be transient and classify as locked.

No existing test was changed. Ran tests/hermes_state/, tests/conformance/persistence/, tests/hermes_cli/test_web_server.py and the other test files that use the touched predicates: 1251 passed. Two timing tests failed under host load (~155) and pass when re-run alone. Also clean: ruff, check_no_tmp_literals, check-windows-footguns --all, git diff --check.

Related

• fix(state): dashboard and hermes sessions list no longer fail with 'database is locked' in DELETE journal mode #120386 (open) raises the busy timeout for DELETE-mode reads. This PR is complementary: it covers the vtable constructor failed wording and the app-level retry. Swept for competing PRs (fix(state): converge concurrent WAL initialization #101384, fix(state): retry contended session list queries #102621, fix(state): enforce write patience during lock acquisition #79140); none classify lock errors by result code on the open path.

Infographic

[github-actions]

૮ >ﻌ< ა ci review

_{ran on dc01ffe — fix(state): SessionDB open waits out a lock lost inside the}

debug info

CI timings

CI timings · View report · View job

Wall time 5m46s vs 5m57s (-3.1%). 5 job(s) slower, 7 faster, 1 unchanged.

• Check contributors / check-attribution: +216.0s
• Python lints / Windows footguns (blocking): +29.0s
• Check no case-colliding filenames / check-case-collisions: -19.0s
• OS-specific tests / Windows-only tests: +16.0s
• Check no committed infographics / check-no-committed-infographics: +10.0s