fix: SQLite WAL write-lock contention causing 15-20s TUI freeze - #3385
Merged
Conversation
Multiple hermes processes (gateway + CLI sessions + worktree agents) sharing one state.db caused WAL write-lock convoy effects. SQLite's built-in busy handler uses deterministic sleep intervals (up to 100ms) that synchronize competing writers, creating 15-20 second freezes during agent init. Root cause: timeout=30.0 with 7+ concurrent connections meant: - WAL never checkpointed (294MB, readers always blocked it) - Bloated WAL slowed all reads and writes - Deterministic backoff caused convoy effects under contention Fix: - Replace 30s SQLite timeout with 1s + app-level retry (15 attempts, random 20-150ms jitter between retries to break convoys) - Use BEGIN IMMEDIATE for explicit write-lock acquisition (fail fast) - Set isolation_level=None for manual transaction control - PASSIVE WAL checkpoint on close() and every 50 writes - All 12 write methods converted to _execute_write() helper Before: 15-20s frozen at create_session during agent init After: <1s to API call, WAL stays at ~4MB Tested: 4355 tests pass, 3 concurrent live sessions with simultaneous writes showed zero contention on every py-spy sample.
angelburgosrosado
pushed a commit
to angelburgosrosado/hermes-agent
that referenced
this pull request
Apr 27, 2026
…Research#3385) Multiple hermes processes (gateway + CLI sessions + worktree agents) sharing one state.db caused WAL write-lock convoy effects. SQLite's built-in busy handler uses deterministic sleep intervals (up to 100ms) that synchronize competing writers, creating 15-20 second freezes during agent init. Root cause: timeout=30.0 with 7+ concurrent connections meant: - WAL never checkpointed (294MB, readers always blocked it) - Bloated WAL slowed all reads and writes - Deterministic backoff caused convoy effects under contention Fix: - Replace 30s SQLite timeout with 1s + app-level retry (15 attempts, random 20-150ms jitter between retries to break convoys) - Use BEGIN IMMEDIATE for explicit write-lock acquisition (fail fast) - Set isolation_level=None for manual transaction control - PASSIVE WAL checkpoint on close() and every 50 writes - All 12 write methods converted to _execute_write() helper Before: 15-20s frozen at create_session during agent init After: <1s to API call, WAL stays at ~4MB Tested: 4355 tests pass, 3 concurrent live sessions with simultaneous writes showed zero contention on every py-spy sample.
02356abc
pushed a commit
to 02356abc/hermes-agent
that referenced
this pull request
May 14, 2026
…Research#3385) Multiple hermes processes (gateway + CLI sessions + worktree agents) sharing one state.db caused WAL write-lock convoy effects. SQLite's built-in busy handler uses deterministic sleep intervals (up to 100ms) that synchronize competing writers, creating 15-20 second freezes during agent init. Root cause: timeout=30.0 with 7+ concurrent connections meant: - WAL never checkpointed (294MB, readers always blocked it) - Bloated WAL slowed all reads and writes - Deterministic backoff caused convoy effects under contention Fix: - Replace 30s SQLite timeout with 1s + app-level retry (15 attempts, random 20-150ms jitter between retries to break convoys) - Use BEGIN IMMEDIATE for explicit write-lock acquisition (fail fast) - Set isolation_level=None for manual transaction control - PASSIVE WAL checkpoint on close() and every 50 writes - All 12 write methods converted to _execute_write() helper Before: 15-20s frozen at create_session during agent init After: <1s to API call, WAL stays at ~4MB Tested: 4355 tests pass, 3 concurrent live sessions with simultaneous writes showed zero contention on every py-spy sample.
gweeteve
pushed a commit
to gweeteve/hermes-agent
that referenced
this pull request
Jun 2, 2026
…Research#3385) Multiple hermes processes (gateway + CLI sessions + worktree agents) sharing one state.db caused WAL write-lock convoy effects. SQLite's built-in busy handler uses deterministic sleep intervals (up to 100ms) that synchronize competing writers, creating 15-20 second freezes during agent init. Root cause: timeout=30.0 with 7+ concurrent connections meant: - WAL never checkpointed (294MB, readers always blocked it) - Bloated WAL slowed all reads and writes - Deterministic backoff caused convoy effects under contention Fix: - Replace 30s SQLite timeout with 1s + app-level retry (15 attempts, random 20-150ms jitter between retries to break convoys) - Use BEGIN IMMEDIATE for explicit write-lock acquisition (fail fast) - Set isolation_level=None for manual transaction control - PASSIVE WAL checkpoint on close() and every 50 writes - All 12 write methods converted to _execute_write() helper Before: 15-20s frozen at create_session during agent init After: <1s to API call, WAL stays at ~4MB Tested: 4355 tests pass, 3 concurrent live sessions with simultaneous writes showed zero contention on every py-spy sample.
waefrebeorn
pushed a commit
to waefrebeorn/slermes
that referenced
this pull request
Jul 2, 2026
…Research#3385) Multiple hermes processes (gateway + CLI sessions + worktree agents) sharing one state.db caused WAL write-lock convoy effects. SQLite's built-in busy handler uses deterministic sleep intervals (up to 100ms) that synchronize competing writers, creating 15-20 second freezes during agent init. Root cause: timeout=30.0 with 7+ concurrent connections meant: - WAL never checkpointed (294MB, readers always blocked it) - Bloated WAL slowed all reads and writes - Deterministic backoff caused convoy effects under contention Fix: - Replace 30s SQLite timeout with 1s + app-level retry (15 attempts, random 20-150ms jitter between retries to break convoys) - Use BEGIN IMMEDIATE for explicit write-lock acquisition (fail fast) - Set isolation_level=None for manual transaction control - PASSIVE WAL checkpoint on close() and every 50 writes - All 12 write methods converted to _execute_write() helper Before: 15-20s frozen at create_session during agent init After: <1s to API call, WAL stays at ~4MB Tested: 4355 tests pass, 3 concurrent live sessions with simultaneous writes showed zero contention on every py-spy sample.
OmarB97
pushed a commit
to OmarB97/hermes-agent
that referenced
this pull request
Jul 11, 2026
…erable The macOS in-app updater's detached swap script ran with plain 'set -u' and a success-gated ditto: when ditto or the destination move failed, the script fell through silently -- the app had already quit, the old bundle stayed (or was left moved aside), and 'open "$DST"' either relaunched the stale build or nothing at all. This matches the reported "Update now does not actually update" behavior. Make the swap fail-fast and recoverable: - set -euo pipefail so unexpected failures stop the script instead of compounding. - Detect ditto and destination-replace failures explicitly; on either, fall back to opening the freshly rebuilt bundle directly so the user is never left with a dead quit. - Clean up the .hermes-update-old copy via an EXIT trap so it is removed on every exit path. Recovered from the pre-force-push head of desktop-mac-swap-fix (NousResearch#38410, which lost this work to a force-push and was closed). The sqlite busy_timeout half of that PR is intentionally not revived: the BEGIN IMMEDIATE + jitter-retry redesign (NousResearch#3385) deliberately keeps the connection busy handler short, and a 5s busy_timeout would override that design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Multiple hermes processes (gateway + CLI sessions + worktree agents) sharing one
state.dbcaused WAL write-lock convoy effects. SQLite's built-in busy handler uses deterministic sleep intervals (up to 100ms between retries) that synchronize competing writers, creating 15-20 second freezes during agent init.Root cause chain:
timeout=30.0— SQLite retries the write lock for up to 30 seconds with deterministic backoffFix
BEGIN IMMEDIATEfor explicit write-lock acquisition (fail-fast on contention)isolation_level=Nonefor manual transaction control (prevents Python's implicit transaction management from conflicting with BEGIN IMMEDIATE)close()and every 50 writes — keeps the WAL from growing unbounded_execute_write()helperResults
Testing
_execute_writecontention on every py-spy sampleended_at+end_reason