Skip to content

fix: SQLite WAL write-lock contention causing 15-20s TUI freeze - #3385

Merged
teknium1 merged 1 commit into
mainfrom
hermes/hermes-f8f80cce
Mar 27, 2026
Merged

fix: SQLite WAL write-lock contention causing 15-20s TUI freeze#3385
teknium1 merged 1 commit into
mainfrom
hermes/hermes-f8f80cce

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Problem

Multiple hermes processes (gateway + CLI sessions + worktree agents) sharing one state.db caused WAL write-lock convoy effects. SQLite's built-in busy handler uses deterministic sleep intervals (up to 100ms between retries) that synchronize competing writers, creating 15-20 second freezes during agent init.

Root cause chain:

  1. timeout=30.0 — SQLite retries the write lock for up to 30 seconds with deterministic backoff
  2. 7+ concurrent connections prevent WAL checkpointing → WAL grew to 294MB
  3. Bloated WAL slows all reads/writes (hash index traversal on every operation)
  4. Deterministic backoff causes convoy effects — all processes wake and collide at the same time

Fix

  • Replace 30s SQLite timeout with 1s + application-level retry (15 attempts, random 20-150ms jitter between retries to break convoys)
  • Use BEGIN IMMEDIATE for explicit write-lock acquisition (fail-fast on contention)
  • Set isolation_level=None for manual transaction control (prevents Python's implicit transaction management from conflicting with BEGIN IMMEDIATE)
  • PASSIVE WAL checkpoint on close() and every 50 writes — keeps the WAL from growing unbounded
  • All 12 write methods converted to _execute_write() helper

Results

Metric Before After
Init freeze 15-20s <1s
WAL size 294MB (growing) ~4MB (stable)
Write contention Visible on every py-spy dump Zero across 3 concurrent sessions

Testing

  • 4,355 unit tests pass (hermes_state, gateway, tools, CLI)
  • Live tested: 3 concurrent CLI sessions with simultaneous messages — zero _execute_write contention on every py-spy sample
  • Gateway restarted with new code, verified clean operation (zero SQLite errors)
  • Sessions properly persisted: message counts, token tracking, clean exit with ended_at + end_reason
  • WAL truncated from 294MB → 0, stayed at 4MB through all concurrent testing

Multiple hermes processes (gateway + CLI sessions + worktree agents) sharing
one state.db caused WAL write-lock convoy effects. SQLite's built-in busy
handler uses deterministic sleep intervals (up to 100ms) that synchronize
competing writers, creating 15-20 second freezes during agent init.

Root cause: timeout=30.0 with 7+ concurrent connections meant:
- WAL never checkpointed (294MB, readers always blocked it)
- Bloated WAL slowed all reads and writes
- Deterministic backoff caused convoy effects under contention

Fix:
- Replace 30s SQLite timeout with 1s + app-level retry (15 attempts,
  random 20-150ms jitter between retries to break convoys)
- Use BEGIN IMMEDIATE for explicit write-lock acquisition (fail fast)
- Set isolation_level=None for manual transaction control
- PASSIVE WAL checkpoint on close() and every 50 writes
- All 12 write methods converted to _execute_write() helper

Before: 15-20s frozen at create_session during agent init
After:  <1s to API call, WAL stays at ~4MB

Tested: 4355 tests pass, 3 concurrent live sessions with simultaneous
writes showed zero contention on every py-spy sample.
@teknium1
teknium1 merged commit b7bcae4 into main Mar 27, 2026
2 checks passed
angelburgosrosado pushed a commit to angelburgosrosado/hermes-agent that referenced this pull request Apr 27, 2026
…Research#3385)

Multiple hermes processes (gateway + CLI sessions + worktree agents) sharing
one state.db caused WAL write-lock convoy effects. SQLite's built-in busy
handler uses deterministic sleep intervals (up to 100ms) that synchronize
competing writers, creating 15-20 second freezes during agent init.

Root cause: timeout=30.0 with 7+ concurrent connections meant:
- WAL never checkpointed (294MB, readers always blocked it)
- Bloated WAL slowed all reads and writes
- Deterministic backoff caused convoy effects under contention

Fix:
- Replace 30s SQLite timeout with 1s + app-level retry (15 attempts,
  random 20-150ms jitter between retries to break convoys)
- Use BEGIN IMMEDIATE for explicit write-lock acquisition (fail fast)
- Set isolation_level=None for manual transaction control
- PASSIVE WAL checkpoint on close() and every 50 writes
- All 12 write methods converted to _execute_write() helper

Before: 15-20s frozen at create_session during agent init
After:  <1s to API call, WAL stays at ~4MB

Tested: 4355 tests pass, 3 concurrent live sessions with simultaneous
writes showed zero contention on every py-spy sample.
02356abc pushed a commit to 02356abc/hermes-agent that referenced this pull request May 14, 2026
…Research#3385)

Multiple hermes processes (gateway + CLI sessions + worktree agents) sharing
one state.db caused WAL write-lock convoy effects. SQLite's built-in busy
handler uses deterministic sleep intervals (up to 100ms) that synchronize
competing writers, creating 15-20 second freezes during agent init.

Root cause: timeout=30.0 with 7+ concurrent connections meant:
- WAL never checkpointed (294MB, readers always blocked it)
- Bloated WAL slowed all reads and writes
- Deterministic backoff caused convoy effects under contention

Fix:
- Replace 30s SQLite timeout with 1s + app-level retry (15 attempts,
  random 20-150ms jitter between retries to break convoys)
- Use BEGIN IMMEDIATE for explicit write-lock acquisition (fail fast)
- Set isolation_level=None for manual transaction control
- PASSIVE WAL checkpoint on close() and every 50 writes
- All 12 write methods converted to _execute_write() helper

Before: 15-20s frozen at create_session during agent init
After:  <1s to API call, WAL stays at ~4MB

Tested: 4355 tests pass, 3 concurrent live sessions with simultaneous
writes showed zero contention on every py-spy sample.
gweeteve pushed a commit to gweeteve/hermes-agent that referenced this pull request Jun 2, 2026
…Research#3385)

Multiple hermes processes (gateway + CLI sessions + worktree agents) sharing
one state.db caused WAL write-lock convoy effects. SQLite's built-in busy
handler uses deterministic sleep intervals (up to 100ms) that synchronize
competing writers, creating 15-20 second freezes during agent init.

Root cause: timeout=30.0 with 7+ concurrent connections meant:
- WAL never checkpointed (294MB, readers always blocked it)
- Bloated WAL slowed all reads and writes
- Deterministic backoff caused convoy effects under contention

Fix:
- Replace 30s SQLite timeout with 1s + app-level retry (15 attempts,
  random 20-150ms jitter between retries to break convoys)
- Use BEGIN IMMEDIATE for explicit write-lock acquisition (fail fast)
- Set isolation_level=None for manual transaction control
- PASSIVE WAL checkpoint on close() and every 50 writes
- All 12 write methods converted to _execute_write() helper

Before: 15-20s frozen at create_session during agent init
After:  <1s to API call, WAL stays at ~4MB

Tested: 4355 tests pass, 3 concurrent live sessions with simultaneous
writes showed zero contention on every py-spy sample.
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
…Research#3385)

Multiple hermes processes (gateway + CLI sessions + worktree agents) sharing
one state.db caused WAL write-lock convoy effects. SQLite's built-in busy
handler uses deterministic sleep intervals (up to 100ms) that synchronize
competing writers, creating 15-20 second freezes during agent init.

Root cause: timeout=30.0 with 7+ concurrent connections meant:
- WAL never checkpointed (294MB, readers always blocked it)
- Bloated WAL slowed all reads and writes
- Deterministic backoff caused convoy effects under contention

Fix:
- Replace 30s SQLite timeout with 1s + app-level retry (15 attempts,
  random 20-150ms jitter between retries to break convoys)
- Use BEGIN IMMEDIATE for explicit write-lock acquisition (fail fast)
- Set isolation_level=None for manual transaction control
- PASSIVE WAL checkpoint on close() and every 50 writes
- All 12 write methods converted to _execute_write() helper

Before: 15-20s frozen at create_session during agent init
After:  <1s to API call, WAL stays at ~4MB

Tested: 4355 tests pass, 3 concurrent live sessions with simultaneous
writes showed zero contention on every py-spy sample.
OmarB97 pushed a commit to OmarB97/hermes-agent that referenced this pull request Jul 11, 2026
…erable

The macOS in-app updater's detached swap script ran with plain 'set -u'
and a success-gated ditto: when ditto or the destination move failed,
the script fell through silently -- the app had already quit, the old
bundle stayed (or was left moved aside), and 'open "$DST"' either
relaunched the stale build or nothing at all. This matches the reported
"Update now does not actually update" behavior.

Make the swap fail-fast and recoverable:
- set -euo pipefail so unexpected failures stop the script instead of
  compounding.
- Detect ditto and destination-replace failures explicitly; on either,
  fall back to opening the freshly rebuilt bundle directly so the user
  is never left with a dead quit.
- Clean up the .hermes-update-old copy via an EXIT trap so it is
  removed on every exit path.

Recovered from the pre-force-push head of desktop-mac-swap-fix
(NousResearch#38410, which lost this work to a force-push
and was closed). The sqlite busy_timeout half of that PR is
intentionally not revived: the BEGIN IMMEDIATE + jitter-retry redesign
(NousResearch#3385) deliberately keeps the connection busy handler short, and a 5s
busy_timeout would override that design.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant