fix(memory): share one SQLite connection per holographic store database - #61726
Merged
Conversation
alt-glitch
added
type/bug
teknium1
force-pushed
the
salvage/43819-holographic-shared-conn
branch
from
July 10, 2026 00:56
ea5d3e9 to
6ab00f3
Compare
teknium1
added a commit
that referenced
this pull request
Jul 10, 2026
…ort (#61734) test_accepted_at_every_position spawned 11 separate 'python -m hermes_cli.main' subprocesses, each cold-importing the full CLI module tree under a 15s TimeoutExpired deadline. On a loaded CI worker the import alone can exceed that (slice 2/8 flaked exactly here on PR #61726's run, TimeoutExpired at subprocess.py:1253), failing PRs that never touched the CLI. Replace with ONE driver subprocess that imports hermes_cli.main once and parses all 11 argvs in-process (catching SystemExit per argv), reporting JSON results. Same assertions per argv, identical semantics (verified the --help-before-unknown-flag exit behavior matches the old method), ~11x less import work, and the 180s timeout only trips on a genuine hang.
Every MemoryStore instance opened its own SQLite connection guarded by its own RLock. Several providers coexist in one process (the main agent plus every delegate_task subagent), so instances pointing at the same memory_store.db raced as independent WAL writers. Combined with writes that were not rolled back on error, one connection could leave an open write transaction that pinned the write lock and made every other connection's writes fail with "database is locked" for the full busy timeout. Instances for the same database now share ONE process-wide connection and ONE re-entrant lock, so access is fully serialized and cross-connection contention is impossible. The shared connection is refcounted: closing one instance never tears it out from under a live sibling, and the last close releases it. The connection runs in autocommit (isolation_level=None) so a write that raises mid-method can never leave a dangling transaction holding the write lock; the existing explicit commit() calls become harmless no-ops. The provider's shutdown() now calls the refcount-guarded close() instead of just dropping the reference: leaving finalization to GC kept the connection (and its write lock) alive indefinitely on long-running gateways, prolonging the exact contention this fix removes. The last provider now releases the connection deterministically while siblings stay live; regression tests fail without the wiring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-ups for salvaged PR #43819: the registry key was str(Path(db_path).expanduser()) — a symlinked or relative path to the same DB file got its own connection, silently reintroducing the exact multi-writer contention the registry prevents. Key on Path.resolve() (OSError-tolerant fallback). Adds a symlink regression test and the AUTHOR_MAP entry for adambiggs.
teknium1
force-pushed
the
salvage/43819-holographic-shared-conn
branch
from
July 10, 2026 01:08
6ab00f3 to
2f909a6
Compare
This was referenced Jul 10, 2026
santhreal
pushed a commit
to santhreal/hermes-agent
that referenced
this pull request
Jul 13, 2026
…ort (NousResearch#61734) test_accepted_at_every_position spawned 11 separate 'python -m hermes_cli.main' subprocesses, each cold-importing the full CLI module tree under a 15s TimeoutExpired deadline. On a loaded CI worker the import alone can exceed that (slice 2/8 flaked exactly here on PR NousResearch#61726's run, TimeoutExpired at subprocess.py:1253), failing PRs that never touched the CLI. Replace with ONE driver subprocess that imports hermes_cli.main once and parses all 11 argvs in-process (catching SystemExit per argv), reporting JSON results. Same assertions per argv, identical semantics (verified the --help-before-unknown-flag exit behavior matches the old method), ~11x less import work, and the 180s timeout only trips on a genuine hang.
justemu
pushed a commit
to justemu/hermes-agent
that referenced
this pull request
Jul 18, 2026
…ort (NousResearch#61734) test_accepted_at_every_position spawned 11 separate 'python -m hermes_cli.main' subprocesses, each cold-importing the full CLI module tree under a 15s TimeoutExpired deadline. On a loaded CI worker the import alone can exceed that (slice 2/8 flaked exactly here on PR NousResearch#61726's run, TimeoutExpired at subprocess.py:1253), failing PRs that never touched the CLI. Replace with ONE driver subprocess that imports hermes_cli.main once and parses all 11 argvs in-process (catching SystemExit per argv), reporting JSON results. Same assertions per argv, identical semantics (verified the --help-before-unknown-flag exit behavior matches the old method), ~11x less import work, and the 180s timeout only trips on a genuine hang.
Gravezzz
pushed a commit
to Gravezzz/hermes-agent
that referenced
this pull request
Jul 21, 2026
…ort (NousResearch#61734) test_accepted_at_every_position spawned 11 separate 'python -m hermes_cli.main' subprocesses, each cold-importing the full CLI module tree under a 15s TimeoutExpired deadline. On a loaded CI worker the import alone can exceed that (slice 2/8 flaked exactly here on PR NousResearch#61726's run, TimeoutExpired at subprocess.py:1253), failing PRs that never touched the CLI. Replace with ONE driver subprocess that imports hermes_cli.main once and parses all 11 argvs in-process (catching SystemExit per argv), reporting JSON results. Same assertions per argv, identical semantics (verified the --help-before-unknown-flag exit behavior matches the old method), ~11x less import work, and the 180s timeout only trips on a genuine hang.
leewenjie
pushed a commit
to leewenjie/hermes-agent
that referenced
this pull request
Aug 7, 2026
…ort (NousResearch#61734) test_accepted_at_every_position spawned 11 separate 'python -m hermes_cli.main' subprocesses, each cold-importing the full CLI module tree under a 15s TimeoutExpired deadline. On a loaded CI worker the import alone can exceed that (slice 2/8 flaked exactly here on PR NousResearch#61726's run, TimeoutExpired at subprocess.py:1253), failing PRs that never touched the CLI. Replace with ONE driver subprocess that imports hermes_cli.main once and parses all 11 argvs in-process (catching SystemExit per argv), reporting JSON results. Same assertions per argv, identical semantics (verified the --help-before-unknown-flag exit behavior matches the old method), ~11x less import work, and the 180s timeout only trips on a genuine hang.
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…ort (NousResearch#61734) test_accepted_at_every_position spawned 11 separate 'python -m hermes_cli.main' subprocesses, each cold-importing the full CLI module tree under a 15s TimeoutExpired deadline. On a loaded CI worker the import alone can exceed that (slice 2/8 flaked exactly here on PR NousResearch#61726's run, TimeoutExpired at subprocess.py:1253), failing PRs that never touched the CLI. Replace with ONE driver subprocess that imports hermes_cli.main once and parses all 11 argvs in-process (catching SystemExit per argv), reporting JSON results. Same assertions per argv, identical semantics (verified the --help-before-unknown-flag exit behavior matches the old method), ~11x less import work, and the 180s timeout only trips on a genuine hang.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The holographic memory store now shares one SQLite connection (+ one RLock) per database file across all
MemoryStoreinstances in a process, ending thedatabase is lockedcontention and retry burns reported on 24/7 gateways (#55503).Root cause: each instance opened its own connection; a mid-method exception could leave an open write transaction pinning the WAL write lock, failing every sibling's writes for the 10s busy timeout — cascading into model-visible errors and token-burning retry loops.
Salvages PR #43819 by @adambiggs (cherry-picked, authorship preserved).
Changes
plugins/memory/holographic/store.py: process-wide refcounted registry (_shared+_shared_guard) — one connection + RLock per DB path, autocommit (isolation_level=None) so a raise can never dangle a transaction, schema init once, last-close teardown (contributor commit)Path.resolve()so symlinked/relative paths to the same file share one connection instead of silently reintroducing the contention; symlink regression test; AUTHOR_MAP entrytests/plugins/memory/test_holographic_store.py: 14 tests — sharing, refcounting, 8-thread concurrent writers, failed-write lock release, provider shutdown, symlink identityValidation
Plugin-only diff — no core files touched. Note: this removes in-process contention; cross-process cron contention is mitigated (write lock held per-statement under autocommit) but not eliminated.
Infographic