fix: stop holding pooled DB connections across embedder/LLM/reranker calls

[zommiommy]

Problem

Several memory-engine paths held a pooled PostgreSQL connection checked out for the entire duration of a slow external call — an embedder, reranker, or LLM round-trip.

The pools are already bounded (asyncpg, default max_size=20) and per-process — the API and the background worker each build their own MemoryEngine/pool — so this is not a leak or unbounded growth. The failure mode is saturation:

• The background consolidation worker is the main source. _process_one_llm_batch held one connection for a whole batch — across the consolidation recall, the consolidation LLM call, per-action embeddings, and the dedup recall/embed/LLM. Under load, enough concurrent batches park the worker's entire pool on these multi-second calls; the poller and remaining batches then block until acquire_timeout. The worker starves itself.
• Two API-side edit paths embedded while holding a connection: update_memory_unit re-embedded mid-transaction, and update_mental_model embedded inside the acquired connection.
• Because the parked connections sit idle mid-call, this also inflates pressure on the server's global max_connections (the sum of every process's pool) for no useful work.

(The hot recall/rerank path was already clean: the query is embedded before the connection is acquired, and reranking runs after retrieval returns.)

Why a bounded / bulkhead pool wasn't enough

The obvious smaller fix — bound or bulkhead the pool — doesn't address the real failure:

• The pools are already bounded; capping them tighter just makes saturation arrive sooner, and a slow call still parks its connection for its full duration.
• A dedicated consolidation sub-pool (bulkhead) was considered. It would stop batch work from starving the worker's other duties, but it only self-throttles consolidation — slow LLMs still park the sub-pool, so throughput is capped at sub-pool size — and it leaves the API-side update_memory_unit / update_mental_model hold-across-embed paths untouched.

To keep consolidation throughput decoupled from connection count, and to shrink the connection footprint everywhere, the connection has to be released across the slow call. That's what this PR does.

Approach

Invariant enforced across the changed paths: no pooled DB connection is held across an embedder / reranker / LLM await.

• Connections are acquired short-lived, only around SQL. Embeddings, the LLM call, and dedup adjudication run with no connection held.
• Consolidation: action executors and dedup helpers take the backend (pool) and self-acquire a short connection around their SQL. Each source-liveness check is paired with its write in one tiny transaction (this strengthens PG — previously autocommit-per-statement — and normalizes Oracle). Dedup folds are RETURNING-gated and re-filter live sources inside the fold transaction (FOR SHARE, sources-before-observation lock order, matching the normal write paths), so a twin or source deleted during the now connection-free window can't drop a CREATE or fold a dead source id.
• update_memory_unit: split into a read/resolve + embed phase (off-connection, into typed _MemoryEditPlan / _MemoryRevertPlan) and a short write transaction that re-locks the row (FOR UPDATE), filters surviving entities, and applies the precomputed embedding — aborting (rollback) if a concurrent edit changed an embedding-input column between the phases.
• update_mental_model: embeds before acquiring (its text depends only on its arguments).

This is a large change — and why

Moving the embed/LLM out of the held connection widens the read→write window, so this isn't a one-line "release the connection." The held transaction used to provide serialization for free; replacing it requires new guards to preserve the same correctness:

• atomic check-then-write transactions,
• RETURNING-gated, live-source-filtered dedup folds with a consistent (sources-first) lock order,
• a two-phase update_memory_unit with re-lock, abort/retry on inter-phase edits, and orphan-entity reclaim.

That's why the diff is substantial (~1.5k lines), concentrated in the two core engine modules and their test suites.

One deliberate exception

A few rare recovery paths re-embed inside the Phase-2 transaction: when a concurrent graph-maintenance prune (or a same-unit entity-only edit) lands between the phases, update_memory_unit / revert re-embed under the lock so the stored vector stays consistent with unit_entities. These are bounded to a single re-embed and pinned by tests. There is no retry wrapper around update_memory_unit (a RuntimeError surfaces as HTTP 500), so converting them to abort/retry would regress interactive edits — the bounded in-txn re-embed is the intentional fallback. It can be converted to strict abort/retry later if a "never embed in-transaction" guarantee is preferred.

Verification

• ruff + ty clean (changed files).
• Deterministic no-DB unit suites pass — they exercise the dedup decision and fold guards (RETURNING-gating, live-source filtering, created/skipped propagation) and the pre-embed guards directly.
• Real PostgreSQL (pgvector pg18): the consolidation, memory-curation, observation-invalidation, mental-model (mock-LLM), and document-transfer suites pass against a live DB. The only non-passes are real-LLM tests that require provider credentials, unrelated to this change.

[nicoloboschi]

superseded by #3082 and #3083