fix(pgvector): reconnect once when the server drops the pooled connection

[messelink]

Problem

_PgVectorClient holds one psycopg connection for the life of the process and reuses it. When Postgres restarts — a package upgrade, a failover, an operator's pg_ctl restart — that handle looks alive until the next statement runs on it, and that statement is the one that surfaces the drop:

BackendError: pgvector query failed: terminating connection due to administrator command

The call after it reconnects and succeeds, so every restart costs each connected client exactly one spurious error. On a shared palace with several long-lived MCP servers, one systemctl restart postgresql produces one of these per client.

Observed in a multi-machine setup where unattended-upgrades restarts Postgres for library updates: each restart surfaces the error once per peer, then self-heals.

Verified live: pg_terminate_backend() against a client's own backend PID reproduces the error on develop and is transparently absorbed with this patch, with the client landing on a new backend PID.

Fix

Retry that first statement once on a fresh connection — what a short-lived client would have done implicitly.

The retry is gated on the failure being about the connection rather than the statement:

• SQLSTATE class 57 (operator intervention): 57P01 admin shutdown, 57P02 crash shutdown, 57P03 cannot connect now.
• A driver error on a handle psycopg already marks closed or broken — how a mid-query TCP drop arrives, with no SQLSTATE because no server response came back to carry one. Consulted only when no SQLSTATE arrived, so a live connection that merely rejected the statement is never mistaken for a lost one.

A statement-level failure (bad SQL, constraint violation, type error) leaves the connection usable and propagates unchanged; retrying would run the same failing statement twice and report the second one. A second drop on the fresh connection is a real outage rather than a stale pooled handle, and surfaces as the BackendError callers already expect.

Safety

Retrying is safe for the write path: upserts are by id and callers already run an idempotency check, so a statement that never reached the server is replayed, not duplicated. many=True (upsert_rows) is covered.

Tests

Five new cases in tests/test_pgvector_backend.py:

• 57P01 → reconnect → success, stale connection discarded
• closed handle with no SQLSTATE → still retries
• 23505 constraint violation → propagates on first attempt, connection kept
• two drops → BackendError, exactly one retry, no retry storm
• executemany path covered

Related

• fix(mcp): self-heal writer lease instead of latching read-only for life #1960 established the same self-heal-once-rather-than-latch-broken shape for the MCP writer lease.

[igorls]

Thanks for this contribution, and apologies for the slow turnaround.

develop has moved a fair way since this was opened and the branch no longer merges cleanly. If you're still interested in landing it, could you rebase onto current develop? Once it merges cleanly and CI is green I'll get it reviewed for the 3.8.0 cycle.

If you'd rather not pick it back up, no problem at all — just say so and I'll close it out, and thanks either way for taking the time to send it.

[messelink]

Picked it back up. Rebased onto current develop and force-pushed, now at fa2ed2c.

The only conflict was in tests/test_pgvector_backend.py. This branch and 811d6c9 (the facet_counts work) both append their tests at the same spot, so it was a positional collision with no real overlap. Kept both sets. mempalace/backends/pgvector.py merged cleanly, since facet_counts and the _execute retry touch different methods.

CI is green across all legs.

Let me know if anything needs changing in review.