feat(backends): pgvector substrate via #665 cherry-pick + smoke tests

.github/workflows/ci.yml

@@ -53,3 +53,41 @@ jobs:
       - run: pip install "ruff>=0.4.0,<0.5"
       - run: ruff check .
       - run: ruff format --check .
+
+  test-postgres:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: pgvector/pgvector:pg16
+        env:
+          POSTGRES_PASSWORD: postgres
+          POSTGRES_DB: mempalace_test
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 10
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.11"
+          cache: 'pip'
+      - run: pip install -e ".[dev,postgres]"
+      - name: Create pgvector extension
+        env:
+          PGURL: postgresql://postgres:postgres@localhost:5432/mempalace_test
+        run: |
+          python - <<'PY'
+          import os, psycopg2
+          conn = psycopg2.connect(os.environ["PGURL"])
+          conn.autocommit = True
+          conn.cursor().execute("CREATE EXTENSION IF NOT EXISTS vector")
+          print("vector extension ready")
+          PY
+      - name: Run postgres backend tests
+        env:
+          TEST_POSTGRES_DSN: postgresql://postgres:postgres@localhost:5432/mempalace_test
+        run: python -m pytest tests/test_backends_postgres.py -v

docs/internal/pgvector-665-decision.md

@@ -0,0 +1,98 @@
+# Decision: #665 composition stance — WAIT (with Plan-B trigger)
+
+**Date:** 2026-05-11 PDT
+**Plan task:** Phase 0, Task 0.2 of `docs/superpowers/plans/2026-05-10-pgvector-age-migration-impl.md`
+**Author:** Claude (Opus 4.7, 1M context) for JP
+
+---
+
+## Decision
+
+**WAIT for upstream PR #665 to merge, then cherry-pick / sync.** Phase 1 starts at **Task 1.A.1** ("Wait" path), not Task 1.B.1 ("Fork-port" path).
+
+**Plan-B trigger (revisit decision when ANY fires):**
+- #665 sees no maintainer activity past **2026-06-08** (4 weeks from today), OR
+- skuznetsov stops responding to the open review thread (currently engaged), OR
+- A blocking review concern remains unresolved 14+ days after we open implementation work on the "wait" path, OR
+- We hit a blocker in Phase 1 that requires patching #665's internals to land our AGE-layer work.
+
+If any trigger fires, switch to fork-port path. The 22 days of staleness *so far* are within tolerance because skuznetsov rewrote the PR on 2026-04-19 in response to #995 landing — that's an active author, not an abandoned PR.
+
+## State of #665 as of 2026-05-11
+
+| Field | Value |
+|---|---|
+| State | OPEN |
+| Author | skuznetsov (Sergey Kuznetsov) |
+| Last update | 2026-04-19 (22 days ago) |
+| Last commit | 2026-04-19 — full rewrite on top of #995/RFC 001 backend contract |
+| Size | +1839 / -11, 12 files |
+| Base | `develop` |
+| Mergeable (gh API) | UNKNOWN (matches our local 4-conflict observation) |
+| Labels | area/mcp, area/search, storage, area/install |
+| Open concerns | dekoza raised `pg_sorted_heap` bus-factor concern 2026-04-13; skuznetsov acknowledged 2026-04-13, rewrote PR on 2026-04-19 to make pg_sorted_heap optional with pgvector fallback. Thread is not closed. |
+
+## Conflict probe on our `main` (commit `6827b97`)
+
+```
+git merge --no-commit --no-ff pr-665 → 4 conflicts:
+  README.md                 (trivial — fork README vs upstream README; always-fork-wins per memory)
+  mempalace/palace.py       32 LOC of conflict markers — substantive but small
+  tests/test_backends.py    19 LOC of conflict markers — substantive but small
+  uv.lock                   (trivial — regenerate)
+```
+
+Total substantive conflict surface: ~51 LOC across 2 code files. **Moderate**, not heavy. Resolvable in 15–30 min when we cherry-pick.
+
+## Architecture summary of #665 (from PR description)
+
+- Implements `BaseBackend` + `BaseCollection` from the merged #995/RFC 001 contract.
+- Backend registry discovery + env/config selection. ChromaDB stays default.
+- PostgreSQL collection supports **two paths gated by extension availability**:
+  - **Preferred:** `pg_sorted_heap` (niche, bus-factor concern flagged)
+  - **Fallback:** `pgvector` (what we'll actually run on `apache/age:release_PG16_1.6.0` since `pg_sorted_heap` isn't in apt or the AGE image)
+- INSERT … SELECT FROM unnest() + ON CONFLICT for batch writes.
+- First-wins `add()`, last-wins `upsert()`.
+- `vector` index created lazily after a threshold.
+- Reuses Chroma's default local embedding function — postgres extra only adds `psycopg2-binary`, no new ML stack.
+- Tests: 1044 passed on author's machine. Files: `tests/test_backends.py` updated; new tests for filter translation, batch upsert, typed result shapes.
+
+## Why "Wait"
+
+1. **Conflict surface is moderate.** ~51 LOC across 2 code files, plus 2 trivial files. Not heavy enough to justify reimplementing 1839 LOC.
+2. **#665 is comprehensive.** It already covers BaseBackend conformance, registry discovery, env routing, batch insert, lazy indexes, typed results. Rewriting that surface area duplicates work we'd want anyway.
+3. **`pg_sorted_heap` is opt-in, not invasive.** Gated by `if extension_installed → use it; else → use pgvector`. Our `apache/age:release_PG16_1.6.0` image won't have pg_sorted_heap, so we'd silently run the pgvector fallback — the exact path we wanted.
+4. **Upstream flow is preserved.** Building on top of #665 keeps our AGE-layer work proposable as clean follow-on PRs that compose on the BaseBackend contract, rather than a parallel impl that diverges from upstream's evolution.
+5. **Author is engaged, not abandoning.** skuznetsov rewrote the PR on 2026-04-19 specifically to compose on the merged backend contract. That's a thoughtful, responsive maintainer.
+6. **Staleness isn't yet alarming.** 22 days from the rewrite is within tolerance for a substantial 1839-LOC PR awaiting maintainer review. We bake in a trigger to revisit if it stretches.
+
+## Why NOT "Fork-port"
+
+- We'd be reimplementing roughly 1200–1500 LOC (pg_sorted_heap path stripped, plus tests).
+- Risk of duplicating subtle bugs #665 has already fixed (the UNKNOWN mergeable state in part reflects that #665 is moving target, but that's a feature: skuznetsov is refining it).
+- When #665 eventually merges, we'd have to reconcile two parallel implementations — net cost is higher than the savings from skipping the cherry-pick.
+- The dekoza-vs-skuznetsov pg_sorted_heap thread is open and pertinent — being a real-world consumer of the pgvector fallback path is *valuable signal* to attach to that thread later (per `feedback_upstream_comment_timing`: defer the comment until we have working code).
+
+## Phase 1 implications
+
+Phase 1 starts at **Task 1.A.1** ("Wait" path):
+
+1. Cherry-pick #665's commit range onto `feat/pgvector-age-impl`.
+2. Resolve the 4 conflicts (README — fork-wins; palace.py — manual; test_backends.py — manual; uv.lock — regenerate).
+3. Run the full existing test suite (`pytest -x -q` against ChromaDB default) — must stay green; any regression in non-postgres code paths is fixed before moving on.
+4. Write the smoke test `tests/test_backends_postgres.py::test_postgres_backend_smoke` (already specified in plan Task 1.A.1 Step 3).
+5. Spin a `mempalace-db` service in `/opt/mediaserver/docker-compose.yml` on `disks` with the `apache/age:release_PG16_1.6.0`-based image. Bind on `disks` LAN IP (chosen during Phase 1 setup; defaults to ssh tunnel if LAN-bind unsafe).
+6. Run smoke test against the running container.
+
+The "Task 1.A.1 → 1.A.5 → onward" sequence in the plan stands as written. Phase 1's remaining tasks (1.2 index conformance, 1.3 where-clause translation, 1.4 CI matrix) are common to both decision branches.
+
+## Tracking obligations
+
+Adding to `scratch/promises.md`:
+1. Watch #665 for merge or activity-stall (trigger date 2026-06-08).
+2. After Phase 1 lands and smoke test is green, draft a comment for #665's dekoza-vs-skuznetsov thread offering us as a real-world data point for the pgvector-fallback path. **Do not file before code lands**, per `feedback_upstream_comment_timing`.
+3. If we make non-trivial improvements to the cherry-picked backend (better error handling, indexes, etc.), file as a follow-on PR composing on top of #665 — don't graft into our cherry-pick.
+
+## Final note
+
+This decision document moves to `docs/internal/pgvector-665-decision.md` per Task 0.2 Step 4 and is committed on `feat/pgvector-age-impl` (not on main per `feedback_own_projects_pr_workflow`).

docs/postgres_backend.md

@@ -0,0 +1,210 @@
+# PostgreSQL Backend
+
+MemPalace uses ChromaDB by default. The PostgreSQL backend is optional and is
+intended for larger, long-lived, or team/server deployments where a local
+Chroma directory is not the right storage boundary.
+
+The backend supports two database extension paths:
+
+- `pg_sorted_heap` (optional optimized self-managed path): uses `sorted_heap`,
+  `svec`, and `sorted_hnsw`.
+- `pgvector` (broadly available managed-service path): uses a regular heap
+  table, `vector`, and `hnsw`.
+
+When both extensions are available in the target database, MemPalace selects
+`pg_sorted_heap` to exercise the optimized storage/index path. Operators who
+prefer a simpler supply-chain profile or run on managed PostgreSQL services
+should install only `vector` and use the `pgvector` path. Both paths require a
+PostgreSQL extension to be available to the server and created in the target
+database. MemPalace does not vendor or install database extensions at Python
+package install time.
+
+`pg_sorted_heap` is listed in the
+[PostgreSQL Software Catalogue](https://www.postgresql.org/download/products/6/)
+and on [PGXN](https://pgxn.org/dist/pg_sorted_heap/), but it is not bundled with
+managed PostgreSQL providers; use this path only when you control extension
+installation for the target PostgreSQL server.
+
+## Install MemPalace Dependencies
+
+```bash
+pip install "mempalace[postgres]"
+```
+
+The PostgreSQL extra installs the Python driver. Text queries and writes that do
+not pass embeddings directly reuse the same Chroma default local embedding
+function that MemPalace already depends on for the default backend.
+
+## Optional: Install `pg_sorted_heap`
+
+Requirements:
+
+- PostgreSQL 17 or 18.
+- `pg_config` for the PostgreSQL version you want to use.
+- Standard PGXS build tools (`make`, compiler toolchain, PostgreSQL server
+  development files).
+- Database privileges to run `CREATE EXTENSION`.
+
+Automated helper (from a source checkout of the MemPalace repository):
+
+```bash
+scripts/install_pg_backend.sh --dsn "postgresql://mempalace_user@localhost:5432/mempalace"
+```
+
+The helper clones `https://github.com/skuznetsov/pg_sorted_heap.git`, runs
+`make`, runs `make install`, verifies the installed control/library files, and
+then creates the extension in the database if `--dsn` is supplied.
+
+Use an explicit PostgreSQL installation when multiple versions are installed:
+
+```bash
+scripts/install_pg_backend.sh \
+  --pg-config /usr/lib/postgresql/18/bin/pg_config \
+  --dsn "postgresql://mempalace_user@localhost:5432/mempalace"
+```
+
+Build from an existing checkout instead of cloning:
+
+```bash
+scripts/install_pg_backend.sh \
+  --source /path/to/pg_sorted_heap \
+  --dsn "postgresql://mempalace_user@localhost:5432/mempalace"
+```
+
+Manual installation:
+
+```bash
+git clone https://github.com/skuznetsov/pg_sorted_heap.git
+cd pg_sorted_heap
+make
+make install
+psql "postgresql://mempalace_user@localhost:5432/mempalace" \
+  -c "CREATE EXTENSION IF NOT EXISTS pg_sorted_heap;"
+```
+
+If `make install` needs elevated permissions for your PostgreSQL installation,
+run the helper with `--sudo`, or run the manual `make install` step with the
+appropriate privilege escalation for your environment.
+
+## Fallback: Install `pgvector`
+
+If `pg_sorted_heap` is not installed but the `vector` extension is available in
+the target database, MemPalace will fall back to `pgvector` automatically.
+`pgvector` is still a PostgreSQL extension, so it must be installed or exposed by
+your PostgreSQL distribution/provider and created in the database.
+
+Check whether the server exposes the extension:
+
+```sql
+SELECT name, default_version, installed_version
+FROM pg_available_extensions
+WHERE name = 'vector';
+```
+
+Create the extension in the target database:
+
+```bash
+psql "postgresql://mempalace_user@localhost:5432/mempalace" \
+  -c "CREATE EXTENSION IF NOT EXISTS vector;"
+```
+
+The helper can also install `pgvector` from source for self-managed PostgreSQL:
+
+```bash
+scripts/install_pg_backend.sh \
+  --extension vector \
+  --dsn "postgresql://mempalace_user@localhost:5432/mempalace"
+```
+
+Manual source installation:
+
+```bash
+git clone https://github.com/pgvector/pgvector.git
+cd pgvector
+make
+make install
+psql "postgresql://mempalace_user@localhost:5432/mempalace" \
+  -c "CREATE EXTENSION IF NOT EXISTS vector;"
+```
+
+For managed PostgreSQL services, including AWS RDS/Aurora PostgreSQL versions
+that expose `vector` as a supported extension, do not run local `make install`
+against the managed server. Use the provider-supported extension mechanism and
+verify availability with `pg_available_extensions`.
+
+Use `pg_sorted_heap` when you control the PostgreSQL installation and want the
+optional sorted storage plus planner-integrated `sorted_hnsw` path. Use
+`pgvector` when you need a managed-database setup, a more common extension, or a
+simpler supply-chain profile.
+
+## Configure MemPalace
+
+Environment variables:
+
+```bash
+export MEMPALACE_BACKEND=postgres
+export MEMPALACE_POSTGRES_DSN="postgresql://mempalace_user@localhost:5432/mempalace"
+
+# optional, defaults to mempalace_drawers
+export MEMPALACE_COLLECTION_NAME=mempalace_drawers
+```
+
+Equivalent `~/.mempalace/config.json`:
+
+```json
+{
+  "backend": "postgres",
+  "postgres_dsn": "postgresql://mempalace_user@localhost:5432/mempalace",
+  "collection_name": "mempalace_drawers"
+}
+```
+
+Then use MemPalace normally:
+
+```bash
+mempalace mine ~/projects/myapp
+mempalace search "why did we change the auth flow"
+```
+
+## Verify The Backend
+
+Check the selected PostgreSQL extension:
+
+```sql
+SELECT extname
+FROM pg_extension
+WHERE extname IN ('pg_sorted_heap', 'vector')
+ORDER BY extname;
+```
+
+For a `pg_sorted_heap` collection, the MemPalace table should use
+`sorted_heap`:
+
+```sql
+SELECT am.amname
+FROM pg_class c
+JOIN pg_am am ON am.oid = c.relam
+WHERE c.relname = 'mempalace_drawers';
+```
+
+Expected:
+
+```text
+sorted_heap
+```
+
+For fallback `pgvector`, the table access method is the regular heap access
+method and the `vector` extension should be present.
+
+## Operational Notes
+
+- The PostgreSQL backend creates the collection table on first write when
+  `create=True`.
+- For `pg_sorted_heap`, MemPalace stores drawers with primary key
+  `(wing, room, id)` so wing/room locality is preserved in the sorted table.
+- Vector indexes are created lazily after the collection reaches the backend's
+  index threshold; small collections use exact vector ordering.
+- `count()` is exact and may scan large tables; use `estimated_count()` for
+  progress/status paths where PostgreSQL catalog statistics are acceptable.
+- ChromaDB remains the zero-config default and is still the benchmarked raw-mode
+  path in the public README.

mempalace/backends/__init__.py

@@ -12,6 +12,7 @@
 * Registry: :func:`get_backend`, :func:`register`, :func:`available_backends`,
   :func:`resolve_backend_for_palace`.
 * In-tree Chroma default: :class:`ChromaBackend`, :class:`ChromaCollection`.
+* Optional PostgreSQL backend: :class:`PostgresBackend`, :class:`PostgresCollection`.
 """
 
 from .base import (
@@ -29,6 +30,7 @@
     UnsupportedFilterError,
 )
 from .chroma import ChromaBackend, ChromaCollection
+from .postgres import PostgresBackend, PostgresCollection
 from .registry import (
     available_backends,
     get_backend,
@@ -52,6 +54,8 @@
     "HealthStatus",
     "PalaceNotFoundError",
     "PalaceRef",
+    "PostgresBackend",
+    "PostgresCollection",
     "QueryResult",
     "UnsupportedFilterError",
     "available_backends",