feat: add HttpChromaBackend + Postgres KG backend for stateless deployments by shockstricken · Pull Request #1337 · MemPalace/mempalace

shockstricken · 2026-05-03T22:35:23Z

Context

This PR adds support for deploying MemPalace as a fully stateless workload: external ChromaDB (HTTP) for drawer/closet storage and Postgres for the knowledge graph, with no local PVC. Currently both stores must live on the same filesystem as the mempalace process, which prevents single-replica StatefulSet pods from being moved without dragging gigabytes of palace state along.

The implementation respects the existing RFC 001 backend abstraction (mempalace.backends.{base,registry,chroma}) and is additive — defaults are unchanged, all 1521 existing tests pass on this branch with no regressions.

What it adds

KG abstraction layer (`mempalace.kg`)

BaseKnowledgeGraph ABC mirrors the historical KnowledgeGraph 8-method surface (add_entity, add_triple, invalidate, query_entity, query_relationship, timeline, stats, seed_from_entity_facts, close).
SqliteKnowledgeGraph — original implementation lifted verbatim behind the ABC; behavior byte-for-byte identical.
PostgresKnowledgeGraph — SQLAlchemy 2.x Core, dialect-native ON CONFLICT upserts, JSONB on Postgres / JSON elsewhere via a TypeDecorator.
mempalace/knowledge_graph.py becomes a backwards-compat shim re-exporting the factory; existing callers don't need changes.

Selection priority: url= kwarg → MEMPALACE_DATABASE_URL env → db_path= kwarg → MEMPALACE_KG_PATH env → ~/.mempalace/knowledge_graph.sqlite3 (default).

HTTP ChromaDB backend (`mempalace.backends.chroma_http`)

HttpChromaBackend(BaseBackend) wraps chromadb.HttpClient. One cached client per process; multiple palaces share the same chromadb server via namespace-prefixed collection names derived from PalaceRef.namespace (or id fallback).
New env vars: MEMPALACE_CHROMA_URL / _HOST / _PORT / _SSL / _AUTH_TOKEN / _AUTH_HEADER / _TENANT / _DATABASE.
detect() returns True iff any HTTP-mode env var is set, so resolve_backend_for_palace picks it automatically.
Registered in backends/registry.py and [project.entry-points].

Backend-aware searcher / repair / CLI

A new mempalace._runtime module centralizes the "are we local or HTTP?" question (using_local_chroma(), using_http_chroma()).

searcher._bm25_only_via_api mirrors the sqlite BM25 fallback through where_document={"$contains": tok}. Same result shape so _merge_bm25_union_candidates and search_memories work uniformly. The new _bm25_only dispatcher routes by backend.
mcp_server._refresh_vector_disabled_flag is a no-op in HTTP mode (segment health is server-side; the client never sees _vector_disabled flip).
mcp_server._get_client returns a cached HttpClient in HTTP mode (no inode/mtime watching).
repair.status and repair --mode max-seq-id early-exit with a clear refusal in HTTP mode — they manipulate ChromaDB's internal sqlite/max_seq_id table which physically lives on the remote server.
repair.rebuild_index in HTTP mode swaps the chroma.sqlite3 cross-check (mempalace repair silently truncates drawers to 10,000 — data loss on palaces > 10K #1208 guard) for a collection.count() cross-check, skips the file-system backup, and surfaces "snapshot the chromadb server before rebuild" as a warning. Extract / recreate / re-upsert all flow through the public chromadb API.
cli.cmd_repair delegates to repair.rebuild_index in HTTP mode, skipping the CLI's own palace-dir copytree backup (which requires local FS).

Config surface

MempalaceConfig grows database_url, chroma_url, chroma_host/port/ssl/tenant/database, and backend_mode properties so a single config object answers both "where is my palace?" and "which backend will I use?" without callers cross-importing _runtime.

Architectural note on "feature parity"

Two operator-only diagnostic commands physically cannot be implemented client-side over HTTP:

Command	Local	HTTP
`mempalace repair-status`	✅	refuse — reads `index_metadata.pickle` from disk
`mempalace repair --mode max-seq-id`	✅	refuse — direct UPDATE on chromadb's internal `max_seq_id` table

Both operate on storage that lives on the remote chromadb server in HTTP mode. There is no chromadb HTTP API for either. The CLI prints a clear refusal pointing at the chromadb server. Every end-user path (mine, search, KG, all MCP tools, scan, prune, rebuild) has full parity.

Tests

tests/test_kg_conformance.py — 13 behavioral tests parametrized over [SqliteKnowledgeGraph, PostgresKnowledgeGraph (sqlite-via-sqlalchemy), PostgresKnowledgeGraph (real Postgres via testcontainers)]. Real-Postgres tier is gated behind @pytest.mark.postgres and skipped when docker is unavailable.
tests/test_chroma_http_backend.py — 28 unit tests (URL parsing, env resolution, namespace qualification, registry membership, detect(), close() idempotency) plus a @pytest.mark.chroma_http integration test against either CHROMADB_TEST_URL or a chromadb/chroma:0.5.20 testcontainer.
One existing test updated for the KnowledgeGraph-becomes-factory change (test_kg_thread_safety.py now asserts against SqliteKnowledgeGraph.close); README badge regex test URL-decodes the version-segment +.

pytest -q → 1521 passed, 14 skipped, 106 deselected, no regressions.

Packaging

Version bumped to 3.3.4+stateless.1 (PEP 440 local-version segment so Renovate keeps tracking upstream while this fork is identifiable).
New optional dependencies: [postgres] and [remote] pull in sqlalchemy>=2,<3, psycopg[binary]>=3.2, alembic>=1.13.
New entry point: chroma_http = "mempalace.backends.chroma_http:HttpChromaBackend" under mempalace.backends.

How to use

pip install "mempalace[postgres]"

export MEMPALACE_DATABASE_URL="postgresql+psycopg://user:pw@db.svc:5432/mempalace"
export MEMPALACE_CHROMA_URL="https://chroma.svc:8000"
mempalace mcp        # talks to remote services, no /palace volume needed

Or invoked from a Kubernetes single-replica StatefulSet behind a ToolHive MCPServer with the same env vars. Reference container images: ghcr.io/shockstruck/mempalace-mcp (CPU + OpenVINO variants).

Out of scope (explicit)

Migration tool from existing local palaces to remote stores (one-time mempalace migrate is a follow-up).
Performance benchmarks against remote backends.
Server-side ChromaDB tuning (HNSW params, multi-tenant config).

Refactor the temporal knowledge graph so it works against either local SQLite (the historical implementation) or a remote Postgres instance. This is the first half of the "stateless palace" effort: drawer/closet data lives in ChromaDB which already has a backend abstraction; the KG was the remaining local-only state. Now both can be externalized. Layout ------ * ``mempalace/kg/base.py`` — ``BaseKnowledgeGraph`` ABC with the historical 8-method surface (``add_entity``, ``add_triple``, ``invalidate``, ``query_entity``, ``query_relationship``, ``timeline``, ``stats``, ``seed_from_entity_facts``, ``close``). * ``mempalace/kg/sqlite.py`` — ``SqliteKnowledgeGraph``: the original ``KnowledgeGraph`` implementation lifted verbatim behind the ABC. Behavior is byte-for-byte unchanged. * ``mempalace/kg/postgres.py`` — ``PostgresKnowledgeGraph``: SQLAlchemy 2.x Core, dialect-native ``ON CONFLICT`` upserts, JSONB on Postgres / JSON elsewhere via a custom ``TypeDecorator``. * ``mempalace/kg/_schema.py`` — single ``MetaData`` instance shared between in-process ``create_all`` and future Alembic migrations. * ``mempalace/kg/factory.py`` — ``KnowledgeGraph(...)`` is now a factory that picks the right backend based on its arguments and the ``MEMPALACE_DATABASE_URL`` env var. * ``mempalace/knowledge_graph.py`` — backwards-compat shim re-exporting the factory and ``DEFAULT_KG_PATH``. Existing ``from mempalace.knowledge_graph import KnowledgeGraph`` callers keep working without modification. Selection priority (first non-empty wins): 1. Explicit ``url=`` keyword argument 2. ``MEMPALACE_DATABASE_URL`` env var 3. Explicit ``db_path=`` keyword argument 4. ``MEMPALACE_KG_PATH`` env var 5. Default ``~/.mempalace/knowledge_graph.sqlite3`` (SQLite) Tests ----- * New ``tests/test_kg_conformance.py`` parametrizes the same behavioral test set across SQLite, SQLAlchemy-on-SQLite (catches SQLAlchemy-layer bugs without docker), and Postgres-via-testcontainers (skipped when ``testcontainers`` is unavailable; tagged with ``@pytest.mark.postgres``). * ``tests/test_kg_thread_safety.py`` updated to assert against ``SqliteKnowledgeGraph.close`` since ``KnowledgeGraph`` is now a factory function. * ``tests/test_readme_claims.py`` URL-decodes the version-badge value before comparing, since shields.io requires ``+`` to be ``%2B`` in PEP 440 local-version segments. All 1492 existing tests pass on this branch with the new shim; 99 KG-related tests pass (28 historical + 13×3 conformance + version/ thread-safety) with 13 postgres-marked tests skipped without docker. Packaging --------- * Bump version to ``3.3.4+stateless.1`` (PEP 440 local-version segment — preserves upstream-version visibility for Renovate while marking fork-only changes). * New ``[postgres]`` and ``[remote]`` extras pull in ``sqlalchemy>=2``, ``psycopg[binary]>=3.2``, ``alembic>=1.13``. * Pre-register ``chroma_http`` entry point in ``mempalace.backends`` for the next phase (HTTP-mode ChromaDB backend).

Implement the ``chroma_http`` backend so MemPalace can run against an external ChromaDB server (``chromadb run`` or a managed service) instead of an embedded ``PersistentClient``. Lays the groundwork for truly stateless palace pods. Design ------ * ``mempalace/backends/chroma_http.py`` — new ``HttpChromaBackend`` extending ``BaseBackend`` (RFC 001 §2). Owns a single cached ``chromadb.HttpClient`` per process. Multiple palaces share the same chromadb server but use distinct collection-name prefixes derived from ``PalaceRef.namespace`` (falling back to the palace ``id``). * Reuses the existing ``ChromaCollection`` adapter from ``backends/chroma.py`` — the per-collection wire format is identical whether the underlying client is ``PersistentClient`` or ``HttpClient``. * HTTP mode skips every filesystem-coupled operation: no ``quarantine_stale_hnsw``, no ``_fix_blob_seq_ids``, no ``os.makedirs`` on the palace path. Those operations target ChromaDB's internal sqlite/HNSW state, which physically lives on the remote chromadb server in HTTP mode. * ``_qualify`` sanitizes the collection-name prefix to chromadb's ``[a-zA-Z0-9._-]`` rule and ensures the first character is alphanumeric (``p`` prefix when needed). Configuration ------------- All env-driven, all optional (constructor accepts each as a kwarg for tests): * ``MEMPALACE_CHROMA_URL`` — full URL, preferred form. * ``MEMPALACE_CHROMA_HOST`` / ``MEMPALACE_CHROMA_PORT`` / ``MEMPALACE_CHROMA_SSL`` — split form. * ``MEMPALACE_CHROMA_AUTH_TOKEN`` / ``MEMPALACE_CHROMA_AUTH_HEADER`` — bearer-token auth, header name defaults to ``Authorization`` but can override (``X-Api-Key`` for proxies). * ``MEMPALACE_CHROMA_TENANT`` / ``MEMPALACE_CHROMA_DATABASE`` — multi-tenant chromadb routing. ``HttpChromaBackend.detect()`` returns True iff any URL/HOST/PORT env var is set, so :func:`registry.resolve_backend_for_palace` picks HTTP mode automatically when configured. Registry + entry point ---------------------- * ``mempalace/backends/registry.py::_register_builtins`` — register the new backend alongside ``chroma``. * ``pyproject.toml`` already declares the ``chroma_http`` entry point under ``mempalace.backends`` (added in the previous commit for forward compat); now it's pointing at a real class. Tests ----- * ``tests/test_chroma_http_backend.py`` — 28 unit tests covering URL parsing, env resolution, namespace qualification, registry membership, ``detect()`` behavior, ``close()`` idempotence. * Integration round-trip test gated behind ``@pytest.mark.chroma_http`` — runs against ``CHROMADB_TEST_URL`` if set, otherwise spins up a ``chromadb/chroma:0.5.20`` testcontainer. Skipped without docker.

Stateless mempalace pods need to talk to a centralized ChromaDB server (``chromadb run`` / managed) instead of an embedded ``PersistentClient``. This adds the second half of the "stateless palace" effort: drawer and closet collections live on the remote server, and the per-pod PVC becomes optional. Architecture ------------ * ``HttpChromaBackend`` extends ``BaseBackend`` exactly the same shape as ``ChromaBackend``, so callers using the kwargs-only ``palace=PalaceRef`` contract from RFC 001 work unchanged. * The ``ChromaCollection`` adapter is reused as-is — it wraps a ``chromadb.Collection`` object, which is identical between ``PersistentClient`` and ``HttpClient``. Zero duplication. * Multiple palaces can share one chromadb server; collection names are prefixed with ``palace_ref.namespace`` (or ``palace_ref.id`` if no namespace is set), sanitized to chromadb's allowed character class. * Capability set advertises ``http_mode`` (replacing ``local_mode``) so ``searcher.py`` and ``repair.py`` can gate the local-only fallback paths in a follow-up commit. Configuration env vars (all optional, all overridable from the constructor): * ``MEMPALACE_CHROMA_URL`` — full URL, preferred form. * ``MEMPALACE_CHROMA_HOST``, ``_PORT``, ``_SSL`` — split form. * ``MEMPALACE_CHROMA_AUTH_TOKEN`` (+ ``_AUTH_HEADER`` to override ``Authorization`` for proxies that prefer ``X-Api-Key``). * ``MEMPALACE_CHROMA_TENANT``, ``_DATABASE`` — chroma multi-tenant routing. What's deliberately NOT in this commit -------------------------------------- * Wiring ``mcp_server._get_client`` to use this backend — that lives in the config-selection commit so the wiring decision is a single place. * Refusing local-only operations (``mempalace repair-status``, ``--mode max-seq-id``) when running in HTTP mode — that's the next commit (searcher/repair backend-aware refactor). Tests ----- * ``tests/test_chroma_http_backend.py`` — 28 unit tests with ``chromadb.HttpClient`` mocked. Covers env resolution, URL parsing, connection caching, namespace prefixing, sanitization, health probe paths, registry integration, and ``make_client`` legacy shim. * Full repo test suite still green (1521 passed, 14 skipped) after registering ``chroma_http`` as a built-in backend. * New pytest marker ``chroma_http`` reserved for the live-server integration tests that ship in the next commit.

Splits the BM25-only candidate path into two implementations chosen at runtime by the active chromadb backend: * ``_bm25_only_via_sqlite`` — unchanged. Reads ``chroma.sqlite3`` + ``embedding_fulltext_search`` directly. Active when local backend. * ``_bm25_only_via_api`` — new. Uses ``collection.get(where_document= {"$contains": tok}, where=metadata, ...)`` for candidate selection and BM25-ranks the materialized docs in Python. Active when HTTP backend (no client-side sqlite). Same return shape as the sqlite path so ``_merge_bm25_union_candidates`` and ``search_memories`` don't have to special-case. The ``_bm25_only`` dispatcher is the new single entry point; both historical callers (``search_memories(vector_disabled=True)`` and ``_merge_bm25_union_candidates``) updated. ``mempalace/_runtime.py`` (new) centralizes the backend-mode question behind ``using_local_chroma()`` / ``using_http_chroma()`` with a process-local cache and a ``reset_backend_mode_cache()`` for tests. This keeps the routing decision in one place — searcher today, repair and the MCP server's HNSW capacity probe in follow-ups. Tests ----- * All 65 existing searcher + hybrid-candidate-union tests still pass. * The HTTP-mode ``_bm25_only_via_api`` round-trip test will land with the chroma_http integration test (``@pytest.mark.chroma_http``).

The remaining call sites that read ``chroma.sqlite3`` or ``index_metadata.pickle`` directly (BM25 fallback, HNSW capacity probe, ``mempalace repair``) needed to learn whether the active backend is local or HTTP. Add a single source of truth and wire every consumer through it. * ``mempalace/_runtime.py`` — new module. ``resolve_backend_mode()``, ``using_local_chroma()``, ``using_http_chroma()`` resolve from ``MEMPALACE_BACKEND`` (explicit) or ``MEMPALACE_CHROMA_*`` env vars (auto). Cached per process, ``reset_backend_mode_cache()`` for tests. * ``mempalace/palace.py::_resolve_backend()`` — ``get_collection()`` now picks ``ChromaBackend`` or ``HttpChromaBackend`` per call. The HTTP backend instance is cached so the chromadb ``HttpClient`` connection pool is shared across the searcher/miner/MCP paths. * ``mempalace/searcher.py``: * New ``_bm25_only(...)`` dispatcher routes to either the existing ``_bm25_only_via_sqlite`` (local) or the new ``_bm25_only_via_api`` (HTTP). * ``_bm25_only_via_api`` implements the BM25-only fallback using chromadb's public API: ``where_document={"$contains": tok}`` per query token (≥3 chars to mirror the FTS5 trigram tokenizer), union the results, BM25-rank in Python, recency-fallback when no token survives. Same result shape as the sqlite path so ``_merge_bm25_union_candidates`` and ``search_memories`` consume it unchanged. * ``_merge_bm25_union_candidates`` and the ``vector_disabled`` branch of ``search_memories`` now call the dispatcher. * ``mempalace/mcp_server.py``: * ``_get_client()`` builds an ``HttpClient`` once (no inode/mtime watching) when in HTTP mode; falls through to the existing ``PersistentClient`` cache for local mode. * ``_refresh_vector_disabled_flag()`` is a no-op in HTTP mode (segment health is server-side; client never sees divergence). * ``mempalace/repair.py``: * New ``BackendUnsupportedError`` for clean refusals. * ``status()`` returns ``status="unsupported"`` and prints the HTTP refusal hint instead of probing local files. * ``rebuild_index()`` swaps the sqlite-count cross-check (MemPalace#1208 guard) for a ``collection.count()`` cross-check in HTTP mode, skips the ``chroma.sqlite3`` backup, and surfaces the "snapshot the chromadb server before rebuild" warning. * ``repair_max_seq_id()`` early-exits with a clear message — the table lives in the chromadb server's internal sqlite and has no remote API. * ``mempalace/cli.py::cmd_repair`` — HTTP mode delegates to ``repair.rebuild_index`` directly, skipping the CLI's own palace-dir copytree backup (which requires local filesystem access). Result ------ Every end-user feature path (mine, search, KG, all MCP tools, scan, prune, rebuild) works against either backend. Two operator-only diagnostic ops (``repair-status``, ``repair --mode max-seq-id``) refuse cleanly in HTTP mode with a hint pointing at the chromadb server — those manipulate ChromaDB's *internal* storage, which is physically inaccessible from a remote client. 1521 tests pass on this branch, no regressions vs. main.

Surface the new env vars (``MEMPALACE_DATABASE_URL``, ``MEMPALACE_CHROMA_URL`` / ``_HOST`` / ``_PORT`` / ``_SSL`` / ``_TENANT`` / ``_DATABASE``) as properties on ``MempalaceConfig`` and add ``backend_mode`` so a single config object answers both "where is my palace?" and "which backend will I use?" without callers cross-importing :mod:`mempalace._runtime`. Also adds the legacy ``delete_collection`` / ``create_collection`` helpers on :class:`HttpChromaBackend` so :func:`repair.rebuild_index` works against either backend through a uniform interface.

jphein · 2026-05-11T21:47:35Z

Architectural-neighbor signal: the jphein/mempalace fork is exploring a different stateless-deployment substrate that lands the storage layer in Postgres entirely — pgvector for embeddings (composing on @skuznetsov's open #665) and Apache AGE for the knowledge graph (graph as a co-located Postgres extension rather than a sibling Postgres database).

The two approaches answer different questions: this PR's HttpChromaBackend says "ChromaDB stays the vector store; make the storage layer remote so pods can move freely." Our fork's pgvector + AGE says "consolidate everything into one Postgres engine — single connection, single backup story, KG traversals adjacent to vector similarity scores."

Both respect the RFC 001 backend contract, which means they don't collide at the abstraction layer. A deployment could in principle pick the right backend for its operational shape — stateless-with-Chroma-HTTP for ephemeral cloud pods, monolithic-Postgres for homelab-shaped boxes — and the rest of mempalace stays unchanged.

Spec for the pgvector + AGE direction is at docs/superpowers/specs/2026-05-10-pgvector-age-migration-design.md on jphein/mempalace if you're curious. Worth knowing about each other's parallel exploration.

shockstruck added 6 commits May 3, 2026 16:41

shockstricken requested review from bensig, igorls and milla-jovovich as code owners May 3, 2026 22:35

igorls added enhancement New feature or request storage area/kg Knowledge graph labels May 6, 2026

TengJoe mentioned this pull request May 13, 2026

[Feature] Support standalone ChromaDB HTTP Client to prevent SQLite concurrency crashes #832

Open

igorls changed the base branch from main to develop May 17, 2026 19:35

messelink mentioned this pull request May 18, 2026

feat(plugin): add mempalace-remote — SSH-proxied palace for remote clients #1190

Closed

cschnatz mentioned this pull request Jun 6, 2026

Add pluggable vector backends #1679

Merged

Navigation Menu

Uh oh!

feat: add HttpChromaBackend + Postgres KG backend for stateless deployments - #1337

feat: add HttpChromaBackend + Postgres KG backend for stateless deployments#1337
shockstricken wants to merge 6 commits into
MemPalace:developfrom
shockstruck:feat/remote-backends

shockstricken commented May 3, 2026

Uh oh!

jphein commented May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

shockstricken commented May 3, 2026

Context

What it adds

KG abstraction layer (mempalace.kg)

HTTP ChromaDB backend (mempalace.backends.chroma_http)

Backend-aware searcher / repair / CLI

Config surface

Architectural note on "feature parity"

Tests

Packaging

How to use

Out of scope (explicit)

Uh oh!

jphein commented May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

KG abstraction layer (`mempalace.kg`)

HTTP ChromaDB backend (`mempalace.backends.chroma_http`)