feat(chroma): opt-in HTTP client/server mode for multi-process palaces

[ahabshamaa]

What does this PR do?

Adds an opt-in HTTP client/server mode for the ChromaDB backend, as requested in #832 and #1096 — and implements the "ChromaDB HTTP server mode" option named in the architectural decision epic #1963.

Problem. Embedded chromadb.PersistentClient is not process-safe. When more than one process writes the same palace — multiple MCP server instances, stop-hook mines firing alongside an interactive session, a manual mempalace mine — writers contest the same sqlite/HNSW files and can corrupt the index (the #1963 cluster). Worse, a blocked writer waits on the file lock with no timeout, so palace calls can hang the MCP server indefinitely.

Fix. With MEMPALACE_CHROMA_MODE=http, one standalone chroma run server exclusively owns the palace directory and every MemPalace process is a thin HTTP client constructed in one place (mempalace/palace_client.py). The server serializes writes where ChromaDB actually supports concurrency; clients get:

• Health probe — construction hits /api/v2/heartbeat (3 s timeout) and raises PalaceBackendUnreachableError naming host:port instead of hanging.
• Hard per-operation timeout — every wire call runs under a 15 s ceiling (MEMPALACE_OP_TIMEOUT) via a TimeoutProxy wrapping the client and every collection obtained from it; timeouts raise PalaceOperationTimeoutError naming the operation.
• One error shape — transport failures mid-call (httpx.TransportError, ConnectionError) are translated into the same structured unreachable error the probe gives; the MCP server returns these as structured tool errors, never hangs, never retries (the bound already happened).

Default unchanged. embedded remains the default: zero change and no external service for single-process installs. Config surface: MEMPALACE_CHROMA_MODE, CHROMA_HOST/CHROMA_PORT (default 127.0.0.1:8801), MEMPALACE_OP_TIMEOUT.

Embedded-only mechanisms gated off in HTTP mode (each is correct for a process that owns the files, wrong for a thin client):

• the pre-open repair pass — client-side repair against a live server's files is a corruption risk;
• the inode-keyed client cache and mtime reconnect — file identity is meaningless over HTTP, and the server bumps mtime on every write;
• the HNSW capacity probe — HNSW max_elements frozen at 16K while collection grows to 100K+ entries — MCP server segfaults on every tool call #1222 guards an embedded-only segfault; left on it would wrongly route search to BM25;
• the collection _write_lock flock — SIGSEGV in HNSW parallel inserts — missing hnsw:num_threads on collection creation #974/Critical Data write loop, HNSW compaction bug #965 guarded embedded multi-threaded corruption; kept on it turns legitimate concurrent writers into MineAlreadyRunning errors;
• the MCP peer-writer lease (MCP server should detect a peer writer on the same palace (concurrent stdio writers corrupt the Chroma backend) #1818) — in HTTP mode it would force peer servers read-only, the same incompatibility Peer-writer lock (#1818/#1823) is incompatible with the multi-session daemon topology (#1270) + external hook/manual mines #1888 reports for daemon topologies.

Mining passes still take mine_palace_lock for application-level atomicity in both modes.

Docs: docs/chroma-client-server.md — topology, config, error contract, launchd + systemd service examples, heartbeat/backup/vacuum runbook (#1096 asked for exactly this).

Relationship to the daemon+bridge direction (#1270/#1976): complementary, not competing — this is opt-in, changes nothing by default, and gives multi-process users a supported deployment today at the layer where ChromaDB itself supports concurrency.

Closes #832. Closes #1096. Related to #1963, #1581, #1799, #1888.

How to test

python -m pytest tests/ -v                      # suite forces embedded mode; 17 new tests in tests/test_palace_client.py
# live smoke test:
chroma run --path /tmp/palace-http --port 8801 &
MEMPALACE_CHROMA_MODE=http mempalace init ~/somewhere && MEMPALACE_CHROMA_MODE=http mempalace mine ~/somewhere
# kill the server mid-session → structured PalaceBackendUnreachableError, no hang

Checklist

• Tests pass (python -m pytest tests/ -v) — 3284 passed; the 2 failures in test_repair.py (_errors_are_isolated_fts5 string matching) fail identically on clean develop in this environment
• No hardcoded paths
• Linter passes (ruff check .)

🤖 Generated with Claude Code

[fatkobra]

Blocking: this is a timeout on the caller's wait, not a timeout or cancellation of the Chroma operation. The implementation explicitly leaves a timed-out call running and then reports failure. For add, upsert, update, or delete, the caller can retry while the original request later commits, producing duplicate, reordered, or overlapping effects.

Repeated hangs also occupy all eight executor workers. Subsequent healthy calls can remain queued and time out before they ever start, making the failure self-sustaining.

Please use transport-level connect/read/write/pool timeouts enforced by the HTTP client. An ambiguous write timeout must be reported as "outcome unknown" and reconciled with an idempotency key or status check rather than treated as a clean failed operation. Add tests for commit-after-timeout and executor saturation.

[igorls]

Thanks for this contribution, and apologies for the slow turnaround.

develop has moved a fair way since this was opened and the branch no longer merges cleanly. If you're still interested in landing it, could you rebase onto current develop? Once it merges cleanly and CI is green I'll get it reviewed for the 3.8.0 cycle.

If you'd rather not pick it back up, no problem at all — just say so and I'll close it out, and thanks either way for taking the time to send it.