feat(backends): add sqlite_vec backend, fixes UAF crash on macOS/ARM64 (smaller footprint)

[MohamedAbdallah-14]

Summary

Alternate vector backend at mempalace.backends.sqlite_vec.SqliteVecBackend implementing the full BaseBackend / BaseCollection contract with sqlite-vec's vec0 virtual table. Targets the platforms where chromadb_rust_bindings is unsafe — primarily macOS 26 / ARM64 where the rust bindings have an intra-process UAF in the recursive segment walker (chroma-core/chroma#6852, #1355, #1376).

The backend has no Rust extension and no chromadb dependency; this UAF vector does not exist in the call graph.

Storage

One sqlite_vec.db file per palace. Per-collection vec0 virtual table sized to the collection's dimension. Schema:

CREATE TABLE collections (
    name TEXT PRIMARY KEY,
    dimension INTEGER NOT NULL,
    created_at TEXT DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE drawers (
    rowid INTEGER PRIMARY KEY AUTOINCREMENT,
    collection TEXT NOT NULL,
    drawer_id TEXT NOT NULL,
    doc TEXT,
    meta TEXT,         -- JSON; chroma-style filters compile to json_extract(meta, ?)
    UNIQUE(collection, drawer_id)
);
CREATE INDEX idx_drawers_collection ON drawers(collection);

-- Per-collection vec0 (rowid 1:1 with drawers.rowid)
CREATE VIRTUAL TABLE vec_<collection> USING vec0(
    rowid INTEGER PRIMARY KEY,
    embedding FLOAT[<dim>]
);

PRAGMAs at open: journal_mode=WAL, synchronous=NORMAL, foreign_keys=ON.

Supported where operators: $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $and, $or, plus bare-scalar equality. Supported where_document operators: $contains, $not_contains, $and, $or. Unknown operators raise UnsupportedFilterError per RFC 001 §1.4.

Selection

Optional dependency — opt in via pip install mempalace[sqlite-vec]. Registered through the existing mempalace.backends entry-point group, so selection goes through the standard registry: get_backend("sqlite_vec"). The mempalace.backends.chroma:ChromaBackend registration is unchanged, so default behaviour is identical for users not opting in.

Migration

examples/migrate_chroma_to_sqlite_vec.py reads chroma.sqlite3 directly via stdlib sqlite3 (no chromadb code is loaded, so the UAF cannot fire even on affected platforms) and re-embeds documents via the existing get_embedding_function. Stock hnswlib cannot load chromadb's segment envelope, hence re-embed rather than vector copy.

Resumable on drawer_id uniqueness — interrupted runs pick up where they left off. Validated on a 664k-drawer palace; final counts matched the source exactly (drawers 648700/648700, closets 15368/15368). Took ~60min at ~90 records/sec on M4 Pro.

Disk footprint

Side benefit on my corpus (664k drawers, 384-dim embeddings): the sqlite_vec store ended up roughly 3x smaller than the chromadb palace it replaced.

	chromadb	sqlite_vec
metadata sqlite	4.2 GB	—
HNSW segment (drawers)	1.3 GB	—
HNSW segment (closets)	0.2 MB	—
sqlite_vec.db	—	1.8 GB
total	5.5 GB	1.8 GB

Where it comes from:

• No precomputed HNSW index on disk — vec0 does brute-force SIMD scan, which is fast enough at this scale.
• Vectors stored once. chromadb writes each vector both as a serialized blob in embedding_metadata and inside the HNSW segment.
• No segment-management tables (segments, segment_metadata, embeddings_queue, max_seq_id).
• Part of the gain is a one-time defrag effect from soft-delete tombstones dropped on migration. A fresh palace migrated at insert-time would show less compression than 3x.

Numbers are from one corpus; mileage will vary, especially on smaller or less-churned palaces.

Tests

39 new cases in tests/test_sqlite_vec_backend.py covering backend lifecycle, write paths (add / upsert / update / delete-by-ids / delete-by-where), query paths (with and without metadata filters, dimension-mismatch raises, dict-compat access), the where compiler parametrized over every supported operator, where_document filters, get pagination, and registry-side discovery. Skip cleanly when the sqlite-vec extra isn't installed.

Full test suite: 904 pass / 1 pre-existing failure (test_mcp_stdio_protection, also fails on main without my changes — unrelated).

Out of scope for this PR

This PR is the contract impl only. The downstream selection plumbing (palace.py _DEFAULT_BACKEND and mcp_server.py _get_collection) is intentionally not touched. Follow-up PR if this looks good.