feat: many palaces, one machine — config-dir lever, source-adapter registry

[joshuafontany]

Addresses #2110. Stacked on #2107 — ② calls _hnsw_creation_metadata(), introduced there, so this
branch is cut from fix/2106-hnsw-guard-corrupts-what-it-protects and its diff includes #2107 until
that lands.

Who is asking

Same context as #2107 and #2109. We create instances of your code as libraries in ours:

  sensoriums/memory/content          ← does the job an upstream install does
  sensoriums/mark-twin/content
  sensoriums/kumulipo/content
  ~/.mempalace                       ← the operator's own install, when present

Three sovereign content palaces, one per sensorium (extendable), plus whatever the operator already
runs. Peers, never satellites. We consume mempalace as a library from our own NDJSON holder
processes, with the embedding contract inverted: vectors arrive on the wire, so no model loads in the
store process.

Everything below answers one question — what does a second, third, fourth instance need?

What this donates

① MEMPALACE_CONFIG_DIR. MempalaceConfig.__init__ takes config_dir documented "useful for
testing", with no env or CLI lever, while palace_path already honors MEMPALACE_PALACE_PATH. Every
instance on a machine therefore reads the same ~/.mempalace/config.json — backend, collection_name,
embedding_model, write_routing, the milvus/qdrant/pgvector sets, ~24 keys — so one install configures
the others. Resolution mirrors palace_path exactly: explicit arg > env > default. Unset changes
nothing.

② Caller-supplied embedding function. get_collection(options.embedding_function) passes the
value verbatim: a function is used, None skips the model entirely and survives reopen, absent leaves
today's path unchanged. Running many palaces, a model per store process costs more than the stores do;
a host that embeds upstream hands the vector over and asks the store to hold it.

A finding worth having even if ② does not land: the modern CreateHNSWConfiguration API leaves
collection.metadata empty — parameters go to configuration_json — which silently blinds the
divergence guard, the cosine detector (ChromaCollection.distance_metric), and
_read_sync_threshold, all of which read collection_metadata. Legacy metadata= stays correct.

③ Source-adapter registry, ndjson adapter, daemon-queue routing. Your miners find the content. A
host running its own extraction already holds it, in its own shape, and --source NAME lets it
deliver records directly. The ndjson adapter proves the shape on a JSON-Lines spool. Pre-extracted
records carry the original words — the verbatim promise holds end to end.

④ Stored metadata on search hits (+5 lines), so one call answers completely.

Each alone reads as a convenience. Together they answer the question a reviewer actually rules on:
does mempalace support standing more than one of itself?

Status: draft

feat: route mine --source NAME through the daemon queue leaves state that makes
tests/test_sync.py::TestSyncMcpTool fail when tests/test_mine_source_registry.py runs first — 7
failures in a full run, none when either file runs alone. Dropping that one commit makes the branch
green; I would rather fix it. Tell me which you prefer.

Verification

• ruff check . clean · ruff format --check . → 197 files already formatted
• pytest tests/ --ignore=tests/benchmarks → 3423 passed, 31 skipped, 12 failed: the 7 above, plus 5
  test_init_filters_sys_path_from_leaked_pythonpath that fail identically on the base in the same
  worktree (a leaked PYTHONPATH here — what that test asserts about).

Local ruff reads 0.15.20 against your pinned 0.15.14; let CI have the final word on lint.

[joshuafontany]

Dropping the caller-supplied embedding-function commit — it fails to do what it claims.

embedding_function=None reaches only the client-level function. chromadb carries a second at the schema level, defaulting to a live DefaultEmbeddingFunction(), and _embed() falls through to it. On a fresh collection that commit stores a 384-dim ONNX vector — a model loading in the store process, the one outcome it existed to prevent.

The shape that holds needs an EF-less schema at creation. But create_collection rejects metadata= and schema= together, and this backend passes metadata= for the HNSW parameters, so the two cannot coexist. That goes as an issue referencing #1261, which meets the same wall.

--source through the daemon queue comes out too — it touches the single-writer seam while #1976 and #2033 run live there.

Remaining: MEMPALACE_CONFIG_DIR, the source-adapter registry, the ndjson adapter, stored metadata on search hits.

[joshuafontany]

The caller-vector constraint: #2121.

[joshuafontany]

Split into two, so each piece meets a reviewer at the size it deserves.

#2173 — mine --source through the RFC 002 registry, ready for review. It closes #2062 against that issue's acceptance criteria, including the --source × explicit --mode exclusivity.

#2174 — draft, holding the config-dir lever, the ndjson adapter, and stored metadata on hits until #2173 lands.

The HNSW defaults that opened this branch merged in #2107, with the test follow-up in #2159. #2110 stays open as the umbrella; the caller-supplied-vectors face of it is filed separately as #2121.