fix(backends): return Python floats from _embed_texts for NumPy 2.x

[mbeacom]

Fixes MemPalace#2190

What does this PR do?

Fixes a total ingest failure on the default Chroma backend. Every write raised:

ValueError: Expected embeddings to be a list of floats or ints, a list of lists,
a numpy array, or a list of numpy arrays, got [[np.float32(-0.08368583), ...]]

Root cause — two independently-correct changes that combined into a regression

1. mempalace/backends/chroma.py — ChromaBackend.capabilities began declaring requires_explicit_embeddings (commit 0e79797, "fix: stabilize Chroma embeddings on Windows"). This routed the default backend through EmbeddingCollection for the first time.

2. mempalace/backends/embedding_wrapper.py::_embed_texts — converted rows with list(...):

   vectors = ef(input=texts)
   return [list(v) for v in vectors]

   Iterating a NumPy row yields np.float32 scalars, so this returned list[list[np.float32]], not the annotated list[list[float]].

ChromaDB's normalize_embeddings (chromadb/api/types.py) gates on isinstance(target[0][0], (int, float)). Under NumPy 2.x, np.float32 is not a float subclass — so the batch falls straight through to raise ValueError. Confirmed locally on numpy 2.4.4 / chromadb 1.5.7:

np.float32 is float subclass: False
np.float64 is float subclass: True   # note: float64 still is, which is why this hid for so long

Neither change is wrong on its own. #2 was latent: only pgvector / sqlite_exact / milvus / qdrant used _embed_texts, and they all accept np.float32. Because pyproject.toml pins numpy>=1.24 with no upper bound, fresh installs resolve NumPy 2.x and hit this by default.

The fix

Convert rows with .tolist() instead of list(...) — it returns genuine Python floats, which every backend accepts. The hasattr guard keeps embedding functions that already return plain Python lists working unchanged.

Why CI didn't catch this

tests/conftest.py has an autouse fixture that monkeypatches _embed_texts away for every test module except the _REAL_EMBEDDING_TEST_MODULES allowlist ({"test_embedding", "test_embeddinggemma"}). Every existing test exercised the stub, not the real function — so the suite stayed green while ingest was broken in the field.

The regression tests are therefore placed in tests/test_embedding.py deliberately. I verified the failure mode: tests placed in tests/test_backends.py silently hit the conftest stub and pass even against the buggy code.

How to test

1. Reproduce on develop (pre-fix) — fails:

uv run python -c "
from mempalace.backends.embedding_wrapper import _embed_texts
from chromadb.api.types import normalize_embeddings
v=_embed_texts(['hello world'])
print(type(v[0][0]), isinstance(v[0][0], float))
normalize_embeddings(v)"

<class 'numpy.float32'> False → ValueError ❌

2. Same snippet with this branch — passes:

<class 'float'> True, normalize_embeddings OK, dim = 384 ✅

3. The new tests genuinely catch the regression. I reverted the one-line fix back to return [list(v) for v in vectors] and re-ran:

FAILED tests/test_embedding.py::test_embed_texts_returns_python_floats_so_chroma_accepts
E  AssertionError: assert [<class 'numpy.float32'>, ...] == [<class 'float'>, ...]
E  At index 0 diff: <class 'numpy.float32'> != <class 'float'>
1 failed, 1 passed

The fix was then restored — the revert is not in this branch.

4. Real end-to-end mine (not a dry-run — this actually exercises the Chroma write path) over 3 real Claude transcripts into a throwaway palace:

	Files	Drawers filed
develop (unpatched)	3	0 — ValueError in _validate_and_prepare_upsert_request
this branch	3	86 ✅

mempalace search against the resulting palace returns correct verbatim content with sensible cosine_sim / bm25 scores, so the read path is healthy too.

5. Full suite / lint

uv run pytest tests/ -q --ignore=tests/benchmarks   →  3852 passed, 31 skipped
uv run ruff check .                                 →  All checks passed!
uv run ruff format --check .                        →  212 files already formatted

Test-count accounting: develop collects 3881, this branch collects 3883 (exactly +2), and 3852 passed + 31 skipped = 3883. ✅

Checklist

• Tests pass (python -m pytest tests/ -v) — 3852 passed, 31 skipped
• No hardcoded paths
• Linter passes (ruff check .) — plus ruff format --check . clean

Notes

• Scope is deliberately surgical: 3 files, +58/−2. No behavioural change for backends that were already working.
• The docstring on _embed_texts now records why .tolist() is required, so the list(...) form isn't reintroduced.
• Worth considering as a follow-up (out of scope here): the conftest stub means the EmbeddingCollection → real-backend seam is effectively untested outside two allowlisted modules. That blind spot is what let this reach users.

[mbeacom]

Superseded by MemPalace#2191 — this fix belongs upstream against MemPalace/mempalace base develop, not on the fork.