docs(investigate): root-cause metadata reshape bug (#32)

[jphein]

Fixes investigation portion of #32.

TL;DR

The chokepoint sanitizer
ChromaCollection._sanitize_metadatas_for_chromadb
(mempalace/backends/chroma.py:1094-1111) returns a new outer list but
aliases each non-empty caller dict by reference. The
comprehension [m if (...) else sentinel for m in metadatas]
returns m itself whenever the guard passes, so the sanitized list
still points at the caller's dict. Any in-place mutation between
sanitize and chromadb's validate_metadata empties the dict from
under the validator's feet — which matches the failure mode reported
at ~120K of a 151K rebuild.

The same aliasing pattern appears in repair.py:_extract_drawers and
repair.py:_rebuild_one_collection.

What this PR contains

• docs/investigations/metadata-reshape-root-cause.md — full
  walkthrough of the chromadb internal flow (Collection.add →
  normalize_insert_record_set → validate_metadatas →
  validate_metadata), confirmation that chromadb does not reshape
  metadata in our code path, the aliasing diagnosis with empirical
  reproduction, candidate mutators at scale, and a minimal-fix
  proposal.

• tests/test_metadata_reshape_bug.py — 9 tests. Two of them
  (test_sanitizer_inner_dicts_alias_caller_dicts and
  test_sanitized_dict_empty_after_caller_clear_passes_into_chromadb)
  reproduce the failure deterministically on a small palace, against
  the real chromadb 1.5.9 validator. The remaining 7 establish the
  upstream contract ({} rejected, None accepted) and verify the
  current sanitizer behaviour the fix must preserve.

This is investigation-only. The actual fix (change m → dict(m)
in three places) is recorded in the investigation doc and proposed as
a follow-up PR so review can focus on the diagnosis first.

Verification

$ python -m pytest tests/test_metadata_reshape_bug.py -v
====================== 9 passed in 1.47s =======================

$ python -m pytest tests/ -x -q --ignore=tests/benchmarks
====================== 2606 passed, 35 skipped =================

Architectural follow-ups

Recorded at the end of the investigation doc:

1. Land the non-aliasing fix in a separate small PR.
2. After that ships and a real rebuild confirms the fix, drop the
   redundant repair-layer sanitizers and keep the chokepoint as the
   sole owner of the "no empty dict reaches chromadb" contract — the
   chokepoint is the structurally-correct layer for this invariant.

🤖 Generated with Claude Code

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.