fix: pin chromadb<1 + resilient mining against HNSW corruption

[jonashertner]

Summary

Two related fixes for the SIGSEGV crash when mining (#96, #100):

1. Pin chromadb>=0.4.0,<1 in pyproject.toml — the published PyPI package had no upper bound, so pip install mempalace pulled chromadb 1.0.12 (Rust HNSW backend). On macOS arm64, interrupted mine runs cause link_lists.bin to bloat catastrophically (297 GB for ~3,300 drawers) and subsequent collection.get() / collection.count() calls SIGSEGV. Downgrading to chromadb 0.6.3 (Python HNSW backend, within the <1 pin) resolves all crashes — 10,000+ drawers mine without incident on the same hardware.

2. Crash-resilient mining pipeline (defence-in-depth against future regressions in any chromadb version):

   • Batch writes with checkpointing — drawers flushed in batches of 50 instead of one-at-a-time. A checkpoint file (mine-checkpoint.json) tracks fully processed files, so interrupted runs resume without querying ChromaDB (which may be corrupted).
   • Palace health check on startup — before mining, verify the palace is readable. If corrupted, auto-detect bloated HNSW index files and remove them so ChromaDB rebuilds on next access.
   • Graceful signal handling — SIGINT/SIGTERM flush the pending batch and save the checkpoint before exiting.
   • mempalace repair CLI command — detect and fix corrupted palaces (--force to rebuild unconditionally).

Files changed

File	Change
pyproject.toml	Pin chromadb>=0.4.0,<1
mempalace/checkpoint.py	New — atomic JSON checkpoint for mine progress
mempalace/miner.py	Batch writes, health check, repair, signal handling
mempalace/cli.py	mempalace repair [--force] subcommand
tests/test_miner_resilience.py	9 new tests covering all resilience paths

Test plan

• All 18 tests pass (9 existing + 9 new)
• Confirmed: chromadb 0.6.3 mines 10,000+ drawers without corruption on macOS arm64
• Manual: mempalace mine interrupted with Ctrl-C resumes correctly
• Manual: mempalace repair recovers a corrupted palace
• Manual: mempalace repair --force rebuilds index unconditionally

Closes #96, closes #100

🤖 Generated with Claude Code

[daedalus]

I experienced the same issue, this would be awesome to have it.

[bensig]

Thanks — chromadb pin landed in #114 and repair command in #119. Closing as duplicate.