fix: serialize ChromaCollection writes through palace lock

[imtylervo]

#976 protects mempalace mine, but MCP/direct backend writers still call ChromaCollection.add/upsert/update/delete without the palace lock. This PR moves the lock boundary to the Chroma backend seam so all Chroma writes share the same palace-level serialization, with a re-entrant guard for miner paths that already hold the lock.

Why this matters

mine_palace_lock(palace_path) closes the mine fan-out hole, but its current coverage is the miner.mine() pipeline. Other writers can still reach Chroma directly through the backend adapter, including MCP write tools and third-party callers that use ChromaCollection.

When one of those writers runs concurrently with mempalace mine or another direct writer, it can still enter ChromaDB's HNSW write path concurrently. That is the gap this PR closes.

What changed

• ChromaCollection now accepts an optional palace_path.
• ChromaCollection.add/upsert/update/delete acquire mine_palace_lock(palace_path) when a palace path is available.
• ChromaBackend.get_collection/create_collection and MCP _get_collection() pass the palace path through.
• mine_palace_lock is now re-entrant for the same thread and palace key, so miner.mine() can hold the outer lock while internal collection writes pass through safely.
• The re-entrant holder state is tagged with the current PID so forked children do not inherit lock-holder credit from the parent.

palace_path=None preserves the previous no-lock behavior for legacy/tests/direct construction.

Production evidence

An equivalent local patch on a large production palace caught the gap under live load:

• A mempalace mine subprocess held the palace lock while ingesting project transcripts.
• A sibling MCP diary_write attempted to write during that window and was blocked by the patched ChromaCollection lock instead of entering Chroma concurrently.
• Before this lock layer, the same workload had produced HNSW index bloat in link_lists.bin on the order of tens of GB.
• After the patch, the palace stayed healthy under the same workload: no new Chroma/Rust segfaults, no new auto-quarantine batch, and link_lists.bin remained in the KB range.

The intended trade-off is: fail/skip a concurrent write loudly rather than corrupting the HNSW index.

Tests

$ pytest tests/test_chroma_collection_lock.py tests/test_palace_locks.py tests/test_backends.py -q
....................................................                     [100%]
52 passed in 1.09s

Coverage added:

• palace_path=None skips locking for backward compatibility.
• Write methods raise MineAlreadyRunning while another process holds the palace lock, without calling the underlying Chroma collection.
• Re-entrant writes inside an existing mine_palace_lock do not deadlock.
• Concurrent direct writers serialize through the palace lock.
• Read paths do not acquire the write lock.

Scope

This PR intentionally only covers the correctness layer: serializing Chroma writes through the existing palace lock.

Deferred follow-up: a durable pending-write queue for MCP/autosave writes that encounter a busy palace lock, so agent autosaves can be replayed later instead of being dropped.

Refs #1161.

[bensig]

Hey @imtylervo — heads up that #976 merged to develop on 2026-04-25 and includes the chromadb 1.5.x `hnsw:num_threads=1` retrofit you correctly identified as needed. It uses `collection.modify(configuration=UpdateCollectionConfiguration(hnsw=UpdateHNSWConfiguration(num_threads=1)))` — the API that does persist on 1.5.x — so the runtime-retrofit half of this PR is now redundant.

The `palace_write_lock` half might still add value beyond #976's `mine_global_lock` — different scope (collection-level vs miner-process-wide cross-process). Worth a focused look at whether the queue-pattern lock catches anything that the global mine lock doesn't.

If you want to keep this open, suggest rebasing against develop and scoping down to just the lock layer. If the retrofit was the main motivator, fine to close — and thank you for the precise diagnosis in #1161; that's what made #976's retrofit correct.

[imtylervo]

@bensig Updated per your review:

• Rebased onto develop.
• Dropped the HNSW retrofit half now covered by fix: HNSW graph corruption, PreCompact deadlock, mine fan-out (closes #974, #965, #955) #976 / develop.
• Scoped this PR down to the backend-seam lock layer only: ChromaCollection.add/upsert/update/delete now acquire mine_palace_lock(palace_path).
• Added a per-thread, pid-tagged re-entrant guard so miner.mine() can hold the outer lock while inner Chroma writes pass through without self-deadlock.
• Added tests for writer-blocks-during-mine, re-entrant-inside-mine, two-process serialization, palace_path=None backward compatibility, and read paths not acquiring the write lock.

Production evidence is summarized in the PR body. Happy to iterate if anything still looks off.

[igorls]

@imtylervo please check the tests on Windows

[igorls]

Pushed a follow-up commit (d1e27b8) that runs ruff format (matching the CI-pinned 0.4.x) on the two new test files — that was the only failing check. Pure whitespace/line-collapse changes; no behavioral diff. CI should now go green; will re-review afterwards.