fix(compress): drop and recreate mempalace_compressed to avoid HNSW link_lists.bin inflation

[guilhermefriol]

Context

Adds a localized fix for one disk-fill vector reported in #1092 (and reproduced in my #1272, now closed as duplicate): the mempalace compress code path.

In my case the mempalace_compressed collection's link_lists.bin grew to 1.7 TB physical / 17 TB logical (sparse) for ~13K entries on a 1.8 TB disk, after running mempalace compress twice in one day with the MCP server and Stop/PreCompact hooks active in parallel. This PR doesn't address the underlying chromadb 1.5.8 concurrent-writer issue (that's broader — see #1092 for the full picture), but it removes one practical way the disk fills.

Fix

Drop and recreate the mempalace_compressed collection at the start of each compress run, before the upsert loop. Each run starts with a fresh HNSW index, so accumulation in link_lists.bin cannot happen.

Diff is small (~5 lines added) in cmd_compress (mempalace/cli.py).

Trade-off

Re-vectorizing all compressed entries on every compress run. With the local ONNX embedder and ~10K entries this is on the order of minutes, well within the budget for a weekly cron job. The alternative (silent disk fill) is much worse.

This is consistent with what the miner already does: miner.py:718-726 deletes by source_file before re-inserting to bypass hnswlib's updatePoint path that triggers the same kind of behavior. The compress path was the missing parallel.

Testing

Patched a local pipx install with this change and re-ran mempalace compress against the same palace several times in a row. du -sh ~/.mempalace/palace/<compressed_uuid> stayed proportional to the actual entry count across runs — no growth.

Notes

• Doesn't fix the underlying chromadb 1.x sparse-file behavior. A separate upstream report in chroma-core/chroma would be the right path for that.
• verbatim always is preserved: the user-facing data lives in mempalace_drawers, untouched. mempalace_compressed is a derived index of AAAK summaries, regenerated from those originals on each compress.

Related: #1092

[igorls]

Attempted to rebase onto develop but the conflict surfaces a semantic problem we should resolve before merging:

Context: #1244 (merged) renamed the compress-target collection from mempalace_compressed to the shared mempalace_closets. After that change, the same collection is now also written to by mempalace mine and regenerate_closets (#1107).

Problem: This PR's fix — backend.delete_collection(palace_path, "mempalace_compressed") followed by recreate — was safe when the collection was a dedicated per-run scratch space. Translating it to mempalace_closets (the new target) would silently destroy entries from mining and regenerate_closets, which is much worse than the HNSW link_lists.bin inflation we're trying to fix.

Suggested re-scoping: instead of drop+recreate, delete only the IDs this compress run is about to upsert, then upsert. That preserves the GC behavior locally without affecting other writers' data:

ids_to_write = [doc_id for doc_id, *_ in compressed_entries]
try:
    comp_col.delete(ids=ids_to_write)
except Exception:
    pass
comp_col.upsert(...)

Happy to push that as a follow-up commit if you want, but flagging here in case there's a reason to take a different approach (e.g., a separate dedicated collection for compress run scratch data, with a periodic compaction job).

[igorls]

Thanks for the careful repro and the diff @guilhermefriol — wanted to walk through the current state before any rebase, because the trade-off has shifted under this PR since it was filed.

The collection rename (#1244) makes drop-and-recreate unsafe now

PR #1244 merged on 2026-05-02 (after this PR was filed) and redirected the cmd_compress writer from mempalace_compressed → mempalace_closets. Before that rename, mempalace_compressed was an isolated, derivable collection that nothing else wrote to, so dropping it on each run was safe. After the rename, mempalace_closets is shared infrastructure:

• mempalace/cli.py:911 — cmd_compress (this PR's site)
• mempalace/miner.py:1090 — every mempalace mine writes closets as it goes
• mempalace/diary_ingest.py:108 — diary save path
• mempalace/closet_llm.py:226 — LLM-assisted closet writer
• read by searcher.py:777, palace.get_closets_collection, repair/HNSW diagnostics

So this PR as-currently-written — even after a literal rebase that updates the collection name — would erase closets populated by the miner / diary / closet_llm whenever a user runs mempalace compress. That's a verbatim-data-loss regression and would conflict with the "incremental only" rule in CLAUDE.md.

Two structural fixes that may already cover the original trigger

Your repro was concurrent writers (compress run 2x in one day with MCP server + Stop/PreCompact hooks active in parallel). Two landings since target exactly that trigger:

1. _HNSW_BLOAT_GUARD (2026-04-25, commit 88a53b2) — sets hnsw:batch_size=50_000 and hnsw:sync_threshold=50_000 at collection-create time. Breaks the resize+persistDirty feedback loop that produces the sparse-file accumulation, for newly-created collections.
2. fix: serialize ChromaCollection writes through palace lock #1162 (merged 2026-05-06) — serializes ChromaCollection.add/upsert/update/delete through mine_palace_lock(palace_path). The MCP server, hooks, and CLI now share the same palace-level write lock, so the concurrent-writer race that triggered the bloat in your case is closed.

Could you re-run your reproducer against develop tip (the rename is in v3.3.4, the lock is in upcoming v3.3.5)? Specifically: same machine, run mempalace compress twice in one day with MCP server + Stop/PreCompact hooks active, and check whether link_lists.bin still grows on the closets collection. If it does, the right follow-up is targeted — for example, deleting only the IDs cmd_compress is about to upsert before the upsert (so other writers' rows are untouched), or a heuristic-triggered HNSW reset gated on detected bloat. Happy to draft either of those once we know whether the structural fixes alone close the gap.

If the bloat doesn't reproduce on develop tip, this PR can be closed as superseded with thanks. The diagnostic value of your report is real even if the patch shape changes.

[igorls]

Thanks for chasing this — a 1.7 TB physical / 17 TB sparse link_lists.bin is genuinely alarming and worth a careful look. I tried to reproduce the inflation locally and want to share what I found, because the result changes how I'd approach the fix.

Reproduction attempt

On develop (commit ef8d83c) with chromadb==1.5.7 (project pin is >=1.5.4,<2):

• 200 entries × 6 close-and-reopen cycles, identical content
• 200 entries × 6 cycles, varying content per run (different vectors via the embedder, same ids — i.e. the updatePoint path)
• 2000 entries × 10 cycles, varying content

In every case, after each close-and-reopen:

• link_lists.bin: 0 bytes
• data_level0.bin: 167,600 bytes (constant across all cycles)
• header.bin / length.bin: 100 / 400 bytes (constant)
• chroma.sqlite3 is the file that grows with content

Where I got stuck

In chromadb 1.5.x, embeddings persist into chroma.sqlite3 (the embeddings_queue.vector column), not into the HNSW .bin files. The HNSW files look preallocated/vestigial in this version — single-process upserts don't grow link_lists.bin at all. That makes the "rebuild the HNSW index each compress run" framing hard to verify on current chromadb: the file the PR targets isn't on the write path I can see.

I'm not doubting your du reading — but it would help a lot to know which file actually held the bytes. Could you share:

1. python -c "import chromadb; print(chromadb.__version__)"
2. du -sh and ls -la inside the bloated collection directory from your reproduction
3. Whether the palace had a prior mempalace_compressed collection from before Closets not backfilled for non-mined palaces — mempalace compress writes to wrong collection (_compressed vs _closets) #1244 renamed it to mempalace_closets — a stranded collection from the old name wouldn't be cleaned up by either the old code or this PR

Two patch-level issues if we wanted to land this

• Collection name is stale. Closets not backfilled for non-mined palaces — mempalace compress writes to wrong collection (_compressed vs _closets) #1244 (cbd6e5d) renamed the compress output to mempalace_closets. The diff still calls delete_collection(\"mempalace_compressed\") and get_or_create_collection(\"mempalace_compressed\") — names cmd_compress no longer uses on develop. Needs a rebase + name swap.
• Failure semantics change. Dropping the closets collection at the start of every run creates a window where compressed data is gone before the new run finishes. If the dialect or embedder fails mid-loop, the user has lost their entire closets index and must recompress from drawers. The current upsert path degrades gracefully (partial-write recovery is just "rerun, the rest will land"). Worth spelling out in the description if we keep the approach.

If the real mechanism turns out to be elsewhere (concurrent writers from #1092, or stranded mempalace_compressed segments from before #1244), the fix likely lives there rather than in cmd_compress. Happy to keep digging once we have the version + ls -la data — moving this off the 3.3.5 milestone for now.

[igorls]

Thanks! Drop-and-recreate on the shared compressed collection risks data loss on the renamed collection, and the bloat trigger is being handled structurally (lock #1162 + the HNSW guard work). Closing in favor of the non-destructive path.