fix(backends): keep post-deletion dim-None HNSW segments instead of quarantining (#1710)

[mvalentsev]

What does this PR do?

quarantine_invalid_hnsw_metadata renames an HNSW segment to *.corrupt-* when its
index_metadata.pickle has dimensionality: None together with labels, unless
_missing_dimensionality_appears_recoverable judges the pickle safe to keep. develop
already keeps such a segment when its bookkeeping is internally consistent, but the helper
required total_elements_added == len(id_to_label), and that equality only holds for a
segment that has never had a delete.

total_elements_added is a monotonic counter of every element ever added, while
id_to_label and label_to_id track only the live elements. After any deletion
total_elements_added exceeds the live label count, so the helper classified the segment
as unrecoverable and quarantined it. When the WAL had already been pruned (as it is after
the flush that writes the dim-None pickle), that segment was the only copy of the vectors,
so a later cold start rebuilds an empty index: the rows still count, but vector search
returns nothing for them. That is the silent erosion QuiqueMH reported in #1710 (forensics:
total_elements_added: 4826 against 4792 labels).

The change relaxes the counter check from == to >= (one operator, != to <). The
label-map size check and the bijection check are untouched, so inconsistent label maps are
still quarantined, and an impossible total_elements_added < len(id_to_label) is still
rejected.

This closes the remaining reproducible case in #1710; the total == labels variant is
already kept by the helper on develop.

How to test

New regression test:

python -m pytest tests/test_backends.py -k post_deletion -q

It writes a dim-None pickle with total_elements_added=5 over two live labels (three
deleted) and asserts the segment is kept rather than quarantined. It fails on develop and
passes with this change.

Checked end to end against real ChromaDB 1.5.7, 1.5.8 and 1.5.9: a collection that is
built, then has rows deleted and rows added, persists a pickle with dimensionality=None
and total_elements_added > len(id_to_label). Before this change the guard quarantines
that segment and a later cold start finds it gone (count still reports the rows, but vector
search returns nothing); after it the segment is kept and a fresh process opens and queries
it with no error and no SIGSEGV on any of the three versions.

Full suite (python -m pytest tests/ --ignore=tests/benchmarks): 2431 passed, 5 skipped.

Checklist

• Tests pass (python -m pytest tests/ -v)
• No hardcoded paths
• Linter passes (ruff check .)

[gemini-code-assist]

Code Review

This pull request fixes an issue where post-deletion segments with missing dimensionality were incorrectly quarantined. It relaxes the check in _missing_dimensionality_appears_recoverable to allow total elements to be greater than or equal to label_count, since total_elements_added is monotonic and can exceed the live label count after deletions. A corresponding unit test has been added to verify this behavior. I have no feedback to provide as there are no review comments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.