fix(storage): quarantine partial HNSW flush without metadata - #1342
Merged
igorls merged 2 commits intoMay 7, 2026
Merged
Conversation
Two conflicts, both because MemPalace#1339 (bloated link payloads) merged into develop after this branch was authored: - mempalace/backends/chroma.py: _segment_appears_healthy now stacks three checks — bloated-link from MemPalace#1339 (top), missing-metadata-with- data-floor from this branch (middle), pickle format sniff (bottom). All three are complementary; MemPalace#1339 catches structural payload corruption, this branch catches pickle truncation, the original catches pickle protocol-byte corruption. - tests/test_backends.py: kept both new imports (_segment_appears_healthy from this branch, quarantine_invalid_hnsw_metadata from MemPalace#1285). Local: 1618 tests pass, ruff lint+format clean against 0.4.x CI pin.
Member
|
Maintainer-edit: merged develop in to resolve conflicts opened by #1339 (bloated link payloads, merged earlier today). `_segment_appears_healthy` now stacks three complementary checks:
Each catches a different shape of broken-on-disk segment. Local: 1618 tests pass on the merge, ruff lint+format clean against the 0.4.x CI pin. |
This was referenced May 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #1274.
This tightens
_segment_appears_healthy()so a missingindex_metadata.pickleis treated as healthy only when the HNSW segment still looks fresh or effectively empty.If
data_level0.binalready has non-trivial payload butindex_metadata.pickleis missing, the segment is now treated as partially flushed corruption andquarantine_stale_hnsw()renames it before Chroma opens the native HNSW reader.What changed
_HNSW_MISSING_METADATA_DATA_FLOOR._segment_appears_healthy():index_metadata.pickle+ tiny/emptydata_level0.binremains healthyindex_metadata.pickle+ non-trivialdata_level0.binis unhealthyOSErrorwhile checking the payload is treated as unsafe_segment_appears_healthyfalse-negative for partially-flushed segments #1274 partial-flush shapeWhy
#1274 reports a real palace with this segment state: