[BUG]: Delete collection resource leak (single-node Chroma)

[tazarov]

Description of changes

Closes #3296

The delete collection logic slightly changes to accomodate the fix without breaking the transactional integrity of self._sysdb.delete_collection. The chromadb.segment.SegmentManager.delete_segments had to change to accept the list of segments to delete instead of collection_id.

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • Fixes the resource leak when deleting a collection

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

N/A

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[tazarov]

• [BUG]: Local segment manager memory leak fix #3340
• [BUG]: Delete collection resource leak (single-node Chroma) #3297 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[rohitcpbot]

Thanks for identifying the leak and raising the fix. I did not see this earlier so did not review earlier. my miss. Reviewed it now.

[rohitcpbot]

I feel we should try not to call sysdb for getting segments. It adds extra call to the backend for distributed chroma.

Seeing the current code, I see we are already calling sysdb.get_segments() from the manager, so you are simply moving that line here, and not adding extra calls. But i feel we can do better.

Do you think we should just call delete_segment() from delete_collection() ?
So we can add this snippet back -

   for s in self._manager.delete_segments(existing[0]["id"]):
       self._sysdb.delete_segment(s)

and do a no-op inside delete_segments() in db/impl/grpc/client.py
Will that fix the leak ?

[tazarov]

@rohitcpbot,

using this snippet:

for s in self._manager.delete_segments(existing[0]["id"]): 
   self._sysdb.delete_segment(s)

Makes sense however we revert back to a non-atomic deletion of sysdb resources. In the above snippet we'd delete the segments separately from deleting the collection, which I wanted to avoid on purpose which is why I pulled the get of the segments here before the were atomically deleted as part of self._sysdb.delete_collection.

Why do you think that this would cause extra calls in the distributed backend?

[tazarov]

@rohitcpbot, is this your concern about the call to distributed sysdb?