core: add new clean up strategy "scoped_full" to indexing #28505
+384
−33
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dosubot
bot
added
the
size:M
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
KeiichiHirobe
commented
Dec 4, 2024
eyurtsev
reviewed
Dec 11, 2024
| @@ -259,6 +256,11 @@ def index( | |||
| specify a custom vector_field: | |||
| upsert_kwargs={"vector_field": "embedding"} | |||
| .. versionadded:: 0.3.10 | |||
| scoped_full_cleanup: This argument will be valid only when `claneup` is Full. | |||
There was a problem hiding this comment.
How about we turn this into another clean up mode so we don't introduce another parameter that only works conditionally?
This looks like cleanup == 'full_scoped' or something like that.
And we'd document:
- In the important section -- that full_scoped is a solution to problem w/ batches being a best effort
- This solution keeps track of the source ids in memory (probably fine for most use cases in terms of memory consumption) -- would require parallelizing for 10M+ docs anyway
dosubot
bot
added
size:L
eyurtsev
reviewed
Dec 13, 2024
eyurtsev
reviewed
Dec 13, 2024
KeiichiHirobe
changed the title
KeiichiHirobe
changed the title
|
Pushed a minor doc-string update |
eyurtsev
approved these changes
Dec 13, 2024
dosubot
bot
added
the
lgtm
|
@KeiichiHirobe thank you looks great! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note that this PR is now Draft, so I didn't add change toaindexfunction and didn't add test codes for my change.After we have an agreement on the direction, I will add commits.
batch_sizeis very difficult to decide because setting a large number like >10000 will impact VectorDB and RecordManager, while setting a small number will delete records unnecessarily, leading to redundant work, as theIMPORTANTsection says.On the other hand, we can't use
fullbecause the loader returns just a subset of the dataset in our use case.I guess many people are in the same situation as us.
So, as one of the possible solutions for it, I would like to introduce a new argument,
scoped_full_cleanup.This argument will be valid only when
claneupis Full. If True, Full cleanup deletes all documents that haven't been updated AND that are associated with source ids that were seen during indexing. Default is False.This change keeps backward compatibility.