Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA]: Add bulk ingest to vdb task #164

Open
jperez999 opened this issue Oct 15, 2024 · 0 comments · May be fixed by #190
Open

[FEA]: Add bulk ingest to vdb task #164

jperez999 opened this issue Oct 15, 2024 · 0 comments · May be fixed by #190
Assignees
Labels
feature request New feature or request

Comments

@jperez999
Copy link
Contributor

jperez999 commented Oct 15, 2024

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Currently preventing usage

Please provide a clear description of problem this feature solves

This feature will allow for bulk ingest to milvus using the vdb task when adding a large corpus of data to an index. This will decrease the time it takes to ingest large amounts of data into an index, as compared to the currently used streaming process. This will also require a minio sink addition where we save the embeddings to minio to be uploaded during bulk ingest.

Describe the feature, and optionally a solution or implementation and any alternatives

We will add a boolean to the VDBTask that will activate the use of bulk ingest if the users want to deploy using this strategy. Otherwise the current ingestion strategy will continue to be used.

i.e.
VDBTask(bulk=True)

Additional context

No response

@jperez999 jperez999 added the feature request New feature or request label Oct 15, 2024
This was referenced Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant