Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support automatically updating index when content is published #6

Open
tomusher opened this issue Nov 3, 2023 · 3 comments
Open

Support automatically updating index when content is published #6

tomusher opened this issue Nov 3, 2023 · 3 comments
Assignees

Comments

@tomusher
Copy link
Member

tomusher commented Nov 3, 2023

At the moment the only way to update the indexes is to run the update_vector_indexes management command. This gives developers control over when embedding API costs are incurred.

It would be nice if there was an option to add/update items in the index when they are published/saved.

Some considerations:

  • As this could get expensive on edit-heavy sites, it might make sense to have this feature disabled by default
  • There may be cause to only auto-update certain models rather than every registered model/index.
@tm-kn
Copy link
Member

tm-kn commented Dec 11, 2023

Should we have a way to run this on Celery given that this is an external API call and advise not to use it in the request/response cycle?

@Morsey187
Copy link
Collaborator

I've opened up a draft PR here #30, however theres a few points that need resolving before this can be completed.

Issue: Signals are part of the request cycle and updating indexes can be time consuming, we should add support for a task queue and consider whether we'd want to allow using these signals without one at all.
Issue: Currently requires rebuilding the whole index, instead of updating, we'd need to figure out:
Which indexes a model is in (so we can update the right indexes)
A way to remove documents from an index that match a given set of metadata (the object id and content type ID in this case)
An easier way to generate embeddings on a per-document level, instead of at the rebuild index stage

@tm-kn
Copy link
Member

tm-kn commented Feb 12, 2024

For reference, @mgax implemented this as part of his project, but we'd likely have the same issues as above. I don't think @mgax's solution rebuilds the whole index.

Maybe we need to make it a bit less automatic or perfect so we can merge this in some form?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants