Distributed Execution of index.search(query_vector, k) Without Loading Large Faiss Index in Memory

[ivishalanand]

I'm working with a large parquet file containing several million embeddings and I'm trying to perform index.search(query_vector, k) in a distributed fashion. The issue arises when dealing with the size of the faiss index, which is approximately 16GB. Given the size of the index, loading it entirely into memory is inefficient and presents a performance bottleneck.

In addition, I want to avoid having to convert the embeddings into pandas or npy formats for performance reasons.

I am looking for an efficient way to conduct the search without the need to load the entire faiss index into memory or convert the embeddings into pandas or npy.

Any guidance or suggestions would be greatly appreciated. Thank you.

[mlomeli1]

@ivishalanand please have a look at this page in the wiki: https://github.com/facebookresearch/faiss/wiki/Indexes-that-do-not-fit-in-RAM