Recommended RAG setup for EXO

[sudosar]

Hi there

Excited to get EXO working finally.

what's the easiest way to setup local RAG with exo (e.g. on MacOS) across multiple devices ? Any recommendations?

[dtnewman]

EXO is compatible with the OpenAI API, so any advice you come across for running RAG against OpenAI's text models will similarly apply to EXO. As far as running "across multiple devices", that depends what you mean. The inference is done across multiple devices, but most likely, for RAG, you'd want a single computer to query for the context to insert into the prompts, since that process is not very computationally expensive. The advantage of multiple devices is that you can spread the model across them, so for large models that use lots of memory, you can run them across multiple devices.

[justushar]

In such cases, for the digested corpus can be kept on a single machine in the required db and get calls can be made to use any retriever to put the information into the network. This would allow for exo to share the inferences between all other machines and utilize RAG architecture.