[Query] How to set up for local models

[bharattrader]

Hi ,
Sorry for this maybe n00b question, I generally serve models using llama.cpp, ollama or textgenui on OpenAI compatible interface. How can I set up e11, to make it hit these services?

Thanks,

[bharattrader]

Got, and example examples/ollama_example.py, Thanks

[QueryType]

Just for completeness, there is a problem with the example code. So you should try:

def set_api():
    _ = load_dotenv(find_dotenv()) # read local .env file
    return os.environ['OPENAI_API_KEY']

set_api()

ell.models.ollama.register(base_url="http://10.0.0.4:7862/v1")
@ell.complex(model="mistral-nemo-q6-k:latest", temperature=0.7)
def chat_bot(message_history: List[Message]) -> List[Message]:
    return [
        ell.system("You are a friendly chatbot. Engage in casual conversation."),
    ] + message_history

[rosmur]

@QueryType why does the OPENAI_API_KEY need to be loaded for ollama to work??

[QueryType]

@QueryType why does the OPENAI_API_KEY need to be loaded for ollama to work??

You can just pass a dummy key.

import os
import ell.models
import ell.models.ollama
from openai import OpenAI
from dotenv import load_dotenv, find_dotenv
import ell


def set_api():
    _ = load_dotenv(find_dotenv()) # read local .env file
    return os.environ['OPENAI_API_KEY']

set_api()
LOCAL_MODEL = "llama3.1:latest"
LOCAL_URL = "http://10.0.0.4:7862/v1"
ell.models.ollama.register(base_url=LOCAL_URL)
ollama_local_client = OpenAI(base_url=LOCAL_URL, api_key="local-llm")

@ell.simple(model=LOCAL_MODEL, client=ollama_local_client, temperature=0.7)
def hello(name: str):
    """You are a helpful assistant.""" # System prompt
    return f"Say hello to {name}!" # User prompt

greeting = hello("QueryType")
print(greeting)

[rosmur]

Hmm that still doesnt explain why you need load anything at all for OPENAI when you are using Ollama..