Ability to set ollama serve url

[thawkins]

Feature request

The ollama integration assumes that all models are served on "localhost:11434", if the ollama service is hosted on a different machine, the integration will fail.

Can we add an environment variable that if present overrides this url, so the correct url for the ollama server can be set.

Motivation

In My setup ollama sits on a separate machine that is resourced for serving LLMs.

Your contribution

I'm afraid I don't have any knowledge of python, go, cpp and rust only.

[dosubot]

🤖

Hello @thawkins,

Nice to meet you! I'm Dosu, a bot here to assist you with LangChain related issues and questions. While we await a human maintainer, feel free to ask me anything about the project. I'm here to make your contributor journey smoother. Let's solve some bugs together!

Thank you for your feature request, thawkins. I understand that you want to set a custom URL for the ollama server when it's hosted on a different machine. This is indeed a valid use case and I appreciate you bringing it up.

Currently, the LangChain framework allows setting custom URLs for external services like ollama by setting the base_url attribute of the _OllamaCommon class. This attribute is used to construct the API URL for the ollama service. Here is the relevant code:

class _OllamaCommon(BaseLanguageModel):
    base_url: str = "http://localhost:11434"
    """Base url the model is hosted under."""

However, I understand that this might not be the most convenient way to set the URL, especially if you're not familiar with Python.

To address your request, we can add an environment variable to the LangChain framework to set a custom URL for the ollama server. Here's how it can be done:

import os

class _OllamaCommon(BaseLanguageModel):
    base_url: str = os.getenv('OLLAMA_SERVER_URL', "http://localhost:11434")
    ...

In this code, os.getenv('OLLAMA_SERVER_URL', "http://localhost:11434") will try to get the value of the OLLAMA_SERVER_URL environment variable. If it's not set, it will default to "http://localhost:11434".

You can set the OLLAMA_SERVER_URL environment variable in your operating system. The way to do this depends on your operating system. For example, in a Unix-based system like Linux or MacOS, you can set it in the terminal like this:

export OLLAMA_SERVER_URL="http://your_custom_url"

In Windows, you can set it in the command prompt like this:

set OLLAMA_SERVER_URL=http://your_custom_url

Remember to replace "http://your_custom_url" with the actual URL of your ollama server.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Best regards,
Dosu

Sources

• libs/community/langchain_community/llms/ollama.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[chyroc]

try:

ChatOllama(base_url='your ollama server host')

[NatEvory]

You can use the base_url parameter when instantiating the model, something like this
ollama_llm = Ollama( model="llama2", base_url="http://OLLAMA_HOST:PORT" )

It's not documented but I found it in the code and tried it out, it worked for me. There are a bunch of other undocumented parameters in that file as well (top_k, temperature, etc)
https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/llms/ollama.py

[mfreeman451]

this should be closed/resolved..

[tdimitch]

I'm using
from langchain_community.llms.ollama import Ollama Ollama(model="llama3.1", base_url="http://localhost:11436")
But I'm getting:
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=11436): Max retries exceeded with url: /api/generate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001EDB6ACBE30>: Failed to establish a new connection: [WinError 10061] No se puede establecer una conexión ya que el equipo de destino denegó expresamente dicha conexión'))

When I try to serve ollama from PowerShell in Windows, if I want to modify the host i have to:

$env:OLLAMA_HOST="0.0.0.0:11436"
ollama serve

In order to modify the default host, since I see no way of modify it with flags.

I'm trying to see if I can set the OLLAMA_HOST variable from the Python script itself, but with no luck so far. Nor I'm able to check how langchain is serving the api.