Wildcard not expanding for ollama_chat to find models dynamically.

[sakoht]

It appears that the ollama model list is hard-coded to just one model, which might-or-might-not actually be locally available.

I'll follow with a patch that detects an Ollama install and instead uses the actual list available, and has an empty list if none are availaable.

My example config demonstrates the problem. For this config, the first three expand as expected, and the 4th explicity ollama model works, but for the final entry it doesn't expand:

model_list:
  - model_name: perplexity/*
    litellm_params:
      model: perplexity/*
      api_key: os.environ/PERPLEXITY_API_KEY

  - model_name: anthropic/*
    litellm_params:
      model: anthropic/*
      api_key: os.environ/ANTHROPIC_API_KEY
  
  - model_name: openai/*
    litellm_params:
      model: openai/*
      api_key: os.environ/OPENAI_API_KEY
 
  - model_name: ollama_chat/deepseek-r1:14b
    litellm_params:
      model: ollama_chat/deepseek-r1:14b
      api_base: "http://localhost:11434"
  
  - model_name: ollama_chat/*
    litellm_params:
      model: ollama_chat/*
      api_base: "http://localhost:11434"

Full launch output with --detailed_debug:

> litellm --config litellm-config/litellm-config.yaml --detailed_debug
(venv-openwebui) ssmith@Scott-Laptop-2024 ~/open-webui (0.5.7-ss1)
> litellm --config litellm-config/litellm-config.yaml --detailed_debug
/Users/ssmith/open-webui/backend/venv-openwebui/lib/python3.11/site-packages/pydantic/_internal/_config.py:341: UserWarning: Valid config keys have changed in V2:
* 'fields' has been removed
  warnings.warn(message, UserWarning)
INFO:     Started server process [24689]
INFO:     Waiting for application startup.
10:35:37 - LiteLLM Proxy:DEBUG: proxy_server.py:442 - litellm.proxy.proxy_server.py::startup() - CHECKING PREMIUM USER - False
10:35:37 - LiteLLM Proxy:DEBUG: litellm_license.py:98 - litellm.proxy.auth.litellm_license.py::is_premium() - ENTERING 'IS_PREMIUM' - LiteLLM License=None
10:35:37 - LiteLLM Proxy:DEBUG: litellm_license.py:107 - litellm.proxy.auth.litellm_license.py::is_premium() - Updated 'self.license_str' - None
10:35:37 - LiteLLM Proxy:DEBUG: proxy_server.py:453 - worker_config: {"model": null, "alias": null, "api_base": null, "api_version": "2024-07-01-preview", "debug": false, "detailed_debug": true, "temperature": null, "max_tokens": null, "request_timeout": null, "max_budget": null, "telemetry": true, "drop_params": false, "add_function_to_prompt": false, "headers": null, "save": false, "config": "litellm-config/litellm-config.yaml", "use_queue": false}

#------------------------------------------------------------#
#                                                            #
#               'A feature I really want is...'               #
#        https://github.com/BerriAI/litellm/issues/new        #
#                                                            #
#------------------------------------------------------------#

 Thank you for using LiteLLM! - Krrish & Ishaan



Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new


10:35:37 - LiteLLM Proxy:DEBUG: proxy_server.py:1500 - loaded config={
    "model_list": [
        {
            "model_name": "perplexity/*",
            "litellm_params": {
                "model": "perplexity/*",
                "api_key": "os.environ/PERPLEXITY_API_KEY"
            }
        },
        {
            "model_name": "anthropic/*",
            "litellm_params": {
                "model": "anthropic/*",
                "api_key": "os.environ/ANTHROPIC_API_KEY"
            }
        },
        {
            "model_name": "openai/*",
            "litellm_params": {
                "model": "openai/*",
                "api_key": "os.environ/OPENAI_API_KEY"
            }
        },
        {
            "model_name": "ollama_chat/deepseek-r1:14b",
            "litellm_params": {
                "model": "ollama_chat/deepseek-r1:14b",
                "api_base": "http://localhost:11434"
            }
        },
        {
            "model_name": "ollama_chat/*",
            "litellm_params": {
                "model": "ollama_chat/*",
                "api_base": "http://localhost:11434"
            }
        }
    ]
}
LiteLLM: Proxy initialized with Config, Set models:
    perplexity/*
    anthropic/*
    openai/*
    ollama_chat/deepseek-r1:14b
    ollama_chat/*
10:35:37 - LiteLLM:DEBUG: utils.py:4323 - Error getting model info: This model isn't mapped yet. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json
10:35:37 - LiteLLM Router:DEBUG: client_initalization_utils.py:468 - Initializing OpenAI Client for perplexity/*, Api Base:https://api.perplexity.ai, Api Key:pplx-5LC***************
10:35:37 - LiteLLM:DEBUG: utils.py:4323 - Error getting model info: This model isn't mapped yet. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json
10:35:37 - LiteLLM Router:DEBUG: client_initalization_utils.py:468 - Initializing OpenAI Client for openai/*, Api Base:None, Api Key:sk-proj-***************
10:35:37 - LiteLLM:DEBUG: utils.py:4323 - Error getting model info: OllamaError: Error getting model info for *. Set Ollama API Base via `OLLAMA_API_BASE` environment variable. Error: Server error '500 Internal Server Error' for url 'http://localhost:11434/api/show'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
10:35:37 - LiteLLM Router:DEBUG: router.py:4038 - 
Initialized Model List ['perplexity/*', 'anthropic/*', 'openai/*', 'ollama_chat/deepseek-r1:14b', 'ollama_chat/*']
10:35:37 - LiteLLM Router:INFO: router.py:613 - Routing strategy: simple-shuffle
10:35:37 - LiteLLM Router:DEBUG: router.py:505 - Intialized router with Routing strategy: simple-shuffle

Routing enable_pre_call_checks: False

Routing fallbacks: None

Routing content fallbacks: None

Routing context window fallbacks: None

Router Redis Caching=None

10:35:37 - LiteLLM Proxy:DEBUG: proxy_server.py:518 - prisma_client: None
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)

The key error seems to be:

10:35:37 - LiteLLM:DEBUG: utils.py:4323 - Error getting model info: OllamaError: Error getting model info for *. Set Ollama API Base via `OLLAMA_API_BASE` environment variable. Error: Server error '500 Internal Server Error' for url 'http://localhost:11434/api/show'

Note that the error about the env var was different at first, and I set that value:

export OLLAMA_API_BASE=http://localhost:11434

The error message is misleading b/c that URL does return correct model info, just not for "*".

[GTez]

I wrote a little script to get around this issue while the pull request works it's way through the system.

import requests
import yaml

# List your Ollama server base URLs here.
ollama_servers = [
    "http://ollamaserver01.local:11434",
    "http://ollamaserver02.local:11434"
]

rpm_value = 60  # Set desired rate limit (requests per minute)
model_entries = []

for server in ollama_servers:
    url = f"{server}/api/tags"
    print(f"Polling {url}...")
    try:
        response = requests.get(url)
        response.raise_for_status()
        data = response.json()
        # The endpoint returns a JSON with a key "models" that is a list of model objects.
        models = data.get("models", [])
        if not models:
            print(f"No models found at {url}")
        for model_obj in models:
            model_name_raw = model_obj.get("name")
            if not model_name_raw:
                continue
            # Prepend the required "ollama_chat/" prefix.
            full_model_name = f"ollama_chat/{model_name_raw}"
            entry = {
                "model_name": full_model_name,
                "litellm_params": {
                    "model": full_model_name,
                    "api_base": server,
                    "rpm": rpm_value
                }
            }
            model_entries.append(entry)
    except Exception as e:
        print(f"Error polling {url}: {e}")

# Build final config dictionary without master_key (handled via .env)
config = {
    "model_list": model_entries,
    "router_settings": {
        "routing_strategy": "least-busy",
        "num_retries": 2,
        "timeout": 30
    }
}

# Dump the YAML to stdout.
yaml_output = yaml.dump(config, sort_keys=False)
print("\nGenerated YAML config:")
print(yaml_output)

[sakoht]

I am guesing that the problem is known, and because there is not clarity on a more ideal solution, there is some concern about doing an incremental one?

My guess is that, long-term, we don't want the list of models hard-coded in the modules. But it's a way to move forward and handle the nuance later.

Like any change, there are many possibilities to do something bigger and better. My hope is to do the minimally intrusive change to fix the problem, then let folks who are responsible for core architecture to address.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[sakoht]

Before closing this, there is a PR to fix it I submitted just after. I need to sign a contributor agreement it seems. I'll do that and circle back.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.