Skip to content

feat: litellm-add-models #395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yxtay merged 1 commit into from
Jul 31, 2026
Merged

feat: litellm-add-models #395

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions agent/litellm/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ router_settings:
default_fallbacks: ["deepseek-v4-flash:free"]
fallbacks:
- deepseek-v4-flash:free: ["mimo-v2.5:free"]
- hy3:free: ["deepseek-v4-flash:free"]

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Files:"
git ls-files | rg '(^|/)agent/litellm/config\.yaml$|litellm' || true

echo
echo "config excerpt:"
if [ -f agent/litellm/config.yaml ]; then
  nl -ba agent/litellm/config.yaml | sed -n '1,120p'
fi

echo
echo "Search fallback keys:"
rg -n "fallbacks|fallback|deepseek-v4-flash|hy3:free|mimo-v2.5" .

Repository: yxtay/docker-stacks

Length of output: 248

🌐 Web query:

LiteLLM config fallbacks keys list semantics primary model fallback models

💡 Result:

In LiteLLM, fallbacks provide a mechanism for automatic failover, allowing the system to switch to alternative models when a primary model request fails [1][2]. Configuration is typically handled within the Router settings or directly in completion calls [3][4]. Key Fallback Types and Semantics LiteLLM supports specialized fallbacks mapped to specific error types [4][5]: 1. fallbacks: Handles general errors, such as rate limits (429), server errors (5xx), and connection timeouts [1][5]. 2. context_window_fallbacks: Triggered specifically by ContextWindowExceededError when input exceeds a model's token limit (requires enable_pre_call_checks: true) [5]. 3. content_policy_fallbacks: Triggered by ContentPolicyViolationError [1][5]. Configuration Structure These fallbacks are defined as lists of dictionaries in your configuration (YAML or Python) [1][5]. Each dictionary maps a primary model name to a list of fallback model names [5]. Example YAML structure: litellm_settings: fallbacks: [{"primary-model-name": ["fallback-model-1", "fallback-model-2"]}] context_window_fallbacks: [{"primary-model-name": ["larger-context-model"]}] content_policy_fallbacks: [{"primary-model-name": ["safe-alternative-model"]}] Operational Semantics - Primary Model Execution: LiteLLM first attempts the request with the primary model [3][2]. - Retry Logic: If the request fails, the system first exhausts configured num_retries within the primary model's deployment group [1][2]. - Fallback Trigger: If all retries fail, the router sequentially attempts the models defined in the respective fallback list [1][5]. - Model Availability: Every model referenced in a fallback configuration must exist within your model_list [5]. - Granularity: Fallback configurations can be applied globally via proxy settings, or more granularly at the Team or API Key level [6][7]. Additionally, you can use the /v1/models endpoint with the include_metadata=true parameter to inspect configured fallbacks for specific models [8].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "config excerpt:"
awk '{printf "%6d  %s\n", NR, $0}' agent/litellm/config.yaml | sed -n '1,130p'

echo
echo "Search fallback keys:"
rg -n "fallbacks|fallback|deepseek-v4-flash|hy3:free|mimo-v2.5" agent/litellm/config.yaml .

Repository: yxtay/docker-stacks

Length of output: 7861

Add hy3:free to the DeepSeek fallback chain.

Line 22 makes hy3:free the primary fallback target and only falls it back to DeepSeek. LiteLLM fallback entries use the mapping key as the primary model and the list as fallback models.

If deepseek-v4-flash:free should fall back to hy3:free, use this mapping:

Proposed configuration 
-    - deepseek-v4-flash:free: ["mimo-v2.5:free"]
-    - hy3:free: ["deepseek-v4-flash:free"]
+    - deepseek-v4-flash:free: ["mimo-v2.5:free", "hy3:free"]
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@agent/litellm/config.yaml` at line 22, Update the fallback mapping in the
configuration so deepseek-v4-flash:free is the primary model and hy3:free is
included in its fallback list. Remove the inverse hy3:free-to-DeepSeek mapping
and preserve the existing LiteLLM mapping structure.

- mimo-v2.5:free: ["deepseek-v4-flash:free"]

model_list:
Expand Down Expand Up @@ -50,6 +51,50 @@ model_list:
litellm_params:
model: groq/openai/gpt-oss-120b

- model_name: hy3:free
litellm_params:
model: openai/hy3:free
litellm_credential_name: nous
Comment on lines +54 to +57

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

check_model() {
  local api_key="$1"
  local endpoint="$2"
  local model="$3"

  curl -fsS \
    -H "Authorization: Bearer ${api_key}" \
    "${endpoint}" |
    jq -e --arg model "${model}" '.data[]? | select(.id == $model)'
}

check_model "${NOUS_API_KEY:?}" \
  "https://inference-api.nousresearch.com/v1/models" \
  "hy3:free"

check_model "${OPENCODE_GO_API_KEY:?}" \
  "https://opencode.ai/zen/go/v1/models" \
  "hy3"

Repository: yxtay/docker-stacks

Length of output: 212

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "== files =="
git ls-files | rg '(^|/)config\.ya?ml$|litellm|agent/|README|docs|Dockerfile|compose|environment' || true

echo
echo "== agent/litellm/config.yaml outline/sections =="
if [ -f agent/litellm/config.yaml ]; then
  wc -l agent/litellm/config.yaml
  cat -n agent/litellm/config.yaml | sed -n '1,140p'
fi

echo
echo "== references to hy3, nous, openrouter, opencode, config provider model =="
rg -n -C 3 'hy3|hy3:free|tencent/hy3|Nous|nou|opencode|openrouter|litellm_credential_name|model_name:|litellm_params:' . --glob '!node_modules' --glob '!dist' --glob '!build' 2>/dev/null || true

Repository: yxtay/docker-stacks

Length of output: 12744

🌐 Web query:

OpenRouter Tencent Hy3 free model tencent/hy3:free

💡 Result:

The tencent/hy3:free model on OpenRouter was a time-limited promotional offering [1][2][3]. It provided free access to the 295B-parameter Tencent Hy3 model for approximately two weeks, with the promotional window ending on July 21, 2026 [2][3]. Following the conclusion of this free period, the tencent/hy3:free route is no longer available as a free tier. Access to the standard Hy3 model (tencent/hy3) continues to be available through OpenRouter at standard paid rates, which are currently set at $0.14 per million input tokens and $0.58 per million output tokens [2][4]. Tencent Hy3 itself remains an open-weights model, with weights available on platforms such as Hugging Face for those who wish to host the model independently [5][6]. For users continuing to use the OpenRouter API, standard integration methods remain the same, requiring only the updated model slug and appropriate billing configuration [2][4].

Citations:

🌐 Web query:

LiteLLM config provider model model_name model credential_name compatibility

💡 Result:

In LiteLLM, you can decouple model configurations from sensitive API credentials by using named credentials. This allows you to define credentials once and reuse them across multiple models [1]. ### Configuration using litellm_credential_name In your config.yaml file, you can reference a predefined credential using the litellm_credential_name parameter within the litellm_params block [2]. Example config.yaml: yaml model_list: - model_name: gpt-4o litellm_params: model: azure/gpt-4o litellm_credential_name: my-azure-credential # Reference name defined in the credentials table credential_list: - credential_name: my-azure-credential credential_values: api_key: os.environ/AZURE_API_KEY api_base: os.environ/AZURE_API_BASE api_version: "2024-02-01" credential_info: custom_llm_provider: "azure" ### Key Components 1. model_list: Contains your model definitions. By including litellm_credential_name, LiteLLM retrieves the associated keys and base URLs from the credential store [2]. 2. credential_list: Defines the named credentials, their sensitive values (which support os.environ/ for environment variable injection), and provider-specific metadata (e.g., custom_llm_provider) [2][3]. ### Credential Management * Persistence: When models and credentials are saved to the LiteLLM proxy database (via the UI or API), credentials are encrypted at rest using a secret key (configured via LITELLM_SALT_KEY) [1][4]. * CLI Support: You can manage these credentials using the LiteLLM CLI. For example, to create a credential via CLI: bash lite credentials create azure-prod \ --info '{"custom_llm_provider": "azure"}' \ --values '{"api_key": "sk-123", "api_base": "https://prod.azure.openai.com"}' * Advanced Routing: LiteLLM supports complex credential routing where credentials can be overridden per team or project using model_config metadata in the proxy's team/project settings [5][6]. This allows different teams to use the same model name while routing to different underlying endpoints or credentials [5][6]. Note: Ensure your LITELLM_SALT_KEY is set before adding credentials to your database, as changing this key later will prevent decryption of existing credentials [1][4].

Citations:

🌐 Web query:

LiteLLM proxy config.yaml model_list litellm_params model prefix provider model_name hy3:free

💡 Result:

In the LiteLLM proxy config.yaml, the model_list section defines how external requests map to specific model deployments [1][2]. Here is how the requested parameters function: model_name: This is the user-facing alias or model name that clients use when making requests to the LiteLLM proxy [1][3]. litellm_params: This section contains all parameters accepted by the litellm.completion function [1][4]. The sub-parameter model within litellm_params specifies the actual model string (often including a provider prefix) sent to the underlying API [1][3]. Provider Prefix: LiteLLM uses prefixes in the litellm_params.model field to identify the provider and ensure correct routing [1][5]. For example, using openai/gpt-4 tells LiteLLM to use the OpenAI provider, while azure/gpt-4 routes to Azure OpenAI [1][4]. Regarding the specific model name hy3:free, if you intend to use this as a model name, you would configure it in the model_list as follows: model_list: - model_name: hy3:free litellm_params: model: / api_key: os.environ/<YOUR_API_KEY_ENV_VAR> Example Configuration: To route a request to a hypothetical provider (e.g., vllm) using your model name, your configuration would look like this [5][4]: model_list: - model_name: hy3:free litellm_params: model: vllm/facebook/opt-125m api_base: http://your-api-endpoint:4000/v1 Key Points: - model_name is the alias clients use [1]. - litellm_params.model is the actual model identifier with the required provider prefix [1]. - You can use environment variables for sensitive data like API keys by prefixing them with os.environ/ [1][6].

Citations:

🌐 Web query:

headroom service git yxtay docker-stacks headroom 8787 hy3:free

💡 Result:

Your query combines three distinct, unrelated technical topics: a Headroom service configuration, a GitHub repository for Docker stacks, and a specific model identifier for Tencent's Hy3. Headroom Service and Port 8787 Headroom is a tool for compressing and managing AI agent tool outputs, logs, and files [1]. It commonly runs a proxy service on port 8787 to manage these agent sessions [2][3]. You can run this service via a Docker-native setup using the repository's provided docker-compose.native.yml file [4][3]. This setup ensures the proxy remains stable and restarts automatically [2][3]: export HEADROOM_HOST_HOME="$HOME" export HEADROOM_WORKSPACE="$PWD" docker compose -f docker/docker-compose.native.yml up -d proxy yxtay/docker-stacks This GitHub repository (github.com/yxtay/docker-stacks) contains a collection of automation scripts and configurations for managing various services using Docker stacks [5]. The repository focuses on home-lab infrastructure, including reverse proxying via Caddy, authentication via TinyAuth, and container management utilities like Portainer [5]. It is not directly related to Headroom's internal logic. hy3:free "hy3:free" refers to a specific model identifier for Tencent's Hy3, a 295B parameter, open-weights Mixture-of-Experts (MoE) model [6][7]. As of July 2026, it was made available via OpenRouter with a limited-time free tier [8][7]. Users can access it by configuring an OpenAI-compatible API client with the following settings [8][7]: Model: tencent/hy3:free Provider/Base URL: https://openrouter.ai/api/v1

Citations:

Route the free Hy3 route to the provider that exposes it.

model: openai/hy3:free with litellm_credential_name: nous should only be kept if the headroom proxy forwards that model to an endpoint that returns complete completions for hy3:free. Otherwise change the model to the provider-specific free model ID and use the matching credential, or remove the route before merge.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@agent/litellm/config.yaml` around lines 54 - 57, Update the hy3:free entry in
the LiteLLM model configuration so it targets the provider-specific free model
ID exposed by the configured provider and uses that provider’s matching
credential. If no provider endpoint returns complete hy3:free completions,
remove the route instead of retaining the openai/hy3:free mapping with the nous
credential.


- model_name: deepseek-v4-flash:high
litellm_params:
model: openai/deepseek-v4-flash
reasoning_effort: high
litellm_credential_name: opencode_go

- model_name: deepseek-v4-flash:max
litellm_params:
model: openai/deepseek-v4-flash
reasoning_effort: max
litellm_credential_name: opencode_go

- model_name: deepseek-v4-pro:high
litellm_params:
model: openai/deepseek-v4-pro
reasoning_effort: high
litellm_credential_name: opencode_go

- model_name: deepseek-v4-pro:max
litellm_params:
model: openai/deepseek-v4-pro
reasoning_effort: max
litellm_credential_name: opencode_go

- model_name: hy3
litellm_params:
model: openai/hy3
litellm_credential_name: opencode_go

- model_name: mimo-v2.5
litellm_params:
model: openai/mimo-v2.5
litellm_credential_name: opencode_go

- model_name: mimo-v2.5-pro
litellm_params:
model: openai/mimo-v2.5-pro
litellm_credential_name: opencode_go

- model_name: deepseek-v4-flash:free
litellm_params:
model: openai/deepseek-v4-flash-free
Expand Down
Loading