feat: litellm-add-models - #395
Conversation
📝 WalkthroughWalkthroughThe LiteLLM configuration adds ChangesLiteLLM model routing
Estimated code review effort: 2 (Simple) | ~10 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@agent/litellm/config.yaml`:
- Line 22: Update the fallback mapping in the configuration so
deepseek-v4-flash:free is the primary model and hy3:free is included in its
fallback list. Remove the inverse hy3:free-to-DeepSeek mapping and preserve the
existing LiteLLM mapping structure.
- Around line 54-57: Update the hy3:free entry in the LiteLLM model
configuration so it targets the provider-specific free model ID exposed by the
configured provider and uses that provider’s matching credential. If no provider
endpoint returns complete hy3:free completions, remove the route instead of
retaining the openai/hy3:free mapping with the nous credential.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c3002a17-29b0-4986-b333-74d271277e29
📒 Files selected for processing (1)
agent/litellm/config.yaml
| default_fallbacks: ["deepseek-v4-flash:free"] | ||
| fallbacks: | ||
| - deepseek-v4-flash:free: ["mimo-v2.5:free"] | ||
| - hy3:free: ["deepseek-v4-flash:free"] |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files:"
git ls-files | rg '(^|/)agent/litellm/config\.yaml$|litellm' || true
echo
echo "config excerpt:"
if [ -f agent/litellm/config.yaml ]; then
nl -ba agent/litellm/config.yaml | sed -n '1,120p'
fi
echo
echo "Search fallback keys:"
rg -n "fallbacks|fallback|deepseek-v4-flash|hy3:free|mimo-v2.5" .Repository: yxtay/docker-stacks
Length of output: 248
🌐 Web query:
LiteLLM config fallbacks keys list semantics primary model fallback models
💡 Result:
In LiteLLM, fallbacks provide a mechanism for automatic failover, allowing the system to switch to alternative models when a primary model request fails [1][2]. Configuration is typically handled within the Router settings or directly in completion calls [3][4]. Key Fallback Types and Semantics LiteLLM supports specialized fallbacks mapped to specific error types [4][5]: 1. fallbacks: Handles general errors, such as rate limits (429), server errors (5xx), and connection timeouts [1][5]. 2. context_window_fallbacks: Triggered specifically by ContextWindowExceededError when input exceeds a model's token limit (requires enable_pre_call_checks: true) [5]. 3. content_policy_fallbacks: Triggered by ContentPolicyViolationError [1][5]. Configuration Structure These fallbacks are defined as lists of dictionaries in your configuration (YAML or Python) [1][5]. Each dictionary maps a primary model name to a list of fallback model names [5]. Example YAML structure: litellm_settings: fallbacks: [{"primary-model-name": ["fallback-model-1", "fallback-model-2"]}] context_window_fallbacks: [{"primary-model-name": ["larger-context-model"]}] content_policy_fallbacks: [{"primary-model-name": ["safe-alternative-model"]}] Operational Semantics - Primary Model Execution: LiteLLM first attempts the request with the primary model [3][2]. - Retry Logic: If the request fails, the system first exhausts configured num_retries within the primary model's deployment group [1][2]. - Fallback Trigger: If all retries fail, the router sequentially attempts the models defined in the respective fallback list [1][5]. - Model Availability: Every model referenced in a fallback configuration must exist within your model_list [5]. - Granularity: Fallback configurations can be applied globally via proxy settings, or more granularly at the Team or API Key level [6][7]. Additionally, you can use the /v1/models endpoint with the include_metadata=true parameter to inspect configured fallbacks for specific models [8].
Citations:
- 1: https://docs.litellm.ai/docs/proxy/reliability
- 2: https://docs.litellm.ai/docs/router_architecture
- 3: https://docs.litellm.ai/docs/completion/reliable_completions
- 4: https://deepwiki.com/BerriAI/litellm/7.1-fallbacks-and-retries
- 5: https://gingerlabs.ai/blog/litellm-router-setup-guide
- 6: https://docs.litellm.ai/docs/proxy/keys_teams_router_settings
- 7: https://docs.litellm.ai/docs/routing-load-balancing
- 8: https://docs.litellm.ai/docs/proxy/model_access
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "config excerpt:"
awk '{printf "%6d %s\n", NR, $0}' agent/litellm/config.yaml | sed -n '1,130p'
echo
echo "Search fallback keys:"
rg -n "fallbacks|fallback|deepseek-v4-flash|hy3:free|mimo-v2.5" agent/litellm/config.yaml .Repository: yxtay/docker-stacks
Length of output: 7861
Add hy3:free to the DeepSeek fallback chain.
Line 22 makes hy3:free the primary fallback target and only falls it back to DeepSeek. LiteLLM fallback entries use the mapping key as the primary model and the list as fallback models.
If deepseek-v4-flash:free should fall back to hy3:free, use this mapping:
Proposed configuration
- - deepseek-v4-flash:free: ["mimo-v2.5:free"]
- - hy3:free: ["deepseek-v4-flash:free"]
+ - deepseek-v4-flash:free: ["mimo-v2.5:free", "hy3:free"]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@agent/litellm/config.yaml` at line 22, Update the fallback mapping in the
configuration so deepseek-v4-flash:free is the primary model and hy3:free is
included in its fallback list. Remove the inverse hy3:free-to-DeepSeek mapping
and preserve the existing LiteLLM mapping structure.
| - model_name: hy3:free | ||
| litellm_params: | ||
| model: openai/hy3:free | ||
| litellm_credential_name: nous |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
check_model() {
local api_key="$1"
local endpoint="$2"
local model="$3"
curl -fsS \
-H "Authorization: Bearer ${api_key}" \
"${endpoint}" |
jq -e --arg model "${model}" '.data[]? | select(.id == $model)'
}
check_model "${NOUS_API_KEY:?}" \
"https://inference-api.nousresearch.com/v1/models" \
"hy3:free"
check_model "${OPENCODE_GO_API_KEY:?}" \
"https://opencode.ai/zen/go/v1/models" \
"hy3"Repository: yxtay/docker-stacks
Length of output: 212
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)config\.ya?ml$|litellm|agent/|README|docs|Dockerfile|compose|environment' || true
echo
echo "== agent/litellm/config.yaml outline/sections =="
if [ -f agent/litellm/config.yaml ]; then
wc -l agent/litellm/config.yaml
cat -n agent/litellm/config.yaml | sed -n '1,140p'
fi
echo
echo "== references to hy3, nous, openrouter, opencode, config provider model =="
rg -n -C 3 'hy3|hy3:free|tencent/hy3|Nous|nou|opencode|openrouter|litellm_credential_name|model_name:|litellm_params:' . --glob '!node_modules' --glob '!dist' --glob '!build' 2>/dev/null || trueRepository: yxtay/docker-stacks
Length of output: 12744
🌐 Web query:
OpenRouter Tencent Hy3 free model tencent/hy3:free
💡 Result:
The tencent/hy3:free model on OpenRouter was a time-limited promotional offering [1][2][3]. It provided free access to the 295B-parameter Tencent Hy3 model for approximately two weeks, with the promotional window ending on July 21, 2026 [2][3]. Following the conclusion of this free period, the tencent/hy3:free route is no longer available as a free tier. Access to the standard Hy3 model (tencent/hy3) continues to be available through OpenRouter at standard paid rates, which are currently set at $0.14 per million input tokens and $0.58 per million output tokens [2][4]. Tencent Hy3 itself remains an open-weights model, with weights available on platforms such as Hugging Face for those who wish to host the model independently [5][6]. For users continuing to use the OpenRouter API, standard integration methods remain the same, requiring only the updated model slug and appropriate billing configuration [2][4].
Citations:
- 1: https://www.ayautomate.com/free-models/tencent-hy3
- 2: https://www.techtimes.com/articles/320292/20260713/tencent-hy3-tops-openrouter-charts-base-url-swap-runs-it-free-codex-cli.htm
- 3: https://blockchain.news/ainews/tencent-hy3-launches-free-access-on-openrouter
- 4: https://openrouter.ai/tencent/hy3
- 5: https://hy.tencent.com/research/hy3?langVersion=en
- 6: https://huggingface.co/tencent/Hy3
🌐 Web query:
LiteLLM config provider model model_name model credential_name compatibility
💡 Result:
In LiteLLM, you can decouple model configurations from sensitive API credentials by using named credentials. This allows you to define credentials once and reuse them across multiple models [1]. ### Configuration using litellm_credential_name In your config.yaml file, you can reference a predefined credential using the litellm_credential_name parameter within the litellm_params block [2]. Example config.yaml: yaml model_list: - model_name: gpt-4o litellm_params: model: azure/gpt-4o litellm_credential_name: my-azure-credential # Reference name defined in the credentials table credential_list: - credential_name: my-azure-credential credential_values: api_key: os.environ/AZURE_API_KEY api_base: os.environ/AZURE_API_BASE api_version: "2024-02-01" credential_info: custom_llm_provider: "azure" ### Key Components 1. model_list: Contains your model definitions. By including litellm_credential_name, LiteLLM retrieves the associated keys and base URLs from the credential store [2]. 2. credential_list: Defines the named credentials, their sensitive values (which support os.environ/ for environment variable injection), and provider-specific metadata (e.g., custom_llm_provider) [2][3]. ### Credential Management * Persistence: When models and credentials are saved to the LiteLLM proxy database (via the UI or API), credentials are encrypted at rest using a secret key (configured via LITELLM_SALT_KEY) [1][4]. * CLI Support: You can manage these credentials using the LiteLLM CLI. For example, to create a credential via CLI: bash lite credentials create azure-prod \ --info '{"custom_llm_provider": "azure"}' \ --values '{"api_key": "sk-123", "api_base": "https://prod.azure.openai.com"}' * Advanced Routing: LiteLLM supports complex credential routing where credentials can be overridden per team or project using model_config metadata in the proxy's team/project settings [5][6]. This allows different teams to use the same model name while routing to different underlying endpoints or credentials [5][6]. Note: Ensure your LITELLM_SALT_KEY is set before adding credentials to your database, as changing this key later will prevent decryption of existing credentials [1][4].
Citations:
- 1: https://docs.litellm.ai/docs/proxy/model_management
- 2: https://docs.litellm.ai/docs/proxy/configs
- 3: https://github.com/BerriAI/litellm/blob/main/litellm/proxy/client/cli/README.md
- 4: https://docs.litellm.ai/docs/proxy/docker_quick_start
- 5: https://docs.litellm.ai/docs/proxy/credential_routing
- 6: feat(proxy): add credential overrides per team/project via model_config metadata BerriAI/litellm#24438
🌐 Web query:
LiteLLM proxy config.yaml model_list litellm_params model prefix provider model_name hy3:free
💡 Result:
In the LiteLLM proxy config.yaml, the model_list section defines how external requests map to specific model deployments [1][2]. Here is how the requested parameters function: model_name: This is the user-facing alias or model name that clients use when making requests to the LiteLLM proxy [1][3]. litellm_params: This section contains all parameters accepted by the litellm.completion function [1][4]. The sub-parameter model within litellm_params specifies the actual model string (often including a provider prefix) sent to the underlying API [1][3]. Provider Prefix: LiteLLM uses prefixes in the litellm_params.model field to identify the provider and ensure correct routing [1][5]. For example, using openai/gpt-4 tells LiteLLM to use the OpenAI provider, while azure/gpt-4 routes to Azure OpenAI [1][4]. Regarding the specific model name hy3:free, if you intend to use this as a model name, you would configure it in the model_list as follows: model_list: - model_name: hy3:free litellm_params: model: / api_key: os.environ/<YOUR_API_KEY_ENV_VAR> Example Configuration: To route a request to a hypothetical provider (e.g., vllm) using your model name, your configuration would look like this [5][4]: model_list: - model_name: hy3:free litellm_params: model: vllm/facebook/opt-125m api_base: http://your-api-endpoint:4000/v1 Key Points: - model_name is the alias clients use [1]. - litellm_params.model is the actual model identifier with the required provider prefix [1]. - You can use environment variables for sensitive data like API keys by prefixing them with os.environ/ [1][6].
Citations:
- 1: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/configs.md
- 2: https://docs.litellm.ai/docs/proxy/configs
- 3: https://github.com/BerriAI/litellm/blob/v1.82.6.dev2/docs/my-website/docs/proxy/configs.md
- 4: https://berriai-litellm.mintlify.app/proxy/configs
- 5: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/providers/vllm.md
- 6: https://github.com/BerriAI/litellm/blob/fb39683521702145833a502c90127b0600886a9e/proxy_server_config.yaml
🌐 Web query:
headroom service git yxtay docker-stacks headroom 8787 hy3:free
💡 Result:
Your query combines three distinct, unrelated technical topics: a Headroom service configuration, a GitHub repository for Docker stacks, and a specific model identifier for Tencent's Hy3. Headroom Service and Port 8787 Headroom is a tool for compressing and managing AI agent tool outputs, logs, and files [1]. It commonly runs a proxy service on port 8787 to manage these agent sessions [2][3]. You can run this service via a Docker-native setup using the repository's provided docker-compose.native.yml file [4][3]. This setup ensures the proxy remains stable and restarts automatically [2][3]: export HEADROOM_HOST_HOME="$HOME" export HEADROOM_WORKSPACE="$PWD" docker compose -f docker/docker-compose.native.yml up -d proxy yxtay/docker-stacks This GitHub repository (github.com/yxtay/docker-stacks) contains a collection of automation scripts and configurations for managing various services using Docker stacks [5]. The repository focuses on home-lab infrastructure, including reverse proxying via Caddy, authentication via TinyAuth, and container management utilities like Portainer [5]. It is not directly related to Headroom's internal logic. hy3:free "hy3:free" refers to a specific model identifier for Tencent's Hy3, a 295B parameter, open-weights Mixture-of-Experts (MoE) model [6][7]. As of July 2026, it was made available via OpenRouter with a limited-time free tier [8][7]. Users can access it by configuring an OpenAI-compatible API client with the following settings [8][7]: Model: tencent/hy3:free Provider/Base URL: https://openrouter.ai/api/v1
Citations:
- 1: https://github.com/headroomlabs-ai/headroom
- 2: https://github.com/voly-codes/voly/blob/main/headroom/wiki/persistent-installs.md
- 3: https://headroomlabs-ai.github.io/headroom/persistent-installs/
- 4: https://github.com/voly-codes/voly/blob/main/headroom/docker/docker-compose.native.yml
- 5: https://github.com/yxtay/docker-stacks
- 6: https://hy.tencent.com/research/hy3?langVersion=en
- 7: https://www.ayautomate.com/free-models/tencent-hy3
- 8: https://agentos.guide/free-codex
Route the free Hy3 route to the provider that exposes it.
model: openai/hy3:free with litellm_credential_name: nous should only be kept if the headroom proxy forwards that model to an endpoint that returns complete completions for hy3:free. Otherwise change the model to the provider-specific free model ID and use the matching credential, or remove the route before merge.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@agent/litellm/config.yaml` around lines 54 - 57, Update the hy3:free entry in
the LiteLLM model configuration so it targets the provider-specific free model
ID exposed by the configured provider and uses that provider’s matching
credential. If no provider endpoint returns complete hy3:free completions,
remove the route instead of retaining the openai/hy3:free mapping with the nous
credential.
1 participant
Summary by CodeRabbit