Skip to content

feat: litellm auto router #376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yxtay merged 5 commits into from
Jul 29, 2026
Merged

feat: litellm auto router #376

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cd10fd4
feat: litellm-auto-router
yxtay Jul 29, 2026
80855ea
feat: auto router
yxtay Jul 29, 2026
17f46fe
Update config.yaml
yxtay Jul 29, 2026
75684c7
Update config.yaml
yxtay Jul 29, 2026
24b96d6
Update config.yaml
yxtay Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion agent/.env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ GROQ_API_KEY=
MISTRAL_API_KEY=
NOUS_API_KEY=
NVIDIA_API_KEY=
NVIDIA_NIM_API_KEY=
OPENCODE_ZEN_API_KEY=
OPENROUTER_API_KEY=
PARALLEL_API_KEY=
Expand Down
72 changes: 57 additions & 15 deletions agent/litellm/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,67 @@
general_settings:
store_model_in_db: true
enable_health_check_routing: true
background_health_checks: true
health_check_interval: 60
health_check_ignore_transient_errors: true
cancel_on_disconnect: true

litellm_settings:
drop_params: true
request_timeout: 60
json_logs: true
request_timeout: 60
drop_params: true

router_settings:
cooldown_time: 60
allowed_fails: 3
ttft_timeout: 30
enable_pre_call_checks: true
optional_pre_call_checks: [prompt_caching]
enable_anthropic_prompt_caching: true
default_litellm_params:
cache_control_injection_points:
- location: message
role: system
default_fallbacks: [complex, medium, simple]
fallbacks:
- reasoning: complex
- complex: medium
- medium: simple

Comment on lines +26 to 30

@coderabbitai coderabbitai Bot Jul 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== candidate file =="
if [ -f agent/litellm/config.yaml ]; then
  cat -n agent/litellm/config.yaml
else
  echo "agent/litellm/config.yaml not found"
fi

echo
echo "== fallback config occurrences =="
rg -n "fallbacks:|fallback:" . -g '*.yaml' -g '*.yml' -g '*.py' -g '*.js' -g '*.ts' || true

echo
echo "== litellm mentions =="
rg -n "litellm|LiteLLM|model_fallback|fallbacks" pyproject.toml requirements*.txt uv.lock poetry.lock agent .github 2>/dev/null || true

echo
echo "== package version declarations =="
for f in pyproject.toml requirements.txt requirements-dev.txt uv.lock poetry.lock; do
  [ -f "$f" ] && { echo "--- $f"; rg -n "litellm|version|python-dependencies" "$f" || true; }
done

Repository: yxtay/docker-stacks

Length of output: 7316

🌐 Web query:

LiteLLM proxy config fallbacks scalar list model_fallback documentation

💡 Result:

In LiteLLM Proxy, fallbacks are configured within the litellm_settings section of the configuration file. The fallbacks parameter accepts a list of objects, where each object maps a primary model (or model group) to a list of fallback models [1][2]. Configuration Structure The fallbacks are defined as a list of dictionaries [1][2]: litellm_settings: fallbacks: - primary-model-name: ["fallback-model-1", "fallback-model-2"] In this structure: - The key (e.g., primary-model-name) represents the model group being requested [1][3]. - The value is a scalar list (an array of strings) representing the models to attempt if the primary request fails [1][3]. Key Fallback Types LiteLLM supports different types of fallbacks, which can be configured similarly: - General Fallbacks: Standard failover if a model or provider fails (e.g., after retries) [4][5]. - Context Window Fallbacks: Specific fallbacks triggered when a context window limit is exceeded [2][4]. - Content Policy Fallbacks: Fallbacks triggered by content policy violations [4]. Example Configuration In your proxy config.yaml, you can define multiple types of fallbacks as follows [2]: litellm_settings: fallbacks: [{"gpt-4": ["gpt-3.5-turbo"]}] context_window_fallbacks: [{"gpt-4": ["gpt-4-32k"]}] Behavior and Management - Sequential Execution: Fallbacks are attempted in the order they are defined in the list [4][3]. - Retries: Fallbacks typically trigger after the configured number of retries (num_retries) for the primary model have been exhausted [1][6][4]. - Management Endpoints: You can also manage fallbacks dynamically via the Proxy API without modifying the config file directly, provided STORE_MODEL_IN_DB=True is set in your environment [4]. This includes endpoints like POST /fallback, GET /fallback/{model}, and DELETE /fallback/{model} [4]. Note: Earlier versions of LiteLLM used a deprecated model_fallbacks parameter in the code; modern proxy configurations should use the fallbacks key within litellm_settings [1][7].

Citations:

🌐 Web query:

LiteLLM v1.94.0 model_list_router model_fallbacks Router model_list_router yaml parser

💡 Result:

LiteLLM manages model routing and failover through the Router class and Proxy configuration, which allows for advanced control over model selection and reliability [1][2][3]. Router and Fallback Mechanisms The LiteLLM Router provides automatic failover across different model groups [1][4]. When a model deployment fails after reaching the configured num_retries, the Router initiates a fallback process [2][4]. 1. Deployment-Level Ordering: You can configure the order parameter within the model_list of your YAML configuration [2]. The Router will attempt deployments in ascending order (order 1, then order 2, etc.) before escalating to broader model-level fallbacks [2]. 2. Model-Level Fallbacks: Defined under router_settings.fallbacks in the YAML configuration, these specify which model groups to try if all deployments for the current model group fail [2][5]. The syntax typically maps a primary model group to a list of fallback groups, such as - gpt-4: ["gpt-3.5-turbo"] [2][5]. 3. Weighted Failover: When routing_strategy is set to simple-shuffle (the default), the Router can perform weighted failover, excluding failing deployments and retrying within the same model group before triggering cross-group fallbacks [1]. Configuration via YAML The YAML configuration supports detailed control over routing behavior [3][6]. Key sections include: - model_list: Contains the list of supported models, their deployment parameters (e.g., api_base, api_key), and order for prioritized failover [2][5]. - router_settings: Configures routing behavior, including: - routing_strategy: Strategies such as simple-shuffle, least-busy, usage-based-routing, or latency-based-routing [5][6]. - fallbacks: Defines the mapping for model-level failover [5]. - num_retries, timeout, allowed_fails, and cooldown_time: Control retry logic and deployment health tracking [5][6]. Internal Handling LiteLLM utilizes utility functions (e.g., within litellm/router_utils/fallback_event_handlers.py) to parse these configurations [7]. The internal parser distinguishes between standard fallback formats (model group mapping) and non-standard formats (such as parameter overrides) to ensure the failover chain is processed correctly [7]. For the SDK, completion() calls can also accept an explicit fallbacks parameter to implement similar logic programmatically [4][8][9].

Citations:

🌐 Web query:

github BerriAI/litellm v1.94.0 fallbacks scalar list Router model_fallbacks

💡 Result:

In LiteLLM (including v1.94.0), the Router's fallbacks configuration is defined as a list of dictionaries, where each dictionary maps a primary model_group (the model that failed) to a list of fallback model_group names to be attempted sequentially [1][2][3]. Example configuration: fallbacks=[{"gpt-4": ["gpt-3.5-turbo", "claude-2"]}] Key details regarding Router fallbacks: 1. Structure and Execution: Fallbacks are processed in the order they appear in the list [4][5]. If a request to a model group fails (and after any configured internal retries are exhausted), the Router attempts the fallback models sequentially until one succeeds or the max_fallbacks limit is reached [4][5][6][7]. The default max_fallbacks value is 5 [8]. 2. Configuration Types: - General Fallbacks: The standard fallbacks parameter handles generic provider or connection errors [4][5]. - Context Window Fallbacks: Configured via context_window_fallbacks, specifically triggered by ContextWindowExceededError [1][2][5]. - Content Policy Fallbacks: Configured via content_policy_fallbacks, triggered by ContentPolicyViolationError [4][1][5]. 3. Management: - Initialization: Fallbacks are typically passed during Router initialization [1][2]. - Proxy Management: If running the LiteLLM Proxy, fallback configurations can also be managed dynamically via dedicated endpoints (e.g., POST /fallback) when STORE_MODEL_IN_DB is enabled [5]. - Internal vs. Public Names: When using team-scoped models or aliases, Ensure that fallback keys match the internal model group names recognized by the router, as lookup issues can occur if using unresolved public names [9]. 4. Relation to Retries: Fallbacks operate at a higher level than internal retries. Retries occur within the same model group (e.g., trying a different deployment), whereas fallbacks redirect the request to an entirely different model group [6][7].

Citations:

🌐 Web query:

site:github.com/BerriAI/litellm litellm/router.py model_fallbacks "fallbacks" "v1.94.0"

💡 Result:

In LiteLLM v1.94.0, model fallbacks are a core reliability feature used to automatically switch to backup models when a primary request fails [1][2]. The mechanism is handled by the litellm.Router for proxy environments or the fallbacks parameter in standard completion() calls [2][3][4]. Key concepts and implementation details for fallbacks as of v1.94.0 include: Types of Fallbacks LiteLLM distinguishes between different categories of failure-based fallbacks, which can be configured independently [2][5]: 1. Fallbacks: Used for general errors, such as rate limits or connectivity issues [2]. 2. Context Window Fallbacks: Specifically triggered by ContextWindowExceededError [2][5]. 3. Content Policy Fallbacks: Triggered by ContentPolicyViolationError [2][5]. 4. Default Fallbacks: A fallback mechanism used if a specific model group is misconfigured or inaccessible [2][5]. Configuration in Proxy (config.yaml) In a LiteLLM Proxy setup, these are defined under litellm_settings in your configuration file [2][4]: litellm_settings: fallbacks: [{"primary-model": ["fallback-model-1", "fallback-model-2"]}] context_window_fallbacks: [{"primary-model": ["larger-context-model"]}] default_fallbacks: ["fallback-model"] Implementation in Code (SDK) When using the SDK directly, you can pass a fallbacks list to the completion() function [1][3]. The library attempts the primary model first, and if it fails, iterates through the provided fallback list [1][3]. from litellm import completion response = completion( model="primary-model", messages=[...], fallbacks=["fallback-model-1", "fallback-model-2"]) Changes and Deprecation Note that model_fallbacks has been deprecated in favor of litellm.fallbacks in the core configuration [6]. LiteLLM actively manages these through router.py and dedicated fallback utilities, ensuring that retries and model switching are handled according to the logic defined in your configuration [7][2][3].

Citations:

🌐 Web query:

site:github.com/BerriAI/litellm litellm/proxy/proxy_server.py yaml config fallbacks v1.94.0

💡 Result:

In LiteLLM Proxy v1.94.0, fallback configurations are defined within the config.yaml file, primarily under router_settings or litellm_settings [1][2]. Fallbacks are designed to improve reliability by allowing the proxy to automatically switch to alternative models or deployments when a request fails [2][3]. Key Configuration Patterns: 1. Standard Fallbacks: Configured under router_settings to handle general failures (e.g., rate limits, connection errors) [1][2]. yaml router_settings: fallbacks: [{"model-a": ["model-b"]}] 2. Error-Specific Fallbacks: LiteLLM supports specialized fallback paths for specific error types [2][4]: - Content Policy Violations: content_policy_fallbacks [4] - Context Window Errors: context_window_fallbacks [2][4] - Default Fallbacks: default_fallbacks can be set as a safety net if a specific model group is misconfigured or fails [2][4]. 3. Load Balancing Integration: When using multiple deployments for the same model name, you can use the order parameter in litellm_params to set priority [5][3]. If all deployments for a specific order level fail, the router exhausts its retries before falling through to the model-level fallbacks defined in router_settings [5][3]. 4. Recent Enhancements: As of recent updates leading into and including v1.94.0, LiteLLM has expanded fallback capabilities: - Rate Limit Handling: Fallbacks now trigger on local rate limit errors (ProxyRateLimitError) encountered during pre-call hooks [6]. - Budget Fallbacks: Key-level budget_fallbacks allow requests to be rerouted to a different model if the current model's budget has been exceeded, rather than blocking the request entirely [7]. For detailed implementation, refer to the official LiteLLM documentation on reliability and routing [2][3].

Citations:

🌐 Web query:

"lru_cache" "_parse_fallback" "fallbacks" "model_list_router" "v1.94.0" github.com/BerriAI/litellm

💡 Result:

In LiteLLM version v1.94.0 [1], the components mentioned function as part of the library's routing and reliability infrastructure. LRU Cache and Performance: LiteLLM utilizes functools.lru_cache for several internal hot-path functions, such as get_model_info and _cached_get_model_group_info [2]. Recent optimizations, including those relevant to v1.94.0, increased the DEFAULT_MAX_LRU_CACHE_SIZE from 16 to 64 [2]. This change was implemented to reduce cache eviction thrashing in multi-model production environments [2]. Fallback Mechanism and Router: The model_list_router (typically referenced as Router) [3][4] handles request routing and automatic failover [5][6]. Fallbacks are configured within the Router to ensure reliability [5]. The parsing of these fallbacks, including internal functions like _parse_fallback (often associated with internal validation/initialization logic in litellm/router.py or related utilities) [7][8], ensures that fallback configurations are correctly mapped to model groups [8][9]. Key Architecture points: - Configuration: Fallbacks can be set via the Router initialization (e.g., fallbacks=[{"model-a": ["model-b"]}]) [5]. - Parsing: The router validates and parses these definitions to handle various types of fallbacks, including general, context-window, and content-policy fallbacks [7][10][8]. - Recent Enhancements: Version 1.94.0 included various fixes and updates to the proxy, authentication, and guardrail components [1], building on established mechanisms for model routing and fallback event handling [8][9]. For specific implementation details, refer to litellm/router.py [7] and the utilities located in litellm/router_utils/ [8].

Citations:

🌐 Web query:

github.com/BerriAI/litellm v1.94.0 "fallbacks" "model_list_router"

💡 Result:

In LiteLLM version 1.94.0, the Router manages failover using a hierarchical approach to ensure high reliability. The primary mechanisms for handling failures are retries (within the same model group) and fallbacks (escalation to different model groups) [1][2]. Router Fallback Logic When a request is made through the Router, it follows this execution flow: 1. Retries: If a call fails, the Router first attempts to retry the request within the same model group using available deployments (e.g., based on weights or order) [3][1]. 2. Fallbacks: If all retries within the current group are exhausted, the Router checks the configured fallbacks list [1][2]. It will then attempt to call the next model group specified in that configuration [4][5]. Configuration and Types You can configure fallbacks by passing them to the Router constructor [4][6]. It supports several specialized fallback types: - General Fallbacks (fallbacks): Handles generic provider/model errors, such as 429 rate limits or 500 server errors [5][2]. - Context Window Fallbacks (context_window_fallbacks): Specifically triggers when a ContextWindowExceededError is raised [7][4][2]. - Content Policy Fallbacks (content_policy_fallbacks): Specifically triggers when a ContentPolicyViolationError is raised [4][2]. Configuration Example: router = Router( model_list=[...], fallbacks=[ {"gpt-4": ["gpt-3.5-turbo", "claude-2"]} ], context_window_fallbacks=[ {"gpt-3.5-turbo": ["gpt-3.5-turbo-16k"]} ]) Key Technical Considerations - Prerequisites: Every model group referenced in your fallback configuration must be explicitly defined in your model_list. If a referenced model is missing, the Router will raise a BadRequestError [2]. - Model Group Aliases: As of recent updates in the 1.9x release cycle, there have been fixes (e.g., PR #29378) to ensure that fallback lookups correctly resolve when using model_group_alias. Previously, the fallback lookup path could fail to match if an alias was used, causing the Router to unintentionally bypass configured fallbacks [8]. - Order-based Routing: You can also use the order parameter in litellm_params within your model_list to prioritize specific deployments. If an order=1 deployment fails, the Router automatically escalates to order=2 deployments before falling through to the configured global fallbacks list [9][3].

Citations:

🌐 Web query:

"def _router_setup" "fallbacks" "litellm_settings" "router_settings" litellm proxy server v1.94.0

💡 Result:

In LiteLLM Proxy v1.94.0, fallbacks are managed through the Router configuration, which orchestrates failover logic between models or model groups when requests fail [1][2][3]. ### Fallback Configuration Fallbacks can be defined in the router_settings section of your proxy configuration (usually config.yaml) or set dynamically at the key/team level [4][5]. The available fallback types are [6][2]: * fallbacks: The default mechanism for handling general errors (e.g., RateLimitError) [6][2]. It allows defining a chain of model groups to attempt sequentially [6][7]. * context_window_fallbacks: Specifically for ContextWindowExceededErrors. These often incorporate pre-call checks to avoid sending requests that exceed token limits [6][2][5]. * content_policy_fallbacks: Specifically for ContentPolicyViolationError [6][5]. * default_fallbacks: A fallback list applied if a specific model group is misconfigured or unreachable [6][2][5]. ### Router Setup Logic The internal method _router_setup initializes these settings [8]. In the LiteLLM Proxy, the router processes requests using an internal loop (typically async_function_with_fallbacks) that respects the following hierarchy [7][8]: 1. Deployments within a model group (load balancing/retries) [7][9]. 2. Configured fallbacks defined in router_settings [7][2]. For example, a configuration might look like this [2][5]: router_settings: fallbacks: - "primary-model-group": ["fallback-model-group-1", "fallback-model-group-2"] context_window_fallbacks: - "primary-model-group": ["larger-context-model"] ### Recent Developments in v1.94.x LiteLLM v1.94.0 and its release candidates include several refinements to proxy reliability and router behavior [1][10]: * Rate Limit Handling: Recent updates ensure that ProxyRateLimitError (raised by pre-call hooks like parallel_request_limiter) now correctly triggers the router's fallback logic rather than immediately returning a 429 error [11]. * Granular Control: Fallbacks can now be configured specifically for individual API keys or teams via the proxy UI or configuration, allowing for custom fallback chains that override global defaults [4][3]. * Bug Fixes: Specific fixes have addressed issues where metadata parameters were incorrectly handled during failover in certain routes (e.g., /v1/responses/compact) [12]. For further implementation details, consult the official reliability documentation [2] and the router configuration API references [13].

Citations:

🌐 Web query:

"model_fallbacks" "litellm_settings" "fallbacks" "Router(" litellm proxy_server.py

💡 Result:

In LiteLLM, model fallbacks allow for automatic provider failover when a primary model group fails after all retries are exhausted [1][2]. Configuration Methods 1. Router Initialization (Python) When using the Router class directly, you define fallbacks by mapping the primary model name to a list of backup model names [3][4]. router = Router( model_list=[...], fallbacks=[{"primary-model": ["backup-model-1", "backup-model-2"]}]) 2. Proxy Configuration (litellm_settings) In the LiteLLM Proxy configuration (e.g., config.yaml), you define these under litellm_settings [3][4][5]. litellm_settings: fallbacks: - primary-model: ["backup-model-1", "backup-model-2"] Specialized Fallbacks LiteLLM also supports specific error-handling fallbacks, which can be configured similarly in the config file [2][6]: - content_policy_fallbacks: Used specifically for ContentPolicyViolationError [2][6]. - context_window_fallbacks: Used specifically for ContextWindowExceededError [2][6]. - default_fallbacks: A catch-all list of models to use if a specific group is misconfigured or fails [6][5]. Hierarchy and Management - Resolution Order: Router settings are applied hierarchically: Key-level settings > Team-level settings > Global (Proxy) settings [7]. - Proxy Server: The proxy_server.py logic utilizes these settings to coordinate the Router, which manages load balancing, retries (within a model group), and fallbacks (across model groups) [8][9]. - Requirements: All models referenced in a fallback chain must be defined in the model_list [2]. Key Reliability Settings In your configuration, you can also define: - num_retries: Number of retries within a model group before triggering a fallback [3][2]. - allowed_fails & cooldown_time: Settings to temporarily remove failing models from rotation [3][2]. - max_fallbacks: Sets a limit on the number of fallback attempts (default is 5) [10][11].

Citations:

🌐 Web query:

"litellm_settings" "fallbacks" "Router(" "model_fallbacks" "fallbacks" "None"

💡 Result:

In LiteLLM, fallbacks are used to automatically route a request to a different model group if the primary model deployment fails [1][2]. These can be configured in the Router using either the fallbacks parameter in the Router constructor [3][4] or via litellm_settings in a configuration file (often used with the LiteLLM Proxy) [1][4][5]. Configuration Structure Fallbacks are defined as a list of dictionaries, where each dictionary maps a primary model group to a list of fallback model groups [3][4]. Example (Python Router): router = Router( model_list=[...], fallbacks=[ {"gpt-4": ["gpt-3.5-turbo", "claude-2"]} ]) Example (Config YAML): litellm_settings: fallbacks: - gpt-4: ["gpt-3.5-turbo", "claude-2"] Behavior and Key Details 1. Triggering: Fallbacks are generally triggered after the configured number of retries (num_retries) for a deployment has been exhausted [6][2]. 2. Context Window Fallbacks: There is a separate setting, context_window_fallbacks, specifically for handling ContextWindowExceededError [3][4]. These are triggered immediately when this specific error occurs, rather than waiting for general retries to exhaust [7][8]. 3. Execution: When a failure occurs, the router iterates through the list of fallback model groups in the order specified until one succeeds [7][5]. 4. Management: In the LiteLLM Proxy, you can also use dedicated fallback management API endpoints (/fallback) to manage these configurations dynamically without needing to restart or update the entire proxy configuration, provided STORE_MODEL_IN_DB=True is set [6][9]. Note: "Weighted failover" is a separate mechanism available when using the simple-shuffle routing strategy; it retries across deployments within the same model group before escalating to cross-group fallbacks [8].

Citations:

🌐 Web query:

"https://raw.githubusercontent.com/BerriAI/litellm/v1.94.0/litellm/proxy/proxy_server.py" "fallbacks" "litellm_settings"

💡 Result:

In LiteLLM Proxy, the litellm_settings section in your config.yaml file is used to configure global module-level settings for the LiteLLM library, including reliability features like fallbacks [1][2][3]. The fallbacks configuration under litellm_settings enables automatic provider failover [4][5]. If a model call fails (after exhausting configured retries), LiteLLM can automatically route the request to a fallback model [4][6]. Example Configuration: litellm_settings: num_retries: 3 fallbacks: [{"zephyr-beta": ["gpt-3.5-turbo"]}] When the proxy server starts, it reads these settings from your configuration file within proxy_server.py. The relevant code processes litellm_settings to initialize various components, such as callbacks, caching, and reliability logic, ensuring these configurations are applied consistently across the proxy instance [7][3]. There are three primary types of fallback configurations: 1. fallbacks: For general errors (e.g., RateLimitError) [6]. 2. content_policy_fallbacks: Specifically for content policy violations [6]. 3. context_window_fallbacks: Specifically for context window exceeded errors [6][2]. In addition to litellm_settings, you can manage routing and fallbacks more granularly using router_settings or by setting an order parameter within individual model definitions in model_list for load-balancing scenarios [2][8][5].

Citations:

🌐 Web query:

"https://raw.githubusercontent.com/BerriAI/litellm/v1.94.0/litellm/router.py" "fallbacks" "model_fallbacks"

💡 Result:

In LiteLLM, model fallbacks are a reliability mechanism designed for automatic provider failover [1][2]. When a primary model group fails—after exhausting all configured retries within that group—the Router attempts to route the request to a pre-defined backup model group [1][3]. The Router class (located in litellm/router.py) handles these fallbacks using three primary parameters, each targeting specific error classes [4][5][2]: 1. fallbacks: Catches general operational errors, such as 429 RateLimitErrors, 500 server errors, or connection failures [6][2]. 2. context_window_fallbacks: Specifically handles ContextWindowExceededError when input tokens exceed a model's limit [6][2]. 3. content_policy_fallbacks: Specifically handles ContentPolicyViolationError [6][2]. Configuration details: - Definition: You define fallbacks as a list of dictionaries where the key is the primary model group and the value is a list of fallback model groups to try in order (e.g., [{"gpt-4": ["gpt-3.5-turbo"]}]) [1][7]. - Prerequisites: Every model referenced in a fallback configuration must exist within the model_list provided to the Router, or the router will raise a BadRequestError [2]. - Precedence: Retries occur first within the same model group (load balancing) [3][2]. If those fail, the router escalates to the configured cross-group fallbacks [8][3]. - Customization: These settings can be configured globally in the Router constructor or at the request level via the .completion() call for granular control [1][6]. The underlying implementation in litellm/router.py validates these parameters during initialization and utilizes utility functions (e.g., get_fallback_model_group in litellm/router_utils/fallback_event_handlers.py) to determine the appropriate fallback path when an exception occurs [4][5][9].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== router_settings fallbacks vs litellm_settings fallbacks =="
python3 - <<'PY'
from pathlib import Path
import re
p = Path('agent/litellm/config.yaml')
text = p.read_text()
m = re.search(r'(?s)^router_settings:\n(.*?)(?=\n[a-z]|\Z)', text)
print('router_settings contains fallbacks:', bool(m and re.search(r'(?m)^\s*fallbacks:', m.group(1))))
print('top-level litellm_settings fallbacks present:', bool(re.search(r'(?m)^\s*fallbacks:', text)))
print('router settings excerpt:')
if m:
    lines = m.group(1).splitlines()[:40]
    for n, line in enumerate(lines, 1):
        print(f'{n:4}: {line}')
print('fallbacks excerpt:')
for i, line in enumerate(text.splitlines(), 1):
    if i >= 24 and i <= 32:
        print(f'{i:4}: {line}')
PY

echo
echo "== LiteLLM v1.94.0 source: proxy router setup excerpt =="
python3 - <<'PY'
from urllib.request import urlopen
url = "https://raw.githubusercontent.com/BerriAI/litellm/v1.94.0/litellm/proxy/proxy_server.py"
text = urlopen(url, timeout=20).read().decode()
for needle in ["router_settings", "Router(","model_fallbacks", "fallbacks="]:
    print(f"\n-- occurrences of {needle!r} --")
    for i, line in enumerate(text.splitlines(), 1):
        if needle in line:
            lo, hi = max(1, i-4), min(len(text.splitlines()), i+8)
            lines = text.splitlines()
            print(f"--- around line {i} ---")
            for j in range(lo, hi+1):
                print(f"{j:4}: {lines[j-1]}")
            print()
            break
PY

Repository: yxtay/docker-stacks

Length of output: 2937

🌐 Web query:

"litellm_settings" "fallbacks" "Router" "litellm_settings" "model_fallbacks" "router_settings"

💡 Result:

In LiteLLM, fallbacks (automatic failover) and router configurations are managed through settings in your configuration file (often config.yaml) or direct router initialization [1][2][3]. Key Concepts and Settings: 1. litellm_settings These settings control the behavior of the LiteLLM module globally, affecting how requests are handled across the proxy or application [4]. - fallbacks: Defines a mapping of model groups to fallback models (e.g., [{"primary-model": ["fallback-model"]}]). These are triggered after the configured number of retries fails [1][5]. - context_window_fallbacks: Specifically handles ContextWindowExceededErrors by falling back to specified models [1][5]. - content_policy_fallbacks: Specifically handles ContentPolicyViolationErrors [1][5]. - default_fallbacks: Used as a catch-all if a specific model group is misconfigured [5]. - num_retries: Sets the number of retries attempted on a specific deployment before triggering the fallback path [4][5]. 2. router_settings These settings control the LiteLLM Router, which handles load balancing, retries within a model group, and deployment-level failover [6][7]. - fallbacks: You can define fallback hierarchies here, often used in conjunction with deployment order [7][8]. - routing_strategy: Defines how traffic is distributed (e.g., simple-shuffle, least-busy). Defaults to simple-shuffle [8][9]. - model_group_alias: Allows aliasing one model name to another globally [8][4]. - max_fallbacks: Caps the number of fallback attempts (default is 5) [6]. Relationship and Behavior: - Retries vs. Fallbacks: Retries occur within the same model group (trying different deployments) [2]. If all retries fail, the router escalates to the configured fallback model group [2][7]. - Deployment Priority: By setting the order parameter in litellm_params for individual deployments, you can create a prioritized chain (e.g., order 1 -> order 2). The router will automatically try these in sequence before moving to the general fallback path defined in settings [7][8]. - Fallback Management: Recent versions of the LiteLLM Proxy support dedicated endpoints (e.g., POST /fallback) for managing these configurations dynamically without updating the entire proxy config [10]. Example Configuration Snippet: litellm_settings: num_retries: 3 fallbacks: [{"gpt-4": ["gpt-3.5-turbo"]}] router_settings: routing_strategy: simple-shuffle timeout: 30

Citations:

🌐 Web query:

Github BerriAI/litellm v1.94.0 "fallbacks=" "litellm_settings" Router

💡 Result:

In LiteLLM, you set Router (fallback chain) reliability at the litellm_settings level using the fallbacks key. The fallback mapping is configured as List[Dict[str, List[str]]], where each dict maps a “model_name” (model group) to an ordered list of fallback “model_name” values. Example: litellm_settings: fallbacks: [{"zephyr-beta": ["gpt-3.5-turbo"]}] [1][2]. How to express your query’s idea (“BerriAI/litellm v1.94.0 fallbacks= litellm_settings Router”): 1) Global (proxy-wide) fallbacks (via litellm_settings) YAML: litellm_settings: fallbacks: [{"zephyr-beta": ["gpt-3.5-turbo"]}] This means: when the router tries zephyr-beta and the call fails, it will attempt gpt-3.5-turbo as the fallback [1][2]. 2) If you also want “default” fallbacks for misconfigured/bad model groups default_fallbacks is a separate setting used when a specific model group is misconfigured / bad [3]. 3) Related (optional) router settings (retries/cooldowns) num_retries and cooldown-related settings like allowed_fails / cooldown_time are configured alongside fallbacks under router/module reliability configuration [1][4]. Router settings for keys/teams: LiteLLM also supports setting different router settings (including fallbacks) at the API key / team level with hierarchical resolution (Keys > Teams > Global) [5]. If you share the exact config you’re using (or paste your model_list + current litellm_settings/router_settings), I can rewrite it precisely for the intended fallback behavior in v1.94.0.

Citations:

🌐 Web query:

site:github.com/BerriAI/litellm "Router(" "model_fallbacks" "litellm_settings" "->" "fallbacks"

💡 Result:

In LiteLLM, fallbacks for the Router are configured using the fallbacks parameter, which maps a primary model to a list of fallback models to be tried if the primary model fails [1][2][3]. When configuring a Router, fallbacks can be defined in a few ways depending on your implementation: 1. SDK/Python Initialization: You pass a list of dictionaries to the fallbacks parameter of the Router constructor [1]. Each dictionary maps a primary model name to a list of fallback model names (e.g., fallbacks=[{"gpt-3.5-turbo": ["gpt-4"]}]). 2. Configuration File (Proxy/YAML): When using a config file, you define fallbacks under litellm_settings [1][4][5]. The syntax follows the same mapping structure: litellm_settings: fallbacks: [{"primary-model-name": ["fallback-model-1", "fallback-model-2"]}] Router Execution Flow: - Order-Based Priority: The Router first attempts to fulfill requests using deployments defined with the order parameter in model_list (where lower values represent higher priority) [6][3]. It will cycle through these deployments based on their order [6][3]. - Fallback Execution: If all deployments for a specific model are exhausted or fail, the Router falls through to the models defined in the fallbacks configuration [6][3]. - Context Window Fallbacks: Additionally, you can configure context_window_fallbacks within litellm_settings to specifically handle cases where a model fails due to context window limits [4][5][7]. For robust reliability, the Router also supports settings like num_retries, allowed_fails, and cooldown_time, which work alongside these fallback mechanisms to manage traffic effectively [1][4][3].

Citations:

🌐 Web query:

"Router(" "model_fallbacks=" "fallbacks=" "litellm_settings" ".get"

💡 Result:

In the LiteLLM Router, the construct Router(model_fallbacks=fallbacks=litellm_settings.get(...)) appears to be a syntactically incorrect attempt to configure fallbacks [1][2]. The correct way to configure fallbacks in LiteLLM depends on whether you are using the Router class directly in Python or a configuration file (e.g., config.yaml) for the LiteLLM Proxy [3][4][5]. When using the Router class in Python, fallbacks are passed as a direct argument to the constructor, not through a model_fallbacks parameter [2]. The standard fallbacks parameter accepts a list of dictionaries [2]. Example Python usage: router = Router( model_list=[...], fallbacks=[{"gpt-4": ["gpt-3.5-turbo"]}] # List of dicts mapping model to fallbacks [2]) When using the LiteLLM Proxy (via config.yaml), fallbacks are defined under litellm_settings [3][6]. Example YAML configuration: litellm_settings: fallbacks: - gpt-4: ["gpt-3.5-turbo"] # Fallback mapping [4][6] To access these settings programmatically from a configuration object (like litellm_settings), you would typically use standard dictionary .get() methods, such as litellm_settings.get("fallbacks"), and then pass the resulting list to the fallbacks parameter of the Router [1][4]. The Router initialization code should look more like this: # Correct programmatic approach fallbacks = litellm_settings.get("fallbacks", []) router = Router( model_list=[...], fallbacks=fallbacks) Additionally, LiteLLM supports specialized fallback types, such as context_window_fallbacks and content_policy_fallbacks, which are configured similarly to general fallbacks [4][7]. If you are working with the LiteLLM Proxy, you can also manage these dynamically via dedicated API endpoints (e.g., POST /fallback) instead of modifying the config file directly [7].

Citations:

🌐 Web query:

GitHub BerriAI/litellm "def _router_setup" "fallbacks" "litellm_settings"

💡 Result:

In the LiteLLM library, the Router's fallback mechanisms and litellm_settings are integral to managing reliability and routing behavior [1][2]. ### Router Setup and Fallbacks The Router class in LiteLLM initializes fallback logic during its setup phase [3][4]. When initializing a Router, you can define fallback models to be used if a primary model call fails [2][5]. Configuration can be handled via code or a YAML configuration file [6][1]. #### Programmatic Setup (Python) When initializing the Router, you pass the fallbacks parameter as a list of dictionaries, where each dictionary maps a model (or "*" for all models) to a list of fallback models [3][2]. python from litellm import Router router = Router( model_list=[...], fallbacks=[ {"gpt-4": ["gpt-3.5-turbo", "claude-2"]} ] ) #### Proxy Configuration (YAML) In a LiteLLM Proxy setup, you can define fallbacks within the litellm_settings section of your configuration file [6][7]. yaml litellm_settings: fallbacks: [{"gpt-4": ["gpt-3.5-turbo", "claude-2"]}] ### Key Concepts * litellm_settings: This section in the configuration file controls global LiteLLM module settings, such as num_retries, request_timeout, and fallbacks [6][8]. These settings apply to the Router's behavior when processing requests [6]. * Fallback Types: LiteLLM supports various fallback scenarios, including generic model fallbacks (used when a model fails), context window fallbacks (context_window_fallbacks), and content policy violations [2][9][8]. * Cooldowns: Router settings include allowed_fails and cooldown_time, which work alongside fallbacks to manage deployment health by temporarily disabling models that fail frequently [6][5][8]. * Pre-call checks: The router can perform checks, such as verifying if a request is within a model's context window, before attempting the call [1][8]. For detailed configuration parameters, refer to the litellm/router.py source code or the official LiteLLM documentation [3][6][7].

Citations:

🌐 Web query:

LiteLLM v1.94.0 source router_utils fallback_event_handlers.py model_list_router fallbacks

💡 Result:

In LiteLLM, fallback mechanisms are orchestrated by the Router to ensure reliability when primary model deployments fail [1]. The file litellm/router_utils/fallback_event_handlers.py contains core utility functions that manage these fallback sequences [2][3][4]. Key mechanisms and components include: 1. Fallback Execution Logic: The run_async_fallback function in fallback_event_handlers.py is responsible for iterating through configured fallback model groups [2][4]. If a primary call fails, it catches the exception and attempts to call the next model group in the sequence [2][1]. It logs successes and failures for each step and raises the final exception if all configured fallbacks are exhausted [2][3]. 2. Configuration and Setup: The Router class (located in litellm/router.py) handles the configuration of fallbacks [5][6][7]. Fallbacks can be set at the Router level or passed during specific calls [8]. The router validates these configurations and organizes them into self.fallbacks [5]. 3. Fallback Types: - General Fallbacks: Managed via the fallbacks parameter, used for standard errors like RateLimitError [8]. - Specialized Fallbacks: context_window_fallbacks handle ContextWindowExceededError, and content_policy_fallbacks handle ContentPolicyViolationError [8]. - Weighted Failover: When enable_weighted_failover is active, the router first attempts to retry within the same model group by excluding the failed deployment before escalating to cross-group fallbacks [9]. 4. Integration: The router architecture wraps the initial request in function_with_fallbacks, which handles the logic of shifting to a different model group if retries within the current group (managed by function_with_retries) fail [1]. This system ensures that if a model or provider becomes unavailable, the traffic automatically reroutes to a healthy alternative defined in your fallback configuration [8][1].

Citations:

🌐 Web query:

LiteLLM v1.94.0 source router_utils fallback_event_handlers.py Router fallbacks string list

💡 Result:

In LiteLLM v1.94.0, the litellm/router_utils/fallback_event_handlers.py module manages the logic for cross-group model failover [1][2]. The router allows configuring fallbacks using a list of model groups [3][4]. The configuration format for fallbacks supports strings or dictionaries [1][5]: 1. String List: You can specify a simple list of model groups to fall back to sequentially if the primary call fails (e.g., fallbacks: ["gpt-3.5-turbo", "claude-3-haiku"]) [1]. 2. Dictionary Mapping: For more granular control, dictionaries allow mapping a specific model group to its own set of fallbacks (e.g., fallbacks: [{"gpt-4": ["gpt-3.5-turbo"]}]) [1][3]. Internal Logic and Handling: - The get_fallback_model_group function in fallback_event_handlers.py resolves the appropriate fallback chain by checking for exact model group matches, stripped provider prefixes, or wildcard * catch-alls [1][2]. - The run_async_fallback function iterates through these defined fallback groups recursively until a call succeeds or the max_fallbacks limit is reached [6][1][2]. - LiteLLM maintains specialized handling for dictionary-based fallback entries, including logic to distinguish between standard model-group mapping dictionaries and those containing LiteLLM parameter overrides [7][5]. This ensures that when a model group name happens to collide with a reserved LiteLLM parameter name (like "model"), the router correctly identifies it as a mapping rather than an invalid parameter configuration [7]. These mechanisms allow for robust provider and model failover, separate from the intra-group retries managed by function_with_retries [8][9].

Citations:

Make the LiteLLM proxy use list-valued fallback targets.

LiteLLM proxy fallback mappings define fallback model names as a list; scalar values are not a documented mapping format.

Proposed fix 
 fallbacks:
-  - reasoning: complex
-  - complex: medium
@@ -26,7 +26,7 @@
   fallbacks:
+  - reasoning: [complex]
+  - complex: [medium]
+  - medium: [simple]
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
fallbacks:
- reasoning: complex
- complex: medium
- medium: simple
fallbacks:
- reasoning: [complex]
- complex: [medium]
- medium: [simple]
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@agent/litellm/config.yaml` around lines 26 - 30, Update the fallback mappings
in the LiteLLM configuration so each target value is a list containing its
fallback model name, including the reasoning, complex, and medium mappings.
Preserve the existing fallback order and model names while replacing scalar
values with list-valued entries.

model_list:
- model_name: auto
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: simple
MEDIUM: medium
COMPLEX: complex
REASONING: reasoning
adaptive: true
session_affinity: true
complexity_router_default_model: complex

- model_name: simple
litellm_params:
model: openai/deepseek-v4-flash-free
litellm_credential_name: opencode_zen

- model_name: medium
litellm_params:
model: openai/deepseek-v4-flash-free
litellm_credential_name: opencode_zen

- model_name: complex
litellm_params:
model: openai/tencent/hy3:free
litellm_credential_name: nous

- model_name: reasoning
litellm_params:
model: openai/tencent/hy3:free
litellm_credential_name: nous

- model_name: cerebras/gemma-4-31b
litellm_params:
model: cerebras/gemma-4-31b
Expand Down Expand Up @@ -42,19 +97,6 @@ model_list:
litellm_params:
model: openrouter/openrouter/free

- model_name: smart-router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: opencode-zen/deepseek-v4-flash-free
MEDIUM: opencode-zen/deepseek-v4-flash-free
COMPLEX: nous/hy3:free
REASONING: nous/hy3:free
adaptive: true
session_affinity: true
complexity_router_default_model: nous/hy3:free

credential_list:
- credential_name: nous
credential_values:
Expand Down
Loading