feat: litellm auto router - #376
Conversation
|
Warning Review limit reached
Next review available in: 52 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughLiteLLM health checks, fallback routing, cache behavior, and tiered model selection are reconfigured. The previous smart-router entry is replaced by an auto router and dedicated model tiers. The environment example removes the NVIDIA NIM API key variable. ChangesLiteLLM routing and configuration
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested labels: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
✅MegaLinter analysis: Success
Notices📣 MegaLinter 9.5.0 is out! Discover the new features and security recommendations in the release announcement. (Skip this info by defining See detailed reports in MegaLinter artifacts Your project could benefit from a custom flavor, which would allow you to run only the linters you need, and thus improve runtime performances. (Skip this info by defining
|
✅MegaLinter analysis: Success
Notices📣 MegaLinter 9.5.0 is out! Discover the new features and security recommendations in the release announcement. (Skip this info by defining See detailed reports in MegaLinter artifacts Your project could benefit from a custom flavor, which would allow you to run only the linters you need, and thus improve runtime performances. (Skip this info by defining
|
✅
|
| Descriptor | Linter | Files | Fixed | Errors | Warnings | Elapsed time |
|---|---|---|---|---|---|---|
| ✅ ACTION | actionlint | 4 | 0 | 0 | 0.25s | |
| ✅ ACTION | zizmor | 4 | 0 | 0 | 0 | 0.34s |
| ✅ BASH | bash-exec | 11 | 0 | 0 | 0.03s | |
| ✅ BASH | shellcheck | 11 | 0 | 0 | 0.17s | |
| ✅ BASH | shfmt | 11 | 0 | 0 | 0 | 0.01s |
| jscpd | yes | 1 | no | 0.11s | ||
| ✅ EDITORCONFIG | editorconfig-checker | 107 | 0 | 0 | 0.14s | |
| ✅ JSON | prettier | 1 | 0 | 0 | 0 | 0.24s |
| ✅ JSON | v8r | 1 | 0 | 0 | 1.84s | |
| ✅ MARKDOWN | markdownlint | 5 | 0 | 0 | 0 | 0.74s |
| ✅ MARKDOWN | markdown-table-formatter | 5 | 0 | 0 | 0 | 0.17s |
| ✅ REPOSITORY | betterleaks | yes | no | no | 1.32s | |
| ✅ REPOSITORY | checkov | yes | no | no | 28.1s | |
| ✅ REPOSITORY | gitleaks | yes | no | no | 0.61s | |
| ✅ REPOSITORY | git_diff | yes | no | no | 0.01s | |
| ✅ REPOSITORY | grype | yes | no | no | 66.56s | |
| ✅ REPOSITORY | osv-scanner | yes | no | no | 0.25s | |
| ✅ REPOSITORY | secretlint | yes | no | no | 1.45s | |
| ✅ REPOSITORY | syft | yes | no | no | 2.11s | |
| ✅ REPOSITORY | trivy | yes | no | no | 12.87s | |
| ✅ REPOSITORY | trivy-sbom | yes | no | no | 0.16s | |
| ✅ REPOSITORY | trufflehog | yes | no | no | 6.55s | |
| lychee | 56 | 41 | 0 | 1.61s | ||
| ✅ YAML | prettier | 49 | 0 | 0 | 0 | 1.07s |
| ✅ YAML | v8r | 49 | 0 | 0 | 19.3s | |
| ✅ YAML | yamllint | 49 | 0 | 0 | 1.14s |
Detailed Issues
⚠️ COPYPASTE / jscpd - 1 error
Using config from /action/lib/.automation/.jscpd.json
Clone found (bash)
- bin/oci-rm-stack-create.sh [32:1 - 40:5] (9 lines, 51 tokens)
bin/oci-rm-stack-update.sh [12:1 - 20:5]
┌────────┬────────────────┬─────────────┬──────────────┬──────────────┬──────────────────┬───────────────────┐
│ Format │ Files analyzed │ Total lines │ Total tokens │ Clones found │ Duplicated lines │ Duplicated tokens │
├────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ bash │ 7 │ 214 │ 1120 │ 1 │ 8 (3.74%) │ 51 (4.55%) │
├────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ hcl │ 4 │ 382 │ 1407 │ 0 │ 0 (0.00%) │ 0 (0.00%) │
├────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ txt │ 1 │ 29 │ 379 │ 0 │ 0 (0.00%) │ 0 (0.00%) │
├────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ Total: │ 12 │ 625 │ 2906 │ 1 │ 8 (1.28%) │ 51 (1.75%) │
└────────┴────────────────┴─────────────┴──────────────┴──────────────┴──────────────────┴───────────────────┘
Found 1 clones.
HTML report saved to megalinter-reports/copy-paste/jscpd-report.html
ERROR: jscpd found too many duplicates (1.3%) over threshold (0.0%)
time: 22.866ms
⚠️ SPELL / lychee - 41 errors
📝 Summary
---------------------
🔍 Total...........68
🔗 Unique..........58
✅ Successful......20
⏳ Timeouts.........0
🔀 Redirected.......0
👻 Excluded.........0
❓ Unknown..........0
🚫 Errors..........41
⛔ Unsupported.....41
Errors in agent/compose.yaml
[ERROR] http://localhost:2375/_ping (at 89:32) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:4000/health/liveliness (at 129:70) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:9119/api/status (at 41:33) | Connection refused - server may be down or port blocked
[ERROR] https://hermes/ (at 12:67) | Connection failed. Check network connectivity and firewall settings
Errors in agent/litellm/config.yaml
[ERROR] http://headroom:8787/ (at 124:19) | Connection failed. Check network connectivity and firewall settings
[404] https://inference-api.nousresearch.com/v1 (at 104:17) | Rejected status code: 404 Not Found
[404] https://opencode.ai/zen/go/v1 (at 116:17) | Rejected status code: 404 Not Found
[404] https://opencode.ai/zen/v1 (at 110:17) | Rejected status code: 404 Not Found
Errors in arcane/compose.yaml
[ERROR] http://localhost:3552/ (at 6:27) | Connection refused - server may be down or port blocked
Errors in homeassistant/compose.yaml
[ERROR] http://localhost:10000/health (at 146:33) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:6052/version (at 105:33) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:8095/ (at 66:32) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:8123/ (at 27:33) | Connection refused - server may be down or port blocked
Errors in immich/compose.yaml
[ERROR] http://localhost:8080/ (at 160:32) | Connection refused - server may be down or port blocked
Errors in infra/compose.yaml
[ERROR] http://localhost:2375/_ping (at 60:32) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:9002/healthz (at 95:36) | Connection refused - server may be down or port blocked
Errors in monitoring/compose.yaml
[ERROR] http://localhost:8090/ (at 73:44) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:8090/ (at 94:27) | Connection refused - server may be down or port blocked
Errors in pangolin/compose.yaml
[ERROR] http://gerbil:3004/ (at 63:23) | Connection failed. Check network connectivity and firewall settings
[ERROR] http://localhost/ping (at 121:32) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:3001/api/v1/ (at 51:33) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:3004/healthz (at 87:32) | Connection refused - server may be down or port blocked
[ERROR] http://pangolin:3001/api/v1/ (at 65:24) | Connection failed. Check network connectivity and firewall settings
Errors in pangolin/traefik/dynamic/config.yml
[ERROR] http://pangolin:3000/ (at 80:18) | Connection failed. Check network connectivity and firewall settings
[ERROR] http://pangolin:3002/ (at 85:18) | Connection failed. Check network connectivity and firewall settings
Errors in pangolin/traefik/traefik.template.yml
[ERROR] http://pangolin:3001/api/v1/traefik-config (at 7:15) | Connection failed. Check network connectivity and firewall settings
Errors in proxy/compose.yaml
[ERROR] http://localhost/healthz (at 36:32) | Connection refused - server may be down or port blocked
[ERROR] https://tinyauth/ (at 50:24) | Connection failed. Check network connectivity and firewall settings
Errors in security/compose.yaml
[ERROR] http://localhost:8080/ (at 52:16) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:8080/health (at 34:32) | Connection refused - server may be down or port blocked
Errors in torrent/compose.yaml
[ERROR] http://localhost:6868/ (at 220:33) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:7878/ping (at 140:33) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:8191/health (at 16:33) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:8989/ping (at 182:33) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:9696/ping (at 45:33) | Connection refused - server may be down or port blocked
[ERROR] https://profilarr/ (at 205:15) | Connection failed. Check network connectivity and firewall settings
Errors in usenet/compose.yaml
[ERROR] http://127.0.0.1:7000/ (at 83:32) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:1337/ (at 288:32) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:3000/health (at 26:32) | Connection refused - server may be down or port blocked
[ERROR] http://localhost:7000/health (at 134:32) | Connection refused - server may be down or port blocked
[ERROR] https://aiostreams/ (at 232:18) | Connection failed. Check network connectivity and firewall settings
Hint: You can configure accepted/rejected response codes with `-a` or `--accept`
Notices
📣 MegaLinter 9.5.0 is out! Discover the new features and security recommendations in the release announcement. (Skip this info by defining SECURITY_SUGGESTIONS: false)
See detailed reports in MegaLinter artifacts
Your project could benefit from a custom flavor, which would allow you to run only the linters you need, and thus improve runtime performances. (Skip this info by defining FLAVOR_SUGGESTIONS: false)
- Documentation: Custom Flavors
- Command:
npx mega-linter-runner@9.6.0 --custom-flavor-setup --custom-flavor-linters ACTION_ACTIONLINT,ACTION_ZIZMOR,BASH_EXEC,BASH_SHELLCHECK,BASH_SHFMT,COPYPASTE_JSCPD,EDITORCONFIG_EDITORCONFIG_CHECKER,JSON_V8R,JSON_PRETTIER,MARKDOWN_MARKDOWNLINT,MARKDOWN_MARKDOWN_TABLE_FORMATTER,REPOSITORY_CHECKOV,REPOSITORY_GIT_DIFF,REPOSITORY_GITLEAKS,REPOSITORY_BETTERLEAKS,REPOSITORY_GRYPE,REPOSITORY_OSV_SCANNER,REPOSITORY_SECRETLINT,REPOSITORY_SYFT,REPOSITORY_TRIVY,REPOSITORY_TRIVY_SBOM,REPOSITORY_TRUFFLEHOG,SPELL_LYCHEE,YAML_PRETTIER,YAML_YAMLLINT,YAML_V8R
Show us your support by starring ⭐ the repository
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@agent/litellm/config.yaml`:
- Around line 20-25: Remove the global
default_litellm_params.cache_control_injection_points configuration so
Anthropic-style cache-control markers are not merged into OpenAI-based auto
tiers. Leave prompt-caching configuration scoped to provider-specific Anthropic
or Gemini tiers where applicable, and preserve unrelated settings such as
optional_pre_call_checks and enable_anthropic_prompt_caching.
- Around line 26-30: Update the fallback mappings in the LiteLLM configuration
so each target value is a list containing its fallback model name, including the
reasoning, complex, and medium mappings. Preserve the existing fallback order
and model names while replacing scalar values with list-valued entries.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 9561046b-79a2-4222-97e3-29a2bf77683c
📒 Files selected for processing (2)
agent/.env.exampleagent/litellm/config.yaml
💤 Files with no reviewable changes (1)
- agent/.env.example
| default_litellm_params: | ||
| cache_control_injection_points: | ||
| - location: message | ||
| role: system | ||
| optional_pre_call_checks: ["prompt_caching"] | ||
| enable_anthropic_prompt_caching: true |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== config files =="
git ls-files | rg '(^|/)config\.ya?ml$|litellm|auto_router|auto-routing|prompt_caching|cache_control' || true
echo "== agent/litellm/config.yaml =="
sed -n '1,220p' agent/litellm/config.yaml
echo "== references to default_litellm_params / optional_pre_call_checks / enable_anthropic_prompt_caching =="
rg -n "default_litellm_params|optional_pre_call_checks|enable_anthropic_prompt_caching|prompt_caching|cache_control" agent . --glob '!**/node_modules/**' --glob '!**/.git/**' || trueRepository: yxtay/docker-stacks
Length of output: 4110
🌐 Web query:
LiteLLM proxy auto routing model_list router_settings default_litellm_params merged into selected tier
💡 Result:
In LiteLLM, settings for proxy-based routing and load balancing are organized under distinct configurations in your config file, primarily using model_list and router_settings [1][2]. model_list This section defines your available model deployments [1][2]. Each entry includes a model_name and litellm_params, which specify the underlying provider (e.g., openai/gpt-4o, azure/my-deployment), credentials, and deployment-specific constraints like RPM (requests per minute) [2][3]. router_settings This section configures the Router behavior [1][2]. Key configurations include: - routing_strategy: Defines how traffic is distributed (e.g., simple-shuffle, least-busy, usage-based-routing, latency-based-routing) [2][4]. - default_litellm_params: Sets global default parameters (like timeout or custom metadata) applied to requests if not overridden by specific calls [5][4]. - Advanced Routing: Configurations for auto-routing, semantic matching, and complexity scoring are also defined here (specifically within the complexity_router_config for Auto Router) [6][7]. Merging Behavior Regarding the merging of parameters, LiteLLM has updated its logic to prevent the silent loss of configurations: - Auto Router Parameters: When using the Auto Router, settings such as drop_params or cache_control_injection_points defined on the router alias itself are now correctly merged into the outbound request instead of being dropped [7]. This ensures they are applied alongside the parameters specific to the chosen tier/model [7]. - Admin UI/DB-backed Configs: Recent updates to the Router settings sync path (specifically in Admin UI contexts using a database) ensure that Router.update_settings now merges default_litellm_params into existing settings rather than performing a full replacement [8]. This prevents the loss of crucial defaults like timeout, max_retries, or metadata [8]. For complex setups involving multiple proxy instances, you should ensure redis_host, redis_port, and redis_password are configured under router_settings to maintain shared routing state [1][2].
Citations:
- 1: https://docs.litellm.ai/docs/proxy/configs
- 2: https://docs.litellm.ai/docs/proxy/load_balancing
- 3: https://docs.litellm.ai/docs/routing
- 4: https://github.com/BerriAI/litellm/blob/62920a0c/litellm/types/router.py
- 5: https://github.com/BerriAI/litellm/blob/main/litellm/router.py
- 6: https://docs.litellm.ai/docs/proxy/auto_routing
- 7: https://docs.litellm.ai/blog/autorouter-v2
- 8: feat(router): support default params and expose optional checks in Admin UI BerriAI/litellm#33144
🌐 Web query:
LiteLLM cache_control_injection_points cache_control providers other than Anthropic Gemini ignored
💡 Result:
In LiteLLM, cache_control_injection_points is a utility designed specifically to automatically inject cache_control directives into requests for providers that support Anthropic-style message-level caching, most notably Anthropic (direct) and Amazon Bedrock [1][2]. Regarding your query about whether these points are ignored for other providers, the behavior is as follows: 1. Provider Support for Injection: The cache_control_injection_points feature is explicitly built to handle the Anthropic/Bedrock message-level caching format [2][3]. If you attempt to use this parameter with models or providers that do not natively support or are not configured to process these specific injected message markers, LiteLLM generally does not transform the request to fit those providers' unique caching APIs [4]. 2. Mechanism of Ignoring: For many providers, LiteLLM intentionally strips or fails to process cache_control flags if they are not explicitly supported by the model's backend implementation [4]. Specifically, in the internal code, many provider transformations (such as those for MiniMax, GLM, or Xiaomi) utilize a helper function like remove_cache_control_flag_from_messages_and_tools within their transform_request logic [4]. If a provider is not registered in the internal CacheControlSupportedModels enumeration, any cache_control-related parameters or injected markers are effectively ignored or stripped to prevent API errors [4]. 3. Other Providers: While LiteLLM provides a unified interface for prompt caching, it handles other providers differently: - OpenAI: Uses automatic prompt caching (no manual markers required) [5]. - Gemini: Supports its own context caching mechanism; LiteLLM performs automatic translation of Anthropic-style markers to Gemini's native cachedContents format [5]. If you are seeing cache_control_injection_points ignored, it is likely because that specific provider does not have a corresponding implementation in LiteLLM to interpret those injected markers, leading the library to either drop them or pass them through in a way the provider cannot process [4]. If you believe a specific provider should support this, check the current LiteLLM GitHub repository for open issues, as support for additional providers (like MiniMax or GLM) is frequently requested and updated [4].
Citations:
- 1: https://docs.litellm.ai/docs/tutorials/prompt_caching
- 2: [Feat] Add support for
cache_control_injection_pointsfor Anthropic API, Bedrock API BerriAI/litellm#9996 - 3: [Feature]: Support injecting cache control flags on anthropic requests BerriAI/litellm#9949
- 4: [Bug]: Prompt Caching and Reasoning broken for the most price efficient models including Xiaomi Mimo, Minimax 2.1 and GLM BerriAI/litellm#19923
- 5: https://docs.litellm.ai/docs/completion/prompt_caching
Remove the global prompt-caching injection defaults.
default_litellm_params.cache_control_injection_points gets merged into the selected auto tier, and the tiers are all OpenAI-based. Those injected Anthropic-style cache-control markers don’t enable prompt caching for openai/... models; keep any prompt-caching settings scoped to only Anthropic/Gemini providers/tiers if needed.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@agent/litellm/config.yaml` around lines 20 - 25, Remove the global
default_litellm_params.cache_control_injection_points configuration so
Anthropic-style cache-control markers are not merged into OpenAI-based auto
tiers. Leave prompt-caching configuration scoped to provider-specific Anthropic
or Gemini tiers where applicable, and preserve unrelated settings such as
optional_pre_call_checks and enable_anthropic_prompt_caching.
| fallbacks: | ||
| - reasoning: complex | ||
| - complex: medium | ||
| - medium: simple | ||
|
|
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== candidate file =="
if [ -f agent/litellm/config.yaml ]; then
cat -n agent/litellm/config.yaml
else
echo "agent/litellm/config.yaml not found"
fi
echo
echo "== fallback config occurrences =="
rg -n "fallbacks:|fallback:" . -g '*.yaml' -g '*.yml' -g '*.py' -g '*.js' -g '*.ts' || true
echo
echo "== litellm mentions =="
rg -n "litellm|LiteLLM|model_fallback|fallbacks" pyproject.toml requirements*.txt uv.lock poetry.lock agent .github 2>/dev/null || true
echo
echo "== package version declarations =="
for f in pyproject.toml requirements.txt requirements-dev.txt uv.lock poetry.lock; do
[ -f "$f" ] && { echo "--- $f"; rg -n "litellm|version|python-dependencies" "$f" || true; }
doneRepository: yxtay/docker-stacks
Length of output: 7316
🌐 Web query:
LiteLLM proxy config fallbacks scalar list model_fallback documentation
💡 Result:
In LiteLLM Proxy, fallbacks are configured within the litellm_settings section of the configuration file. The fallbacks parameter accepts a list of objects, where each object maps a primary model (or model group) to a list of fallback models [1][2]. Configuration Structure The fallbacks are defined as a list of dictionaries [1][2]: litellm_settings: fallbacks: - primary-model-name: ["fallback-model-1", "fallback-model-2"] In this structure: - The key (e.g., primary-model-name) represents the model group being requested [1][3]. - The value is a scalar list (an array of strings) representing the models to attempt if the primary request fails [1][3]. Key Fallback Types LiteLLM supports different types of fallbacks, which can be configured similarly: - General Fallbacks: Standard failover if a model or provider fails (e.g., after retries) [4][5]. - Context Window Fallbacks: Specific fallbacks triggered when a context window limit is exceeded [2][4]. - Content Policy Fallbacks: Fallbacks triggered by content policy violations [4]. Example Configuration In your proxy config.yaml, you can define multiple types of fallbacks as follows [2]: litellm_settings: fallbacks: [{"gpt-4": ["gpt-3.5-turbo"]}] context_window_fallbacks: [{"gpt-4": ["gpt-4-32k"]}] Behavior and Management - Sequential Execution: Fallbacks are attempted in the order they are defined in the list [4][3]. - Retries: Fallbacks typically trigger after the configured number of retries (num_retries) for the primary model have been exhausted [1][6][4]. - Management Endpoints: You can also manage fallbacks dynamically via the Proxy API without modifying the config file directly, provided STORE_MODEL_IN_DB=True is set in your environment [4]. This includes endpoints like POST /fallback, GET /fallback/{model}, and DELETE /fallback/{model} [4]. Note: Earlier versions of LiteLLM used a deprecated model_fallbacks parameter in the code; modern proxy configurations should use the fallbacks key within litellm_settings [1][7].
Citations:
- 1: https://docs.litellm.ai/docs/proxy/reliability
- 2: https://docs.litellm.ai/docs/proxy/configs
- 3: https://docs.litellm.ai/docs/proxy/model_access_guide
- 4: https://docs.litellm.ai/docs/proxy/fallback_management
- 5: https://docs.litellm.ai/docs/routing-load-balancing
- 6: https://docs.litellm.ai/docs/proxy/load_balancing
- 7: https://github.com/BerriAI/litellm/blob/main/litellm/__init__.py
🌐 Web query:
LiteLLM v1.94.0 model_list_router model_fallbacks Router model_list_router yaml parser
💡 Result:
LiteLLM manages model routing and failover through the Router class and Proxy configuration, which allows for advanced control over model selection and reliability [1][2][3]. Router and Fallback Mechanisms The LiteLLM Router provides automatic failover across different model groups [1][4]. When a model deployment fails after reaching the configured num_retries, the Router initiates a fallback process [2][4]. 1. Deployment-Level Ordering: You can configure the order parameter within the model_list of your YAML configuration [2]. The Router will attempt deployments in ascending order (order 1, then order 2, etc.) before escalating to broader model-level fallbacks [2]. 2. Model-Level Fallbacks: Defined under router_settings.fallbacks in the YAML configuration, these specify which model groups to try if all deployments for the current model group fail [2][5]. The syntax typically maps a primary model group to a list of fallback groups, such as - gpt-4: ["gpt-3.5-turbo"] [2][5]. 3. Weighted Failover: When routing_strategy is set to simple-shuffle (the default), the Router can perform weighted failover, excluding failing deployments and retrying within the same model group before triggering cross-group fallbacks [1]. Configuration via YAML The YAML configuration supports detailed control over routing behavior [3][6]. Key sections include: - model_list: Contains the list of supported models, their deployment parameters (e.g., api_base, api_key), and order for prioritized failover [2][5]. - router_settings: Configures routing behavior, including: - routing_strategy: Strategies such as simple-shuffle, least-busy, usage-based-routing, or latency-based-routing [5][6]. - fallbacks: Defines the mapping for model-level failover [5]. - num_retries, timeout, allowed_fails, and cooldown_time: Control retry logic and deployment health tracking [5][6]. Internal Handling LiteLLM utilizes utility functions (e.g., within litellm/router_utils/fallback_event_handlers.py) to parse these configurations [7]. The internal parser distinguishes between standard fallback formats (model group mapping) and non-standard formats (such as parameter overrides) to ensure the failover chain is processed correctly [7]. For the SDK, completion() calls can also accept an explicit fallbacks parameter to implement similar logic programmatically [4][8][9].
Citations:
- 1: https://docs.litellm.ai/docs/routing
- 2: https://docs.litellm.ai/docs/proxy/load_balancing
- 3: https://docs.litellm.ai/docs/proxy/configs
- 4: https://docs.litellm.ai/docs/proxy/reliability
- 5: https://mintlify.wiki/BerriAI/litellm/api/router-config
- 6: https://mintlify.wiki/BerriAI/litellm/api/config-schema
- 7: https://github.com/BerriAI/litellm/blob/a05a1eef/litellm/router_utils/fallback_event_handlers.py
- 8: https://docs.litellm.ai/docs/tutorials/fallbacks
- 9: https://docs.litellm.ai/docs/completion/reliable_completions
🌐 Web query:
github BerriAI/litellm v1.94.0 fallbacks scalar list Router model_fallbacks
💡 Result:
In LiteLLM (including v1.94.0), the Router's fallbacks configuration is defined as a list of dictionaries, where each dictionary maps a primary model_group (the model that failed) to a list of fallback model_group names to be attempted sequentially [1][2][3]. Example configuration: fallbacks=[{"gpt-4": ["gpt-3.5-turbo", "claude-2"]}] Key details regarding Router fallbacks: 1. Structure and Execution: Fallbacks are processed in the order they appear in the list [4][5]. If a request to a model group fails (and after any configured internal retries are exhausted), the Router attempts the fallback models sequentially until one succeeds or the max_fallbacks limit is reached [4][5][6][7]. The default max_fallbacks value is 5 [8]. 2. Configuration Types: - General Fallbacks: The standard fallbacks parameter handles generic provider or connection errors [4][5]. - Context Window Fallbacks: Configured via context_window_fallbacks, specifically triggered by ContextWindowExceededError [1][2][5]. - Content Policy Fallbacks: Configured via content_policy_fallbacks, triggered by ContentPolicyViolationError [4][1][5]. 3. Management: - Initialization: Fallbacks are typically passed during Router initialization [1][2]. - Proxy Management: If running the LiteLLM Proxy, fallback configurations can also be managed dynamically via dedicated endpoints (e.g., POST /fallback) when STORE_MODEL_IN_DB is enabled [5]. - Internal vs. Public Names: When using team-scoped models or aliases, Ensure that fallback keys match the internal model group names recognized by the router, as lookup issues can occur if using unresolved public names [9]. 4. Relation to Retries: Fallbacks operate at a higher level than internal retries. Retries occur within the same model group (e.g., trying a different deployment), whereas fallbacks redirect the request to an entirely different model group [6][7].
Citations:
- 1: https://berriai-litellm.mintlify.app/features/fallbacks
- 2: https://mintlify.wiki/BerriAI/litellm/api/router
- 3: https://docs.litellm.ai/docs/proxy/reliability
- 4: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 5: https://docs.litellm.ai/docs/proxy/fallback_management
- 6: https://docs.litellm.ai/docs/routing
- 7: https://docs.litellm.ai/docs/router_architecture
- 8: https://github.com/BerriAI/litellm/blob/main/litellm/router.py
- 9: [Bug]: Fallback lookup fails for team-scoped models — model_group uses public name but fallback config requires internal names BerriAI/litellm#28019
🌐 Web query:
site:github.com/BerriAI/litellm litellm/router.py model_fallbacks "fallbacks" "v1.94.0"
💡 Result:
In LiteLLM v1.94.0, model fallbacks are a core reliability feature used to automatically switch to backup models when a primary request fails [1][2]. The mechanism is handled by the litellm.Router for proxy environments or the fallbacks parameter in standard completion() calls [2][3][4]. Key concepts and implementation details for fallbacks as of v1.94.0 include: Types of Fallbacks LiteLLM distinguishes between different categories of failure-based fallbacks, which can be configured independently [2][5]: 1. Fallbacks: Used for general errors, such as rate limits or connectivity issues [2]. 2. Context Window Fallbacks: Specifically triggered by ContextWindowExceededError [2][5]. 3. Content Policy Fallbacks: Triggered by ContentPolicyViolationError [2][5]. 4. Default Fallbacks: A fallback mechanism used if a specific model group is misconfigured or inaccessible [2][5]. Configuration in Proxy (config.yaml) In a LiteLLM Proxy setup, these are defined under litellm_settings in your configuration file [2][4]: litellm_settings: fallbacks: [{"primary-model": ["fallback-model-1", "fallback-model-2"]}] context_window_fallbacks: [{"primary-model": ["larger-context-model"]}] default_fallbacks: ["fallback-model"] Implementation in Code (SDK) When using the SDK directly, you can pass a fallbacks list to the completion() function [1][3]. The library attempts the primary model first, and if it fails, iterates through the provided fallback list [1][3]. from litellm import completion response = completion( model="primary-model", messages=[...], fallbacks=["fallback-model-1", "fallback-model-2"]) Changes and Deprecation Note that model_fallbacks has been deprecated in favor of litellm.fallbacks in the core configuration [6]. LiteLLM actively manages these through router.py and dedicated fallback utilities, ensuring that retries and model switching are handled according to the logic defined in your configuration [7][2][3].
Citations:
- 1: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/completion/reliable_completions.md
- 2: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 3: https://github.com/BerriAI/litellm/blob/62920a0c/litellm/litellm_core_utils/fallback_utils.py
- 4: https://github.com/BerriAI/litellm/blob/4dc9726d/docs/my-website/docs/proxy/configs.md
- 5: https://github.com/BerriAI/litellm/blob/4dc9726d/docs/my-website/docs/proxy/config_settings.md
- 6: https://github.com/BerriAI/litellm/blob/main/litellm/__init__.py
- 7: BerriAI/litellm@v1.94.0-rc.1...v1.95.0-dev.1
🌐 Web query:
site:github.com/BerriAI/litellm litellm/proxy/proxy_server.py yaml config fallbacks v1.94.0
💡 Result:
In LiteLLM Proxy v1.94.0, fallback configurations are defined within the config.yaml file, primarily under router_settings or litellm_settings [1][2]. Fallbacks are designed to improve reliability by allowing the proxy to automatically switch to alternative models or deployments when a request fails [2][3]. Key Configuration Patterns: 1. Standard Fallbacks: Configured under router_settings to handle general failures (e.g., rate limits, connection errors) [1][2]. yaml router_settings: fallbacks: [{"model-a": ["model-b"]}] 2. Error-Specific Fallbacks: LiteLLM supports specialized fallback paths for specific error types [2][4]: - Content Policy Violations: content_policy_fallbacks [4] - Context Window Errors: context_window_fallbacks [2][4] - Default Fallbacks: default_fallbacks can be set as a safety net if a specific model group is misconfigured or fails [2][4]. 3. Load Balancing Integration: When using multiple deployments for the same model name, you can use the order parameter in litellm_params to set priority [5][3]. If all deployments for a specific order level fail, the router exhausts its retries before falling through to the model-level fallbacks defined in router_settings [5][3]. 4. Recent Enhancements: As of recent updates leading into and including v1.94.0, LiteLLM has expanded fallback capabilities: - Rate Limit Handling: Fallbacks now trigger on local rate limit errors (ProxyRateLimitError) encountered during pre-call hooks [6]. - Budget Fallbacks: Key-level budget_fallbacks allow requests to be rerouted to a different model if the current model's budget has been exceeded, rather than blocking the request entirely [7]. For detailed implementation, refer to the official LiteLLM documentation on reliability and routing [2][3].
Citations:
- 1: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/configs.md
- 2: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 3: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/routing.md
- 4: https://github.com/BerriAI/litellm/blob/4dc9726d/docs/my-website/docs/proxy/config_settings.md
- 5: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/load_balancing.md
- 6: fix(proxy): trigger gateway fallbacks on local rate limit errors BerriAI/litellm#31788
- 7: feat(proxy): add key-level budget_fallbacks to reroute requests when a per-model budget is exceeded BerriAI/litellm#31783
🌐 Web query:
"lru_cache" "_parse_fallback" "fallbacks" "model_list_router" "v1.94.0" github.com/BerriAI/litellm
💡 Result:
In LiteLLM version v1.94.0 [1], the components mentioned function as part of the library's routing and reliability infrastructure. LRU Cache and Performance: LiteLLM utilizes functools.lru_cache for several internal hot-path functions, such as get_model_info and _cached_get_model_group_info [2]. Recent optimizations, including those relevant to v1.94.0, increased the DEFAULT_MAX_LRU_CACHE_SIZE from 16 to 64 [2]. This change was implemented to reduce cache eviction thrashing in multi-model production environments [2]. Fallback Mechanism and Router: The model_list_router (typically referenced as Router) [3][4] handles request routing and automatic failover [5][6]. Fallbacks are configured within the Router to ensure reliability [5]. The parsing of these fallbacks, including internal functions like _parse_fallback (often associated with internal validation/initialization logic in litellm/router.py or related utilities) [7][8], ensures that fallback configurations are correctly mapped to model groups [8][9]. Key Architecture points: - Configuration: Fallbacks can be set via the Router initialization (e.g., fallbacks=[{"model-a": ["model-b"]}]) [5]. - Parsing: The router validates and parses these definitions to handle various types of fallbacks, including general, context-window, and content-policy fallbacks [7][10][8]. - Recent Enhancements: Version 1.94.0 included various fixes and updates to the proxy, authentication, and guardrail components [1], building on established mechanisms for model routing and fallback event handling [8][9]. For specific implementation details, refer to litellm/router.py [7] and the utilities located in litellm/router_utils/ [8].
Citations:
- 1: https://github.com/BerriAI/litellm/releases/tag/v1.94.0
- 2: perf: increase default LRU cache size to reduce multi-model thrash BerriAI/litellm#21139
- 3: https://docs.litellm.ai/docs/routing
- 4: https://mintlify.wiki/BerriAI/litellm/api/router
- 5: https://docs.litellm.ai/docs/proxy/reliability
- 6: https://berriai-litellm.mintlify.app/features/loadbalancing
- 7: https://github.com/BerriAI/litellm/blob/main/litellm/router.py
- 8: https://github.com/BerriAI/litellm/blob/e15b37a1/litellm/router_utils/fallback_event_handlers.py
- 9: fix(router): detect standard fallback format when model_group name collides with LiteLLM param BerriAI/litellm#30526
- 10: feat: extended
/v1/modelsendpoint, now it returns with fallbacks on demand BerriAI/litellm#12811
🌐 Web query:
github.com/BerriAI/litellm v1.94.0 "fallbacks" "model_list_router"
💡 Result:
In LiteLLM version 1.94.0, the Router manages failover using a hierarchical approach to ensure high reliability. The primary mechanisms for handling failures are retries (within the same model group) and fallbacks (escalation to different model groups) [1][2]. Router Fallback Logic When a request is made through the Router, it follows this execution flow: 1. Retries: If a call fails, the Router first attempts to retry the request within the same model group using available deployments (e.g., based on weights or order) [3][1]. 2. Fallbacks: If all retries within the current group are exhausted, the Router checks the configured fallbacks list [1][2]. It will then attempt to call the next model group specified in that configuration [4][5]. Configuration and Types You can configure fallbacks by passing them to the Router constructor [4][6]. It supports several specialized fallback types: - General Fallbacks (fallbacks): Handles generic provider/model errors, such as 429 rate limits or 500 server errors [5][2]. - Context Window Fallbacks (context_window_fallbacks): Specifically triggers when a ContextWindowExceededError is raised [7][4][2]. - Content Policy Fallbacks (content_policy_fallbacks): Specifically triggers when a ContentPolicyViolationError is raised [4][2]. Configuration Example: router = Router( model_list=[...], fallbacks=[ {"gpt-4": ["gpt-3.5-turbo", "claude-2"]} ], context_window_fallbacks=[ {"gpt-3.5-turbo": ["gpt-3.5-turbo-16k"]} ]) Key Technical Considerations - Prerequisites: Every model group referenced in your fallback configuration must be explicitly defined in your model_list. If a referenced model is missing, the Router will raise a BadRequestError [2]. - Model Group Aliases: As of recent updates in the 1.9x release cycle, there have been fixes (e.g., PR #29378) to ensure that fallback lookups correctly resolve when using model_group_alias. Previously, the fallback lookup path could fail to match if an alias was used, causing the Router to unintentionally bypass configured fallbacks [8]. - Order-based Routing: You can also use the order parameter in litellm_params within your model_list to prioritize specific deployments. If an order=1 deployment fails, the Router automatically escalates to order=2 deployments before falling through to the configured global fallbacks list [9][3].
Citations:
- 1: https://docs.litellm.ai/docs/router_architecture
- 2: https://gingerlabs.ai/blog/litellm-router-setup-guide
- 3: https://docs.litellm.ai/docs/routing
- 4: https://berriai-litellm.mintlify.app/features/fallbacks
- 5: https://docs.litellm.ai/docs/proxy/reliability
- 6: https://mintlify.wiki/BerriAI/litellm/api/router
- 7: https://github.com/BerriAI/litellm/blob/1b942568897a48f014fa44618ec3ce54d7570a46/litellm/router.py
- 8: fix(router): resolve model_group_alias on the fallback lookup path BerriAI/litellm#29378
- 9: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/routing.md
🌐 Web query:
"def _router_setup" "fallbacks" "litellm_settings" "router_settings" litellm proxy server v1.94.0
💡 Result:
In LiteLLM Proxy v1.94.0, fallbacks are managed through the Router configuration, which orchestrates failover logic between models or model groups when requests fail [1][2][3]. ### Fallback Configuration Fallbacks can be defined in the router_settings section of your proxy configuration (usually config.yaml) or set dynamically at the key/team level [4][5]. The available fallback types are [6][2]: * fallbacks: The default mechanism for handling general errors (e.g., RateLimitError) [6][2]. It allows defining a chain of model groups to attempt sequentially [6][7]. * context_window_fallbacks: Specifically for ContextWindowExceededErrors. These often incorporate pre-call checks to avoid sending requests that exceed token limits [6][2][5]. * content_policy_fallbacks: Specifically for ContentPolicyViolationError [6][5]. * default_fallbacks: A fallback list applied if a specific model group is misconfigured or unreachable [6][2][5]. ### Router Setup Logic The internal method _router_setup initializes these settings [8]. In the LiteLLM Proxy, the router processes requests using an internal loop (typically async_function_with_fallbacks) that respects the following hierarchy [7][8]: 1. Deployments within a model group (load balancing/retries) [7][9]. 2. Configured fallbacks defined in router_settings [7][2]. For example, a configuration might look like this [2][5]: router_settings: fallbacks: - "primary-model-group": ["fallback-model-group-1", "fallback-model-group-2"] context_window_fallbacks: - "primary-model-group": ["larger-context-model"] ### Recent Developments in v1.94.x LiteLLM v1.94.0 and its release candidates include several refinements to proxy reliability and router behavior [1][10]: * Rate Limit Handling: Recent updates ensure that ProxyRateLimitError (raised by pre-call hooks like parallel_request_limiter) now correctly triggers the router's fallback logic rather than immediately returning a 429 error [11]. * Granular Control: Fallbacks can now be configured specifically for individual API keys or teams via the proxy UI or configuration, allowing for custom fallback chains that override global defaults [4][3]. * Bug Fixes: Specific fixes have addressed issues where metadata parameters were incorrectly handled during failover in certain routes (e.g., /v1/responses/compact) [12]. For further implementation details, consult the official reliability documentation [2] and the router configuration API references [13].
Citations:
- 1: https://github.com/BerriAI/litellm/releases/tag/v1.94.0
- 2: https://docs.litellm.ai/docs/proxy/reliability
- 3: https://docs.litellm.ai/docs/routing-load-balancing
- 4: https://docs.litellm.ai/docs/proxy/keys_teams_router_settings
- 5: https://github.com/BerriAI/litellm/blob/4dc9726d/docs/my-website/docs/proxy/config_settings.md
- 6: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 7: https://docs.litellm.ai/docs/proxy/load_balancing
- 8: https://github.com/BerriAI/litellm/blob/main/litellm/router.py
- 9: https://docs.litellm.ai/docs/routing
- 10: https://github.com/BerriAI/litellm/releases/tag/v1.94.0-rc.1
- 11: fix(proxy): trigger gateway fallbacks on local rate limit errors BerriAI/litellm#31788
- 12: [Bug]: /v1/responses/compact fails during router_settings.fallbacks failover — 'Unknown parameter: metadata' (hardcoded 'metadata' key in run_async_fallback) BerriAI/litellm#25402
- 13: https://mintlify.wiki/BerriAI/litellm/api/router-config
🌐 Web query:
"model_fallbacks" "litellm_settings" "fallbacks" "Router(" litellm proxy_server.py
💡 Result:
In LiteLLM, model fallbacks allow for automatic provider failover when a primary model group fails after all retries are exhausted [1][2]. Configuration Methods 1. Router Initialization (Python) When using the Router class directly, you define fallbacks by mapping the primary model name to a list of backup model names [3][4]. router = Router( model_list=[...], fallbacks=[{"primary-model": ["backup-model-1", "backup-model-2"]}]) 2. Proxy Configuration (litellm_settings) In the LiteLLM Proxy configuration (e.g., config.yaml), you define these under litellm_settings [3][4][5]. litellm_settings: fallbacks: - primary-model: ["backup-model-1", "backup-model-2"] Specialized Fallbacks LiteLLM also supports specific error-handling fallbacks, which can be configured similarly in the config file [2][6]: - content_policy_fallbacks: Used specifically for ContentPolicyViolationError [2][6]. - context_window_fallbacks: Used specifically for ContextWindowExceededError [2][6]. - default_fallbacks: A catch-all list of models to use if a specific group is misconfigured or fails [6][5]. Hierarchy and Management - Resolution Order: Router settings are applied hierarchically: Key-level settings > Team-level settings > Global (Proxy) settings [7]. - Proxy Server: The proxy_server.py logic utilizes these settings to coordinate the Router, which manages load balancing, retries (within a model group), and fallbacks (across model groups) [8][9]. - Requirements: All models referenced in a fallback chain must be defined in the model_list [2]. Key Reliability Settings In your configuration, you can also define: - num_retries: Number of retries within a model group before triggering a fallback [3][2]. - allowed_fails & cooldown_time: Settings to temporarily remove failing models from rotation [3][2]. - max_fallbacks: Sets a limit on the number of fallback attempts (default is 5) [10][11].
Citations:
- 1: https://docs.litellm.ai/docs/routing-load-balancing
- 2: https://gingerlabs.ai/blog/litellm-router-setup-guide
- 3: https://docs.litellm.ai/docs/proxy/reliability
- 4: https://mintlify.wiki/BerriAI/litellm/api/router-config
- 5: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 6: https://github.com/BerriAI/litellm/blob/4dc9726d/docs/my-website/docs/proxy/config_settings.md
- 7: https://docs.litellm.ai/docs/proxy/keys_teams_router_settings
- 8: https://github.com/BerriAI/litellm/blob/main/litellm/proxy/proxy_server.py
- 9: https://docs.litellm.ai/docs/router_architecture
- 10: https://github.com/BerriAI/litellm/blob/e59e34bed3670a6894d43129c2af16af28057d03/litellm/router.py
- 11: https://docs.litellm.ai/docs/routing
🌐 Web query:
"litellm_settings" "fallbacks" "Router(" "model_fallbacks" "fallbacks" "None"
💡 Result:
In LiteLLM, fallbacks are used to automatically route a request to a different model group if the primary model deployment fails [1][2]. These can be configured in the Router using either the fallbacks parameter in the Router constructor [3][4] or via litellm_settings in a configuration file (often used with the LiteLLM Proxy) [1][4][5]. Configuration Structure Fallbacks are defined as a list of dictionaries, where each dictionary maps a primary model group to a list of fallback model groups [3][4]. Example (Python Router): router = Router( model_list=[...], fallbacks=[ {"gpt-4": ["gpt-3.5-turbo", "claude-2"]} ]) Example (Config YAML): litellm_settings: fallbacks: - gpt-4: ["gpt-3.5-turbo", "claude-2"] Behavior and Key Details 1. Triggering: Fallbacks are generally triggered after the configured number of retries (num_retries) for a deployment has been exhausted [6][2]. 2. Context Window Fallbacks: There is a separate setting, context_window_fallbacks, specifically for handling ContextWindowExceededError [3][4]. These are triggered immediately when this specific error occurs, rather than waiting for general retries to exhaust [7][8]. 3. Execution: When a failure occurs, the router iterates through the list of fallback model groups in the order specified until one succeeds [7][5]. 4. Management: In the LiteLLM Proxy, you can also use dedicated fallback management API endpoints (/fallback) to manage these configurations dynamically without needing to restart or update the entire proxy configuration, provided STORE_MODEL_IN_DB=True is set [6][9]. Note: "Weighted failover" is a separate mechanism available when using the simple-shuffle routing strategy; it retries across deployments within the same model group before escalating to cross-group fallbacks [8].
Citations:
- 1: https://docs.litellm.ai/docs/proxy/reliability
- 2: https://docs.litellm.ai/docs/router_architecture
- 3: https://mintlify.wiki/BerriAI/litellm/api/router
- 4: https://mintlify.wiki/BerriAI/litellm/api/router-config
- 5: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 6: https://docs.litellm.ai/docs/proxy/fallback_management
- 7: https://github.com/BerriAI/litellm/blob/1b942568897a48f014fa44618ec3ce54d7570a46/litellm/router.py
- 8: https://docs.litellm.ai/docs/routing
- 9: https://docs.litellm.ai/docs/routing-load-balancing
🌐 Web query:
"https://raw.githubusercontent.com/BerriAI/litellm/v1.94.0/litellm/proxy/proxy_server.py" "fallbacks" "litellm_settings"
💡 Result:
In LiteLLM Proxy, the litellm_settings section in your config.yaml file is used to configure global module-level settings for the LiteLLM library, including reliability features like fallbacks [1][2][3]. The fallbacks configuration under litellm_settings enables automatic provider failover [4][5]. If a model call fails (after exhausting configured retries), LiteLLM can automatically route the request to a fallback model [4][6]. Example Configuration: litellm_settings: num_retries: 3 fallbacks: [{"zephyr-beta": ["gpt-3.5-turbo"]}] When the proxy server starts, it reads these settings from your configuration file within proxy_server.py. The relevant code processes litellm_settings to initialize various components, such as callbacks, caching, and reliability logic, ensuring these configurations are applied consistently across the proxy instance [7][3]. There are three primary types of fallback configurations: 1. fallbacks: For general errors (e.g., RateLimitError) [6]. 2. content_policy_fallbacks: Specifically for content policy violations [6]. 3. context_window_fallbacks: Specifically for context window exceeded errors [6][2]. In addition to litellm_settings, you can manage routing and fallbacks more granularly using router_settings or by setting an order parameter within individual model definitions in model_list for load-balancing scenarios [2][8][5].
Citations:
- 1: https://docs.litellm.ai/docs/proxy/config_settings
- 2: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/configs.md
- 3: https://github.com/BerriAI/litellm/blob/0d803e13798db40aa7463e64a6bafaee386424f5/litellm/proxy/proxy_server.py
- 4: https://docs.litellm.ai/docs/proxy/reliability
- 5: https://docs.litellm.ai/docs/routing-load-balancing
- 6: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 7: https://github.com/BerriAI/litellm/blob/main/litellm/proxy/proxy_server.py
- 8: https://docs.litellm.ai/docs/proxy/load_balancing
🌐 Web query:
"https://raw.githubusercontent.com/BerriAI/litellm/v1.94.0/litellm/router.py" "fallbacks" "model_fallbacks"
💡 Result:
In LiteLLM, model fallbacks are a reliability mechanism designed for automatic provider failover [1][2]. When a primary model group fails—after exhausting all configured retries within that group—the Router attempts to route the request to a pre-defined backup model group [1][3]. The Router class (located in litellm/router.py) handles these fallbacks using three primary parameters, each targeting specific error classes [4][5][2]: 1. fallbacks: Catches general operational errors, such as 429 RateLimitErrors, 500 server errors, or connection failures [6][2]. 2. context_window_fallbacks: Specifically handles ContextWindowExceededError when input tokens exceed a model's limit [6][2]. 3. content_policy_fallbacks: Specifically handles ContentPolicyViolationError [6][2]. Configuration details: - Definition: You define fallbacks as a list of dictionaries where the key is the primary model group and the value is a list of fallback model groups to try in order (e.g., [{"gpt-4": ["gpt-3.5-turbo"]}]) [1][7]. - Prerequisites: Every model referenced in a fallback configuration must exist within the model_list provided to the Router, or the router will raise a BadRequestError [2]. - Precedence: Retries occur first within the same model group (load balancing) [3][2]. If those fail, the router escalates to the configured cross-group fallbacks [8][3]. - Customization: These settings can be configured globally in the Router constructor or at the request level via the .completion() call for granular control [1][6]. The underlying implementation in litellm/router.py validates these parameters during initialization and utilizes utility functions (e.g., get_fallback_model_group in litellm/router_utils/fallback_event_handlers.py) to determine the appropriate fallback path when an exception occurs [4][5][9].
Citations:
- 1: https://docs.litellm.ai/docs/proxy/reliability
- 2: https://gingerlabs.ai/blog/litellm-router-setup-guide
- 3: https://docs.litellm.ai/docs/router_architecture
- 4: https://github.com/BerriAI/litellm/blob/main/litellm/router.py
- 5: https://github.com/BerriAI/litellm/blob/1b942568897a48f014fa44618ec3ce54d7570a46/litellm/router.py
- 6: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 7: https://mintlify.wiki/BerriAI/litellm/api/router-config
- 8: https://docs.litellm.ai/docs/routing
- 9: https://github.com/BerriAI/litellm/blob/a05a1eef/litellm/router_utils/fallback_event_handlers.py
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== router_settings fallbacks vs litellm_settings fallbacks =="
python3 - <<'PY'
from pathlib import Path
import re
p = Path('agent/litellm/config.yaml')
text = p.read_text()
m = re.search(r'(?s)^router_settings:\n(.*?)(?=\n[a-z]|\Z)', text)
print('router_settings contains fallbacks:', bool(m and re.search(r'(?m)^\s*fallbacks:', m.group(1))))
print('top-level litellm_settings fallbacks present:', bool(re.search(r'(?m)^\s*fallbacks:', text)))
print('router settings excerpt:')
if m:
lines = m.group(1).splitlines()[:40]
for n, line in enumerate(lines, 1):
print(f'{n:4}: {line}')
print('fallbacks excerpt:')
for i, line in enumerate(text.splitlines(), 1):
if i >= 24 and i <= 32:
print(f'{i:4}: {line}')
PY
echo
echo "== LiteLLM v1.94.0 source: proxy router setup excerpt =="
python3 - <<'PY'
from urllib.request import urlopen
url = "https://raw.githubusercontent.com/BerriAI/litellm/v1.94.0/litellm/proxy/proxy_server.py"
text = urlopen(url, timeout=20).read().decode()
for needle in ["router_settings", "Router(","model_fallbacks", "fallbacks="]:
print(f"\n-- occurrences of {needle!r} --")
for i, line in enumerate(text.splitlines(), 1):
if needle in line:
lo, hi = max(1, i-4), min(len(text.splitlines()), i+8)
lines = text.splitlines()
print(f"--- around line {i} ---")
for j in range(lo, hi+1):
print(f"{j:4}: {lines[j-1]}")
print()
break
PYRepository: yxtay/docker-stacks
Length of output: 2937
🌐 Web query:
"litellm_settings" "fallbacks" "Router" "litellm_settings" "model_fallbacks" "router_settings"
💡 Result:
In LiteLLM, fallbacks (automatic failover) and router configurations are managed through settings in your configuration file (often config.yaml) or direct router initialization [1][2][3]. Key Concepts and Settings: 1. litellm_settings These settings control the behavior of the LiteLLM module globally, affecting how requests are handled across the proxy or application [4]. - fallbacks: Defines a mapping of model groups to fallback models (e.g., [{"primary-model": ["fallback-model"]}]). These are triggered after the configured number of retries fails [1][5]. - context_window_fallbacks: Specifically handles ContextWindowExceededErrors by falling back to specified models [1][5]. - content_policy_fallbacks: Specifically handles ContentPolicyViolationErrors [1][5]. - default_fallbacks: Used as a catch-all if a specific model group is misconfigured [5]. - num_retries: Sets the number of retries attempted on a specific deployment before triggering the fallback path [4][5]. 2. router_settings These settings control the LiteLLM Router, which handles load balancing, retries within a model group, and deployment-level failover [6][7]. - fallbacks: You can define fallback hierarchies here, often used in conjunction with deployment order [7][8]. - routing_strategy: Defines how traffic is distributed (e.g., simple-shuffle, least-busy). Defaults to simple-shuffle [8][9]. - model_group_alias: Allows aliasing one model name to another globally [8][4]. - max_fallbacks: Caps the number of fallback attempts (default is 5) [6]. Relationship and Behavior: - Retries vs. Fallbacks: Retries occur within the same model group (trying different deployments) [2]. If all retries fail, the router escalates to the configured fallback model group [2][7]. - Deployment Priority: By setting the order parameter in litellm_params for individual deployments, you can create a prioritized chain (e.g., order 1 -> order 2). The router will automatically try these in sequence before moving to the general fallback path defined in settings [7][8]. - Fallback Management: Recent versions of the LiteLLM Proxy support dedicated endpoints (e.g., POST /fallback) for managing these configurations dynamically without updating the entire proxy config [10]. Example Configuration Snippet: litellm_settings: num_retries: 3 fallbacks: [{"gpt-4": ["gpt-3.5-turbo"]}] router_settings: routing_strategy: simple-shuffle timeout: 30
Citations:
- 1: https://docs.litellm.ai/docs/proxy/reliability
- 2: https://docs.litellm.ai/docs/router_architecture
- 3: https://docs.litellm.ai/docs/routing-load-balancing
- 4: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/configs.md
- 5: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 6: https://docs.litellm.ai/docs/routing
- 7: https://docs.litellm.ai/docs/proxy/load_balancing
- 8: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/load_balancing.md
- 9: https://github.com/BerriAI/litellm/blob/4dc9726d/docs/my-website/docs/proxy/config_settings.md
- 10: https://docs.litellm.ai/docs/proxy/fallback_management
🌐 Web query:
Github BerriAI/litellm v1.94.0 "fallbacks=" "litellm_settings" Router
💡 Result:
In LiteLLM, you set Router (fallback chain) reliability at the litellm_settings level using the fallbacks key. The fallback mapping is configured as List[Dict[str, List[str]]], where each dict maps a “model_name” (model group) to an ordered list of fallback “model_name” values. Example: litellm_settings: fallbacks: [{"zephyr-beta": ["gpt-3.5-turbo"]}] [1][2]. How to express your query’s idea (“BerriAI/litellm v1.94.0 fallbacks= litellm_settings Router”): 1) Global (proxy-wide) fallbacks (via litellm_settings) YAML: litellm_settings: fallbacks: [{"zephyr-beta": ["gpt-3.5-turbo"]}] This means: when the router tries zephyr-beta and the call fails, it will attempt gpt-3.5-turbo as the fallback [1][2]. 2) If you also want “default” fallbacks for misconfigured/bad model groups default_fallbacks is a separate setting used when a specific model group is misconfigured / bad [3]. 3) Related (optional) router settings (retries/cooldowns) num_retries and cooldown-related settings like allowed_fails / cooldown_time are configured alongside fallbacks under router/module reliability configuration [1][4]. Router settings for keys/teams: LiteLLM also supports setting different router settings (including fallbacks) at the API key / team level with hierarchical resolution (Keys > Teams > Global) [5]. If you share the exact config you’re using (or paste your model_list + current litellm_settings/router_settings), I can rewrite it precisely for the intended fallback behavior in v1.94.0.
Citations:
- 1: https://docs.litellm.ai/docs/proxy/reliability
- 2: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 3: https://github.com/BerriAI/litellm/blob/4dc9726d/docs/my-website/docs/proxy/config_settings.md
- 4: https://docs.litellm.ai/docs/proxy/config_settings
- 5: https://docs.litellm.ai/docs/proxy/keys_teams_router_settings
🌐 Web query:
site:github.com/BerriAI/litellm "Router(" "model_fallbacks" "litellm_settings" "->" "fallbacks"
💡 Result:
In LiteLLM, fallbacks for the Router are configured using the fallbacks parameter, which maps a primary model to a list of fallback models to be tried if the primary model fails [1][2][3]. When configuring a Router, fallbacks can be defined in a few ways depending on your implementation: 1. SDK/Python Initialization: You pass a list of dictionaries to the fallbacks parameter of the Router constructor [1]. Each dictionary maps a primary model name to a list of fallback model names (e.g., fallbacks=[{"gpt-3.5-turbo": ["gpt-4"]}]). 2. Configuration File (Proxy/YAML): When using a config file, you define fallbacks under litellm_settings [1][4][5]. The syntax follows the same mapping structure: litellm_settings: fallbacks: [{"primary-model-name": ["fallback-model-1", "fallback-model-2"]}] Router Execution Flow: - Order-Based Priority: The Router first attempts to fulfill requests using deployments defined with the order parameter in model_list (where lower values represent higher priority) [6][3]. It will cycle through these deployments based on their order [6][3]. - Fallback Execution: If all deployments for a specific model are exhausted or fail, the Router falls through to the models defined in the fallbacks configuration [6][3]. - Context Window Fallbacks: Additionally, you can configure context_window_fallbacks within litellm_settings to specifically handle cases where a model fails due to context window limits [4][5][7]. For robust reliability, the Router also supports settings like num_retries, allowed_fails, and cooldown_time, which work alongside these fallback mechanisms to manage traffic effectively [1][4][3].
Citations:
- 1: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 2: https://github.com/BerriAI/litellm/blob/main/litellm/router.py
- 3: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/load_balancing.md
- 4: https://github.com/BerriAI/litellm/blob/d251238b/docs/my-website/docs/proxy/configs.md
- 5: https://github.com/BerriAI/litellm/blob/v1.82.6.dev2/docs/my-website/docs/proxy/configs.md
- 6: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/routing.md
- 7: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/completion/reliable_completions.md
🌐 Web query:
"Router(" "model_fallbacks=" "fallbacks=" "litellm_settings" ".get"
💡 Result:
In the LiteLLM Router, the construct Router(model_fallbacks=fallbacks=litellm_settings.get(...)) appears to be a syntactically incorrect attempt to configure fallbacks [1][2]. The correct way to configure fallbacks in LiteLLM depends on whether you are using the Router class directly in Python or a configuration file (e.g., config.yaml) for the LiteLLM Proxy [3][4][5]. When using the Router class in Python, fallbacks are passed as a direct argument to the constructor, not through a model_fallbacks parameter [2]. The standard fallbacks parameter accepts a list of dictionaries [2]. Example Python usage: router = Router( model_list=[...], fallbacks=[{"gpt-4": ["gpt-3.5-turbo"]}] # List of dicts mapping model to fallbacks [2]) When using the LiteLLM Proxy (via config.yaml), fallbacks are defined under litellm_settings [3][6]. Example YAML configuration: litellm_settings: fallbacks: - gpt-4: ["gpt-3.5-turbo"] # Fallback mapping [4][6] To access these settings programmatically from a configuration object (like litellm_settings), you would typically use standard dictionary .get() methods, such as litellm_settings.get("fallbacks"), and then pass the resulting list to the fallbacks parameter of the Router [1][4]. The Router initialization code should look more like this: # Correct programmatic approach fallbacks = litellm_settings.get("fallbacks", []) router = Router( model_list=[...], fallbacks=fallbacks) Additionally, LiteLLM supports specialized fallback types, such as context_window_fallbacks and content_policy_fallbacks, which are configured similarly to general fallbacks [4][7]. If you are working with the LiteLLM Proxy, you can also manage these dynamically via dedicated API endpoints (e.g., POST /fallback) instead of modifying the config file directly [7].
Citations:
- 1: https://github.com/BerriAI/litellm/blob/main/litellm/router.py
- 2: https://mintlify.wiki/BerriAI/litellm/api/router
- 3: https://docs.litellm.ai/docs/proxy/reliability
- 4: https://mintlify.wiki/BerriAI/litellm/api/router-config
- 5: https://github.com/BerriAI/litellm/blob/d251238b/docs/my-website/docs/proxy/configs.md
- 6: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 7: https://docs.litellm.ai/docs/proxy/fallback_management
🌐 Web query:
GitHub BerriAI/litellm "def _router_setup" "fallbacks" "litellm_settings"
💡 Result:
In the LiteLLM library, the Router's fallback mechanisms and litellm_settings are integral to managing reliability and routing behavior [1][2]. ### Router Setup and Fallbacks The Router class in LiteLLM initializes fallback logic during its setup phase [3][4]. When initializing a Router, you can define fallback models to be used if a primary model call fails [2][5]. Configuration can be handled via code or a YAML configuration file [6][1]. #### Programmatic Setup (Python) When initializing the Router, you pass the fallbacks parameter as a list of dictionaries, where each dictionary maps a model (or "*" for all models) to a list of fallback models [3][2]. python from litellm import Router router = Router( model_list=[...], fallbacks=[ {"gpt-4": ["gpt-3.5-turbo", "claude-2"]} ] ) #### Proxy Configuration (YAML) In a LiteLLM Proxy setup, you can define fallbacks within the litellm_settings section of your configuration file [6][7]. yaml litellm_settings: fallbacks: [{"gpt-4": ["gpt-3.5-turbo", "claude-2"]}] ### Key Concepts * litellm_settings: This section in the configuration file controls global LiteLLM module settings, such as num_retries, request_timeout, and fallbacks [6][8]. These settings apply to the Router's behavior when processing requests [6]. * Fallback Types: LiteLLM supports various fallback scenarios, including generic model fallbacks (used when a model fails), context window fallbacks (context_window_fallbacks), and content policy violations [2][9][8]. * Cooldowns: Router settings include allowed_fails and cooldown_time, which work alongside fallbacks to manage deployment health by temporarily disabling models that fail frequently [6][5][8]. * Pre-call checks: The router can perform checks, such as verifying if a request is within a model's context window, before attempting the call [1][8]. For detailed configuration parameters, refer to the litellm/router.py source code or the official LiteLLM documentation [3][6][7].
Citations:
- 1: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/routing.md
- 2: https://berriai-litellm.mintlify.app/features/fallbacks
- 3: https://github.com/BerriAI/litellm/blob/main/litellm/router.py
- 4: https://github.com/BerriAI/litellm/blob/62920a0c/litellm/router.py
- 5: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/reliability.md
- 6: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/configs.md
- 7: https://docs.litellm.ai/docs/proxy/reliability
- 8: https://github.com/BerriAI/litellm/blob/4dc9726d/docs/my-website/docs/proxy/config_settings.md
- 9: https://mintlify.wiki/BerriAI/litellm/api/router-config
🌐 Web query:
LiteLLM v1.94.0 source router_utils fallback_event_handlers.py model_list_router fallbacks
💡 Result:
In LiteLLM, fallback mechanisms are orchestrated by the Router to ensure reliability when primary model deployments fail [1]. The file litellm/router_utils/fallback_event_handlers.py contains core utility functions that manage these fallback sequences [2][3][4]. Key mechanisms and components include: 1. Fallback Execution Logic: The run_async_fallback function in fallback_event_handlers.py is responsible for iterating through configured fallback model groups [2][4]. If a primary call fails, it catches the exception and attempts to call the next model group in the sequence [2][1]. It logs successes and failures for each step and raises the final exception if all configured fallbacks are exhausted [2][3]. 2. Configuration and Setup: The Router class (located in litellm/router.py) handles the configuration of fallbacks [5][6][7]. Fallbacks can be set at the Router level or passed during specific calls [8]. The router validates these configurations and organizes them into self.fallbacks [5]. 3. Fallback Types: - General Fallbacks: Managed via the fallbacks parameter, used for standard errors like RateLimitError [8]. - Specialized Fallbacks: context_window_fallbacks handle ContextWindowExceededError, and content_policy_fallbacks handle ContentPolicyViolationError [8]. - Weighted Failover: When enable_weighted_failover is active, the router first attempts to retry within the same model group by excluding the failed deployment before escalating to cross-group fallbacks [9]. 4. Integration: The router architecture wraps the initial request in function_with_fallbacks, which handles the logic of shifting to a different model group if retries within the current group (managed by function_with_retries) fail [1]. This system ensures that if a model or provider becomes unavailable, the traffic automatically reroutes to a healthy alternative defined in your fallback configuration [8][1].
Citations:
- 1: https://docs.litellm.ai/docs/router_architecture
- 2: https://github.com/BerriAI/litellm/blob/e15b37a1/litellm/router_utils/fallback_event_handlers.py
- 3: https://github.com/BerriAI/litellm/blob/a05a1eef/litellm/router_utils/fallback_event_handlers.py
- 4: https://github.com/BerriAI/litellm/blob/2b2ae4ca/litellm/router_utils/fallback_event_handlers.py
- 5: https://github.com/BerriAI/litellm/blob/4dc9726d/litellm/router.py
- 6: https://github.com/BerriAI/litellm/blob/e59e34bed3670a6894d43129c2af16af28057d03/litellm/router.py
- 7: https://github.com/BerriAI/litellm/blob/1b942568897a48f014fa44618ec3ce54d7570a46/litellm/router.py
- 8: https://docs.litellm.ai/docs/proxy/reliability
- 9: https://docs.litellm.ai/docs/routing
🌐 Web query:
LiteLLM v1.94.0 source router_utils fallback_event_handlers.py Router fallbacks string list
💡 Result:
In LiteLLM v1.94.0, the litellm/router_utils/fallback_event_handlers.py module manages the logic for cross-group model failover [1][2]. The router allows configuring fallbacks using a list of model groups [3][4]. The configuration format for fallbacks supports strings or dictionaries [1][5]: 1. String List: You can specify a simple list of model groups to fall back to sequentially if the primary call fails (e.g., fallbacks: ["gpt-3.5-turbo", "claude-3-haiku"]) [1]. 2. Dictionary Mapping: For more granular control, dictionaries allow mapping a specific model group to its own set of fallbacks (e.g., fallbacks: [{"gpt-4": ["gpt-3.5-turbo"]}]) [1][3]. Internal Logic and Handling: - The get_fallback_model_group function in fallback_event_handlers.py resolves the appropriate fallback chain by checking for exact model group matches, stripped provider prefixes, or wildcard * catch-alls [1][2]. - The run_async_fallback function iterates through these defined fallback groups recursively until a call succeeds or the max_fallbacks limit is reached [6][1][2]. - LiteLLM maintains specialized handling for dictionary-based fallback entries, including logic to distinguish between standard model-group mapping dictionaries and those containing LiteLLM parameter overrides [7][5]. This ensures that when a model group name happens to collide with a reserved LiteLLM parameter name (like "model"), the router correctly identifies it as a mapping rather than an invalid parameter configuration [7]. These mechanisms allow for robust provider and model failover, separate from the intra-group retries managed by function_with_retries [8][9].
Citations:
- 1: https://github.com/BerriAI/litellm/blob/2b2ae4ca/litellm/router_utils/fallback_event_handlers.py
- 2: https://deepwiki.com/BerriAI/litellm/7.1-fallbacks-and-retries
- 3: https://docs.litellm.ai/docs/proxy/reliability
- 4: https://docs.litellm.ai/docs/routing-load-balancing
- 5: Admin UI router settings crashes on dict-based fallback entries that runtime supports BerriAI/litellm#26473
- 6: https://github.com/BerriAI/litellm/blob/e15b37a1/litellm/router_utils/fallback_event_handlers.py
- 7: fix(router): detect standard fallback format when model_group name collides with LiteLLM param BerriAI/litellm#30526
- 8: https://docs.litellm.ai/docs/routing
- 9: https://docs.litellm.ai/docs/router_architecture
Make the LiteLLM proxy use list-valued fallback targets.
LiteLLM proxy fallback mappings define fallback model names as a list; scalar values are not a documented mapping format.
Proposed fix
fallbacks:
- - reasoning: complex
- - complex: medium
@@ -26,7 +26,7 @@
fallbacks:
+ - reasoning: [complex]
+ - complex: [medium]
+ - medium: [simple]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| fallbacks: | |
| - reasoning: complex | |
| - complex: medium | |
| - medium: simple | |
| fallbacks: | |
| - reasoning: [complex] | |
| - complex: [medium] | |
| - medium: [simple] |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@agent/litellm/config.yaml` around lines 26 - 30, Update the fallback mappings
in the LiteLLM configuration so each target value is a list containing its
fallback model name, including the reasoning, complex, and medium mappings.
Preserve the existing fallback order and model names while replacing scalar
values with list-valued entries.
1 participant
Summary by CodeRabbit
New Features
Bug Fixes