[Re-issue: Fix] Keys and Teams Router Setting + Allow Override of Router Settings

[yuneng-jiang]

Relevant issues

Fixes #19693

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🐛 Bug Fix
✅ Test

Changes

This PR fixes search tools not being found when using per-request routers by:

• Avoiding per-request Router instantiation: Instead of creating a new Router instance when a key/team has router_settings (which is expensive and loses search_tools), settings are now passed as router_settings_override kwargs to the main router.

• Creating router even with empty model list: The router is now created even when model_list is empty if search_tools are configured, since the router can function with only search tools.

• Supporting per-request model_group_retry_policy: Added support for model_group_retry_policy as a per-request override in the Router's retry logic.

• Using cached key/team objects for router settings: _get_hierarchical_router_settings reads from the already-authenticated key object and the cached team lookup (get_team_object) instead of making direct DB queries on the hot path.

The affected settings that can be overridden per-request are: fallbacks, context_window_fallbacks, content_policy_fallbacks, num_retries, timeout, and model_group_retry_policy.

Added router_settings field to LiteLLM_VerificationToken to support key-level router settings.

Screenshots

Testing

Testing was done mainly with fallbacks with these 3 mappings:
Key: claude-haiku-4-5 will fallback to gpt-4o
Team: claude-haiku-4-5 will fallback to claude-sonnet-4-5
Global: claude-haiku-4-5 will fallback to grok-4-fast-non-reasoning

Team only fallback (key without router settings):

Key and team with router settings:

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Feb 6, 2026 11:09pm

[greptile-apps]

Greptile Overview

Greptile Summary

This PR fixes a critical performance issue where search_tools were not accessible when using per-request router settings. The fix replaces expensive per-request Router instantiation with a router_settings_override approach that merges settings into the main router's kwargs.

Key Changes:

• Stores hierarchical router settings as router_settings_override instead of creating a new Router with user_config
• Allows Router creation even with empty model_list when search_tools are configured
• Adds per-request model_group_retry_policy override support in the Router's retry logic
• Comprehensive test coverage validates the new approach and ensures request-level settings take precedence over key/team settings

Performance Impact:
This change directly addresses the custom rule about avoiding Router object creation in the critical request path, significantly improving request latency by reusing the main router instance.

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• The changes are well-architected, follow best practices, and include comprehensive test coverage. The approach correctly avoids expensive Router instantiation per-request while maintaining the same functionality. All settings precedence is properly handled (request-level > key/team > global).
• No files require special attention

Important Files Changed

Filename	Overview
litellm/proxy/common_request_processing.py	Replaced per-request Router instantiation with router_settings_override to avoid expensive object creation, following performance guidelines
litellm/proxy/route_llm_request.py	Replaced user_config Router instantiation with router_settings_override merging, respecting request-level precedence
litellm/router.py	Added per-request model_group_retry_policy override support and updated retry logic to use helper function

Sequence Diagram

sequenceDiagram
    participant Client
    participant ProxyServer
    participant CommonRequestProcessing
    participant RouteRequest
    participant Router
    participant LLMProvider

    Client->>ProxyServer: POST /chat/completions
    ProxyServer->>CommonRequestProcessing: process_request()
    
    Note over CommonRequestProcessing: Get hierarchical router settings<br/>(from key/team/global config)
    
    alt Router settings found
        CommonRequestProcessing->>CommonRequestProcessing: Store as router_settings_override
        Note over CommonRequestProcessing: OLD: Created new Router instance<br/>NEW: Store settings for later merge
    end
    
    CommonRequestProcessing->>RouteRequest: route_request(data, router, ...)
    
    alt router_settings_override exists
        RouteRequest->>RouteRequest: Extract override_settings
        loop For each per_request_setting
            RouteRequest->>RouteRequest: Merge into data if not already set
        end
        Note over RouteRequest: Settings: fallbacks, num_retries,<br/>timeout, model_group_retry_policy, etc.
        RouteRequest->>Router: acompletion(**data)
        Note over Router: Uses main router with<br/>merged override settings
    else No override
        RouteRequest->>Router: acompletion(**data)
    end
    
    Router->>Router: async_function_with_retries()
    Note over Router: Extract per-request overrides:<br/>- fallbacks<br/>- model_group_retry_policy<br/>- timeout, num_retries
    
    Router->>LLMProvider: Make API call
    
    alt Success
        LLMProvider-->>Router: Response
        Router-->>RouteRequest: Response
        RouteRequest-->>ProxyServer: Response
        ProxyServer-->>Client: 200 OK
    else Error with retry policy
        LLMProvider-->>Router: Error
        Router->>Router: _get_num_retries_from_retry_policy()
        Note over Router: Uses per-request<br/>model_group_retry_policy<br/>if provided
        Router->>LLMProvider: Retry with fallback
        LLMProvider-->>Router: Response
        Router-->>RouteRequest: Response
        RouteRequest-->>ProxyServer: Response
        ProxyServer-->>Client: 200 OK
    end

Loading

[greptile-apps]

_{3 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[yuneng-jiang]

@greptile

[greptile-apps]

Greptile Overview

Greptile Summary

This PR optimizes router settings handling to avoid expensive per-request Router instantiation. Previously, when keys or teams had custom router settings, a new Router object was created for each request, which was costly and lost search_tools configuration.

Key changes:

• Performance improvement: Router settings from keys/teams are now passed as router_settings_override kwargs to the main router instead of creating new Router instances, adhering to the custom rule about avoiding object creation in the critical request path
• Cached lookups: Uses already-cached key objects and team cache (get_team_object) to retrieve hierarchical settings (key > team > global) without direct DB queries
• Search tools support: Router is now created even with empty model_list when search_tools are configured, fixing the search tools not found issue
• Per-request retry policy: Added support for model_group_retry_policy as a per-request override in the Router's retry logic

The implementation correctly merges only supported per-request settings (fallbacks, context_window_fallbacks, content_policy_fallbacks, num_retries, timeout, model_group_retry_policy) and respects request-level precedence over key/team settings.

Confidence Score: 4/5

• This PR is safe to merge with minimal risk after addressing the exception handling issue
• The PR implements a well-tested performance optimization that aligns with the custom rule about avoiding object creation in the request path. The approach is sound: using cached lookups and passing settings as kwargs instead of creating new Router instances. Tests are comprehensive and cover edge cases. One minor syntax issue exists in exception handling (line 3426) where yaml.safe_load() can't raise json.JSONDecodeError, but this won't cause failures - it just makes the except clause unnecessarily broad. The change is backwards compatible and follows existing patterns in the codebase.
• Pay attention to litellm/proxy/proxy_server.py for the exception handling fix on line 3426

Important Files Changed

Filename	Overview
litellm/proxy/common_request_processing.py	Retrieves hierarchical router settings (key > team) and passes them as router_settings_override instead of creating new Router
litellm/proxy/proxy_server.py	Added _parse_router_settings_value and _get_hierarchical_router_settings methods, removed empty model_list check for Router creation
litellm/proxy/route_llm_request.py	Replaced expensive per-request Router instantiation with settings merge into kwargs for main router
litellm/router.py	Added per-request model_group_retry_policy override support in retry logic using existing helper function

Sequence Diagram

sequenceDiagram
    participant Client
    participant ProxyBaseLLMRequestProcessing
    participant ProxyConfig
    participant UserAPIKeyCache
    participant TeamCache
    participant RouteRequest
    participant Router
    
    Client->>ProxyBaseLLMRequestProcessing: Request with API Key
    ProxyBaseLLMRequestProcessing->>ProxyConfig: _get_hierarchical_router_settings()
    ProxyConfig->>UserAPIKeyCache: Get key object (cached)
    UserAPIKeyCache-->>ProxyConfig: user_api_key_dict with router_settings
    
    alt Key has router_settings
        ProxyConfig-->>ProxyBaseLLMRequestProcessing: Return key-level settings
    else No key settings, check team
        ProxyConfig->>TeamCache: get_team_object() (cached)
        TeamCache-->>ProxyConfig: team object with router_settings
        ProxyConfig-->>ProxyBaseLLMRequestProcessing: Return team-level settings
    end
    
    ProxyBaseLLMRequestProcessing->>ProxyBaseLLMRequestProcessing: Add router_settings_override to data
    ProxyBaseLLMRequestProcessing->>RouteRequest: route_request(data, router)
    
    RouteRequest->>RouteRequest: Extract router_settings_override
    RouteRequest->>RouteRequest: Merge allowed settings into data kwargs
    Note over RouteRequest: Only merge: fallbacks, context_window_fallbacks,<br/>content_policy_fallbacks, num_retries,<br/>timeout, model_group_retry_policy
    
    RouteRequest->>Router: router.acompletion(**data)
    Note over Router: Uses main Router with<br/>per-request overrides<br/>(no new Router created)
    Router-->>RouteRequest: Response
    RouteRequest-->>Client: Response

Loading

[greptile-apps]

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

yaml.safe_load() doesn't raise JSONDecodeError, only YAMLError. The json.JSONDecodeError in the except clause won't catch yaml parsing failures.

Suggested change

	except (yaml.YAMLError, json.JSONDecodeError):
	except yaml.YAMLError: