feat(auto_router): keyword tier overrides and semantic keyword matching for the complexity router - #32859
Merged
akapur99 merged 12 commits intoJul 11, 2026
Conversation
Contributor
Author
|
Generated by Claude Code |
Contributor
Greptile SummaryAdds keyword tier overrides and optional semantic keyword matching to the complexity auto-router, fixes proxy lifecycle bugs in
Confidence Score: 5/5Safe to merge; all new routing paths fall back gracefully to the existing scorer, config validation catches misconfiguration at load time, and the proxy lifecycle fixes are guarded and regression-tested. The keyword tier override and semantic matching paths are well-isolated: a lexical miss or semantic failure each return None and let scoring decide as before. The double-checked async lock plus to_thread build is correct for CPython. The metadata sanitization is verified end-to-end against the actual cost-callback reader. The clear_cache and delete_model fixes address the exact cross-tenant registry bug with precise guards and dedicated regression tests. complexity_router.py — the route-layer build has no failure-state cache, so a persistently-unavailable embedding provider causes every request to retry the build and absorb the full embedding timeout before falling back to scoring.
|
| Filename | Overview |
|---|---|
| litellm/router_strategy/complexity_router/complexity_router.py | Adds keyword tier overrides (lexical + semantic) before the scorer, budget-reservation metadata sanitization for sub-calls, and a lock-guarded lazy SemanticRouter build. Core logic is sound; a minor concern exists around retrying the route-layer build on every request after a failed cold-start embedding call. |
| litellm/router_strategy/complexity_router/config.py | Adds KeywordTierRule (with blank-keyword normalization), TIER_SEVERITY_ORDER, and semantic-matching fields to ComplexityRouterConfig, all guarded by validators that catch misconfiguration at load time. |
| litellm/proxy/management_endpoints/model_management_endpoints.py | Fixes clear_cache (selective auto_router/* eviction instead of blanket .clear()) and delete_model (guards registry pop on auto_router/ prefix so config-defined routers survive DB model deletions); both with matching regression tests. |
| tests/test_litellm/router_strategy/test_complexity_router.py | Comprehensive new test classes for lexical escalation, semantic matching via FakeEmbeddingRouter, MAX-aggregation, metadata propagation and budget-stripping, lock-based cold-start singleton, embedding failure fallback, and config validation. All mock-based, no real network calls. |
| tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py | New test classes cover clear_cache config-router preservation, delete_model registry eviction with and without auto_router/ prefix, and the update_model cache-clear regression. All use MagicMock/AsyncMock. |
| ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts | New utility that builds the complexity router config payload; trims and filters empty keyword rules before submission, mirroring the backend validator. |
| ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.tsx | Refactored to split Auto-Router v2 (recommended) and Semantic Router (legacy) flows; dead custom_embedding_model branch removed. |
| ui/litellm-dashboard/src/components/add_model/handle_add_auto_router_submit.tsx | Simplified submit handler; complexity router path passes complexity_router_config directly; semantic router path preserved for legacy flow. |
| ui/litellm-dashboard/src/components/add_model/KeywordTierRules.tsx | New component for managing keyword tier rule rows with add/remove and keyword tag input. |
| ui/litellm-dashboard/src/components/add_model/SemanticKeywordMatching.tsx | New component exposing the semantic matching toggle, embedding model selector, and match threshold slider. |
| ui/litellm-dashboard/src/components/add_model/ComplexityRouterConfig.tsx | Extended with optional keyword tier rules and semantic matching sections; sections hidden when change handlers are absent. |
Reviews (16): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
Contributor
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 5 · PR risk: 0/10 |
Contributor
Greptile SummaryThis PR adds two new capabilities to the complexity auto-router: deterministic keyword-to-tier overrides (
Confidence Score: 4/5Safe to merge; the backend logic and the clear_cache fix are correct and well-tested. The only rough edges are two deleted frontend tests that should have been migrated and a small dead-code remnant in the submit handler. The lexical escalation, semantic override, and targeted cache-clearing are all implemented correctly and covered by new tests. Two existing tests in ComplexityRouterConfig.test.tsx were deleted rather than updated to the new required-prop signature — the component content they exercised still exists, so this is a test coverage gap rather than a functional problem. A dead ComplexityRouterConfig.test.tsx (deleted tests) and handle_add_auto_router_submit.tsx (dead code branch).
|
| Filename | Overview |
|---|---|
| litellm/router_strategy/complexity_router/complexity_router.py | Adds lexical escalation (_lexical_tier_override, escalates to most-severe matched tier) and semantic override (_semantic_tier_override via SemanticRouter.acall). Both plug into async_pre_routing_hook before the scorer. Logic is sound and follows the existing auto_router pattern for SemanticRouter initialization. |
| litellm/router_strategy/complexity_router/config.py | Adds KeywordTierRule, TIER_SEVERITY_ORDER, and new config fields (keyword_tier_rules, semantic_keyword_matching, embedding_model, match_threshold) with a model_validator that enforces embedding_model and non-empty rules when semantic matching is enabled. Config is clean and well-validated. |
| litellm/proxy/management_endpoints/model_management_endpoints.py | Fixes clear_cache to only pop DB-backed entries from auto_routers and complexity_routers instead of calling .clear() on both; preserves config-defined routers that would otherwise go permanently unroutable until restart. |
| tests/test_litellm/router_strategy/test_complexity_router.py | Adds thorough new test classes for lexical escalation, semantic matching (using FakeEmbeddingRouter — no real network calls), config validation, and edge-case branches. No real network calls, satisfying the test isolation rule. |
| tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py | Converts auto_routers mock to a real dict and adds complexity_routers; new TestClearCachePreservesConfigRouters regression test verifies config-defined routers survive a DB-model reload cycle. |
| ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts | New utility that builds the complexity router config payload; omits optional fields when empty and includes semantic fields only when the toggle is on. Well-covered by co-located unit tests. |
| ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.tsx | Router type renamed from "complexity/semantic" to "recommended/semantic"; submit logic split into two clean handlers. Removes Test Connection button (with TODO comment) and the custom model entry option from the semantic router dropdowns. |
| ui/litellm-dashboard/src/components/add_model/ComplexityRouterConfig.tsx | Adds new required props (keywordTierRules, semanticMatchingEnabled, embeddingModel, matchThreshold) and renders the new KeywordTierRules and SemanticKeywordMatching sub-components below the existing config. |
| ui/litellm-dashboard/src/components/add_model/ComplexityRouterConfig.test.tsx | Updated to use baseProps for the new required props; adds good coverage for KeywordTierRules and SemanticKeywordMatching. Two existing tests for "How Classification Works" were deleted instead of migrated, losing coverage for that section. |
| ui/litellm-dashboard/src/components/add_model/KeywordTierRules.tsx | New component for managing keyword-to-tier override rules with add/remove/update operations. Uses AntdSelect in tags mode with tokenSeparators for keyword entry. |
| ui/litellm-dashboard/src/components/add_model/SemanticKeywordMatching.tsx | New component exposing a Switch to enable semantic matching, plus (when enabled) an embedding model select and match threshold input. DEFAULT_MATCH_THRESHOLD is exported for reuse in the parent tab. |
| ui/litellm-dashboard/src/components/add_model/handle_add_auto_router_submit.tsx | Minor refactor; dead code remains in the custom_embedding_model branch since the "Enter custom model name" option was removed from the upstream Select. |
| ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.test.ts | New unit tests for buildComplexityRouterConfig and getSemanticConfigError covering all conditional branches including the "toggle off with lingering embedding model" edge case. |
Comments Outside Diff (1)
-
ui/litellm-dashboard/src/components/add_model/handle_add_auto_router_submit.tsx, line 742-744 (link)Dead code:
custom_embedding_modelbranch is now unreachableThe
add_auto_router_tab.tsxdiff removes the{ value: "custom", label: "Enter custom model name" }option from the embedding modelAntdSelectin the semantic router path — sovalues.auto_router_embedding_modelcan never be"custom", andvalues.custom_embedding_modelis never set. Both the!== "custom"guard and theelse if (values.custom_embedding_model)branch are therefore dead code after this change.
Reviews (2): Last reviewed commit: "feat(auto_router): keyword tier override..." | Re-trigger Greptile
Contributor
Contributor
Author
|
Generated by Claude Code |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Contributor
Author
|
bugbot run Generated by Claude Code |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for all 3 issues found in the latest run.
- ✅ Fixed: Router pop on name collision
- clear_cache now only pops auto/complexity router entries whose matching DB row is itself an auto_router/* deployment, so a plain DB chat model sharing a model_name with a config-defined router can no longer evict it.
- ✅ Fixed: Embedding errors skip scorer fallback
- _resolve_keyword_tier_override now catches exceptions from the semantic path, falls back to lexical rules, and (on further miss) allows the caller to proceed to the weighted scorer instead of hard-failing the request.
- ✅ Fixed: Keyword rules ignore system text
- async_pre_routing_hook now passes system+user text into the tier override helpers, matching how the weighted scorer already scans that combined text for code, technical, and simple signals.
Or push these changes by commenting:
@cursor push 6ff2274c9d
Preview (6ff2274c9d)
diff --git a/litellm/proxy/management_endpoints/model_management_endpoints.py b/litellm/proxy/management_endpoints/model_management_endpoints.py
--- a/litellm/proxy/management_endpoints/model_management_endpoints.py
+++ b/litellm/proxy/management_endpoints/model_management_endpoints.py
@@ -1715,8 +1715,22 @@
for model_id in db_model_ids:
llm_router.delete_deployment(id=model_id)
- # Clear auto routers
- llm_router.auto_routers.clear()
+ # Clear only DB-backed auto/complexity routers, keyed by model_name. A blanket
+ # .clear() would also drop config-defined routers, which are never re-added below
+ # (add_deployment only reloads DB models) - leaving them permanently unroutable
+ # until a full proxy restart, for every tenant, whenever any team admin updates
+ # any team-owned DB model. Only pop entries whose DB row is itself an
+ # auto/complexity router deployment; otherwise a DB model that happens to
+ # share a model_name with a config-defined router would evict it.
+ db_router_model_names = {
+ model.get("model_name")
+ for model in current_models
+ if model.get("model_info", {}).get("db_model", False)
+ and str(model.get("litellm_params", {}).get("model", "")).startswith("auto_router/")
+ }
+ for model_name in db_router_model_names:
+ llm_router.auto_routers.pop(model_name, None)
+ llm_router.complexity_routers.pop(model_name, None)
# Reload only DB models
await proxy_config.add_deployment(prisma_client=prisma_client, proxy_logging_obj=proxy_logging_obj)
diff --git a/litellm/router_strategy/complexity_router/complexity_router.py b/litellm/router_strategy/complexity_router/complexity_router.py
--- a/litellm/router_strategy/complexity_router/complexity_router.py
+++ b/litellm/router_strategy/complexity_router/complexity_router.py
@@ -20,16 +20,20 @@
DEFAULT_REASONING_KEYWORDS,
DEFAULT_SIMPLE_KEYWORDS,
DEFAULT_TECHNICAL_KEYWORDS,
+ TIER_SEVERITY_ORDER,
ComplexityRouterConfig,
ComplexityTier,
)
if TYPE_CHECKING:
+ from semantic_router.routers import SemanticRouter
+
from litellm.router import Router
from litellm.types.router import PreRoutingHookResponse
else:
Router = Any
PreRoutingHookResponse = Any
+ SemanticRouter = Any
def _append_custom_keywords(base_keywords: list[str], custom_keywords: Optional[list[str]]) -> list[str]:
@@ -104,6 +108,10 @@
)
self.simple_keywords = self.config.simple_keywords or DEFAULT_SIMPLE_KEYWORDS
+ # Lazily built on first semantic request and cached for reuse (route
+ # embeddings are static, only the prompt is embedded per request).
+ self._semantic_routelayer: Optional[SemanticRouter] = None
+
# Pre-compile regex patterns for efficiency
# Use non-greedy .*? to prevent ReDoS on pathological inputs
self._multi_step_patterns = [
@@ -325,6 +333,99 @@
raise ValueError(f"No model configured for tier {tier_key} and no default_model set")
+ def _lexical_tier_override(self, user_message: str) -> Optional[ComplexityTier]:
+ """When keyword_tier_rules match literally, the most-severe matched tier wins.
+
+ Escalating to the highest tier (rather than the first rule in the list) keeps
+ routing independent of the order rules were authored in: a prompt hitting both a
+ SIMPLE and a REASONING keyword routes to REASONING.
+ """
+ rules = self.config.keyword_tier_rules
+ if not rules:
+ return None
+ text = user_message.lower()
+ matched_tiers = [
+ rule.tier for rule in rules if any(self._keyword_matches(text, keyword) for keyword in rule.keywords)
+ ]
+ if not matched_tiers:
+ return None
+ return max(matched_tiers, key=TIER_SEVERITY_ORDER.index)
+
+ def _get_or_create_semantic_routelayer(self) -> "SemanticRouter":
+ """Build (once) a SemanticRouter with one route per tier, utterances = that tier's keywords."""
+ if self._semantic_routelayer is not None:
+ return self._semantic_routelayer
+
+ from semantic_router.routers import SemanticRouter
+ from semantic_router.routers.base import Route
+
+ from litellm.router_strategy.auto_router.litellm_encoder import (
+ LiteLLMRouterEncoder,
+ )
+
+ embedding_model = self.config.embedding_model
+ if embedding_model is None:
+ raise ValueError("embedding_model is required for semantic keyword matching")
+
+ rules = self.config.keyword_tier_rules or []
+ ordered_tiers = tuple(dict.fromkeys(rule.tier.value for rule in rules))
+ routes = [
+ Route(
+ name=tier,
+ utterances=[keyword for rule in rules if rule.tier.value == tier for keyword in rule.keywords],
+ score_threshold=self.config.match_threshold,
+ )
+ for tier in ordered_tiers
+ ]
+ routelayer = SemanticRouter(
+ routes=routes,
+ encoder=LiteLLMRouterEncoder(
+ litellm_router_instance=self.litellm_router_instance,
+ model_name=embedding_model,
+ score_threshold=self.config.match_threshold,
+ ),
+ auto_sync="local",
+ )
+ self._semantic_routelayer = routelayer
+ return routelayer
+
+ async def _semantic_tier_override(self, user_message: str) -> Optional[ComplexityTier]:
+ """Match the prompt against keyword_tier_rules by embedding similarity."""
+ from semantic_router.schema import RouteChoice
+
+ routelayer = self._get_or_create_semantic_routelayer()
+ route_choice = await routelayer.acall(text=user_message)
+
+ if isinstance(route_choice, list):
+ route_choice = route_choice[0] if route_choice else None
+ if not isinstance(route_choice, RouteChoice) or not route_choice.name:
+ return None
+ try:
+ return ComplexityTier(route_choice.name)
+ except ValueError:
+ return None
+
+ async def _resolve_keyword_tier_override(self, keyword_text: str) -> Optional[ComplexityTier]:
+ """Resolve a keyword_tier_rule override, semantically or lexically per config.
+
+ Semantic mode degrades to lexical matching (and, if that also yields no
+ match, to the weighted scorer via the caller returning None) when the
+ embedding call fails, so a transient embedding outage does not turn
+ routing enhancements into hard request failures.
+ """
+ if not self.config.keyword_tier_rules:
+ return None
+ if self.config.semantic_keyword_matching:
+ try:
+ return await self._semantic_tier_override(keyword_text)
+ except Exception as exc:
+ verbose_router_logger.warning(
+ f"ComplexityRouter: semantic keyword matching failed ({exc}), "
+ "falling back to lexical keyword_tier_rules"
+ )
+ return self._lexical_tier_override(keyword_text)
+ return self._lexical_tier_override(keyword_text)
+
def _resolve_messages(
self,
messages: Optional[List[Dict[str, Any]]],
@@ -445,6 +546,18 @@
messages=messages if has_original_messages else None,
)
+ keyword_text = f"{system_prompt or ''} {user_message}".strip()
+ override_tier = await self._resolve_keyword_tier_override(keyword_text)
+ if override_tier is not None:
+ routed_model = self.get_model_for_tier(override_tier)
+ verbose_router_logger.info(
+ f"ComplexityRouter: keyword rule fired, tier={override_tier.value}, routed_model={routed_model}"
+ )
+ return PreRoutingHookResponse(
+ model=routed_model,
+ messages=messages if has_original_messages else None,
+ )
+
tier, score, signals = self.classify(user_message, system_prompt)
routed_model = self.get_model_for_tier(tier)
diff --git a/litellm/router_strategy/complexity_router/config.py b/litellm/router_strategy/complexity_router/config.py
--- a/litellm/router_strategy/complexity_router/config.py
+++ b/litellm/router_strategy/complexity_router/config.py
@@ -8,7 +8,7 @@
from enum import Enum
from typing import Dict, List, Optional
-from pydantic import BaseModel, ConfigDict, Field
+from pydantic import BaseModel, ConfigDict, Field, model_validator
class ComplexityTier(str, Enum):
@@ -20,6 +20,26 @@
REASONING = "REASONING"
+TIER_SEVERITY_ORDER: tuple[ComplexityTier, ...] = (
+ ComplexityTier.SIMPLE,
+ ComplexityTier.MEDIUM,
+ ComplexityTier.COMPLEX,
+ ComplexityTier.REASONING,
+)
+
+
+class KeywordTierRule(BaseModel):
+ """A deterministic override: if any keyword matches, route to this tier."""
+
+ keywords: List[str] = Field(
+ min_length=1,
+ description="Keywords/phrases that trigger this rule (lexical or semantic match)",
+ )
+ tier: ComplexityTier = Field(
+ description="Tier to route to when this rule matches",
+ )
+
+
# ─── Default Keyword Lists ───
# Note: Keywords should be full words/phrases to avoid substring false positives.
# The matching logic uses word boundary detection for single-word keywords.
@@ -257,8 +277,40 @@
description="Default model to use if tier cannot be determined",
)
+ # Deterministic keyword -> tier overrides, evaluated before weighted scoring
+ keyword_tier_rules: Optional[List[KeywordTierRule]] = Field(
+ default=None,
+ description="Rules that force a specific tier when their keywords match the prompt",
+ )
+
+ # Semantic (embedding) matching for keyword_tier_rules instead of literal text matching
+ semantic_keyword_matching: bool = Field(
+ default=False,
+ description="Match keyword_tier_rules by embedding similarity instead of literal text",
+ )
+ embedding_model: Optional[str] = Field(
+ default=None,
+ description="Embedding model (LiteLLM model name) used when semantic_keyword_matching is enabled",
+ )
+ match_threshold: float = Field(
+ default=0.5,
+ ge=0.0,
+ le=1.0,
+ description="Minimum cosine similarity for a semantic keyword match",
+ )
+
model_config = ConfigDict(extra="allow") # Allow additional fields
+ @model_validator(mode="after")
+ def _validate_semantic_matching(self) -> "ComplexityRouterConfig":
+ if not self.semantic_keyword_matching:
+ return self
+ if not self.embedding_model:
+ raise ValueError("embedding_model is required when semantic_keyword_matching is enabled")
+ if not self.keyword_tier_rules:
+ raise ValueError("keyword_tier_rules must be non-empty when semantic_keyword_matching is enabled")
+ return self
+
# Combined default config
DEFAULT_COMPLEXITY_CONFIG = ComplexityRouterConfig()
diff --git a/tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py b/tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py
--- a/tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py
+++ b/tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py
@@ -436,28 +436,32 @@
clear_cache,
)
- # Create mock router with mixed DB and config models
+ # Create mock router with mixed DB and config models. DB-backed router
+ # entries only exist for deployments whose litellm_params.model is an
+ # auto_router/* prefix, matching how init_auto_router_deployment and
+ # init_complexity_router_deployment gate creation.
mock_router = MagicMock()
mock_router.model_list = [
{
"model_name": "gpt-4",
"model_info": {"id": "db-model-1", "db_model": True},
- "litellm_params": {"model": "gpt-4"},
+ "litellm_params": {"model": "auto_router/foo"},
},
{
"model_name": "gpt-3.5-turbo",
"model_info": {"id": "config-model-1", "db_model": False},
- "litellm_params": {"model": "gpt-3.5-turbo"},
+ "litellm_params": {"model": "auto_router/foo"},
},
{
"model_name": "claude-3",
"model_info": {"id": "db-model-2", "db_model": True},
- "litellm_params": {"model": "claude-3"},
+ "litellm_params": {"model": "auto_router/complexity_router"},
},
]
mock_router.delete_deployment = MagicMock(return_value=True)
- mock_router.auto_routers = MagicMock()
- mock_router.auto_routers.clear = MagicMock()
+ # Real dicts (not MagicMock) so we can assert on their actual contents below.
+ mock_router.auto_routers = {"gpt-4": MagicMock(), "gpt-3.5-turbo": MagicMock()}
+ mock_router.complexity_routers = {"claude-3": MagicMock(), "gpt-3.5-turbo": MagicMock()}
mock_config = MagicMock()
mock_config.add_deployment = AsyncMock(return_value=True)
@@ -479,8 +483,14 @@
mock_router.delete_deployment.assert_any_call(id="db-model-1")
mock_router.delete_deployment.assert_any_call(id="db-model-2")
- # Should have cleared auto routers
- mock_router.auto_routers.clear.assert_called_once()
+ # DB-backed router entries (gpt-4, claude-3) are cleared so they can be
+ # re-populated by the reload below; the config-backed entry (gpt-3.5-turbo)
+ # must survive, since add_deployment() only reloads DB models and would
+ # otherwise leave it permanently unroutable (see TestClearCachePreservesConfigRouters).
+ assert "gpt-4" not in mock_router.auto_routers
+ assert "claude-3" not in mock_router.complexity_routers
+ assert "gpt-3.5-turbo" in mock_router.auto_routers
+ assert "gpt-3.5-turbo" in mock_router.complexity_routers
# Should have called add_deployment to reload DB models
mock_config.add_deployment.assert_called_once_with(
@@ -488,6 +498,101 @@
)
+class TestClearCachePreservesConfigRouters:
+ """
+ Regression test: clear_cache() must not wipe config-defined auto/complexity
+ routers.
+
+ clear_cache() runs after any DB model write (e.g. a team admin patching a
+ team-owned model via PATCH /model/{id}/update). Before this fix, it called
+ auto_routers.clear() / complexity_routers.clear() unconditionally, which also
+ dropped routers defined in config.yaml belonging to *other* tenants. Those
+ entries are never restored, because the reload below only re-adds DB models
+ (proxy_config.add_deployment), so a config-defined router would stay
+ permanently unroutable until a full proxy restart - a cross-tenant
+ denial-of-service triggerable by any team admin's unrelated model update.
+ """
+
+ @pytest.mark.asyncio
+ async def test_config_backed_routers_survive_unrelated_db_model_update(self):
+ from litellm.proxy.management_endpoints.model_management_endpoints import (
+ clear_cache,
+ )
+
+ mock_router = MagicMock()
+ mock_router.model_list = [
+ {
+ "model_name": "team-a-db-router",
+ "model_info": {"id": "db-model-1", "db_model": True},
+ "litellm_params": {"model": "auto_router/complexity_router"},
+ },
+ ]
+ mock_router.delete_deployment = MagicMock(return_value=True)
+ mock_router.auto_routers = {"config-semantic-router": MagicMock()}
+ mock_router.complexity_routers = {
+ "team-a-db-router": MagicMock(),
+ "config-defined-complexity-router": MagicMock(),
+ }
+
+ mock_config = MagicMock()
+ mock_config.add_deployment = AsyncMock(return_value=True)
+
+ with (
+ patch("litellm.proxy.proxy_server.llm_router", mock_router),
+ patch("litellm.proxy.proxy_server.proxy_config", mock_config),
+ patch("litellm.proxy.proxy_server.prisma_client", MagicMock()),
+ patch("litellm.proxy.proxy_server.proxy_logging_obj", MagicMock()),
+ patch("litellm.proxy.proxy_server.verbose_proxy_logger"),
+ ):
+ await clear_cache()
+
+ # The DB-backed router for the model that was actually updated is cleared
+ # so the reload below can re-populate it.
+ assert "team-a-db-router" not in mock_router.complexity_routers
+ # Config-defined routers for unrelated tenants must survive untouched.
+ assert "config-defined-complexity-router" in mock_router.complexity_routers
+ assert "config-semantic-router" in mock_router.auto_routers
+
+ @pytest.mark.asyncio
+ async def test_config_router_survives_db_model_with_same_name(self):
+ """A DB-backed chat model that happens to share a model_name with a
+ config-defined router must not evict that router. Only DB rows that are
+ themselves auto/complexity router deployments (litellm_params.model
+ starts with ``auto_router/``) should clear the matching router entry.
+ """
+ from litellm.proxy.management_endpoints.model_management_endpoints import (
+ clear_cache,
+ )
+
+ shared_name = "shared-model-name"
+ mock_router = MagicMock()
+ mock_router.model_list = [
+ {
+ "model_name": shared_name,
+ "model_info": {"id": "db-plain-chat", "db_model": True},
+ "litellm_params": {"model": "openai/gpt-4o-mini"},
+ },
+ ]
+ mock_router.delete_deployment = MagicMock(return_value=True)
+ mock_router.auto_routers = {shared_name: MagicMock()}
+ mock_router.complexity_routers = {shared_name: MagicMock()}
+
+ mock_config = MagicMock()
+ mock_config.add_deployment = AsyncMock(return_value=True)
+
+ with (
+ patch("litellm.proxy.proxy_server.llm_router", mock_router),
+ patch("litellm.proxy.proxy_server.proxy_config", mock_config),
+ patch("litellm.proxy.proxy_server.prisma_client", MagicMock()),
+ patch("litellm.proxy.proxy_server.proxy_logging_obj", MagicMock()),
+ patch("litellm.proxy.proxy_server.verbose_proxy_logger"),
+ ):
+ await clear_cache()
+
+ assert shared_name in mock_router.auto_routers
+ assert shared_name in mock_router.complexity_routers
+
+
class TestUpdateModel:
"""
Tests for the update_model (POST /model/update) handler.
diff --git a/tests/test_litellm/router_strategy/test_complexity_router.py b/tests/test_litellm/router_strategy/test_complexity_router.py
--- a/tests/test_litellm/router_strategy/test_complexity_router.py
+++ b/tests/test_litellm/router_strategy/test_complexity_router.py
@@ -15,6 +15,7 @@
0, os.path.abspath("../../..")
) # Adds the parent directory to the system path
+import litellm
from litellm import Router
from litellm.router_strategy.complexity_router.complexity_router import (
ComplexityRouter,
@@ -26,6 +27,7 @@
ComplexityRouterConfig,
ComplexityTier,
)
+from pydantic import ValidationError
@pytest.fixture
@@ -1131,3 +1133,482 @@
)
assert user_msg is None
assert sys_prompt is None
+
+
+class TestLexicalKeywordTierRules:
+ """Test deterministic (literal) keyword_tier_rules overrides."""
+
+ @pytest.fixture
+ def rule_config(self, basic_config) -> Dict:
+ return {
+ **basic_config,
+ "keyword_tier_rules": [
+ {"keywords": ["deploy to k8s"], "tier": "REASONING"},
+ ],
+ }
+
+ @pytest.mark.asyncio
+ async def test_matching_rule_overrides_scoring(
+ self, mock_router_instance, rule_config
+ ):
+ """A prompt hitting a rule keyword routes to that tier, not the scored tier."""
+ router = ComplexityRouter(
+ model_name="test-router",
+ litellm_router_instance=mock_router_instance,
+ complexity_router_config=rule_config,
+ )
+ prompt = "please deploy to k8s now"
+ # Without the rule this short prompt would not score into REASONING.
+ scored_tier, _, _ = router.classify(prompt)
+ assert scored_tier != ComplexityTier.REASONING
+
+ result = await router.async_pre_routing_hook(
+ model="test-model",
+ request_kwargs={},
+ messages=[{"role": "user", "content": prompt}],
+ )
+ assert result is not None
+ assert result.model == "o1-preview" # REASONING tier model
+
+ @pytest.mark.asyncio
+ async def test_most_severe_tier_wins_regardless_of_rule_order(self, mock_router_instance, basic_config):
+ """When several rules match, the highest-severity tier wins, independent of list order."""
+ config = {
+ **basic_config,
+ "keyword_tier_rules": [
+ {"keywords": ["database"], "tier": "SIMPLE"}, # listed first, lower tier
+ {"keywords": ["database"], "tier": "REASONING"}, # listed later, higher tier
+ ],
+ }
+ router = ComplexityRouter(
+ model_name="test-router",
+ litellm_router_instance=mock_router_instance,
+ complexity_router_config=config,
+ )
+ result = await router.async_pre_routing_hook(
+ model="test-model",
+ request_kwargs={},
+ messages=[{"role": "user", "content": "tell me about the database"}],
+ )
+ assert result is not None
+ assert result.model == "o1-preview" # REASONING wins over the earlier SIMPLE rule
+
+ @pytest.mark.asyncio
+ async def test_distinct_keywords_escalate_to_highest_tier(self, mock_router_instance, basic_config):
+ """A prompt hitting keywords across tiers routes to the most complex one."""
+ config = {
+ **basic_config,
+ "keyword_tier_rules": [
+ {"keywords": ["hi"], "tier": "SIMPLE"},
+ {"keywords": ["advise"], "tier": "COMPLEX"},
+ {"keywords": ["kubernetes"], "tier": "REASONING"},
+ ],
+ }
+ router = ComplexityRouter(
+ model_name="test-router",
+ litellm_router_instance=mock_router_instance,
+ complexity_router_config=config,
+ )
+ result = await router.async_pre_routing_hook(
+ model="test-model",
+ request_kwargs={},
+ messages=[{"role": "user", "content": "hi, advise me on kubernetes"}],
+ )
+ assert result is not None
+ assert result.model == "o1-preview" # REASONING, the highest of SIMPLE/COMPLEX/REASONING
+
+ def test_lexical_override_returns_most_severe_matched_tier(self, mock_router_instance, basic_config):
+ """Unit-level check of the escalation helper across mixed matches."""
+ config = {
+ **basic_config,
+ "keyword_tier_rules": [
+ {"keywords": ["hi"], "tier": "SIMPLE"},
+ {"keywords": ["advise"], "tier": "COMPLEX"},
+ ],
+ }
+ router = ComplexityRouter(
+ model_name="test-router",
+ litellm_router_instance=mock_router_instance,
+ complexity_router_config=config,
+ )
+ assert router._lexical_tier_override("hi there, please advise") == ComplexityTier.COMPLEX
+ assert router._lexical_tier_override("just saying hi") == ComplexityTier.SIMPLE
+ assert router._lexical_tier_override("nothing relevant here") is None
+
+ @pytest.mark.asyncio
+ async def test_no_rule_match_falls_back_to_scoring(
+ self, mock_router_instance, basic_config
+ ):
+ """A prompt that matches no rule is classified by the scorer as usual."""
+ config = {
+ **basic_config,
+ "keyword_tier_rules": [
+ {"keywords": ["zzznomatch"], "tier": "REASONING"},
+ ],
+ }
+ router = ComplexityRouter(
+ model_name="test-router",
+ litellm_router_instance=mock_router_instance,
+ complexity_router_config=config,
+ )
+ result = await router.async_pre_routing_hook(
+ model="test-model",
+ request_kwargs={},
+ messages=[{"role": "user", "content": "Hello!"}],
+ )
+ assert result is not None
+ assert result.model == "gpt-4o-mini" # SIMPLE via scoring, rule did not fire
+
+ def test_word_boundary_avoids_substring_false_positive(
+ self, mock_router_instance, basic_config
+ ):
+ """A single-word rule keyword must not match inside a larger word."""
+ config = {
+ **basic_config,
+ "keyword_tier_rules": [{"keywords": ["k8s"], "tier": "REASONING"}],
+ }
+ router = ComplexityRouter(
+ model_name="test-router",
+ litellm_router_instance=mock_router_instance,
+ complexity_router_config=config,
+ )
+ assert router._lexical_tier_override("running my k8s cluster") == ComplexityTier.REASONING
+ assert router._lexical_tier_override("what is a k8scluster thing") is None
+
+
+def _make_embedding_response(vectors: List[List[float]]) -> "litellm.EmbeddingResponse":
+ return litellm.EmbeddingResponse(
+ model="fake-embed",
+ data=[
+ {"embedding": vec, "index": idx, "object": "embedding"}
+ for idx, vec in enumerate(vectors)
+ ],
+ object="list",
+ )
+
+
+class FakeEmbeddingRouter:
+ """A stand-in router whose embeddings are deterministic 2D unit vectors.
+
+ Any text mentioning a cluster/container concept maps to [1, 0]; everything
+ else maps to [0, 1]. This lets the real SemanticRouter compute exact cosine
+ similarities (1.0 or 0.0) so threshold behavior is testable without a network call.
+ """
+
+ _CLUSTER_MARKERS = ("k8s", "kube", "container", "cluster", "orchestrat")
+
+ def __init__(self):
+ self.async_embedding_calls: List[List[str]] = []
+
+ def _vectors(self, docs: List[str]) -> List[List[float]]:
+ return [
+ [1.0, 0.0] if any(marker in doc.lower() for marker in self._CLUSTER_MARKERS) else [0.0, 1.0]
+ for doc in docs
+ ]
+
+ @staticmethod
+ def _as_list(text) -> List[str]:
+ return text if isinstance(text, list) else [text]
+
+ def embedding(self, input, model, **kwargs):
+ return _make_embedding_response(self._vectors(self._as_list(input)))
+
+ async def aembedding(self, input, model, **kwargs):
+ docs = self._as_list(input)
+ self.async_embedding_calls.append(docs)
+ return _make_embedding_response(self._vectors(docs))
+
+
+class TestSemanticKeywordTierRules:
+ """Test embedding-based keyword_tier_rules matching."""
+
+ @pytest.mark.asyncio
+ async def test_semantic_match_routes_to_rule_tier(self, basic_config):
+ """A paraphrase (no literal keyword) still routes via embedding similarity."""
+ fake_router = FakeEmbeddingRouter()
+ config = {
+ **basic_config,
+ "keyword_tier_rules": [
+ {"keywords": ["kubernetes deployment", "container orchestration"], "tier": "REASONING"},
+ {"keywords": ["hello", "thanks"], "tier": "SIMPLE"},
+ ],
+ "semantic_keyword_matching": True,
+ "embedding_model": "fake-embed",
+ "match_threshold": 0.5,
+ }
+ router = ComplexityRouter(
+ model_name="test-router",
+ litellm_router_instance=fake_router,
+ complexity_router_config=config,
+ )
+ result = await router.async_pre_routing_hook(
+ model="test-model",
+ request_kwargs={},
+ messages=[{"role": "user", "content": "help me roll out my k8s cluster today"}],
+ )
+ assert result is not None
+ assert result.model == "o1-preview" # REASONING via semantic match
+ assert fake_router.async_embedding_calls, "expected an embedding call for the prompt"
+
+ @pytest.mark.asyncio
+ async def test_below_threshold_falls_back_to_scoring(self, basic_config):
+ """When no route clears the threshold, scoring decides the tier."""
+ fake_router = FakeEmbeddingRouter()
+ config = {
+ **basic_config,
+ "keyword_tier_rules": [
+ {"keywords": ["kubernetes deployment"], "tier": "REASONING"},
+ ],
+ "semantic_keyword_matching": True,
+ "embedding_model": "fake-embed",
+ "match_threshold": 0.9,
+ }
+ router = ComplexityRouter(
+ model_name="test-router",
+ litellm_router_instance=fake_router,
+ complexity_router_config=config,
+ )
+ # "hello there friend" embeds orthogonal to the REASONING route (cos 0 < 0.9).
+ result = await router.async_pre_routing_hook(
+ model="test-model",
+ request_kwargs={},
+ messages=[{"role": "user", "content": "hello there friend"}],
+ )
+ assert result is not None
+ assert result.model == "gpt-4o-mini" # SIMPLE via scoring fallback
+
+ @pytest.mark.asyncio
+ async def test_route_embeddings_cached_across_requests(self, basic_config):
+ """The route layer is built once and reused on subsequent requests."""
+ fake_router = FakeEmbeddingRouter()
+ config = {
+ **basic_config,
+ "keyword_tier_rules": [
+ {"keywords": ["kubernetes deployment"], "tier": "REASONING"},
+ ],
+ "semantic_keyword_matching": True,
+ "embedding_model": "fake-embed",
+ "match_threshold": 0.5,
+ }
+ router = ComplexityRouter(
+ model_name="test-router",
+ litellm_router_instance=fake_router,
+ complexity_router_config=config,
+ )
+ assert router._semantic_routelayer is None
+ await router.async_pre_routing_hook(
+ model="test-model",
+ request_kwargs={},
+ messages=[{"role": "user", "content": "roll out my k8s cluster"}],
+ )
+ first_layer = router._semantic_routelayer
+ assert first_layer is not None
+ await router.async_pre_routing_hook(
+ model="test-model",
+ request_kwargs={},
+ messages=[{"role": "user", "content": "scale my container cluster"}],
+ )
+ assert router._semantic_routelayer is first_layer
+
+
+class TestSemanticConfigValidation:
+ """Test config validation for semantic_keyword_matching."""
+
+ def test_semantic_without_embedding_model_raises(self):
+ with pytest.raises(ValidationError):
+ ComplexityRouterConfig(
+ semantic_keyword_matching=True,
+ keyword_tier_rules=[{"keywords": ["k8s"], "tier": "REASONING"}],
+ )
+
+ def test_semantic_without_rules_raises(self):
+ with pytest.raises(ValidationError):
+ ComplexityRouterConfig(
+ semantic_keyword_matching=True,
+ embedding_model="fake-embed",
+ )
+
+ def test_semantic_disabled_needs_no_embedding_model(self):
+ config = ComplexityRouterConfig(
+ keyword_tier_rules=[{"keywords": ["k8s"], "tier": "REASONING"}],
+ )
+ assert config.semantic_keyword_matching is False
+ assert config.match_threshold == 0.5
+
+ def test_rule_with_empty_keywords_raises(self):
+ with pytest.raises(ValidationError):
+ ComplexityRouterConfig(
+ keyword_tier_rules=[{"keywords": [], "tier": "REASONING"}],
+ )
+
+
+class _StubRouteLayer:
+ """Returns a fixed acall result so _semantic_tier_override branches can be exercised."""
+
+ def __init__(self, result):
+ self._result = result
+
+ async def acall(self, text=None):
+ return self._result
+
+
+class TestKeywordOverrideEdgeCases:
+ """Cover the defensive branches of the lexical and semantic override helpers."""
+
+ def _semantic_router(self, mock_router_instance, basic_config):
+ config = {
+ **basic_config,
+ "keyword_tier_rules": [{"keywords": ["kubernetes"], "tier": "REASONING"}],
+ "semantic_keyword_matching": True,
+ "embedding_model": "fake-embed",
+ "match_threshold": 0.5,
+ }
+ return ComplexityRouter(
+ model_name="test-router",
+ litellm_router_instance=mock_router_instance,
+ complexity_router_config=config,
+ )
+
+ def test_lexical_override_none_when_no_rules(self, mock_router_instance, basic_config):
+ """No keyword_tier_rules configured -> lexical override is a no-op."""
+ router = ComplexityRouter(
+ model_name="test-router",
+ litellm_router_instance=mock_router_instance,
+ complexity_router_config=basic_config,
+ )
+ assert router._lexical_tier_override("deploy to k8s and reason step by step") is None
+
+ def test_semantic_routelayer_requires_embedding_model(self, mock_router_instance, basic_config):
+ """Building the route layer without an embedding model raises (defensive invariant)."""
+ config = {**basic_config, "keyword_tier_rules": [{"keywords": ["k8s"], "tier": "REASONING"}]}
+ router = ComplexityRouter(
+ model_name="test-router",
... diff truncated: showing 800 of 2212 linesYou can send follow-ups to the cloud agent here.
mateo-berri
force-pushed
the
litellm_complexity_router_keyword_tiers
branch
from
July 11, 2026 03:04
ec64fb2 to
d8c46e0
Compare
Contributor
Author
|
Generated by Claude Code |
Contributor
Author
|
bugbot run Generated by Claude Code |
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Semantic guard allows empty keywords
- Extended getSemanticConfigError to reject any keyword tier rule with an empty keywords array when semantic matching is enabled, so users can no longer pass client validation with rules that would build empty semantic routes.
Or push these changes by commenting:
@cursor push 29f89ebf3e
Preview (29f89ebf3e)
diff --git a/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.test.ts b/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.test.ts
--- a/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.test.ts
+++ b/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.test.ts
@@ -127,4 +127,22 @@
getSemanticConfigError({ semanticMatchingEnabled: true, embeddingModel: "voyage-3-5", keywordTierRules: [rule] }),
).toBeNull();
});
+
+ it("errors when enabled with a rule that has no keywords, so semantic routing can't silently no-op", () => {
+ const emptyRule = { id: "r2", keywords: [], tier: "SIMPLE" as const };
+ expect(
+ getSemanticConfigError({
+ semanticMatchingEnabled: true,
+ embeddingModel: "voyage-3-5",
+ keywordTierRules: [emptyRule],
+ }),
+ ).toMatch(/keyword/i);
+ expect(
+ getSemanticConfigError({
+ semanticMatchingEnabled: true,
+ embeddingModel: "voyage-3-5",
+ keywordTierRules: [rule, emptyRule],
+ }),
+ ).toMatch(/keyword/i);
+ });
});
diff --git a/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts b/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts
--- a/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts
+++ b/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts
@@ -40,6 +40,8 @@
if (!semanticMatchingEnabled) return null;
if (!embeddingModel) return "Select an embedding model to use semantic keyword matching";
if (keywordTierRules.length === 0) return "Add at least one keyword tier rule to use semantic keyword matching";
+ if (keywordTierRules.some((rule) => rule.keywords.length === 0))
+ return "Add at least one keyword to every tier rule to use semantic keyword matching";
return null;
};You can send follow-ups to the cloud agent here.
mateo-berri
force-pushed
the
litellm_complexity_router_keyword_tiers
branch
from
July 11, 2026 03:25
d8c46e0 to
dc9ee1d
Compare
Contributor
Author
|
Generated by Claude Code |
Contributor
Author
|
bugbot run Generated by Claude Code |
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
There are 4 total unresolved issues (including 2 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Empty keyword rules accepted
- KeywordTierRule now requires min_length=1 and a validator that trims blanks and rejects rules with no non-blank keywords; the UI's buildComplexityRouterConfig sanitizes and drops empty rules, and getSemanticConfigError requires at least one rule with non-blank keywords.
- ✅ Fixed: Deleted routers linger after reload
- clear_cache now pops every auto_routers/complexity_routers entry whose model_name isn't backed by a config model, so DB routers deleted since the last reload no longer linger while config-defined routers survive.
Or push these changes by commenting:
@cursor push 7745cd9f71
Preview (7745cd9f71)
diff --git a/litellm/proxy/management_endpoints/model_management_endpoints.py b/litellm/proxy/management_endpoints/model_management_endpoints.py
--- a/litellm/proxy/management_endpoints/model_management_endpoints.py
+++ b/litellm/proxy/management_endpoints/model_management_endpoints.py
@@ -1719,13 +1719,14 @@
# .clear() would also drop config-defined routers, which are never re-added below
# (add_deployment only reloads DB models) - leaving them permanently unroutable
# until a full proxy restart, for every tenant, whenever any team admin updates
- # any team-owned DB model.
- db_model_names = {
- model.get("model_name") for model in current_models if model.get("model_info", {}).get("db_model", False)
- }
- for model_name in db_model_names:
- llm_router.auto_routers.pop(model_name, None)
- llm_router.complexity_routers.pop(model_name, None)
+ # any team-owned DB model. Anything that is not config-defined is DB-backed
+ # (currently registered or removed since last reload), so pop it and let
+ # add_deployment reregister it below if the DB still has it.
+ config_model_names = {model.get("model_name") for model in config_models}
+ for router_map in (llm_router.auto_routers, llm_router.complexity_routers):
+ for model_name in tuple(router_map.keys()):
+ if model_name not in config_model_names:
+ router_map.pop(model_name, None)
# Reload only DB models
await proxy_config.add_deployment(prisma_client=prisma_client, proxy_logging_obj=proxy_logging_obj)
diff --git a/litellm/router_strategy/complexity_router/config.py b/litellm/router_strategy/complexity_router/config.py
--- a/litellm/router_strategy/complexity_router/config.py
+++ b/litellm/router_strategy/complexity_router/config.py
@@ -8,7 +8,7 @@
from enum import Enum
from typing import Dict, List, Literal, Optional
-from pydantic import BaseModel, ConfigDict, Field, model_validator
+from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
class ComplexityTier(str, Enum):
@@ -32,13 +32,24 @@
"""A deterministic override: if any keyword matches, route to this tier."""
keywords: List[str] = Field(
+ min_length=1,
description="Keywords/phrases that trigger this rule (lexical or semantic match)",
)
tier: ComplexityTier = Field(
description="Tier to route to when this rule matches",
)
+ @field_validator("keywords")
+ @classmethod
+ def _require_non_blank_keywords(cls, keywords: List[str]) -> List[str]:
+ cleaned = [kw.strip() for kw in keywords if kw and kw.strip()]
+ if not cleaned:
+ raise ValueError(
+ "KeywordTierRule.keywords must contain at least one non-blank keyword"
+ )
+ return cleaned
+
# ─── Default Keyword Lists ───
# Note: Keywords should be full words/phrases to avoid substring false positives.
# The matching logic uses word boundary detection for single-word keywords.
diff --git a/tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py b/tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py
--- a/tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py
+++ b/tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py
@@ -523,6 +523,16 @@
"model_info": {"id": "db-model-1", "db_model": True},
"litellm_params": {"model": "auto_router/complexity_router"},
},
+ {
+ "model_name": "config-defined-complexity-router",
+ "model_info": {"id": "config-model-1", "db_model": False},
+ "litellm_params": {"model": "auto_router/complexity_router"},
+ },
+ {
+ "model_name": "config-semantic-router",
+ "model_info": {"id": "config-model-2", "db_model": False},
+ "litellm_params": {"model": "auto_router/semantic-router"},
+ },
]
mock_router.delete_deployment = MagicMock(return_value=True)
mock_router.auto_routers = {"config-semantic-router": MagicMock()}
@@ -550,7 +560,50 @@
assert "config-defined-complexity-router" in mock_router.complexity_routers
assert "config-semantic-router" in mock_router.auto_routers
+ @pytest.mark.asyncio
+ async def test_orphaned_db_router_entries_are_cleared(self):
+ """Regression: after a DB router is deleted (so it no longer appears in
+ model_list), a subsequent clear_cache must pop its stale entry from
+ auto_routers / complexity_routers, otherwise async_pre_routing_hook keeps
+ dispatching to the deleted router until the proxy restarts."""
+ from litellm.proxy.management_endpoints.model_management_endpoints import (
+ clear_cache,
+ )
+ mock_router = MagicMock()
+ mock_router.model_list = [
+ {
+ "model_name": "config-defined-router",
+ "model_info": {"id": "config-1", "db_model": False},
+ "litellm_params": {"model": "auto_router/complexity_router"},
+ },
+ ]
+ mock_router.delete_deployment = MagicMock(return_value=True)
+ mock_router.auto_routers = {
+ "config-defined-router": MagicMock(),
+ "deleted-db-auto-router": MagicMock(),
+ }
+ mock_router.complexity_routers = {
+ "deleted-db-complexity-router": MagicMock(),
+ }
+
+ mock_config = MagicMock()
+ mock_config.add_deployment = AsyncMock(return_value=True)
+
+ with (
+ patch("litellm.proxy.proxy_server.llm_router", mock_router),
+ patch("litellm.proxy.proxy_server.proxy_config", mock_config),
+ patch("litellm.proxy.proxy_server.prisma_client", MagicMock()),
+ patch("litellm.proxy.proxy_server.proxy_logging_obj", MagicMock()),
+ patch("litellm.proxy.proxy_server.verbose_proxy_logger"),
+ ):
+ await clear_cache()
+
+ assert "deleted-db-auto-router" not in mock_router.auto_routers
+ assert "deleted-db-complexity-router" not in mock_router.complexity_routers
+ assert "config-defined-router" in mock_router.auto_routers
+
+
class TestUpdateModel:
"""
Tests for the update_model (POST /model/update) handler.
diff --git a/tests/test_litellm/router_strategy/test_complexity_router.py b/tests/test_litellm/router_strategy/test_complexity_router.py
--- a/tests/test_litellm/router_strategy/test_complexity_router.py
+++ b/tests/test_litellm/router_strategy/test_complexity_router.py
@@ -1453,6 +1453,66 @@
assert router._lexical_tier_override("what is a k8scluster thing") is None
+class TestKeywordTierRuleValidation:
+ """Regression: KeywordTierRule must reject empty/blank keyword lists.
+
+ An empty keywords list produces a `\\b\\b` regex in the lexical matcher that
+ fires on every prompt, and yields a route with zero utterances in the
+ semantic matcher which breaks the first embedding-backed request. Guard at
+ config parse time so bad rules never reach either matcher.
+ """
+
+ def test_empty_keywords_list_rejected(self):
+ from pydantic import ValidationError
+
+ from litellm.router_strategy.complexity_router.config import (
+ ComplexityTier,
+ KeywordTierRule,
+ )
+
+ with pytest.raises(ValidationError):
+ KeywordTierRule(keywords=[], tier=ComplexityTier.REASONING)
+
+ def test_blank_only_keywords_rejected(self):
+ from pydantic import ValidationError
+
+ from litellm.router_strategy.complexity_router.config import (
+ ComplexityTier,
+ KeywordTierRule,
+ )
+
+ with pytest.raises(ValidationError):
+ KeywordTierRule(keywords=["", " "], tier=ComplexityTier.REASONING)
+
+ def test_blank_entries_are_stripped_and_dropped(self):
+ from litellm.router_strategy.complexity_router.config import (
+ ComplexityTier,
+ KeywordTierRule,
+ )
+
+ rule = KeywordTierRule(
+ keywords=[" k8s ", "", " ", "docker"], tier=ComplexityTier.REASONING
+ )
+ assert rule.keywords == ["k8s", "docker"]
+
+ def test_empty_rule_via_router_config_rejected(self, mock_router_instance, basic_config):
+ """Even wrapped inside ComplexityRouterConfig, an empty-keywords rule
+ must fail validation instead of silently building a matcher that
+ classifies every prompt as this rule's tier."""
+ from pydantic import ValidationError
+
+ config = {
+ **basic_config,
+ "keyword_tier_rules": [{"keywords": [], "tier": "REASONING"}],
+ }
+ with pytest.raises(ValidationError):
+ ComplexityRouter(
+ model_name="test-router",
+ litellm_router_instance=mock_router_instance,
+ complexity_router_config=config,
+ )
+
+
def _make_embedding_response(vectors: List[List[float]]) -> "litellm.EmbeddingResponse":
return litellm.EmbeddingResponse(
model="fake-embed",
diff --git a/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.test.ts b/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.test.ts
--- a/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.test.ts
+++ b/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.test.ts
@@ -99,6 +99,29 @@
expect(config.custom_technical_keywords).toBeUndefined();
expect(config.keyword_tier_rules).toBeUndefined();
});
+
+ it("drops keyword tier rules whose keywords are empty or blank", () => {
+ const config = buildComplexityRouterConfig({
+ ...baseParams,
+ keywordTierRules: [
+ { id: "empty", keywords: [], tier: "COMPLEX" },
+ { id: "blank", keywords: ["", " "], tier: "COMPLEX" },
+ { id: "keep", keywords: [" k8s ", "", "docker"], tier: "REASONING" },
+ ],
+ });
+ expect(config.keyword_tier_rules).toEqual([{ keywords: ["k8s", "docker"], tier: "REASONING" }]);
+ });
+
+ it("omits keyword_tier_rules entirely when every rule sanitizes to empty", () => {
+ const config = buildComplexityRouterConfig({
+ ...baseParams,
+ keywordTierRules: [
+ { id: "empty", keywords: [], tier: "COMPLEX" },
+ { id: "blank", keywords: [" "], tier: "REASONING" },
+ ],
+ });
+ expect(config.keyword_tier_rules).toBeUndefined();
+ });
});
describe("getSemanticConfigError", () => {
@@ -122,6 +145,19 @@
).toMatch(/keyword tier rule/i);
});
+ it("errors when enabled with rules that have no non-blank keywords", () => {
+ expect(
+ getSemanticConfigError({
+ semanticMatchingEnabled: true,
+ embeddingModel: "voyage-3-5",
+ keywordTierRules: [
+ { id: "r1", keywords: [], tier: "REASONING" },
+ { id: "r2", keywords: [" ", ""], tier: "REASONING" },
+ ],
+ }),
+ ).toMatch(/keyword/i);
+ });
+
it("returns null when enabled with both an embedding model and rules", () => {
expect(
getSemanticConfigError({ semanticMatchingEnabled: true, embeddingModel: "voyage-3-5", keywordTierRules: [rule] }),
diff --git a/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts b/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts
--- a/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts
+++ b/ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts
@@ -30,6 +30,16 @@
match_threshold?: number;
}
+const sanitizeRuleKeywords = (keywords: string[]): string[] =>
+ keywords.map((kw) => kw.trim()).filter((kw) => kw.length > 0);
+
+const rulesWithKeywords = (
+ keywordTierRules: KeywordTierRule[],
+): { keywords: string[]; tier: KeywordTierRule["tier"] }[] =>
+ keywordTierRules
+ .map((rule) => ({ keywords: sanitizeRuleKeywords(rule.keywords), tier: rule.tier }))
+ .filter((rule) => rule.keywords.length > 0);
+
export const getSemanticConfigError = ({
semanticMatchingEnabled,
embeddingModel,
@@ -39,7 +49,9 @@
| null => {
if (!semanticMatchingEnabled) return null;
if (!embeddingModel) return "Select an embedding model to use semantic keyword matching";
- if (keywordTierRules.length === 0) return "Add at least one keyword tier rule to use semantic keyword matching";
+ if (rulesWithKeywords(keywordTierRules).length === 0) {
+ return "Add at least one keyword tier rule with keywords to use semantic keyword matching";
+ }
return null;
};
@@ -52,17 +64,18 @@
semanticMatchingEnabled,
embeddingModel,
matchThreshold,
-}: BuildComplexityRouterConfigParams): ComplexityRouterConfigPayload => ({
- tiers,
- classifier_type: classifierType,
- ...(classifierType === "llm" && classifierLlmConfig && { classifier_llm_config: classifierLlmConfig }),
- ...(customTechnicalKeywords.length > 0 && { custom_technical_keywords: customTechnicalKeywords }),
- ...(keywordTierRules.length > 0 && {
- keyword_tier_rules: keywordTierRules.map((rule) => ({ keywords: rule.keywords, tier: rule.tier })),
- }),
- ...(semanticMatchingEnabled && {
- semantic_keyword_matching: true,
- embedding_model: embeddingModel,
- match_threshold: matchThreshold,
- }),
-});
+}: BuildComplexityRouterConfigParams): ComplexityRouterConfigPayload => {
+ const sanitizedRules = rulesWithKeywords(keywordTierRules);
+ return {
+ tiers,
+ classifier_type: classifierType,
+ ...(classifierType === "llm" && classifierLlmConfig && { classifier_llm_config: classifierLlmConfig }),
+ ...(customTechnicalKeywords.length > 0 && { custom_technical_keywords: customTechnicalKeywords }),
+ ...(sanitizedRules.length > 0 && { keyword_tier_rules: sanitizedRules }),
+ ...(semanticMatchingEnabled && {
+ semantic_keyword_matching: true,
+ embedding_model: embeddingModel,
+ match_threshold: matchThreshold,
+ }),
+ };
+};You can send follow-ups to the cloud agent here.
akapur99
force-pushed
the
litellm_complexity_router_keyword_tiers
branch
from
July 11, 2026 03:59
dc9ee1d to
664eab7
Compare
Contributor
|
@greptile-apps check and score |
Contributor
|
bugbot run |
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
1 issue from previous review remains unresolved.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 664eab7. Configure here.
…ng for the complexity router Add deterministic keyword-to-tier overrides and optional embedding-based (semantic) keyword matching to the complexity router, and surface both in the Add Auto Router UI behind a Router Type selector: "Auto-Router v2 [Recommended]" (complexity tiers + keyword overrides + semantic matching, the default) and "Semantic Router [to be deprecated]" (the existing utterance-based router, unchanged). Keyword-to-tier overrides resolve to the highest tier matched rather than the first keyword matched, so match order no longer affects the routing decision. Backend: - config: KeywordTierRule model plus keyword_tier_rules, semantic_keyword_matching, embedding_model, and match_threshold on ComplexityRouterConfig, with a validator requiring an embedding model and rules when semantic matching is on - complexity_router: evaluate keyword rules before scoring; lexical matches escalate to the most-severe matched tier (order-independent), and semantic mode reuses LiteLLMRouterEncoder + SemanticRouter to match paraphrases by cosine similarity, falling back to the scorer when nothing matches - model management: clear complexity_routers on cache reload so config edits take effect Frontend: - Add Auto Router tab restores the Router Type radio (Auto-Router v2 recommended by default, Semantic Router still available) and sends keyword_tier_rules plus the semantic settings on the recommended path, instead of flattening keywords into custom_technical_keywords - client-side guard blocks submit when semantic matching is enabled without an embedding model or without any keyword tier rules, mirroring the backend validator - moved the "How Classification Works" explainer below Custom Technical Keywords and above Keyword Tier Overrides - remove the Test Connection action from the recommended flow, which can't build a valid pre-save payload for a router (leaves a TODO for a JSON preview / config test follow-up) Tests cover lexical escalation, semantic matching via the real library with injected embeddings, the semantic config guard, config validation, the reload-clear regression, and the frontend payload builder
akapur99
force-pushed
the
litellm_complexity_router_keyword_tiers
branch
from
July 11, 2026 05:05
664eab7 to
3c714ed
Compare
Contributor
Author
|
Generated by Claude Code |
Contributor
Author
|
Hey no need to stay up doing this but I appreciate the passion! |
Contributor
Summary for reviewThis adds two capabilities to the complexity auto-router and surfaces them in the Add Auto Router UI, plus two proxy router-lifecycle fixes Keyword tier overrides add a Semantic keyword matching (opt-in via Two proxy fixes keep the in-memory router registries consistent. Frontend restores the Router Type selector ("Auto-Router v2 [Recommended]" default vs "Semantic Router [to be deprecated]"). The v2 path sends Config validation requires Tests cover lexical escalation and order-independence, semantic matching through the real library with injected embeddings, metadata propagation, budget-reservation stripping, config validation, the semantic guard, concurrent-build-once, off-event-loop build, below-threshold and embedding-error fallback, and the reload-clear and delete-model name-collision regressions, plus the frontend payload builder Two deliberate non-changes worth noting: semantic mode intentionally uses best-match rather than escalation (cosine similarities are directly comparable, so the highest-similarity cluster wins), and keyword overrides intentionally scope to the last user message rather than system text (a keyword in static system boilerplate would otherwise pin every request to one tier) |
…e scoring SemanticRouter defaults to mean aggregation across a route's utterances. Since each tier's route holds one utterance per configured keyword, a real semantic match on one keyword was averaged together with the tier's other, unrelated keywords and dragged below match_threshold — e.g. a MEDIUM tier with keywords [beep, boop, new york] never fired for a genuine "new york" paraphrase, because mean(sim_to_beep, sim_to_boop, sim_to_new_york) landed well under the threshold even though sim_to_new_york alone cleared it. Pass aggregation="max" so a tier matches if the query is close enough to any one of its keywords, not the average of all of them. Verified against live Voyage embeddings: raw cosine similarity for "new york" vs a paraphrase was 0.54 (above a 0.5 threshold), but the route scored 0.28 under mean aggregation and never matched; max aggregation fixes it. Adds a regression test with a tier holding one matching and two unrelated keywords, asserting the tier still fires; fails without aggregation="max".
…itellm_complexity_router_keyword_tiers # Conflicts: # ui/litellm-dashboard/eslint-metrics.json
Contributor
Author
|
Generated by Claude Code |
Contributor
|
Pushed eb0c966: semantic keyword matching now scores each tier's route with max aggregation instead of the semantic_router library's default mean Each tier's route holds one utterance per configured keyword, so a real match against one keyword was being averaged together with the tier's other, unrelated keywords and dragged below match_threshold. Concretely, a MEDIUM tier configured with keywords [beep, boop, new york] never fired for a genuine "new york" paraphrase like "visiting the big apple next week": raw cosine similarity for that pair was 0.54 against a 0.5 threshold, but mean(sim_to_beep, sim_to_boop, sim_to_new_york) landed around 0.28 since beep/boop are unrelated to the query. Max aggregation fixes this by scoring the tier on its single closest keyword rather than the average across all of them, matching the intended semantics: the tier should fire if the query is close enough to any of its configured keywords, not to all of them at once Verified against live Voyage embeddings and added a regression test with a tier holding one matching and two unrelated keywords, asserting the tier still fires; the test fails without aggregation="max" |
Contributor
Author
|
bugbot run Generated by Claude Code |
Removing user_api_key_auth entirely from classifier/embedding sub-call metadata (as _BUDGET_RESERVATION_METADATA_KEYS previously did) prevented _filter_deployments_by_model_access_groups from scoping those sub-calls to the caller's authorized access groups. An access-group-scoped caller could therefore reach embedding/classifier deployments outside their group. Only strip user_api_key_budget_reservation, which is the actual budget- reservation state that must not reach sub-calls. user_api_key_auth is now kept so access-group filtering works correctly for both the embedding path and the LLM classifier path.
Contributor
|
Concise data point for how mean vs max aggregation affects Ran this against live voyage-3 embeddings. Setup: four tiers (SIMPLE/MEDIUM/COMPLEX/REASONING), COMPLEX holds Per-keyword cosine similarity to the prompt: Mean aggregation across COMPLEX's three keywords lands at 0.233, roughly half the real signal from the one keyword that actually matched. Max aggregation keeps it at 0.450, the true best match. At a threshold around 0.35 (realistic for keyword-vs-paragraph voyage-3 cosines, which run lower than word-vs-word matches), COMPLEX fails under mean and passes under max: the exact scenario this commit fixes Worth noting the fix's scope is narrow: max aggregation only prevents dilution from a tier's own other keywords. It doesn't stop unrelated generic keywords like "explain" or "compare" from picking up stray signal on a long prompt regardless of aggregation method, so tier keyword lists should stay tight and specific |
…user_api_key_auth
Contributor
Author
|
Generated by Claude Code |
…itellm_complexity_router_keyword_tiers
…istry eviction, edit-modal controls - config: KeywordTierRule now strips and drops blank/whitespace keywords (a stray "" makes _keyword_matches match every prompt, silently forcing that tier for all traffic); still requires at least one real keyword to remain - frontend build_complexity_router_config: trim keywords and drop rules left empty so an unfilled "Add keyword rule" row no longer ships a rule the backend rejects with a 400 in the heuristic (non-semantic) flow, where the client-side semantic guard doesn't run - proxy clear_cache / delete_model: the auto_router/ prefix also covers quality_router/ and adaptive_router/, so pop the model_name from all four router registries (no-op where absent) instead of only auto/complexity; otherwise a DB quality_router's stale entry made reload raise "already exists" and abort, and adaptive left a leak - frontend ComplexityRouterConfig: only render the Keyword Tier Overrides and Semantic keyword matching sections when their change handlers are provided, so the edit-auto- router modal (which omits them) no longer shows interactive-but-dead controls
Contributor
Author
|
bugbot run Generated by Claude Code |
Contributor
Author
|
Generated by Claude Code |
Contributor
Author
|
Generated by Claude Code |
Contributor
Author
|
bugbot run Generated by Claude Code |
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 8982e1c. Configure here.
…itellm_complexity_router_keyword_tiers
Contributor
Author
|
Generated by Claude Code |
…itellm_complexity_router_keyword_tiers # Conflicts: # ui/litellm-dashboard/eslint-metrics.json
Contributor
|
@mateo-berri this is merge-ready. The staging conflict (the removed eslint-metrics.json snapshot) is resolved via merge commit 973a86a, and all 128 CI checks are green (the single skipped one is the diff-scoped tests/e2e basedpyright gate, which correctly skips since this PR touches no e2e files). Only REVIEW_REQUIRED remains, so it just needs your approve + merge. Good to squash-merge whenever you are ready |
akapur99
self-requested a review
July 11, 2026 21:18
akapur99
approved these changes
Jul 11, 2026
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Copy of #32829 by @akapur99, pushed to an in-repo litellm_ branch so CircleCI can run. All credit for this work goes to @akapur99. The branch tracks the original PR branch, with follow-up fixes committed here directly
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Captured at commit
5ae72c8676against real Anthropic and Voyage APIs (no mocks)Lexical keyword tier overrides on a router with rules
hi-> SIMPLE andbeep-> COMPLEX (semantic off). Each keyword routes to its tier, and a prompt matching both escalates to the most-severe tier rather than the first rule in the listSemantic keyword matching on a router with
semantic_keyword_matching: true,embedding_model: voyage-3-5,match_threshold: 0.5. Paraphrases with no literal keyword still route via embeddingsCommand shape used for each:
Tested end-to-end from Claude Code pointed at the router as its default model (
ANTHROPIC_BASE_URL-> the proxy,ANTHROPIC_MODEL=smart-router); requests reached the proxy over the Anthropic Messages API (/v1/messages, including streaming) and were classified and routed correctlyUI: Models + Endpoints -> Add Model -> Add Auto Router, Router Type set to "Auto-Router v2 [Recommended]" (default)
Type
🆕 New Feature
Changes
Adds two routing capabilities to the complexity auto-router, fixes a semantic-scoring correctness bug, and hardens the proxy router lifecycle. All surfaced in the Add Auto Router UI behind a Router Type selector
Keyword tier overrides: a new
keyword_tier_ruleslist ({keywords, tier}) onComplexityRouterConfig, evaluated before the weighted/LLM scorer; a match routes straight to that tier. When several rules match, routing escalates to the most-severe matched tier (SIMPLE < MEDIUM < COMPLEX < REASONING) so the decision is independent of the order rules were authored in.keywordsmust contain at least one non-empty entry (validated at config load)Semantic keyword matching (opt-in): with
semantic_keyword_matchingplus anembedding_modelandmatch_threshold, the same rules are matched by embedding similarity so paraphrases route without a literal keyword. Reuses the existingLiteLLMRouterEncoder+SemanticRouterSemantic scoring fix, MAX aggregation (the significant behavioral change): each tier's route holds one utterance per configured keyword.
semantic_router's default MEAN aggregation averaged a genuine match on one keyword together with that tier's other unrelated keywords, dragging the score belowmatch_thresholdso real matches silently missed. Switched to MAX aggregation, so a strong match on any single keyword in a tier is what counts. This is what makes multi-keyword semantic rules actually fire as configured; without it the feature under-matchesRouting-time embedding safety: the query embedding carries the caller's request metadata so its spend is attributed and budget-enforced against the originating key/team. Only the parent request's budget reservation is stripped, not
user_api_key_auth, which access-group filtering needs, so the embedding model selection stays scoped to the caller's authorized access groups. The lazy route index is built once under a per-routerasyncio.Lock, off the event loop viato_thread, so concurrent cold-start requests neither each construct it nor block the worker on synchronous provider I/O; an embedding failure falls back to the scorer rather than failing the requestProxy router lifecycle fixes:
clear_cacheno longer blanket-.clear()s the auto/complexity router registries (which left config-defined routers for other tenants permanently unroutable until a restart); it evicts only DB-backedauto_router/*entries by name.delete_modelapplies the sameauto_router/-prefixed guard when popping a deleted deployment's router entry, so deleting a regular DB model that coincidentally shares a name with a config-defined router doesn't evict the config routerFrontend: restores the Router Type radio, where "Auto-Router v2 [Recommended]" (default) sends
keyword_tier_rulesand the semantic settings directly instead of flattening keywords intocustom_technical_keywords, while "Semantic Router [to be deprecated]" keeps the existing utterance flow unchanged. Client-side guards mirror the backend validators (embedding model required, at least one rule, non-empty keywords). The "How Classification Works" explainer moved below Custom Technical Keywords. Test Connection is dropped from the recommended flow since it can't build a valid pre-save payload for a router (a TODO is left for a JSON preview)Tests cover lexical escalation, semantic matching via the real library with injected embeddings, MAX-aggregation scoring, metadata propagation with budget-reservation stripping and access-group preservation on the embedding call, embedding-failure fallback, concurrent cold-start building the index exactly once, config validation (empty keywords and semantic requirements), the
clear_cacheconfig-router-preservation regression,delete_modelregistry eviction, and the frontend payload builder and guardsNote
High Risk
Touches request routing, embedding sub-calls, and budget-reservation metadata on a hot path, plus multi-tenant router registry behavior in
clear_cache/delete_modelwhere mistakes could break routing or budget accounting.Overview
Adds keyword tier overrides and optional semantic keyword matching to the complexity auto-router, plus proxy lifecycle fixes and dashboard support on the “Auto-Router v2” add flow.
Routing: New
keyword_tier_rulesonComplexityRouterConfigrun before heuristic/LLM scoring; multiple matches escalate to the highest tier (SIMPLE → REASONING). Withsemantic_keyword_matching, the same rules match via embeddings (SemanticRouter+LiteLLMRouterEncoder), using MAX aggregation so one strong keyword match isn’t diluted by other utterances on the tier. Query embeddings forward caller metadata for spend/budget while stripping parent budget reservation (sanitizeduser_api_key_authkept for access groups); semantic failures fall back to scoring. Route index builds once under a lock, off the event loop.Proxy:
clear_cacheanddelete_modelno longer blanket-clear router registries—they evict only DB-backedauto_router/*deployments by name so config-defined routers aren’t left unroutable after unrelated DB model updates/deletes.UI: Add Auto Router exposes keyword rules, semantic toggle, embedding model, and threshold via
buildComplexityRouterConfig/ client validation; dev config adds a sample embedding model for local testing.Reviewed by Cursor Bugbot for commit 8982e1c. Bugbot is set up for automated code reviews on this repo. Configure here.