feat(auto_router): keyword tier overrides and semantic keyword matching for the complexity router

[akapur99]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Captured at commit 5ae72c8676 against real Anthropic and Voyage APIs (no mocks)

Lexical keyword tier overrides on a router with rules hi->SIMPLE and beep->COMPLEX (semantic off). Each keyword routes to its tier, and a prompt matching both escalates to the most-severe tier rather than the first rule in the list

"hi"       -> tier=SIMPLE   served=anthropic/claude-haiku-4-5
"beep"     -> tier=COMPLEX  served=anthropic/claude-sonnet-5
"hi beep"  -> tier=COMPLEX  served=anthropic/claude-sonnet-5   # escalated, not SIMPLE

Semantic keyword matching on a router with semantic_keyword_matching: true, embedding_model: voyage-3-5, match_threshold: 0.5. Paraphrases with no literal keyword still route via embeddings

"help me roll out my container cluster today"  -> REASONING  served=anthropic/claude-sonnet-5   # no literal "kubernetes"/"k8s"
"hey there, good morning"                      -> SIMPLE     served=anthropic/claude-haiku-4-5   # paraphrase of "hi"

Command shape used for each:

curl -s http://localhost:4000/v1/chat/completions -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{"model":"smart-router","messages":[{"role":"user","content":"hi beep"}]}'

Tested end-to-end from Claude Code pointed at the router as its default model (ANTHROPIC_BASE_URL -> the proxy, ANTHROPIC_MODEL=smart-router); requests reached the proxy over the Anthropic Messages API (/v1/messages, including streaming) and were classified and routed correctly.

UI: Models + Endpoints -> Add Model -> Add Auto Router, Router Type set to "Auto-Router v2 [Recommended]" (default)

Type

🆕 New Feature

Changes

Adds two capabilities to the complexity auto-router and surfaces them in the Add Auto Router UI behind a Router Type selector

Backend introduces keyword_tier_rules (deterministic keyword-to-tier overrides evaluated before the weighted scorer) and optional semantic_keyword_matching backed by an embedding model and match threshold. When multiple lexical rules match a prompt, routing escalates to the most-severe matched tier so behavior no longer depends on the order rules were authored in. Semantic mode reuses the existing LiteLLMRouterEncoder and SemanticRouter machinery to match paraphrases by cosine similarity and falls back to the scorer when nothing clears the threshold. The cache-reload path now clears complexity_routers alongside auto_routers so config edits take effect without a restart

Frontend restores the Router Type radio: "Auto-Router v2 [Recommended]" is the default and sends keyword_tier_rules plus the semantic settings directly instead of flattening rule keywords into custom_technical_keywords; "Semantic Router [to be deprecated]" keeps the existing utterance-based flow unchanged. A client-side guard blocks submitting semantic matching without an embedding model or without any keyword tier rules, mirroring the backend validator. The "How Classification Works" explainer moved below Custom Technical Keywords and above Keyword Tier Overrides. The Test Connection action is dropped from the recommended flow since it cannot build a valid pre-save payload for a router (a TODO is left for a future JSON preview or config-test action)

Tests cover lexical escalation, semantic matching through the real library with injected embeddings, the semantic config guard, config validation, the reload-clear regression, and the frontend payload builder

[CLAassistant]

All committers have signed the CLA.

[greptile-apps]

Greptile Summary

Adds two capabilities to the complexity auto-router: deterministic keyword-to-tier overrides (keyword_tier_rules) evaluated before the weighted scorer, and optional embedding-based semantic keyword matching backed by SemanticRouter. A companion fix in clear_cache stops the blanket .clear() on auto_routers/complexity_routers from wiping config-defined routers on every DB-model write; it now pops only the DB-backed entries by name. The frontend restores the Router Type selector, exposes the new fields in dedicated KeywordTierRules and SemanticKeywordMatching components, and adds a client-side guard that mirrors the backend's _validate_semantic_matching validator.

• Backend (complexity_router.py, config.py): _lexical_tier_override escalates to the most-severe matched tier (independent of rule order); _semantic_tier_override bypasses SemanticRouter.acall's internal encoding so the query embedding carries the caller's metadata/litellm_metadata for spend attribution; the _semantic_routelayer is lazily built once and cached on the instance.
• Cache-reload fix (model_management_endpoints.py): clear_cache now targets only DB-model names, preventing a cross-tenant availability regression where any team admin's model update could permanently unroute config-defined routers for other tenants until a proxy restart.
• Tests: 423 lines of new backend tests use FakeEmbeddingRouter (no real network calls) to exercise lexical escalation, semantic routing, metadata propagation, threshold fallback, and route-layer caching; frontend tests cover the payload builder and semantic config guard.

Confidence Score: 5/5

Safe to merge — all changed paths are additive or narrowly scoped fixes with no regressions on existing routing behavior.

The lexical escalation logic is correct for all four ComplexityTier values. The semantic override correctly bypasses the library's internal encoding to carry caller metadata. The cache-clear fix is a well-targeted improvement with a regression test that directly models the previously broken scenario. Backend tests use deterministic fake embeddings with no real network calls, and frontend tests cover both the payload builder and the client-side semantic config guard. No existing tests were weakened.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/router_strategy/complexity_router/complexity_router.py	Adds lexical and semantic keyword-tier override resolution; lazy SemanticRouter init is cached correctly; override is evaluated before the scorer in async_pre_routing_hook
litellm/router_strategy/complexity_router/config.py	Adds KeywordTierRule model, TIER_SEVERITY_ORDER tuple, and three new fields with a second model_validator that guards semantic matching preconditions
litellm/proxy/management_endpoints/model_management_endpoints.py	Fixes clear_cache to pop only DB-model entries from auto_routers/complexity_routers instead of blanket .clear(), preserving config-defined routers
tests/test_litellm/router_strategy/test_complexity_router.py	423 lines of new tests covering lexical escalation, semantic routing via injected fake embeddings, config validation, caching, and edge-case branches
tests/test_litellm/proxy/management_endpoints/test_model_management_endpoints.py	Updated TestClearCache to use real dicts and adds TestClearCachePreservesConfigRouters regression test; mock changes correctly reflect the new targeted-clear behavior
ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.tsx	Refactored into submitRecommendedRouter/submitSemanticRouter; adds keyword tier rule and semantic matching state; client-side guard mirrors backend validator
ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts	New pure builder and guard function; id field stripped from keyword tier rules before sending to backend; semantic fields only included when toggle is on
ui/litellm-dashboard/src/components/add_model/KeywordTierRules.tsx	New component for managing keyword-to-tier override rules; uses Date.now()-based ids stripped before submission
ui/litellm-dashboard/src/components/add_model/SemanticKeywordMatching.tsx	New component for semantic keyword matching settings; correctly exports DEFAULT_MATCH_THRESHOLD for use in parent state initialisation
ui/litellm-dashboard/src/components/add_model/handle_add_auto_router_submit.tsx	Cleans up dead custom-embedding-model branching; unused response variable removed

_{Reviews (4): Last reviewed commit: "feat(auto_router): keyword tier override..." | Re-trigger Greptile}

[greptile-apps]

auto_sync="local" instructs the semantic_router library to persist route embeddings to a local JSON file on first use. In containerised/serverless deployments this can fail outright on read-only filesystems, silently write to ephemeral storage that disappears on restart, or race when multiple replicas start simultaneously. The routes are always reconstructed deterministically from the config, so there is no value in persisting them; setting auto_sync to None or omitting it disables the file-I/O path entirely.

Suggested change

	routelayer = SemanticRouter(
	routes=routes,
	encoder=LiteLLMRouterEncoder(
	litellm_router_instance=self.litellm_router_instance,
	model_name=embedding_model,
	score_threshold=self.config.match_threshold,
	),
	auto_sync="local",
	)
	routelayer = SemanticRouter(
	routes=routes,
	encoder=LiteLLMRouterEncoder(
	litellm_router_instance=self.litellm_router_instance,
	model_name=embedding_model,
	score_threshold=self.config.match_threshold,
	),
	auto_sync=None,
	)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing akapur99:litellm_complexity_router_keyword_tiers (3c714ed) with litellm_internal_staging (735a5bf)}

[akapur99]

@greptileai thanks for the review. Fixed the frontend guard; pushing back on the auto_sync="local" flag with the reasoning below.

Frontend guard (fixed): handleAutoRouterSubmit now calls a getSemanticConfigError helper before submit that mirrors the backend's _validate_semantic_matching constraint (embedding model required, keyword_tier_rules non-empty when semantic matching is enabled), and shows a client-side notification instead of letting the request hit the backend validator. Covered by 4 new unit tests on the helper.

auto_sync="local" (kept, not a defect): verified this does not write anything to the filesystem. semantic_router's LocalIndex._write_config is a no-op:

def _write_config(self, config: ConfigParameter):
    logger.warning("No config is written for LocalIndex.")

auto_sync="local" here means "the routes passed in are the source of truth, sync them into the (in-memory) local index" — it's what populates SemanticRouter's in-memory index from the Route objects we construct fresh from keyword_tier_rules on each first use. It is not persistence to disk; only remote index backends (e.g. Pinecone) actually write anything. I also tested dropping it (setting auto_sync=None, the default): the in-memory index never gets populated and all 3 semantic-matching tests fail (no route ever matches). This exact same auto_sync="local" + default LocalIndex combination is already used in production by the existing auto_router/auto_router.py (AutoRouter), so this PR isn't introducing a new pattern, just reusing it in the complexity router's semantic path.

Happy to add a code comment noting the no-op _write_config behavior if that would help future readers avoid re-flagging this.

[greptile-apps]

Both points are well-argued — withdrawing the auto_sync="local" concern.

You're right that my comment conflated LocalIndex with remote index backends. Since LocalIndex._write_config is a no-op and auto_sync=None leaves the in-memory index unpopulated (breaking matching entirely), keeping auto_sync="local" is correct. The code comment explaining the no-op behavior would be a net win for future readers, but it's not required.

The frontend guard fix (a getSemanticConfigError helper mirroring the backend's _validate_semantic_matching with 4 unit tests) is the right approach. LGTM on both.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

[akapur99]

@greptile-apps whats the updated rating

[greptile-apps]

Updated Confidence Score: 4/5

Both issues that dropped the original score to 3/5 have been resolved:

• auto_sync="local" — withdrawn. Your explanation that LocalIndex._write_config is a no-op and that auto_sync=None breaks in-memory index population is correct. The existing AutoRouter uses the same pattern, so this isn't a new risk.
• Frontend semantic guard — fixed with getSemanticConfigError mirroring backend _validate_semantic_matching, plus 4 unit tests. Clean approach.

The remaining half-point gap: the optional code comment explaining auto_sync="local"'s no-op behavior (you mentioned it yourself) would be a net positive for future readers, but it's cosmetic. The implementation is solid.

[veria-ai]

PR overview

This pull request adds keyword tier overrides and semantic keyword matching to the complexity router, with changes around router configuration and runtime matching behavior. It appears to let routing rules use lexical and semantic keyword criteria to influence model tier selection.

There are still two open security concerns in the new routing path. The highest-risk issue is that semantic index construction can occur synchronously on the first request after a router update, allowing a team admin to trigger unbudgeted embedding calls and block the event loop. A related configuration issue also leaves keyword rules unbounded, which can let very large keyword sets degrade proxy worker availability; one earlier issue has already been addressed, but these remaining denial-of-service and budget-bypass paths keep the PR at elevated risk.

Open issues (2)

• High: Request-time index construction bypasses budgets — litellm/router_strategy/complexity_router/complexity_router.py:476
• Medium: Unbounded keyword matching can exhaust proxy workers — litellm/router_strategy/complexity_router/config.py:34

Fixed/addressed: 1 · PR risk: 7/10

[akapur99]

Thanks for catching this — confirmed and fixed in ae89643957.

You're right that the blanket auto_routers.clear() / complexity_routers.clear() in clear_cache() cleared all in-memory router entries, including config-defined ones belonging to other tenants, and since the subsequent reload only re-adds DB-backed models (proxy_config.add_deployment), a config-defined router would stay unroutable until a full proxy restart. Any team admin's PATCH /model/{id}/update on their own team-owned model would trigger this for everyone.

Fixed by scoping the clear to only the router entries backed by a DB model that's actually part of this reload, keyed by model_name (the same key the router dicts use):

db_model_names = {
    model.get("model_name") for model in current_models if model.get("model_info", {}).get("db_model", False)
}
for model_name in db_model_names:
    llm_router.auto_routers.pop(model_name, None)
    llm_router.complexity_routers.pop(model_name, None)

Config-defined routers are never touched now; only the DB-backed ones get cleared (so they can be correctly re-populated by the reload). Added a dedicated regression test, TestClearCachePreservesConfigRouters::test_config_backed_routers_survive_unrelated_db_model_update, plus updated the existing test_clear_cache_preserve_config_models to assert on real dict contents instead of a mocked .clear() call.

[akapur99]

@greptile-apps check again and gimme a score homie

[akapur99]

@greptile-apps new score?

[veria-ai]

High: Request-time index construction bypasses budgets

SemanticRouter initializes its index synchronously here, calling the encoder for dimensions and all rule utterances without the caller metadata forwarded below. A team admin can update a team-owned router to reset this lazy cache, then send its first request to repeatedly consume unbudgeted embedding spend and block the event loop during provider calls. Build the index asynchronously under a per-router lock with attributed metadata, or initialize it outside the request path; also bound the number and size of rule utterances.

[veria-ai]

Medium: Unbounded keyword matching can exhaust proxy workers

A team admin can save tens of thousands of keywords and send nonmatching requests, forcing _lexical_tier_override to compile and evaluate a regex for every keyword synchronously on the event loop; this can delay requests for other tenants. Add conservative limits on rule count, keywords per rule, and keyword length, and preferably precompile lexical matchers when the router is initialized.