feat: add LAR-1 semantic routing strategy - #31289
Conversation
Optional custom routing selects deployments from agent metadata (confidence, evidence, time) via model_info.type tags. Configurable through routing_strategy: lar1 in router_settings or set_custom_routing_strategy(). Co-authored-by: Cursor <cursoragent@cursor.com>
|
|
Proof of fix (local Ollama, 2026-06-25)Proxy: uv run --extra proxy litellm --config examples/lar1_ollama_config.yaml --detailed_debug 2>&1 | tee lar1_proxy.logNote: Low confidence curl ( curl -s http://127.0.0.1:7979/v1/chat/completions -H "Authorization: Bearer sk-lar1-demo" -H "Content-Type: application/json" -d '{"model":"agent-router","messages":[{"role":"user","content":"hi"}],"max_tokens":5,"metadata":{"lar1":{"confidence":0.2}}}'High confidence curl ( curl -s http://127.0.0.1:7979/v1/chat/completions -H "Authorization: Bearer sk-lar1-demo" -H "Content-Type: application/json" -d '{"model":"agent-router","messages":[{"role":"user","content":"hi"}],"max_tokens":5,"metadata":{"lar1":{"confidence":0.8}}}'Log output: Unit tests: |
Greptile SummaryThis PR adds a new
Confidence Score: 3/5Not ready to merge — the sync routing path will silently return None for any non-async caller, and the strategy selects from the full model list without filtering out deployments that are in cooldown. The strategy's sync get_available_deployment is a no-op stub, so the router's sync completion() path returns None and will crash on any downstream attribute access. Additionally, the strategy reads from self._router.model_list directly, bypassing async_get_healthy_deployments, which means it can route to endpoints actively in cooldown. Both issues exist on the changed code paths and would reproduce in production use. litellm/router_strategy/lar1_routing.py — the sync method stub and direct model_list access need to be fixed before this is safe to enable
|
| Filename | Overview |
|---|---|
| litellm/router_strategy/lar1_routing.py | New LAR-1 routing strategy; sync get_available_deployment is a stub returning None (breaks sync router calls), and the strategy reads model_list directly without health filtering |
| litellm/router.py | Adds lar1 to accepted routing strategies and wires lar1_settings; changes are well-scoped and use the existing set_custom_routing_strategy mechanism |
| litellm/types/lar1.py | Adds LAR1Metadata Pydantic schema and enums; currently unused by routing logic, and evidence field lacks enum constraint |
| tests/local_testing/test_lar1_routing.py | 8 async unit tests covering core routing cases; tests are mock-only with fake API keys — no real network calls |
| examples/lar1_ollama_config.yaml | Example Ollama config demonstrating all four deployment types; master_key is a plaintext demo value intended for local use only |
Reviews (1): Last reviewed commit: "feat: add LAR-1 semantic routing strateg..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| def get_available_deployment(self, *args, **kwargs): | ||
| pass |
There was a problem hiding this comment.
get_available_deployment is a no-op stub — breaks all non-async callers
get_available_deployment returns None (implicit return from pass). When set_custom_routing_strategy is called it binds this method onto the router instance, so any sync routing call — e.g. router.completion(...) — will receive None from get_available_deployment and crash with an AttributeError or TypeError downstream when the caller tries to index into the result.
| ) | ||
| time_dim = "NOW" | ||
|
|
||
| model_list = self._router.model_list if self._router else [] |
There was a problem hiding this comment.
self._router.model_list is the raw list of all registered deployments. The router's normal async flow calls async_get_healthy_deployments first and passes a filtered healthy_deployments to the selector. By reading directly from model_list, the LAR-1 strategy can select a deployment that is actively in cooldown, silently sending traffic to a known-failing endpoint.
| def _select_deployment(self, target_type, model_list): | ||
| if not model_list: | ||
| return None | ||
|
|
||
| for m in model_list: | ||
| if isinstance(m, dict): | ||
| model_type = m.get("model_info", {}).get("type", "") | ||
| if model_type == target_type: | ||
| return m | ||
|
|
||
| return model_list[0] |
There was a problem hiding this comment.
model_list[0] emits a misleading log entry
When no deployment with the requested type exists, _select_deployment returns model_list[0] without any warning. Control then returns to async_get_available_deployment, which logs [LAR-1] confidence=X → cloud-fast (for example) even though the deployment actually returned has a different type. This makes the routing decision look correct in logs when it silently fell back.
| def __init__(self, router_instance=None, thresholds: Optional[Dict] = None): | ||
| self._router = router_instance | ||
| self.thresholds = thresholds or { | ||
| "low": 0.3, | ||
| "medium": 0.5, | ||
| "high": 0.7, | ||
| } |
There was a problem hiding this comment.
If lar1_settings is misconfigured so that low >= medium or medium >= high (e.g. a YAML typo), _classify_request will produce silently incorrect routing — cloud-fast and/or local ranges collapse to zero width and are never selected. A simple assertion low < medium < high at construction time would catch this immediately.
| from enum import Enum | ||
| from typing import List | ||
|
|
||
| from pydantic import BaseModel | ||
|
|
||
|
|
||
| class LAR1Act(str, Enum): | ||
| INF = "INF" # Inference | ||
| OBS = "OBS" # Observation | ||
| RET = "RET" # Retrieval | ||
| GEN = "GEN" # Generation | ||
|
|
||
|
|
||
| class LAR1Time(str, Enum): | ||
| NOW = "NOW" # Current context | ||
| MEM = "MEM" # From memory | ||
| CTX = "CTX" # From context window | ||
| PRE = "PRE" # Predicted/future | ||
|
|
||
|
|
||
| class LAR1Mind(str, Enum): | ||
| REF = "REF" # Reflective | ||
| REC = "REC" # Recognized pattern | ||
| HYP = "HYP" # Hypothesis | ||
| ACT = "ACT" # Recommended action | ||
|
|
||
|
|
||
| class LAR1Metadata(BaseModel): | ||
| act: LAR1Act = LAR1Act.INF | ||
| time: LAR1Time = LAR1Time.NOW | ||
| mind: LAR1Mind = LAR1Mind.REF | ||
| confidence: float = 0.5 # 0.0-1.0 | ||
| evidence: List[str] = [] # "SYNTH", "RETRIEVED", "UNVERIFIED" |
There was a problem hiding this comment.
LAR1Metadata schema is never used by the routing strategy
litellm/router_strategy/lar1_routing.py reads metadata fields directly from the raw dict rather than validating through LAR1Metadata. The schema and routing logic will silently drift — for example LAR1Metadata.evidence is List[str] with no enum constraint, while the routing code accepts only {"SYNTH", "RETRIEVED", "UNVERIFIED", "CONFIRMED"}. The PR description notes Pydantic hot-path validation is a follow-up, but the schema should at minimum be kept structurally consistent with the routing code's valid sets.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Greptile SummaryThis PR introduces an optional LAR-1 semantic routing strategy that selects a deployment based on agent-supplied confidence, evidence, and time metadata rather than latency or cost. The
Confidence Score: 3/5The routing strategy core has three correctness problems that would produce wrong behaviour or silent failures in production before any other issues are addressed. The misleading fallback log, the silent None from the synchronous path, and the unfiltered model_list all occur on the hot request path. Any misconfigured model list silently routes to the wrong backend and logs the wrong type. The synchronous path returning None would propagate as an AttributeError downstream. litellm/router_strategy/lar1_routing.py needs the most attention: misleading log on fallback, silent None from sync path, and routing from the unfiltered model list.
|
| Filename | Overview |
|---|---|
| litellm/router_strategy/lar1_routing.py | New LAR-1 routing strategy with three correctness issues: misleading log when fallback fires, silent None from sync path, and routing from the unfiltered model_list that bypasses health checks. |
| litellm/router.py | Adds lar1 to the routing strategy type union and valid-strategy list, wiring up LAR1RoutingStrategy via set_custom_routing_strategy with a deferred import. |
| litellm/types/lar1.py | Defines LAR1Metadata pydantic schema; evidence field uses List[str] instead of an enum, and CONFIRMED is absent from the type comment. |
| tests/local_testing/test_lar1_routing.py | Eight async unit tests covering main routing branches with mock model lists and no real network calls. |
| examples/lar1_ollama_config.yaml | Example proxy config for local Ollama demonstration; straightforward and consistent with the PR description. |
Reviews (2): Last reviewed commit: "feat: add LAR-1 semantic routing strateg..." | Re-trigger Greptile
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
| selected = self._select_deployment(target, model_list) | ||
|
|
||
| if selected: | ||
| verbose_router_logger.info(f"[LAR-1] confidence={confidence} → {target}") | ||
| return selected | ||
|
|
||
| return model_list[0] if model_list else None |
There was a problem hiding this comment.
_select_deployment always returns model_list[0] as a fallback when no deployment with the target type is found, so selected is always truthy for a non-empty list. The if selected branch will therefore log [LAR-1] confidence=0.8 → deep even when the actual deployment returned is a different type. Operators relying on these logs to verify routing behaviour will see incorrect data every time the fallback fires.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| def get_available_deployment(self, *args, **kwargs): | ||
| pass |
There was a problem hiding this comment.
get_available_deployment silently returns None
get_available_deployment is wired in by set_custom_routing_strategy as the synchronous routing entrypoint. Returning None (via bare pass) means any sync call-path — including retry/fallback flows — receives None and will likely produce an AttributeError downstream. Raising NotImplementedError immediately would surface the gap instead of propagating a silent None.
| def get_available_deployment(self, *args, **kwargs): | |
| pass | |
| def get_available_deployment(self, *args, **kwargs): | |
| raise NotImplementedError( | |
| "[LAR-1] Synchronous routing is not yet implemented. " | |
| "Use async_get_available_deployment or an async Router call." | |
| ) |
| ) | ||
| time_dim = "NOW" | ||
|
|
||
| model_list = self._router.model_list if self._router else [] |
There was a problem hiding this comment.
self._router.model_list contains all configured deployments including those that are rate-limited or degraded. Standard routing strategies operate on healthy_deployments filtered by async_get_healthy_deployments, but set_custom_routing_strategy replaces that method entirely, so the LAR-1 strategy never filters the list. Consider calling await self._router.async_get_healthy_deployments(model=model, ...) or documenting the limitation prominently.
| if model_type == target_type: | ||
| return m | ||
|
|
||
| return model_list[0] |
There was a problem hiding this comment.
_select_deployment silently returns model_list[0] when no deployment matches target_type. Adding a warning here makes it observable when a misconfigured model list causes routing to silently diverge from intent.
| return model_list[0] | |
| verbose_router_logger.warning( | |
| f"[LAR-1] No deployment found for type '{target_type}'. Falling back to model_list[0]." | |
| ) | |
| return model_list[0] | |
| def get_available_deployment(self, *args, **kwargs): |
| t = self.thresholds | ||
| if confidence < t["low"]: | ||
| return "cloud-smart" | ||
| elif confidence < t["medium"]: | ||
| return "cloud-fast" | ||
| elif confidence < t["high"]: | ||
| return "local" | ||
| else: | ||
| return "deep" |
There was a problem hiding this comment.
If a user configures confidence_threshold_low >= confidence_threshold_medium (or medium >= high), the elif chain produces wrong tier assignments silently. A startup-time check asserting 0 < low < medium < high < 1 would surface this misconfiguration immediately.
| time: LAR1Time = LAR1Time.NOW | ||
| mind: LAR1Mind = LAR1Mind.REF | ||
| confidence: float = 0.5 # 0.0-1.0 | ||
| evidence: List[str] = [] # "SYNTH", "RETRIEVED", "UNVERIFIED" |
There was a problem hiding this comment.
evidence field should use a typed enum, and CONFIRMED is undocumented
act, time, and mind are all enum-typed, but evidence is List[str]. The routing strategy accepts {"SYNTH", "RETRIEVED", "UNVERIFIED", "CONFIRMED"} as valid values while the comment here lists only three. Defining a LAR1Evidence enum and using List[LAR1Evidence] would make the schema self-documenting and consistent with the other fields.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Use routing_strategy_args instead of lar1_settings, healthy deployment filtering, pydantic metadata validation, and expanded regression tests. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Superseded by #31295. Branch renamed to litellm_lar1-routing so CircleCI required checks can run. |
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| if self._router is None: | ||
| return None | ||
|
|
||
| lar1 = _parse_lar1_metadata(request_kwargs) |
There was a problem hiding this comment.
Medium: Untrusted metadata controls deployment selection
The proxy preserves client-provided metadata.lar1, so a caller can submit confidence: 1 to route every request to the deep deployment, or UNVERIFIED to force cloud-smart. If these tiers represent cost or trust boundaries, derive this signal server-side or require an authenticated/signed assertion rather than accepting it directly from request metadata.
PR overviewThis PR adds a LAR-1 semantic routing strategy for LiteLLM, introducing logic to choose between configured deployments based on LAR-1 request signals such as confidence or verification state. There is one open security concern: deployment selection can currently be influenced by caller-supplied LAR-1 metadata. That means a client may be able to steer traffic toward specific tiers, which could matter if those deployments differ by cost, trust level, or data handling expectations. No issues have been fixed yet, so the PR still needs a server-side or authenticated source of routing signals before this concern is resolved. Open issues (1)
Fixed/addressed: 0 · PR risk: 5/10 |
|
This issue is addressed in cloudiaspecula#1: Untrusted metadata controls deployment selection — LAR-1 routing now defaults to server-side configuration ( To restore the previous client-driven behaviour for middleware-derived signals, set PR: cloudiaspecula#1 |
3 participants
Summary
This PR adds an optional LAR-1 semantic routing strategy. Instead of routing by latency or cost, it selects a deployment from agent-supplied metadata in
request_kwargs.metadata.lar1(confidence, evidence, time). Deployments are tagged withmodel_info.type(cloud-smart,cloud-fast,local,deep). Enable viarouter_settings.routing_strategy: lar1in proxy config, orrouter.set_custom_routing_strategy(LAR1RoutingStrategy(router))in code. The routing decision is local dict lookups only; no extra LLM calls.LAR-1 complements
complexity_router: that router scores request text; LAR-1 scores agent state from the caller. Reference: SSRN abstract 6981858, RFC v0.9.Files
litellm/router_strategy/lar1_routing.py—LAR1RoutingStrategy, validation, configurable thresholdslitellm/types/lar1.py—LAR1Metadatapydantic schema (types only; hot-path validation is follow-up)litellm/router.py—routing_strategy: lar1,lar1_settingsYAML wiringtests/local_testing/test_lar1_routing.py— 8 unit testsexamples/lar1_ollama_config.yaml— local Ollama proof-of-fix configRouting rules (default thresholds)
evidencecontainsUNVERIFIEDcloud-smarttimeisMEMcloud-fastconfidence< 0.3cloud-smartconfidence< 0.5cloud-fastconfidence< 0.7localconfidence>= 0.7deepThresholds overridable via
lar1_settings.confidence_threshold_low|medium|high.Test plan
pytest tests/local_testing/test_lar1_routing.py -v— 8 passedmake test-unit(CI)Unit tests cover: low/high confidence, UNVERIFIED override, MEM override, invalid confidence fallback, no-metadata default, custom thresholds,
Router(routing_strategy="lar1")init.Screenshots / Proof of Fix
Ollama must be running. From repo root:
Low confidence (
0.2→cloud-smart→ollama/qwen3.5:9b):High confidence (
0.8→deep→ollama/lfm2.5-thinking:latest):Log proof:
grep '\[LAR-1\]' lar1_proxy.logExpected:
Downstream model in debug logs:
model='ollama/qwen3.5:9b'thenmodel='ollama/lfm2.5-thinking:latest'.Out of scope (follow-up)
LAR1Metadatapydantic validation in the hot pathget_available_deployment()implementationprefer_cloud_for_memoryyaml flagType
New Feature