feat: add LAR-1 semantic routing strategy

[carlsonchik]

Relevant issues

N/A (new feature)

Summary

Adds optional LAR-1 semantic routing. Instead of routing by latency or cost, the router picks a deployment from agent metadata in request_kwargs.metadata.lar1 (confidence, evidence, time). Each deployment is tagged with model_info.type: cloud-smart, cloud-fast, local, or deep. All LAR-1 tiers must share the same model_name alias (see examples/lar1_ollama_config.yaml). Routing is local classification only; no extra LLM calls.

LAR-1 complements complexity_router: that router scores request text; LAR-1 scores agent state from the caller. References: SSRN 6981858, LAR-1 RFC v0.9

Enable in proxy config:

router_settings:
  routing_strategy: lar1
  routing_strategy_args:
    confidence_threshold_low: 0.3
    confidence_threshold_medium: 0.5
    confidence_threshold_high: 0.7

Or in code: apply_lar1_routing_strategy(router, routing_strategy_args) / Router(routing_strategy="lar1", ...).

Routing rules (default thresholds)

Signal	Route
evidence contains UNVERIFIED	cloud-smart
time is MEM	cloud-fast
confidence < 0.3	cloud-smart
confidence < 0.5	cloud-fast
confidence < 0.7	local
confidence >= 0.7	deep

Override thresholds via routing_strategy_args.confidence_threshold_low|medium|high (single source: DEFAULT_THRESHOLDS in lar1_routing.py).

Implementation notes

• LAR1RoutingStrategy uses async_get_healthy_deployments (cooldown/health checks)
• Pydantic LAR1Metadata validation on the hot path; invalid metadata falls back to defaults with a warning
• Async-only: sync get_available_deployment() raises NotImplementedError
• Router.update_settings(routing_strategy="lar1", ...) and routing_strategy_args updates re-wire the strategy via apply_lar1_routing_strategy
• Supersedes feat: add LAR-1 semantic routing strategy #31289 (branch renamed to litellm_lar1-routing, base retargeted to litellm_oss_staging for fork CI)
• Single squashed commit on litellm_lar1-routing for review (b0e25089ad)

Files

• litellm/router_strategy/lar1_routing.py — strategy, thresholds, metadata parsing
• litellm/types/lar1.py — LAR1Metadata + enums (LAR1Evidence includes CONFIRMED)
• litellm/router.py — lar1 in valid strategies; init + update_settings wiring
• tests/test_litellm/router_strategy/test_lar1_routing.py — 30 unit tests
• examples/lar1_ollama_config.yaml — local Ollama proof-of-fix config

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all unit tests on make test-unit (GitHub Actions green on latest commit)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review (5/5)

Test plan

• pytest tests/test_litellm/router_strategy/test_lar1_routing.py -v — 30 passed locally
• GitHub Actions on litellm_oss_staging base (fork PR)

Coverage includes: confidence bands, UNVERIFIED/MEM overrides, custom thresholds, invalid metadata, healthy-deployment edge cases, update_settings lar1 wiring, fallback logging, Router(routing_strategy="lar1") init.

Screenshots / Proof of Fix

Ollama must be running. From repo root:

source .venv/bin/activate
uv run litellm --config examples/lar1_ollama_config.yaml --port 4000 --detailed_debug 2>&1 | tee lar1_proxy.log

Low confidence (0.2 -> cloud-smart -> ollama/qwen3.5:9b):

curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-lar1-demo" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "agent-router",
    "messages": [{"role": "user", "content": "Say hi in one word"}],
    "max_tokens": 10,
    "metadata": {
      "lar1": {"confidence": 0.2, "evidence": [], "time": "NOW"}
    }
  }'

High confidence (0.8 -> deep -> ollama/lfm2.5-thinking:latest):

curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-lar1-demo" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "agent-router",
    "messages": [{"role": "user", "content": "Say hi in one word"}],
    "max_tokens": 10,
    "metadata": {
      "lar1": {"confidence": 0.8, "evidence": [], "time": "NOW"}
    }
  }'

Log proof:

grep '\[LAR-1\]' lar1_proxy.log

Expected:

[LAR-1] confidence=0.2 -> cloud-smart
[LAR-1] confidence=0.8 -> deep

Type

New Feature

[greptile-apps]

Greptile Summary

This PR adds a new lar1 semantic routing strategy that selects a deployment (cloud-smart, cloud-fast, local, or deep) based on agent-provided metadata (confidence, evidence, time) rather than latency or cost. The strategy is wired into the Router via the existing set_custom_routing_strategy mechanism and gated behind the routing_strategy="lar1" config key.

• New LAR1RoutingStrategy class uses async_get_healthy_deployments for cooldown/health awareness and _select_deployment with two-pass fallback for resilience when a target type is unavailable.
• apply_lar1_routing_strategy is called both at Router.__init__ and in update_settings, with a relink_lar1_from_args flag ensuring threshold changes propagate even when routing_strategy is unchanged.
• 29 unit tests cover confidence bands, evidence/time overrides, invalid metadata, empty healthy-deployments, and update_settings re-wiring, all using mocked or local objects without real network calls.

Confidence Score: 5/5

This PR is safe to merge; the routing logic, init wiring, and update_settings re-link path all work correctly for normal usage.

The new routing strategy correctly uses set_custom_routing_strategy to override the instance-level async_get_available_deployment, so the class-level fallback guard never fires for LAR-1 requests. Both the init and update_settings paths call apply_lar1_routing_strategy, and the routing_strategy_args re-link flag correctly propagates threshold changes made independently of a strategy switch. No data-path bugs or auth issues were found.

litellm/router_strategy/lar1_routing.py — minor import path and OTEL span propagation gaps worth addressing before this strategy sees heavy production use.

Important Files Changed

Filename	Overview
litellm/router_strategy/lar1_routing.py	New routing strategy; logic is sound, but imports CustomRoutingStrategyBase from litellm.router (a re-exporter) instead of litellm.types.router, and async_get_healthy_deployments is called without propagating parent_otel_span.
litellm/router.py	LAR-1 is correctly wired at both init and update_settings; routing_strategy_args re-link logic handles the independent-args-update case properly. The existing async_get_available_deployment guard (lines 11338-11351) doesn't whitelist "lar1", but this is moot in practice because set_custom_routing_strategy shadows the class method with an instance attribute.
litellm/types/lar1.py	Clean Pydantic model with sensible defaults and ge/le validation on confidence; no issues.
tests/test_litellm/router_strategy/test_lar1_routing.py	29 tests covering all key paths; no real network calls (fake API keys, internal health checks only); mocks used where needed for edge cases.
examples/lar1_ollama_config.yaml	Demo config; correctly maps all four LAR-1 tiers to Ollama models under a shared model_name alias.

_{Reviews (3): Last reviewed commit: "feat: add LAR-1 semantic routing strateg..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[carlsonchik]

@greptileai please re-review

Addressed the prior 3/5 feedback in commit b662cfe:

• Router.update_settings(routing_strategy="lar1") and routing_strategy_args updates now call apply_lar1_routing_strategy
• _select_deployment fallback returns the first dict deployment instead of blind deployments[0]
• threshold defaults come from DEFAULT_THRESHOLDS via lar1_thresholds_from_args (no duplicate literals in router.py)

PR description updated with routing rules, config example, and proof-of-fix. 29 tests in tests/test_litellm/router_strategy/test_lar1_routing.py

[carlsonchik]

Ready for maintainer review.

Checklist

• 30 unit tests in tests/test_litellm/router_strategy/test_lar1_routing.py
• CI green on latest commit (lint, docs, type-check budget, unit shards, codecov/patch)
• Greptile Confidence Score 5/5: feat: add LAR-1 semantic routing strategy #31295 (comment)
• Proof-of-fix (Ollama) in PR description
• Branch history squashed to a single commit for easier review

Scope: additive routing_strategy: lar1 only; existing strategies unchanged.

If helpful, recent router_strategy touches include @Sameerlite @yuneng-berri @mateo-berri — would appreciate a look when you have a moment.

Per contributing guide, also available on LiteLLM Slack #pr-review if that is faster.

[carlsonchik]

@greptileai please re-review after squash to single commit b0e2508 (no logic changes, history cleanup only)

[veria-ai]

Medium: Disabling LAR-1 leaves its selector active

apply_lar1_routing_strategy() shadows both deployment-selection methods on the router instance, but this branch does not remove those attributes when switching away from LAR-1. An authenticated caller can therefore continue forcing cloud-smart or deep deployments with crafted metadata.lar1 after an operator changes the strategy to simple-shuffle or another policy. Restore the class methods before initializing the replacement strategy, or integrate LAR-1 into the normal strategy-selector dispatch rather than monkey-patching the router.

[veria-ai]

Medium: Known master key in executable example

Anyone who can reach a proxy started from this example can authenticate with the repository-known master key and access administrative routes and configured models. Require the operator to provide a secret instead.

Suggested change

	master_key: sk-lar1-demo
	master_key: os.environ/LITELLM_MASTER_KEY

[veria-ai]

PR overview

This pull request adds the LAR-1 semantic routing strategy to the LiteLLM router, including logic for selecting between deployment modes such as cloud-smart and deep. It also includes an example Ollama configuration for using the new routing strategy.

There are two open security concerns. The router can leave LAR-1 selection hooks active after switching to another routing policy, allowing an authenticated caller to keep steering requests to specific deployment classes. The included executable example also contains a repository-known master key, which could grant administrative access if an operator runs it as-is on a reachable proxy. No issues have been addressed yet, so the PR still carries moderate security risk.

Open issues (2)

• Medium: Disabling LAR-1 leaves its selector active — litellm/router.py:10619
• Medium: Known master key in executable example — examples/lar1_ollama_config.yaml:42

Fixed/addressed: 0 · PR risk: 6/10

[cloudiaspecula]

Both issues are addressed in cloudiaspecula#1:

Stale selector after strategy switch — apply_lar1_routing_strategy() now calls router._reset_custom_routing_strategy() at the start of the function, before applying the new strategy. This ensures monkey-patched methods are cleaned up when switching away from LAR-1, preventing stale selectors from remaining active.

Known master key in example — Replaced hardcoded master_key: *** with os.environ/LITELLM_MASTER_KEY in examples/lar1_ollama_config.yaml and added security warnings.

PR: cloudiaspecula#1