Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/proxy/auto_routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,16 @@ curl -X POST http://localhost:4000/v1/chat/completions \
-d '{"model": "smart-router", "messages": [{"role": "user", "content": "What is 2+2?"}]}'
```

## Set it up with your agent

To get started, tell your agent:

```
run curl -fsSL https://docs.litellm.ai/skills/auto-router and follow the instructions
```

It reads the models your proxy already serves, asks how you want the router named and which model should serve each tier, and calls out the defaults it is assuming before it writes anything.

## Full config

Every knob v2 exposes. All fields on `complexity_router_config` are optional except `tiers`.
Expand Down
2 changes: 2 additions & 0 deletions docs/tutorials/claude_code_autorouter.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ model_list:
complexity_router_default_model: claude-sonnet-5
```

To have an agent write that entry against the models your proxy already serves, tell it `run curl -fsSL https://docs.litellm.ai/skills/auto-router and follow the instructions`, and say the router is for Claude Code so it picks an accepted name.

If API callers already use a non-Anthropic name, keep both by declaring a second `model_list` entry with the same `complexity_router_config` under the Claude-facing name. `router_settings.model_group_alias` does not work here, because alias resolution runs after auto-router dispatch and the aliased call fails with `Unmapped LLM provider`.

## Add the router to your organization's allowlist
Expand Down
158 changes: 158 additions & 0 deletions static/skills/auto-router
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
# Skill: set up a LiteLLM auto router

You are helping the user add an auto router to their LiteLLM proxy. An auto router is one
model name that classifies each request and forwards it to the cheapest model that can
handle it, so a trivial question does not pay for a frontier model.

Reference documentation: https://docs.litellm.ai/docs/proxy/auto_routing

This setup is interactive. Every choice below belongs to the user: ask, wait for the
answer, and never pick a router name or a tier model on their behalf. Never reference a
model the proxy does not already serve. Batch the questions into as few messages as you can
rather than asking one at a time, and skip anything the user already told you.

## 1. Find the models the proxy already serves

Locate the config the proxy loads: the file passed to `litellm --config`, commonly
`config.yaml` or `proxy_config.yaml`. Read its `model_list` and note every `model_name`,
along with which are chat models and which are embedding models.

If a running proxy is reachable, `curl -s $LITELLM_PROXY_URL/v1/models -H "Authorization:
Bearer $LITELLM_API_KEY"` lists the same names. If the models are managed in the database
(added through the Admin UI or `/model/new`) and no `model_list` exists in a file, use that
listing and create the router through the management API in step 4.

Show the user every chat model you found as a numbered menu they can pick from later, one
line each: the `model_name` they would call, the underlying provider model behind it from
`litellm_params.model`, and a short note on where it sits, roughly cheapest to strongest.
Say outright that these are their options and that nothing outside the list can be routed
to without adding a deployment first. Keep the embedding models in a separate short list,
since they are only usable for semantic keyword matching, not as tier targets.

If the list is long, still show all of it rather than a sample. If you cannot find any
models, stop and ask where the config lives rather than guessing.

## 2. Ask what to name the router

The name becomes the model name callers request. Ask the user for it and use exactly what
they give you. Do not propose a name, and do not fall back to one if they are undecided;
ask again instead.

Raise one constraint only if the user says the router will be called from a client that
filters model names by vendor, such as Claude Code or Claude Desktop: those accept only
names that read as Anthropic models, meaning the name has to contain `claude`, `anthropic`,
or a family word such as `opus`, `sonnet`, or `haiku`, with no other vendor's name in it.
See https://docs.litellm.ai/docs/tutorials/claude_code_autorouter

## 3. Ask which model serves each tier

Show the four tiers with what each one catches, repeat the numbered model menu next to the
question so the user does not have to scroll for it, and ask them to assign one per tier.
Accept a number or a name. Do not pre-fill an assignment; if they ask you to recommend one,
say what you would pick and why, then still wait for their confirmation.

- `SIMPLE`: greetings, one-line lookups, trivial questions
- `MEDIUM`: ordinary questions and short edits
- `COMPLEX`: multi-step or code-heavy work
- `REASONING`: anything asking to think step by step, prove, or plan at length

Tell them a model may serve several tiers, and that a tier can hold a list of models the
router picks from at random rather than a single name. Also ask which model should serve as
the fallback when classification fails, which is what `complexity_router_default_model`
sets, offering the same menu.

If a tier has no good candidate in what they serve, say so plainly and name the models they
would have to add rather than quietly reusing one that does not fit.

## 4. Write the config

Append one entry to `model_list`, changing nothing else in the file:

```yaml
- model_name: <router name>
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: <simple model>
MEDIUM: <medium model>
COMPLEX: <complex model>
REASONING: <reasoning model>
complexity_router_default_model: <fallback model>
```

For a database-managed proxy, create the same deployment through the management API
instead:

```bash
curl -X POST $LITELLM_PROXY_URL/model/new \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
-d '{"model_name": "<router name>", "litellm_params": {"model": "auto_router/complexity_router", "complexity_router_config": {"tiers": {"SIMPLE": "<simple model>", "MEDIUM": "<medium model>", "COMPLEX": "<complex model>", "REASONING": "<reasoning model>"}}, "complexity_router_default_model": "<fallback model>"}}'
```

## 5. Lay out the defaults and ask what else to change

Everything not written above runs on its default. Before you finish, show the user this
list, say these are the defaults now in force, and ask whether they want to change any of
them. Apply whatever they pick, and leave every value they keep out of the config so the
backend default applies.

| Setting | Default in force | What changing it buys |
| --- | --- | --- |
| Classifier | Heuristic scorer, no API call, sub-millisecond, scores prompt length, code presence, reasoning markers, technical terms | `classifier_type: llm` reads the request more accurately, at one small model call per request |
| Session affinity | Off, so every turn is classified on its own merits | `session_affinity: true` pins a session to its first-turn model, preserving provider prompt caches and avoiding replay of one model's history to another, at the cost of the whole session inheriting the first turn's tier. `session_affinity_ttl_seconds` defaults to 3600 |
| Keyword rules | None | `keyword_tier_rules` forces chosen phrases into a tier before the classifier runs |
| Semantic matching | Off | `semantic_keyword_matching: true` plus an `embedding_model` picked from the embedding models you listed extends those rules to paraphrases |
| Adaptive routing | Off, so a tier with several models picks uniformly at random | `adaptive: true` Thompson-samples within the tier by observed quality |
| Technical keyword list | Built-in list only | `custom_technical_keywords` appends domain terms the built-in list misses |
| Scorer thresholds | Built-in boundaries and weights | `tier_boundaries`, `token_thresholds`, and `dimension_weights` move where the scorer draws each tier |

If the user takes the LLM classifier, show the same model menu again, ask which one should
do the classifying, noting that a small fast model is what this wants since it runs on
every request, and add:

```yaml
classifier_type: llm
classifier_llm_config:
model: <classifier model>
timeout_ms: 2000
```

Then tell them what that turns on by default and ask whether to change it: the classifier
sees the last 3 prior human turns, truncated to 200 characters each
(`classifier_context_window_size: 3`, `classifier_context_per_turn_chars: 200`), so a
follow-up like "now do the same for the streaming path" is rated against what it refers to
instead of on its own length. Assistant replies are excluded
(`classifier_context_include_assistant_turns: false`); turning them on helps when the
assistant states the difficulty and the user only answers "yes". A window size of 0
classifies the current message alone.

## 6. Verify

Restart the proxy, then send one trivial request and one hard one, and show the user both
responses and the model each landed on:

```bash
curl -s $LITELLM_PROXY_URL/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_API_KEY" -H 'Content-Type: application/json' \
-d '{"model":"<router name>","messages":[{"role":"user","content":"what is 2+2?"}]}'

curl -s $LITELLM_PROXY_URL/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_API_KEY" -H 'Content-Type: application/json' \
-d '{"model":"<router name>","messages":[{"role":"user","content":"Think step by step and design a multi-region active-active Postgres failover strategy."}]}'
```

The proxy logs carry one line per routing decision:

```
ComplexityRouter: routing decision cause=complexity_scorer, tier=SIMPLE, score=-0.150, signals=['short (7 tokens)', 'simple (what is)'], routed_model=gpt-4o-mini
```

The two requests should report different tiers. If they do not, show the user what you saw
rather than tuning thresholds on your own.

## 7. Hand back

Tell the user the router name to call, the tier assignments, the settings in force, and
where the remaining knobs live:
https://docs.litellm.ai/docs/proxy/auto_routing#full-config