diff --git a/docs/proxy/auto_routing.md b/docs/proxy/auto_routing.md index b264773bf..d8b09bd29 100644 --- a/docs/proxy/auto_routing.md +++ b/docs/proxy/auto_routing.md @@ -58,6 +58,16 @@ curl -X POST http://localhost:4000/v1/chat/completions \ -d '{"model": "smart-router", "messages": [{"role": "user", "content": "What is 2+2?"}]}' ``` +## Set it up with your agent + +To get started, tell your agent: + +``` +run curl -fsSL https://docs.litellm.ai/skills/auto-router and follow the instructions +``` + +It reads the models your proxy already serves, asks how you want the router named and which model should serve each tier, and calls out the defaults it is assuming before it writes anything. + ## Full config Every knob v2 exposes. All fields on `complexity_router_config` are optional except `tiers`. diff --git a/docs/tutorials/claude_code_autorouter.md b/docs/tutorials/claude_code_autorouter.md index 35de4013c..5f353450d 100644 --- a/docs/tutorials/claude_code_autorouter.md +++ b/docs/tutorials/claude_code_autorouter.md @@ -30,6 +30,8 @@ model_list: complexity_router_default_model: claude-sonnet-5 ``` +To have an agent write that entry against the models your proxy already serves, tell it `run curl -fsSL https://docs.litellm.ai/skills/auto-router and follow the instructions`, and say the router is for Claude Code so it picks an accepted name. + If API callers already use a non-Anthropic name, keep both by declaring a second `model_list` entry with the same `complexity_router_config` under the Claude-facing name. `router_settings.model_group_alias` does not work here, because alias resolution runs after auto-router dispatch and the aliased call fails with `Unmapped LLM provider`. ## Add the router to your organization's allowlist diff --git a/static/skills/auto-router b/static/skills/auto-router new file mode 100644 index 000000000..85bdacf72 --- /dev/null +++ b/static/skills/auto-router @@ -0,0 +1,158 @@ +# Skill: set up a LiteLLM auto router + +You are helping the user add an auto router to their LiteLLM proxy. An auto router is one +model name that classifies each request and forwards it to the cheapest model that can +handle it, so a trivial question does not pay for a frontier model. + +Reference documentation: https://docs.litellm.ai/docs/proxy/auto_routing + +This setup is interactive. Every choice below belongs to the user: ask, wait for the +answer, and never pick a router name or a tier model on their behalf. Never reference a +model the proxy does not already serve. Batch the questions into as few messages as you can +rather than asking one at a time, and skip anything the user already told you. + +## 1. Find the models the proxy already serves + +Locate the config the proxy loads: the file passed to `litellm --config`, commonly +`config.yaml` or `proxy_config.yaml`. Read its `model_list` and note every `model_name`, +along with which are chat models and which are embedding models. + +If a running proxy is reachable, `curl -s $LITELLM_PROXY_URL/v1/models -H "Authorization: +Bearer $LITELLM_API_KEY"` lists the same names. If the models are managed in the database +(added through the Admin UI or `/model/new`) and no `model_list` exists in a file, use that +listing and create the router through the management API in step 4. + +Show the user every chat model you found as a numbered menu they can pick from later, one +line each: the `model_name` they would call, the underlying provider model behind it from +`litellm_params.model`, and a short note on where it sits, roughly cheapest to strongest. +Say outright that these are their options and that nothing outside the list can be routed +to without adding a deployment first. Keep the embedding models in a separate short list, +since they are only usable for semantic keyword matching, not as tier targets. + +If the list is long, still show all of it rather than a sample. If you cannot find any +models, stop and ask where the config lives rather than guessing. + +## 2. Ask what to name the router + +The name becomes the model name callers request. Ask the user for it and use exactly what +they give you. Do not propose a name, and do not fall back to one if they are undecided; +ask again instead. + +Raise one constraint only if the user says the router will be called from a client that +filters model names by vendor, such as Claude Code or Claude Desktop: those accept only +names that read as Anthropic models, meaning the name has to contain `claude`, `anthropic`, +or a family word such as `opus`, `sonnet`, or `haiku`, with no other vendor's name in it. +See https://docs.litellm.ai/docs/tutorials/claude_code_autorouter + +## 3. Ask which model serves each tier + +Show the four tiers with what each one catches, repeat the numbered model menu next to the +question so the user does not have to scroll for it, and ask them to assign one per tier. +Accept a number or a name. Do not pre-fill an assignment; if they ask you to recommend one, +say what you would pick and why, then still wait for their confirmation. + +- `SIMPLE`: greetings, one-line lookups, trivial questions +- `MEDIUM`: ordinary questions and short edits +- `COMPLEX`: multi-step or code-heavy work +- `REASONING`: anything asking to think step by step, prove, or plan at length + +Tell them a model may serve several tiers, and that a tier can hold a list of models the +router picks from at random rather than a single name. Also ask which model should serve as +the fallback when classification fails, which is what `complexity_router_default_model` +sets, offering the same menu. + +If a tier has no good candidate in what they serve, say so plainly and name the models they +would have to add rather than quietly reusing one that does not fit. + +## 4. Write the config + +Append one entry to `model_list`, changing nothing else in the file: + +```yaml + - model_name: + litellm_params: + model: auto_router/complexity_router + complexity_router_config: + tiers: + SIMPLE: + MEDIUM: + COMPLEX: + REASONING: + complexity_router_default_model: +``` + +For a database-managed proxy, create the same deployment through the management API +instead: + +```bash +curl -X POST $LITELLM_PROXY_URL/model/new \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \ + -d '{"model_name": "", "litellm_params": {"model": "auto_router/complexity_router", "complexity_router_config": {"tiers": {"SIMPLE": "", "MEDIUM": "", "COMPLEX": "", "REASONING": ""}}, "complexity_router_default_model": ""}}' +``` + +## 5. Lay out the defaults and ask what else to change + +Everything not written above runs on its default. Before you finish, show the user this +list, say these are the defaults now in force, and ask whether they want to change any of +them. Apply whatever they pick, and leave every value they keep out of the config so the +backend default applies. + +| Setting | Default in force | What changing it buys | +| --- | --- | --- | +| Classifier | Heuristic scorer, no API call, sub-millisecond, scores prompt length, code presence, reasoning markers, technical terms | `classifier_type: llm` reads the request more accurately, at one small model call per request | +| Session affinity | Off, so every turn is classified on its own merits | `session_affinity: true` pins a session to its first-turn model, preserving provider prompt caches and avoiding replay of one model's history to another, at the cost of the whole session inheriting the first turn's tier. `session_affinity_ttl_seconds` defaults to 3600 | +| Keyword rules | None | `keyword_tier_rules` forces chosen phrases into a tier before the classifier runs | +| Semantic matching | Off | `semantic_keyword_matching: true` plus an `embedding_model` picked from the embedding models you listed extends those rules to paraphrases | +| Adaptive routing | Off, so a tier with several models picks uniformly at random | `adaptive: true` Thompson-samples within the tier by observed quality | +| Technical keyword list | Built-in list only | `custom_technical_keywords` appends domain terms the built-in list misses | +| Scorer thresholds | Built-in boundaries and weights | `tier_boundaries`, `token_thresholds`, and `dimension_weights` move where the scorer draws each tier | + +If the user takes the LLM classifier, show the same model menu again, ask which one should +do the classifying, noting that a small fast model is what this wants since it runs on +every request, and add: + +```yaml + classifier_type: llm + classifier_llm_config: + model: + timeout_ms: 2000 +``` + +Then tell them what that turns on by default and ask whether to change it: the classifier +sees the last 3 prior human turns, truncated to 200 characters each +(`classifier_context_window_size: 3`, `classifier_context_per_turn_chars: 200`), so a +follow-up like "now do the same for the streaming path" is rated against what it refers to +instead of on its own length. Assistant replies are excluded +(`classifier_context_include_assistant_turns: false`); turning them on helps when the +assistant states the difficulty and the user only answers "yes". A window size of 0 +classifies the current message alone. + +## 6. Verify + +Restart the proxy, then send one trivial request and one hard one, and show the user both +responses and the model each landed on: + +```bash +curl -s $LITELLM_PROXY_URL/v1/chat/completions \ + -H "Authorization: Bearer $LITELLM_API_KEY" -H 'Content-Type: application/json' \ + -d '{"model":"","messages":[{"role":"user","content":"what is 2+2?"}]}' + +curl -s $LITELLM_PROXY_URL/v1/chat/completions \ + -H "Authorization: Bearer $LITELLM_API_KEY" -H 'Content-Type: application/json' \ + -d '{"model":"","messages":[{"role":"user","content":"Think step by step and design a multi-region active-active Postgres failover strategy."}]}' +``` + +The proxy logs carry one line per routing decision: + +``` +ComplexityRouter: routing decision cause=complexity_scorer, tier=SIMPLE, score=-0.150, signals=['short (7 tokens)', 'simple (what is)'], routed_model=gpt-4o-mini +``` + +The two requests should report different tiers. If they do not, show the user what you saw +rather than tuning thresholds on your own. + +## 7. Hand back + +Tell the user the router name to call, the tier assignments, the settings in force, and +where the remaining knobs live: +https://docs.litellm.ai/docs/proxy/auto_routing#full-config