diff --git a/docs/proxy/auto_routing.md b/docs/proxy/auto_routing.md index d8b09bd29..99c82ff25 100644 --- a/docs/proxy/auto_routing.md +++ b/docs/proxy/auto_routing.md @@ -84,11 +84,17 @@ Every knob v2 exposes. All fields on `complexity_router_config` are optional exc COMPLEX: claude-sonnet-5 REASONING: gpt-5.5 + # Optional display names; omit to keep SIMPLE/MEDIUM/COMPLEX/REASONING everywhere + # tier_labels: + # SIMPLE: Cheap + # LLM classifier instead of the heuristic scorer classifier_type: llm classifier_llm_config: model: claude-haiku-4-5-20251001 timeout_ms: 2000 + # system_prompt: # replaces the built-in rubric entirely; omit for the default + classifier_fallback: heuristic # default; or default_model # Prior conversation the classifier sees (LLM classifier only) classifier_context_window_size: 3 # default 3; 0 disables classifier_context_per_turn_chars: 200 # default 200 @@ -107,6 +113,9 @@ Every knob v2 exposes. All fields on `complexity_router_config` are optional exc # Append to the built-in technical keyword list custom_technical_keywords: [kafka, redis, postgresql, udp, dns] + # Marker pair whose blocks are stripped before classification + reminder_markers: ["", ""] # default + # Thompson-sample within the tier's pool adaptive: true @@ -136,7 +145,7 @@ Every knob v2 exposes. All fields on `complexity_router_config` are optional exc ## Classification -Three ways to pick a tier. Pick one; the router falls back to the heuristic scorer if the LLM classifier errors or if no keyword rule matches. +Three ways to pick a tier. Pick one; the router falls back to the heuristic scorer if no keyword rule matches, and, unless `classifier_fallback` says otherwise, if the LLM classifier errors. **Heuristic scorer (default).** Zero API calls, sub-millisecond. Scores each request across seven dimensions and maps the score to a tier. @@ -152,7 +161,7 @@ Three ways to pick a tier. Pick one; the router falls back to the heuristic scor Two or more reasoning markers auto-routes to `REASONING` regardless of the weighted score. -**LLM classifier.** Uses a small fast model (Haiku, gpt-4o-mini, whatever you point it at) with structured output. Goes through the same `Router` instance, so credentials, budgets, and fallbacks apply. Timeout, empty content, or schema mismatch falls back to the heuristic scorer. +**LLM classifier.** Uses a small fast model (Haiku, gpt-4o-mini, whatever you point it at) with structured output. Goes through the same `Router` instance, so credentials, budgets, and fallbacks apply. Timeout, empty content, or schema mismatch falls back to the heuristic scorer, or to `complexity_router_default_model` with `classifier_fallback: default_model`, which is what a classifier grading something other than complexity wants since a complexity score would produce a tier unrelated to its taxonomy. ```yaml classifier_type: llm @@ -186,7 +195,7 @@ Context-window support ships in **v1.96.x** ([PR #35185](https://github.com/Berr The LLM classifier does not see the request in isolation. By default it also receives the last 3 prior turns of the conversation, truncated to 200 characters each, so a referring follow-up like "now do the same for the streaming path" is rated against what it refers to rather than on its own length. Without that context a hard follow-up mid-session classifies as whatever landed last, which in an agentic harness is often a `` blob that barely varies across the session and pins every turn to one tier. -Only turns carrying text a human wrote count toward the window. Tool output never qualifies (`tool_result` blocks on the Messages surface, the `tool` role on chat completions), complete `` blocks are stripped before a turn is considered, and a turn left empty after stripping is skipped rather than spending a slot. A turn whose text equals the ask being classified is excluded so the ask is never quoted twice. Prior turns are sent oldest first and numbered `[1]`, `[2]`, `[3]`, and a turn cut at the character limit gets a trailing `...` so the classifier can tell it was clipped. When prior conversation exists, a single depth line (`Conversation so far: ~N tokens across the request`) is included as well. +Only turns carrying text a human wrote count toward the window. Tool output never qualifies (`tool_result` blocks on the Messages surface, the `tool` role on chat completions), complete reminder blocks are stripped before a turn is considered (`` ... `` by default, another pair with `reminder_markers`), and a turn left empty after stripping is skipped rather than spending a slot. A turn whose text equals the ask being classified is excluded so the ask is never quoted twice. Prior turns are sent oldest first and numbered `[1]`, `[2]`, `[3]`, and a turn cut at the character limit gets a trailing `...` so the classifier can tell it was clipped. When prior conversation exists, a single depth line (`Conversation so far: ~N tokens across the request`) is included as well. The call is split so the system role carries only the operator's rubric, byte-identical across sessions and therefore prompt-cacheable, while everything caller-supplied (their system prompt, the prior turns, the ask) is quoted as labeled sections of the user turn. A three-turn conversation on the defaults produces: @@ -329,11 +338,13 @@ response = await router.acompletion( ## UI -Models + Endpoints > Add Model > Auto Router tab. Router Type defaults to "Auto-Router v2 [Recommended]". Configure the four tier model groups, optionally enable Semantic Keyword Matching, LLM Classifier, or Adaptive, then click **Test Connection**. Test Connection runs a minimal `/v1/chat/completions` or `/v1/embeddings` per distinct tier model group, so a green row means the tier is genuinely reachable and a red row shows the real provider error. +Models + Endpoints > Add Model > Auto Router tab. The form opens on the two things every router needs, a name and a **Template**: pick one of the bundled templates to prefill all four tiers, or **Custom Configuration** to fill them in yourself. A template whose models this proxy does not serve is greyed out with the missing names, so anything selectable is applicable. Everything else lives under **Detailed Configuration**, collapsed by default with a one-line summary of the tiers it currently holds; expand it to set the tier model groups, tier display names, Semantic Keyword Matching, LLM Classifier, escalation keywords, or Adaptive. + +**Test Routing** sends one prompt through the classifier for the config in the form, without creating the router, and shows the model it would pick with the same routing-decision card the logs drawer uses. Nothing is sent to the model it routes to, so a heuristic config spends nothing, while an LLM classifier or semantic matching bills its classifier or embedding call to your key. **Test Connection** instead runs a minimal `/v1/chat/completions` or `/v1/embeddings` per distinct tier model group, so a green row means the tier is genuinely reachable and a red row shows the real provider error. Tier and classifier dropdowns exclude embedding-mode models; the semantic embedding dropdown lists only embedding-mode models. All four tiers are required on submit; missing tiers are flagged inline. -Selecting **LLM Classifier** reveals the classifier context settings alongside the classifier model and timeout: **Context Window Size** (`classifier_context_window_size`), **Context Per-Turn Character Limit** (`classifier_context_per_turn_chars`), and an **Include Assistant Turns** toggle (`classifier_context_include_assistant_turns`). They are written only when the classifier type is LLM, and a value left at the default is omitted from the saved config so the backend default applies. +Selecting **LLM Classifier** reveals, alongside the classifier model and timeout, a **Classifier Prompt** editor (`classifier_llm_config.system_prompt`, prefilled with the built-in rubric for the router's context window size and tier names, and sent only once you edit it), an **If the classifier fails** choice between scoring with the heuristic and routing to the default model (`classifier_fallback`, the second option available only once the router has a default model), and the classifier context settings: **Context Window Size** (`classifier_context_window_size`), **Context Per-Turn Character Limit** (`classifier_context_per_turn_chars`), and an **Include Assistant Turns** toggle (`classifier_context_include_assistant_turns`). They are written only when the classifier type is LLM, and a value left at the default is omitted from the saved config so the backend default applies. **Advanced > Session Affinity** holds the session pin, off to match the config default. Both the create tab and the edit modal write the value explicitly, so a router built in the UI records what it does rather than inheriting whatever the default happens to be. diff --git a/static/skills/auto-router b/static/skills/auto-router index 85bdacf72..7cf96647a 100644 --- a/static/skills/auto-router +++ b/static/skills/auto-router @@ -100,12 +100,14 @@ backend default applies. | Setting | Default in force | What changing it buys | | --- | --- | --- | | Classifier | Heuristic scorer, no API call, sub-millisecond, scores prompt length, code presence, reasoning markers, technical terms | `classifier_type: llm` reads the request more accurately, at one small model call per request | +| Tier names | The canonical `SIMPLE`, `MEDIUM`, `COMPLEX`, `REASONING` | `tier_labels` puts the deployment's own vocabulary (for example Cheap/Standard/Premium/Deep) on the tiers in the dashboard, the spend logs, and the LLM classifier's rubric. Display only: `tiers`, `keyword_tier_rules[].tier`, and `tier_boundaries` stay written with the canonical names, and callers never see the labels. A partial map is fine; two tiers cannot share a label and a label cannot be another tier's canonical name | | Session affinity | Off, so every turn is classified on its own merits | `session_affinity: true` pins a session to its first-turn model, preserving provider prompt caches and avoiding replay of one model's history to another, at the cost of the whole session inheriting the first turn's tier. `session_affinity_ttl_seconds` defaults to 3600 | | Keyword rules | None | `keyword_tier_rules` forces chosen phrases into a tier before the classifier runs | | Semantic matching | Off | `semantic_keyword_matching: true` plus an `embedding_model` picked from the embedding models you listed extends those rules to paraphrases | | Adaptive routing | Off, so a tier with several models picks uniformly at random | `adaptive: true` Thompson-samples within the tier by observed quality | | Technical keyword list | Built-in list only | `custom_technical_keywords` appends domain terms the built-in list misses | | Scorer thresholds | Built-in boundaries and weights | `tier_boundaries`, `token_thresholds`, and `dimension_weights` move where the scorer draws each tier | +| Harness reminder blocks | Text wrapped in `` ... `` is stripped before classification, so an agent harness's boilerplate does not decide the tier | `reminder_markers: ["", ""]` names a different pair when the harness in front of the proxy wraps its injected reminders in something else | If the user takes the LLM classifier, show the same model menu again, ask which one should do the classifying, noting that a small fast model is what this wants since it runs on @@ -118,6 +120,32 @@ every request, and add: timeout_ms: 2000 ``` +Ask what should happen when that call errors, times out, or comes back unparseable. +`classifier_fallback` defaults to `heuristic`, which scores the request locally and is what a +classifier grading complexity wants. `classifier_fallback: default_model` skips scoring and routes +to `complexity_router_default_model` instead, which is what a classifier grading anything other +than complexity wants, since a complexity score would produce a tier unrelated to what they +configured. It requires a default model to be set. + +Mention that the rubric itself is replaceable, but only offer it, never do it unasked. Leaving +`classifier_llm_config.system_prompt` unset keeps the built-in complexity rubric. Setting it +replaces the classifier's entire system role, so the prompt owns the whole taxonomy and the four +tier names become whatever buckets it defines: a prompt grading data sensitivity routes on that +instead of on difficulty. If the user wants one, fetch the built-in prompt first so they are editing +the real text rather than a guess: + +```bash +curl -s "$LITELLM_PROXY_URL/auto_router/classifier/default_prompt?context_window_size=3" \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" +``` + +Tell them two things about replacing it. The default rubric's closing paragraph is the classifier's +prompt-injection defense, telling it that the caller's quoted system prompt and prior turns are +material to judge and never instructions; a replacement that drops that lets a caller ask for a tier +and get it. And the heuristic fallback still scores complexity, so a prompt on some other taxonomy +wants `classifier_fallback: default_model` alongside it. An empty string is rejected; omit the key +for the default. + Then tell them what that turns on by default and ask whether to change it: the classifier sees the last 3 prior human turns, truncated to 200 characters each (`classifier_context_window_size: 3`, `classifier_context_per_turn_chars: 200`), so a @@ -129,7 +157,20 @@ classifies the current message alone. ## 6. Verify -Restart the proxy, then send one trivial request and one hard one, and show the user both +If a proxy is already running, classify a prompt against the config before restarting anything. +`/auto_router/test_routing` reports where a prompt would land without creating the router, and the +prompt is only classified, never sent to the model it routes to: + +```bash +curl -s $LITELLM_PROXY_URL/auto_router/test_routing \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \ + -d '{"prompt": "what is 2+2?", "router_name": "", "default_model": "", "complexity_router_config": {"tiers": {"SIMPLE": "", "MEDIUM": "", "COMPLEX": "", "REASONING": ""}}}' +``` + +A heuristic config spends nothing this way; an LLM classifier or semantic matching bills its own +classifier or embedding call to the key you used. + +Then restart the proxy, and send one trivial request and one hard one, and show the user both responses and the model each landed on: ```bash