feat(ui): simplify auto-router setup and clarify feature limits by moe-berri · Pull Request #42625 · BerriAI/litellm

moe-berri · 2026-09-22T23:49:16Z

TLDR

Problem this solves:

Setup hides classifier choices and available allowances
Choosing models incorrectly consumes Rule-based tuning allowance
Blocked features offer no next step

How it solves it:

Group setup into Heuristics, LLM, and Jev
Show descriptions, availability, and a contact link beside choices
Keep model selection separate from scoring customization
Preserve existing routers, Hybrid modes, and advanced controls
Keep allowances consistent during deletion and background refreshes

New routers default to Rule-based, and switching to Heuristics also selects Rule-based. Unlimited approaches appear first. Updated model templates apply to new selections, while saved routers retain their models and historical Fuse preset definitions

User Flow

Before: an admin must dig through setup, and ordinary model selection can trigger a tuning-license error

Open http://localhost:3020/models-and-endpoints/, choose Auto Routers, then Add Auto Router
See Complexity, Capability, and Fuse v2 tabs without allowance information
Expand Detailed Configuration to find classification controls and select tier models
After another Rule-based router uses the allowance, saving a second router can return 403 for changed scorer settings or tier models

After: the admin can see available choices immediately and save ordinary model selections

Open http://localhost:3019/models-and-endpoints/, choose Auto Routers, then Add Auto Router
See Heuristics, LLM, and Jev cards, explanatory dropdown options, and available counts
Select tier models directly; open Advanced settings for Hybrid, Heuristic first, and other tuning
Save multiple Rule-based routers with chosen models. Exhausted classifier choices are disabled, with a blue “Talk to our team” link directly below the count

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

I have added meaningful tests
The focused test files covering my change pass locally
My PR passes all required CI/CD checks
My PR's scope is as isolated as possible
Greptile confidence is at least 4/5 before maintainer review

Latest local validation: make check passes without budget changes. The focused create, edit, and classifier UI runs passed 190 tests. The backend run passed 852 tests covering model management, availability, proxy configuration, and lifecycle behavior. Scoped Vitest runs reported no type errors

Create and edit forms block Save and form submission while a draft is debouncing, loading, or cannot be checked. Exhausted dropdown choices remain disabled while a background refresh is pending or fails, and become selectable when a successful response reports a free slot

Availability reads a sanitized auto-router snapshot maintained by the existing database-model refresh. It includes saved routers that are blocked or not loaded and preserves ownership checks, without querying the model table or decrypting provider credentials on each availability request. Individual and team deletion now remove committed deletions from this snapshot under the existing reconciliation lock. Failed deletes retain their allowance, and surviving routers keep theirs. Existing repository notifications continue to update other workers

Regression coverage includes pending and failed refreshes, a newly freed slot, failed deletion, unloaded routers, team deletion, and a stale reconciliation completing during deletion. Restoring the previous dropdown condition makes all three new UI regressions fail. Restoring the previous delete paths makes six deletion and reconciliation regressions fail. The fixes pass the same tests

The prior UI CI failure was four stale uppercase JEV expectations after changing the displayed name to Jev. That job passed on 42ea34240f; current-tip CI and bot reviews are rerunning

The deletion bug was also reproduced against the live proxy at 42ea34240f: deletion returned 200, but the next availability response still showed zero slots. At e794f86c0f, the same sequence immediately reports one free slot and accepts a replacement router with 200. Earlier live UI checks covered existing Hybrid routers, owned allowances, custom-definition blocking, restore-defaults recovery, keyboard contact-link access, and a 390px viewport without horizontal overflow

Screenshots / Proof of Fix

The live proxies use a private local configuration and an isolated PostgreSQL database without an auto-router license. fast-model and capable-model are configured aliases. These requests exercise configuration saving, so no provider inference call is needed. The baseline proxy is port 4038 and the updated proxy is port 4039

The model-selection requests use this body with two distinct names, saved as router-a.json and router-b.json:

{
  "model_name": "setup-review-model-before-a",
  "litellm_params": {
    "model": "auto_router/complexity_router",
    "complexity_router_config": {
      "classifier_type": "heuristic",
      "tiers": {
        "SIMPLE": ["fast-model"],
        "MEDIUM": ["fast-model"],
        "COMPLEX": ["capable-model"],
        "REASONING": ["capable-model"]
      },
      "escalation_keywords": ["LITELLM ESCALATE"]
    }
  },
  "model_info": {}
}

To replay, save each body as router-a.json and router-b.json, and set LITELLM_MASTER_KEY for the local proxy. The HTTP requests below are equivalent to the recorded live requests

Before (`238f434`)

Setup and routing choices

Open http://localhost:3020/models-and-endpoints/, select Auto-Routers, then Add Auto Router
The original setup shows Complexity, Capability, and Fuse v2 tabs without allowance labels. Classification controls and tier models are inside Detailed Configuration

Model selection and allowance checks

Create the first standard Rule-based router with selected models:

curl -sS -w '\nHTTP %{http_code}\n' http://localhost:4038/model/new \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' --data-binary @router-a.json

Observed: 200, model ID d528fc24-bef2-45d9-9c08-52aded841f68

Send the same configuration with the second name:

curl -sS -w '\nHTTP %{http_code}\n' http://localhost:4038/model/new \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' --data-binary @router-b.json

Observed: 403, At most 1 auto-router(s) with changed heuristic scorer settings or tier models can be modified without an auto-router license

After (`e794f86`)

Setup and routing choices

Open http://localhost:3019/models-and-endpoints/, select Auto-Routers, then Add Auto Router
The updated setup shows Heuristics, LLM, and Jev, with Rule-based selected by default and marked Unlimited
Open the Heuristic dropdown. With the single unlicensed Heuristic v2 allowance already used by another saved router, Heuristic v2 is disabled and shows 0 of 1 available, with a blue Talk to our team link below it. Rule-based remains available and Unlimited
Select LLM and open Routing approach. Complexity appears first as Unlimited, followed by Capability and Fuse v2 with descriptions and remaining allowances

Model selection and allowance checks

Create the first standard Rule-based router with selected models, using review-followup-model-a:

curl -sS -w '\nHTTP %{http_code}\n' http://localhost:4039/model/new \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' --data-binary @router-a.json

Observed: 200, model ID 8ab33509-66ed-486d-a592-cd1b54dd9aa3

Send the same configuration with the second name, review-followup-model-b:

curl -sS -w '\nHTTP %{http_code}\n' http://localhost:4039/model/new \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' --data-binary @router-b.json

Observed: 200, model ID ab998099-8af4-4c1e-b6d6-295a84e53b6c

Create an additional Rule-based router with "code_keywords": ["LOCAL_DELETE_REGRESSION"], then immediately check its tuning allowance, delete it, and check again:

curl -sS http://localhost:4039/auto_router/availability \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' --data-binary @tuned-availability.json
curl -sS http://localhost:4039/model/delete \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"id":"88909f14-9123-4256-8193-f4748784d1b4"}'
curl -sS http://localhost:4039/auto_router/availability \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' --data-binary @tuned-availability.json

tuned-availability.json contains {"complexity_router_config": <the tuned router configuration>}. Observed: heuristic_tuning.remaining changes from 0 to 1, and error is null immediately after successful deletion

Create a replacement with the same tuned configuration. Observed: 200, model ID 45175dae-5eb1-47b5-8399-97daa59d22af. All temporary proof routers were removed afterward

Type

New Feature, Bug Fix

Caveats (if any)

Medium

Upgrade grandfathers existing scoring customizations once
Existing v2 hashes combine models and scoring rules
v3 records a fresh scoring-only baseline at startup
Later restarts reuse v3; saved routing configurations remain unchanged
Availability refreshes with model reloads; concurrent saves use atomic enforcement

Low

Full repository TypeScript check has unrelated pre-existing errors
Scoped Vitest runs reported no type errors
Inference quality was not benchmarked; runtime routing remains unchanged

Final Attestation

Tests cover the identified setup, gating, and compatibility regressions

Note

Medium Risk
Touches license enforcement, model deletion/reconcile ordering, and a baseline migration that changes which configs count as “tuned”; mis-synced catalog could briefly show wrong allowances until the next model refresh.

Overview
Adds POST /auto_router/availability so the dashboard can preview license-gated auto-router slots (Heuristic v2, Capability, Fuse v2, custom tiers/prompts, rule-based tuning) against a sanitized in-memory catalog built when models are loaded from the DB—without per-request model-table queries or leaking router names.

Deletion and refresh: ProxyConfig now keeps auto_router_db_catalog and strips deleted IDs under MODEL_RECONCILE_LOCK (single and team deletes), so freed allowances show up immediately and stale reconciles cannot resurrect removed routers in the catalog.

Tuning quota: Heuristic tuning fingerprints move to auto_router_tuning_baseline_v3—only scoring-rule fields count toward the allowance; tier/model picks no longer consume rule-based tuning. Startup migrates from v2 with a one-time baseline snapshot.

UI: Auto-router setup is reorganized into Heuristics / LLM / Jev with debounced availability checks, limit labels, save blocking on quota errors, sales contact links, and advanced settings grouped in collapsible sections. Preset JSON and Fuse catalog entries are updated for newer model IDs.

^{Reviewed by Cursor Bugbot for commit e794f86. Bugbot is set up for automated code reviews on this repo. Configure here.}

moe-berri · 2026-09-22T23:52:36Z

bugbot run

codspeed · 2026-09-22T23:55:31Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_autorouter_setup_flow (e794f86) with main (9bad2c3)¹}

No successful run was found on main (7688f56) during the generation of this report, so 9bad2c3 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

greptile-apps · 2026-09-22T23:57:13Z

Confidence Score: 5/5

The PR appears safe to merge, with the prior stale deletion allowance issue fixed and no new actionable failures identified

Findings

Deleted Routers Remain Counted ▶

Summary

This PR simplifies auto-router setup, exposes classifier and tuning allowances, separates model selection from scoring customization, and keeps the cached availability catalog consistent with successful model deletions

Adds an authenticated availability endpoint backed by the model-reconciliation catalog
Introduces scoring-only tuning baselines while preserving saved router configurations
Reorganizes dashboard choices into Heuristics, LLM, and Jev with allowance and recovery guidance
Removes successfully deleted routers from the availability catalog under the reconciliation lock
Keeps exhausted choices disabled through pending or failed background refreshes

_{Reviews (3) · Last reviewed commit: "fix: keep auto-router allowances consist..."}

cursor

Stale Bugbot comment from a previous run.

cursor · 2026-09-22T23:58:26Z

+  const changeFamily = (next: unknown) => {
+    if (next === family) return;
+    if (next === "heuristics") {
+      changeType(v2?.limit != null && v2.remaining === 0 && !v2.used_by_this_router ? "heuristic" : "heuristic_v2");


Heuristics defaults to exhausted v2

Low Severity

Switching to Heuristics treats a missing heuristic_v2 allowance as available and selects heuristic_v2. Pending or failed availability leaves v2 undefined, so the exhausted-slot check never runs and the family lands on a gated classifier the write path can still refuse.

^{Reviewed by Cursor Bugbot for commit 9e6c502. Configure here.}

veria-ai · 2026-09-22T23:59:38Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

codecov · 2026-09-23T00:00:03Z

Codecov Report

❌ Patch coverage is 98.91304% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...roxy/management_endpoints/auto_router_endpoints.py	94.44%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

moe-berri · 2026-09-23T00:18:49Z

bugbot run

moe-berri · 2026-09-23T00:19:01Z

@greptileai

cursor

Stale Bugbot comment from a previous run.

moe-berri · 2026-09-23T00:30:51Z

@greptileai

moe-berri · 2026-09-23T00:30:54Z

bugbot run

moe-berri · 2026-09-23T00:42:48Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e794f86. Configure here.}

…ability debounce #42625 keeps Save disabled while a 300ms-debounced availability check runs. This test waits for Save right after the change, so the whole debounce lands inside waitFor's 1s default and it times out under CI load. It is the recurring UI Unit Tests failure on main since #42625 landed

* test(ci): add used_client_oauth_token to the GCS pub/sub spend-log golden #43063 stamps used_client_oauth_token into spend-log metadata, so test_async_gcs_pub_sub_v1 failed on main with an extra metadata key * test(ui): give the auto-router threshold save wait room for the availability debounce #42625 keeps Save disabled while a 300ms-debounced availability check runs. This test waits for Save right after the change, so the whole debounce lands inside waitFor's 1s default and it times out under CI load. It is the recurring UI Unit Tests failure on main since #42625 landed * test(e2e): expect no pricing tier on bills for streamed calls OpenAI served at default #42870 added both the rule that a served default or standard tier bills at base pricing and records no service_tier, and streamed tests expecting the row to record 'default'. They have failed on every scheduled litellm-e2e run since. The tests now map the served tier to the pricing basis the bill must record and check input is billed at that basis's rate; the messages case registers custom rates so the rate check has something to compare against * test(e2e-ui): wait for the call-id search before hovering the logs row The row the spec hovers is already on the unfiltered first page, so it was found before the search request returned. The search response then re-rendered the table under the mouse, and the Base UI tooltip never opened. Reproduced with Playwright against a local proxy: hovering right after the fill never shows the tooltip, hovering after the search response shows the call id every time * test(e2e): run the Together structured-output case on the hybrid Qwen with reasoning off The case picked the cheapest Together row flagged supports_response_schema. DeepSeek-V4-Flash-0731 hit its cost-map deprecation date on 2026-09-29, so the pick moved to GLM-5.3-Flash, a reasoning-only model that spends the 1024-token budget thinking and returns content=None. Qwen3.5-9B is the pinned hybrid model the reasoning_effort=none case already exercises, and Together lists it with structured output support * test(integration): read the agent 365 guardrail status by its own name in spend logs The MCP shard runs under xdist against one database, and a sibling file creates a default_on pre_mcp_call content filter there. The owned proxy reloads DB guardrails, so that filter's 'success' entry could land first in guardrail_information and the test read it instead of the agent 365 verdict * test(unit): ignore asyncio's leaked-task records in the budget limiter push-failure log check gc.collect() inside the caplog window can collect a pending task an earlier test left on a closed loop, and asyncio logs 'Task was destroyed but it is pending' into this test's records. The check still counts every LiteLLM logger, and unretrieved task exceptions on this loop still go through the asserted exception handler * test(e2e-ui): fill the create-tag fields inside the dialog #42949 added 'Filter by tag name' and 'Filter by description' inputs to the Tag Management page, so page-wide getByLabel('Tag Name') and getByLabel('Description') match two elements and Playwright's strict mode fails the create step * test(integration): run integration proxies with the CI license Multi-worker proxies start each uvicorn worker in a fresh process, so every worker reads the license from its environment. Forward LITELLM_LICENSE into the proxy and test runner environments * ci: save GitHub Actions caches only from main and bump codecov-action to 5.5.5 Every pull request saved its own uv, maturin, Rust and Prisma caches, about 4.5 GB per PR, so the repository's 10 GB cache budget evicted main's entries within minutes. Pull request jobs then missed every cache, downloaded all dependencies from PyPI and hit the install step timeouts. Pull requests now restore only, and main keeps the caches warm for them. test-linting and check-ui-api-types run only on pull requests and keep saving codecov-action 5.5.4 imports its signing key from the deleted codecovsecurity keybase account, so every upload failed signature verification. 5.5.5 reads it from codecovsecops; the key ID matches the one signing the current CLI * test(unit): join the session-minting thread before collecting the handler asyncio.to_thread resumes the test as soon as the worker sets its result, while the pool thread can still hold the work item and through it the handler. gc.collect() then cannot finalize the handler and the session stays open. A pool that shuts down before the test continues drops that reference * test(integration): relaunch owned proxies that lose their port, expire idle gateway connections early owned_proxy_process released its reserved port and the proxy bound it only after full startup, so another xdist worker or an outgoing connection could take it first and the proxy exited with 'address already in use'. The launch now retries on a fresh port when that happens and stops every failed attempt. uvicorn closes idle keep-alive connections after 5 seconds and httpx expired them at the same 5 seconds, so a request sent right at that mark could reuse a socket the server was closing and get 'Connection reset by peer'. Gateway clients now drop idle connections after 2 seconds * ci(circleci): give the base SDK wheel build the same 30 minute no-output window as the Windows build The release profile builds with fat LTO and one codegen unit, so the final link of litellm-cache-s3 runs silently for minutes. Successful builds take 711 to 749 seconds, right at the default 10 minute no-output limit, and about 30% of recent runs were killed there * test(integration): model the budget-reset database outage as 10 seconds instead of 5 refused connections The proxy retries the database about every 30 seconds and each retry opens roughly one connection, so a 5-connection outage took 3 to 4 retries to clear and recovery landed between 60 and 90 seconds, straddling the test's 80 second reset window. A fixed 10 second outage still refuses the immediate reconnect and recovers on the next retry * ci: move the unit-test uv cache split into a composite action check_workflow_startup_safety sums every setup step's timeout, so the save and restore variants each counted 5 minutes although only one runs. One composite step keeps the setup ceiling at 35 minutes * test(unit): point tiktoken at the bundled cache for every unit test The rust_bridge tokenizer tests loaded o200k_base before any test in their xdist worker had imported default_encoding, so tiktoken fell back to the temp cache and tried to download under pytest-socket. Move the session fixture from litellm_core_utils/conftest.py to the root unit conftest. * test(integration): answer model discovery probes in the hosted_vllm wire tests The router's periodic upstream model info refresh sends GET /v1/models to hosted_vllm deployments, so a wire server that is live during a refresh sees an extra request. Answer the probe with an empty model list and leave it out of the provider-call assertions, matching the responses bridge tests.

feat(ui): simplify auto-router setup and clarify feature limits

9e6c502

moe-berri had a problem deploying to e2e-changed September 22, 2026 23:49 — with GitHub Actions Error

moe-berri marked this pull request as ready for review September 22, 2026 23:50

moe-berri requested a review from a team September 22, 2026 23:50

moe-berri requested review from ryan-crabbe-berri and yuneng-berri as code owners September 22, 2026 23:50

greptile-apps Bot reviewed Sep 22, 2026

View reviewed changes

Comment thread ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.tsx Outdated

Comment thread litellm/proxy/management_endpoints/auto_router_endpoints.py Outdated

cursor Bot reviewed Sep 22, 2026

View reviewed changes

veria-ai Bot reviewed Sep 22, 2026

View reviewed changes

Comment thread litellm/proxy/management_endpoints/auto_router_endpoints.py Outdated

fix(ui): validate auto-router drafts before saving

42ea342

moe-berri had a problem deploying to e2e-changed September 23, 2026 00:15 — with GitHub Actions Error

greptile-apps Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread litellm/proxy/proxy_server.py

cursor Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread ui/litellm-dashboard/src/components/add_model/AutoRouterAvailability.tsx

fix: keep auto-router allowances consistent after deletes and refreshes

e794f86

moe-berri requested a deployment to e2e-changed September 23, 2026 00:30 — with GitHub Actions Waiting

This comment was marked as resolved.

Sign in to view

cursor Bot reviewed Sep 23, 2026

View reviewed changes

yuneng-berri approved these changes Sep 23, 2026

View reviewed changes

moe-berri merged commit 2157351 into main Sep 23, 2026
100 of 102 checks passed

moe-berri deleted the litellm_autorouter_setup_flow branch September 23, 2026 01:03

Conversation

moe-berri commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

Screenshots / Proof of Fix

Before (238f434)

Setup and routing choices

Model selection and allowance checks

After (e794f86)

Setup and routing choices

Model selection and allowance checks

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

moe-berri commented Sep 22, 2026

Uh oh!

codspeed Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

greptile-apps Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Findings

Summary

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Sep 22, 2026

Choose a reason for hiding this comment

Heuristics defaults to exhausted v2

Uh oh!

Uh oh!

veria-ai Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Security review

Uh oh!

codecov Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

moe-berri commented Sep 23, 2026

Uh oh!

moe-berri commented Sep 23, 2026

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

moe-berri commented Sep 23, 2026

Uh oh!

moe-berri commented Sep 23, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

moe-berri commented Sep 23, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch is waiting to be deployed

Reviewers

Assignees

moe-berri commented Sep 22, 2026 •

edited

Loading

Before (`238f434`)

After (`e794f86`)

codspeed Bot commented Sep 22, 2026 •

edited

Loading

greptile-apps Bot commented Sep 22, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

veria-ai Bot commented Sep 22, 2026 •

edited

Loading

codecov Bot commented Sep 23, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading