feat(proxy): serve the auto-router preset catalog at runtime - #39412
Conversation
Greptile SummaryThe PR moves the auto-router preset catalog behind a runtime proxy endpoint and updates the dashboard to query and hydrate that catalog.
Confidence Score: 3/5The PR should not merge until remote fetching uses LiteLLM's configured HTTP infrastructure and runtime catalog entries cannot crash the Add Auto Router dialog. The new fetch path bypasses shared outbound HTTP configuration, and the intentionally weak proxy schema passes nested data into dashboard code that unconditionally dereferences the expected tier structure. Files Needing Attention: litellm/proxy/public_endpoints/public_endpoints.py, litellm/types/proxy/public_endpoints/public_endpoints.py, ui/litellm-dashboard/src/lib/autorouter_presets.ts, ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.tsx
|
| Filename | Overview |
|---|---|
| litellm/proxy/public_endpoints/public_endpoints.py | Adds the runtime fetch, cache, fallback, and public endpoint, but bypasses LiteLLM's shared HTTP client handling. |
| litellm/types/proxy/public_endpoints/public_endpoints.py | Adds a forward-compatible response envelope whose nested config validation is too weak for the dashboard's unguarded consumers. |
| ui/litellm-dashboard/src/lib/autorouter_presets.ts | Replaces the static catalog with shallow runtime hydration while retaining accessors that assume a complete nested tier shape. |
| ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.tsx | Integrates asynchronous presets and retry messaging, but evaluates unvalidated remote configurations during render. |
| ui/litellm-dashboard/src/app/(dashboard)/hooks/autoRouter/useAutoRouterPresets.ts | Adds the React Query catalog hook with matching cache timing; its explanatory comment conflicts with repository policy. |
| tests/test_litellm/proxy/public_endpoints/test_public_endpoints.py | Covers source-tree fallback, caching, envelope rejection, and unknown-field forwarding but not malformed nested tier data. |
| ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.test.tsx | Adds useful fetch-failure and retry coverage without covering an envelope-valid malformed runtime preset. |
Reviews (1): Last reviewed commit: "feat(proxy): serve the auto-router prese..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| async with httpx.AsyncClient(timeout=5) as client: | ||
| response: Final = await client.get(url) |
There was a problem hiding this comment.
If outbound HTTPS depends on LiteLLM's configured certificate, proxy, TLS, redirect, or shared transport behavior, this bare httpx.AsyncClient bypasses that configuration. The remote request then falls back to the bundled catalog, leaving the dashboard with stale presets; concurrent cache misses also create redundant clients and upstream requests.
Rule Used: What: Block any PR which writes it's own custom ht... (source)
There was a problem hiding this comment.
Fixed in 5af5e21: the fetch now goes through get_async_httpx_client(httpxSpecialProvider.UI), and ensure_async_clients_test passes locally
| model_config = ConfigDict(extra="allow") | ||
|
|
||
| label: str | ||
| description: str |
There was a problem hiding this comment.
When a remote or overridden catalog contains an envelope-valid preset with missing or malformed tiers, this mapping accepts it and the dashboard later calls Object.values or models.map on the invalid values while rendering the dialog. One malformed entry therefore crashes the Add Auto Router form instead of being rejected so the bundled catalog can be served.
Knowledge Base Used: Dashboard and enterprise UI
There was a problem hiding this comment.
Fixed in 5af5e21: tiers is now validated per preset, so a malformed catalog rejects and the bundled one serves; pinned by adapter tests
| // running proxy (that is the point of serving it at runtime), so no release-length staleTime. | ||
| const options = { |
There was a problem hiding this comment.
This comment restates the cache duration already expressed by staleTime rather than explaining complex business logic or directing a tool. Similar explanatory comments were added throughout the preset-fetch path, increasing maintenance surface that can drift from the code.
Context Used: CLAUDE.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Fixed in 5af5e21: removed that comment and the other restating comments across the preset fetch path
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
a8fdfb2 to
5af5e21
Compare
|
CI is green except documentation and code-quality, which fail only on the env-keys doc gate pending BerriAI/litellm-docs#1143. @greptileai can you reply here with a score and why that score? |
|
bugbot run |
|
Confidence score: 4/5. The implementation is well-scoped and addresses the runtime-catalog flow end to end: the proxy exposes the public endpoint, performs a single-flight process-lifetime fetch with a bundled fallback, validates the required catalog shape while preserving unknown fields, and the dashboard hydrates the server response with an explicit retry/Custom Configuration fallback. The added tests cover local mode, caching, concurrent cold starts, remote failures, malformed payloads, unknown fields, and remote parsing. I’m holding at 4/5 because the documentation and code-quality checks are still red on the env-key documentation gate. Based on the PR context, that appears to be an external dependency on BerriAI/litellm-docs#1143 rather than a functional issue in this change; once that documentation change lands and the gates rerun successfully, I’d expect this to be 5/5. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
5af5e21 to
6e994ef
Compare
|
CI is green on 6e994ef except documentation and code-quality, both failing only on the env-keys doc gate pending BerriAI/litellm-docs#1143. @greptileai can you reply here with a score and why that score? |
|
bugbot run |
|
Confidence score: 4/5. The implementation is well-scoped and the core behavior is covered: runtime proxy serving, single-flight/process-lifetime caching, bundled fallback, schema validation, unknown-field preservation, local-only mode, and dashboard loading/error states with retry. CI is green for the code and tests. I’m holding back one point because the documentation and code-quality jobs remain blocked by the external env-key documentation gate pending BerriAI/litellm-docs#1143; once that dependency lands and those checks pass, this should be a 5/5. |
|
@greptileai can you reply here with a score and why that score? |
|
Score: 4/5 (high confidence). The implementation cleanly addresses the stated problem: the catalog is moved to a package-bundled source, exposed through the new public endpoint, resolved once per process with async single-flight behavior, and safely falls back to the bundled catalog when the remote is unavailable. The Pydantic models enforce the required tier shape while preserving unknown fields for forward compatibility. On the dashboard side, React Query integration, loading/error/retry states, and removal of the build-time catalog are consistent with the intended runtime flow. The added tests cover local mode, caching, concurrent cold starts, fallback behavior, malformed catalogs, unknown fields, and UI fetch states. I’m not assigning 5/5 because the review is based on the change and reported checks rather than independently running the complete proxy and dashboard suites, and the PR still has the separately noted documentation/CI dependency. I found no blocking correctness or security issues in the reviewed changes. |
The dashboard's template picker imported autorouter_presets.json at build time, so every catalog change needed a dashboard rebuild and artifacts refresh. The catalog now lives in litellm/proxy/public_endpoints/ and GET /public/autorouter_presets serves it, fetching litellm.autorouter_presets_url (GitHub raw on main, 1h in-process cache, bundled fallback) so a merged catalog change propagates to running proxies like the model cost map does. The dashboard fetches it at runtime via useAutoRouterPresets and keeps no local copy. Resolves LIT-6764
6e994ef to
a36254d
Compare
|
CI is green on a36254d except documentation and code-quality, both failing only on the env-keys doc gate pending BerriAI/litellm-docs#1143. @greptileai can you reply here with a score and why that score? |
|
bugbot run |
|
Score: 4/5 The implementation is well-scoped and addresses the requested flow end to end:
The score is not 5/5 only because the documentation/code-quality gates are still red pending the external |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit a36254d. Configure here.
|
Bugbot is clean on a36254d with no open threads. Greptile has not re-scored after three asks; every finding from its 3/5 is fixed and answered |
|
@greptile review |
|
QA'd E2E at a36254d against a live proxy (Postgres, store_model_in_db) plus the dashboard dev server, with LITELLM_AUTOROUTER_PRESETS_URL pointed at a local catalog carrying a relabeled preset and a brand new one. Unit tests pass (44 python, 139 vitest) Proxy: GET /public/autorouter_presets works without auth, 10 concurrent cold requests produce exactly one upstream fetch, the default GitHub URL 404s pre merge and falls back to the bundled catalog with one warning, a schema invalid remote also falls back, and LITELLM_LOCAL_AUTOROUTER_PRESETS=True skips the fetch entirely. Unknown fields (icon, future_knob) pass through verbatim UI: the Template dropdown renders the live catalog, a router created from the new preset serves 200s, and after editing the catalog and restarting the proxy the existing router keeps its stored tiers while the dialog shows the updated preset. With the proxy down the dropdown falls back to Custom Configuration with a working Retry Minor, not blocking: the dashboard needs a full page reload to pick up a new catalog (24h staleTime), the failure state takes about 15s to show while react-query retries, and after a failure the templates Retry and the models Retry are separate so presets stay disabled until both are clicked |
2 participants
TLDR
Problem this solves:
How it solves it:
litellm/proxy/public_endpoints/autorouter_presets.json, served byGET /public/autorouter_presetsUser Flow
Before: a proxy admin opening the Add Auto Router dialog sees whatever preset catalog was baked into their build, and a preset update published after that build never reaches them
After: the same dialog renders the current published catalog
GET /public/autorouter_presets, resolved from the published catalog on main when the proxy startedRelevant issues
Linear ticket
Resolves LIT-6764
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Rig: proxy from this tree on port 4620/4000, config with one model and a master key, no DB. The remote leg serves a modified copy of the catalog (relabeled
anthropic_family, plus abrand_new_presetcarrying aniconfield and afuture_knobconfig key this proxy version does not declare) fromhttp://127.0.0.1:4621/autorouter_presets.jsonBefore (2ade3e1)
The catalog is baked into the bundle and no runtime route serves it
curl -s "http://127.0.0.1:4620/ui/_next/static/chunks/1ffshjz5d4_3s.js" | grep -o anthropic_family | head -1returnsanthropic_family; the catalog ships inside the served JS chunkcurl -H "Authorization: Bearer sk-rig-1234" http://127.0.0.1:4620/public/autorouter_presetsreturns 404 (as do/get/autorouter_presetsand/autorouter/presets)A catalog change cannot reach a running deployment
After (a36254d)
The catalog is served at runtime
curl http://127.0.0.1:4000/public/autorouter_presetsreturns 200 withanthropic_family,gemini_family,lite,openai_familyA catalog change reaches a running deployment with no rebuild
LITELLM_AUTOROUTER_PRESETS_URL=http://127.0.0.1:4621/autorouter_presets.jsonpointing at the modified catalogcurl http://127.0.0.1:4000/public/autorouter_presetsnow returnsbrand_new_presetand the relabeledAnthropic Family (updated remotely), with the undeclarediconandfuture_knobfields passed through verbatim; the envelope and the four built-in tier lists validate (tiersaccepts exactly those four names), everything else keeps unknown fields (extra="allow"), so a future catalog survives an old proxy unless it carries presets this dashboard could not apply, which reject in favor of the bundled catalogREASONINGdeleted from the new preset's tiers and restart: the whole catalog rejects and the four bundled presets serve, so the dialog never receives a preset it cannot applynpm run dev, http://localhost:3000, Models + Endpoints, Auto-Routers, Add Auto Router) renders both in the Template dropdown: "Anthropic Family (updated remotely)" and "Brand New Preset, added on the remote after this proxy shipped". Screenshot belowUI steps to reproduce the screenshot
npm run devinui/litellm-dashboardLITELLM_AUTOROUTER_PRESETS_URLto it before starting the proxyType
🆕 New Feature
Caveats (if any)
Medium
/public/complexity_router/scorer_defaultsshipped)Low
Final Attestation
Note
Low Risk
Public read-only metadata endpoint with bundled fallback; main caveat is older proxies without the route leaving the UI on Custom Configuration only.
Overview
Moves the auto-router template catalog out of the dashboard bundle and serves it from the proxy at
GET /public/autorouter_presets, so preset updates can ship without rebuilding the UI.The proxy adds
litellm.autorouter_presets_url(andLITELLM_AUTOROUTER_PRESETS_URL/LITELLM_LOCAL_AUTOROUTER_PRESETS) and resolves the catalog once per process: optional remote fetch with single-flight caching, validation via newAutoRouterPresetRecordtypes (strict four-tier names, forward-compatible extra fields), and bundled JSON fallback on failure. OpenAPIschema.d.tsis updated for the new route.The dashboard drops the static
getAllPresets/getPresetByKeyhelpers in favor ofhydratePresets,useAutoRouterPresets(24h react-query cache), andgetAutoRouterPresetsnetworking. Add Auto Router shows loading, keeps cached templates on refetch errors, and degrades to Custom Configuration with retry when the catalog never loads.Reviewed by Cursor Bugbot for commit a36254d. Bugbot is set up for automated code reviews on this repo. Configure here.