Repository navigation
Conversation
Greptile SummaryThis PR adds a Switchyard-style capability forecaster to the complexity router, including strict verdict parsing, threshold-based two-tier selection, fail-closed routing, dependency discovery, logging metadata, generated API types, documentation, and tests.
Confidence Score: 3/5The PR is not yet safe to merge because dashboard edits can invalidate capability routers and valid integer-form thresholds are rejected. The backend routing implementation is coherent and well tested, but the dashboard omits capability from its classifier gate and consequently drops the required classifier model during edits; configuration validation also rejects valid numeric boundary values represented as integers. Files Needing Attention: ui/litellm-dashboard/src/lib/http/schema.d.ts, ui/litellm-dashboard/src/components/add_model/ComplexityRouterConfig.tsx, ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts, litellm/router_strategy/complexity_router/config.py
|
| Filename | Overview |
|---|---|
| litellm/router_strategy/complexity_router/capability_classifier.py | Adds the packaged prompt, strict verdict schema, rule-boundary validation, and deterministic threshold policy. |
| litellm/router_strategy/complexity_router/complexity_router.py | Integrates capability classification, fail-closed behavior, shared classifier calls, and forecast logging into routing. |
| litellm/router_strategy/complexity_router/config.py | Adds capability configuration and coherence validation, but strict float fields reject valid integer-form thresholds. |
| litellm/router_utils/auto_router_model_naming.py | Recognizes capability classifiers as dependencies through the shared LLM classifier type set. |
| litellm/types/utils.py | Adds capability routing causes and forecast fields with appropriate prompt-redaction classification. |
| ui/litellm-dashboard/src/lib/http/schema.d.ts | Exposes capability configuration in generated API types without corresponding support in the dashboard's handwritten classifier type and serialization gate. |
| tests/test_litellm/router_strategy/test_complexity_router.py | Covers thresholds, prompt construction, invalid verdicts, failure fallback, metadata, and redaction, but not integer-form threshold inputs or dashboard round trips. |
Reviews (1): Last reviewed commit: "feat(router): add Switchyard capability ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| * @enum {string} | ||
| */ | ||
| classifier_type: "heuristic" | "heuristic_v2" | "llm" | "custom" | "heuristic_first" | "hybrid"; | ||
| classifier_type: "heuristic" | "heuristic_v2" | "llm" | "capability" | "custom" | "heuristic_first" | "hybrid"; |
There was a problem hiding this comment.
The generated API type now includes classifier_type: "capability", but the dashboard's hand-written ClassifierType and usesLlmClassifier check do not. When an existing capability router is edited, the dashboard treats it as a non-LLM classifier and omits its required classifier_llm_config from the saved payload. The backend then rejects the edited configuration. Update the dashboard type and gate, and add an edit round-trip test for capability routers.
| base_threshold: StrictFloat = Field( | ||
| ge=0.0, | ||
| le=1.0, | ||
| description="Lowest p_solve that routes a supported task to efficient_tier", | ||
| ) | ||
| threshold_step: StrictFloat = Field( |
There was a problem hiding this comment.
base_threshold and threshold_step use StrictFloat, so valid JSON or YAML values such as base_threshold: 1 or threshold_step: 0 are rejected even though they satisfy the documented ranges and appear as ordinary number fields in the public schema. This prevents the router from starting with otherwise valid numeric configuration. Accept integer numeric inputs while continuing to reject booleans and nonnumeric strings, and cover integer-form values in the tests.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
tin-berri
changed the title
tin-berri
changed the title
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| except Exception as e: # noqa: BLE001 -- every unavailable or invalid judge verdict must fail closed | ||
| if breaker is not None and permit is not None: | ||
| breaker.record_failure(permit, is_timeout=_is_classifier_timeout(e)) | ||
| return self._capability_classifier_failure_outcome(f"capability classifier failed ({e})") |
There was a problem hiding this comment.
Low: Classifier failures bypass prompt-log redaction
An authenticated caller can induce an invalid rule/boundary verdict whose Pydantic exception includes the rejected crux, causing prompt-derived text to be written to the warning log regardless of turn_off_message_logging. Log only the exception type or another fixed diagnostic rather than the exception contents.
| return self._capability_classifier_failure_outcome(f"capability classifier failed ({e})") | |
| return self._capability_classifier_failure_outcome( | |
| f"capability classifier failed ({type(e).__name__})" | |
| ) |
PR overviewThis pull request adds calibrated capability forecasting to the complexity router, including classifier verdict handling and fallback outcomes. One security issue remains open, and none have yet been addressed. A classifier validation failure can place prompt-derived text into warning logs even when message logging is disabled, creating a limited confidentiality risk that depends on access to those logs. Open issues (1)
Fixed/addressed: 0 · PR risk: 4/10 |
|
Closing: this branch has no commits beyond main, so GitHub refuses to retarget it. litellm_internal_staging is being deleted. |
2 participants
TLDR
Problem this solves:
How it solves it:
User Flow
Before: a developer cannot benchmark capability-based routing through the gateway
autorouter-calibratedwith efficient and capable solver tiersPOST http://localhost:4000/auto_router/test_routingcapabilityis not a supported classifier typeAfter: the developer can test a calibrated forecast and run the chosen solver
autorouter-calibratedwith efficient and capable solver tiersPOST http://localhost:4000/auto_router/test_routingPOST http://localhost:4000/v1/chat/completions,/v1/responses, or/v1/messagesusing that aliasRelevant issues
Linear ticket
Pre-Submission checklist
make checkpasses at the current tip, including Python lint and type budgets, test quality, dashboard lint, and API schema checks. An additional 96 focused routing checks pass after the latest staging merge. No budget files or thresholds were changedScreenshots / Proof of Fix
This is real local Qwen inference through the gateway, with no mocked classifier or solver responses. Both solver aliases use the same local model for transport verification. This does not measure fresh Haiku/Opus success or paid-provider behavior. The earlier Anthropic account check reported no credits
Use
mlx-community/Qwen3-4B-Instruct-2507-4bitsnapshot50d427756c6b1b2fe0c0a10f67fbda1fc8e82c1b. SetJUDGE_SNAPSHOTto its local directory and run:Save this as
smoke.yaml. The before checkout loads only the first three entries because it cannot configure the capability router. The after checkout loads all four. Run each checkout's proxy on port 4000 in turn, withLITELLM_MASTER_KEY=sk-local-benchmarkand--config smoke.yamlSave this exact recorded SWE-bench task and configuration as
preview.json:{ "prompt": "Execution setup: the efficient solver is Claude Haiku 4.5 (claude-haiku-4-5-20251001), with the published high-reasoning setting, running mini-SWE-agent v2.0.0, a bash-only coding agent. It can inspect and edit the checked-out repository and execute shell commands. SUCCESS means resolving the issue under the SWE-bench verifier in one fresh attempt. No solution, verifier outcome, or completed trajectory is provided. The exact execution budget is not supplied; do not invent it.\n\nRepository: matplotlib/matplotlib\n\nIssue:\nhist() no longer respects range=... when density=True\n<!--To help us understand and resolve your issue, please fill out the form to the best of your ability.-->\r\n<!--You can feel free to delete the sections that do not apply.-->\r\n\r\n### Bug report\r\n\r\n**Bug summary**\r\n\r\n<!--A short 1-2 sentences that succinctly describes the bug-->\r\n\r\n**Code for reproduction**\r\n\r\n<!--A minimum code snippet required to reproduce the bug.\r\nPlease make sure to minimize the number of dependencies required, and provide\r\nany necessary plotted data.\r\nAvoid using threads, as Matplotlib is (explicitly) not thread-safe.-->\r\n\r\n```python\r\n_, bins, _ = plt.hist(np.random.rand(10), \"auto\", range=(0, 1), density=True)\r\nprint(bins)\r\n```\r\n\r\n**Actual outcome**\r\n\r\n<!--The output produced by the above code, which may be a screenshot, console output, etc.-->\r\n\r\n```\r\n[0.00331535 0.18930174 0.37528813 0.56127453 0.74726092 0.93324731]\r\n```\r\n\r\n**Expected outcome**\r\n\r\nSome array where the first value is 0 and the last one is 1.\r\n\r\nNote that this bug doesn't happen if density=False.\r\n\r\nBisects to https://github.com/matplotlib/matplotlib/pull/8638/commits/239be7b18e311c57a1393b6eeefc62b7cc629339 (#8638).\r\n\r\n**Matplotlib version**\r\n<!--Please specify your platform and versions of the relevant libraries you are using:-->\r\n * Operating system: linux\r\n * Matplotlib version: master\r\n * Matplotlib backend (`print(matplotlib.get_backend())`): any\r\n * Python version: 37\r\n * Jupyter version (if applicable): no\r\n * Other libraries: numpy 1.16.2\r\n\r\n<!--Please tell us how you installed matplotlib and python e.g., from source, pip, conda-->\r\n<!--If you installed from conda, please specify which channel you used if not the default-->\r\n\r\n\n", "router_name": "autorouter-calibrated", "complexity_router_config": { "classifier_type": "capability", "classifier_llm_config": { "model": "benchmark-judge", "timeout_ms": 30000, "circuit_breaker_enabled": false }, "tiers": { "SIMPLE": [ "efficient-solver" ], "REASONING": [ "capable-solver" ] }, "adaptive": false, "session_affinity": true, "session_affinity_ttl_seconds": 86400, "plan_mode_min_tier": null, "route_housekeeping_to_cheapest_tier": false, "stall_escalation_enabled": false, "enable_context_window_escalation": false, "modality_routing": false, "escalation_keywords": [], "keyword_tier_rules": [], "return_raw_model_name": true, "capability_classifier_config": { "efficient_tier": "SIMPLE", "capable_tier": "REASONING", "base_threshold": 0.66, "threshold_step": 0.1, "max_output_tokens": 512, "response_format": "json_object", "calibration": { "version": "qwen3-haiku45-mini-swe-v1", "slope": 0.1482462649948327, "intercept": 0.1895438369492216 } } } }Build serving requests from the same task:
The Responses bridge flag is needed because MLX only serves Chat Completions. The schema-appended prompt matches experiment hash
25e2604dcb4fa05350e42c925caa95d2b52a7056b7de0d0d693cd7442abe8cacBefore (a426dc4)
Capability forecast preview
curl -sS -w '\nHTTP %{http_code}\n' http://localhost:4000/auto_router/test_routing -H 'Authorization: Bearer sk-local-benchmark' -H 'Content-Type: application/json' --data-binary @preview.jsoncapabilityChat Completions
curl -sS -w '\nHTTP %{http_code}\n' http://localhost:4000/v1/chat/completions -H 'Authorization: Bearer sk-local-benchmark' -H 'Content-Type: application/json' --data-binary @chat-request.jsonResponses
curl -sS -w '\nHTTP %{http_code}\n' http://localhost:4000/v1/responses -H 'Authorization: Bearer sk-local-benchmark' -H 'Content-Type: application/json' --data-binary @responses-request.jsonAnthropic Messages
curl -sS -w '\nHTTP %{http_code}\n' http://localhost:4000/v1/messages -H 'x-api-key: sk-local-benchmark' -H 'anthropic-version: 2023-06-01' -H 'Content-Type: application/json' --data-binary @messages-request.jsonAfter (1450ffe)
Capability forecast preview
curl -sS -w '\nHTTP %{http_code}\n' http://localhost:4000/auto_router/test_routing -H 'Authorization: Bearer sk-local-benchmark' -H 'Content-Type: application/json' --data-binary @preview.jsonrouted_model: capable-solver, rawp_solve: 0.95, calibratedp_solve: 0.651592346378068, threshold0.66, and calibration versionqwen3-haiku45-mini-swe-v1Chat Completions
curl -sS -w '\nHTTP %{http_code}\n' http://localhost:4000/v1/chat/completions -H 'Authorization: Bearer sk-local-benchmark' -H 'Content-Type: application/json' --data-binary @chat-request.jsonchat.completionresponse, generated text, and token usageResponses
curl -sS -w '\nHTTP %{http_code}\n' http://localhost:4000/v1/responses -H 'Authorization: Bearer sk-local-benchmark' -H 'Content-Type: application/json' --data-binary @responses-request.jsonresponseresponse, generated text, and token usageAnthropic Messages
curl -sS -w '\nHTTP %{http_code}\n' http://localhost:4000/v1/messages -H 'x-api-key: sk-local-benchmark' -H 'anthropic-version: 2023-06-01' -H 'Content-Type: application/json' --data-binary @messages-request.jsonmessageresponse, generated text, and token usageWith calibration omitted and the original 0.5 threshold, the same live forecast routes to
efficient-solver. A matched raw threshold of 0.9606720864 routes tocapable-solver, just like the calibrated armType
New Feature
Caveats (if any)
Medium
Low
Final Attestation