Repository navigation
chore(backport): add the remaining jev changes to stable/1.100.x - #42668
devin-ai-integration[bot] wants to merge 19 commits into
Conversation
(cherry picked from commit 4bfac88)
…ions pass-through (#42301)
Merge debris cleanup and 1.102.x-only imports fixed; usesClassifierContext re-exported, JEV custom-tier emission, and preset test adapted to the static preset registry.
The type check gate flagged six new reportArgumentType errors and one reportGeneralTypeIssues error over base: an outcome Final rebinding in the fallback path, a Mapping handed to the dict-typed request_kwargs parameter, optional message sequences passed to a non-optional parameter, DatabaseClient where PrismaClient is expected on two access-group lookups, the effective-config helper object return passed to the Mapping-typed validator, the project row passed to can_project_access_model, and the Response or None from AsyncHTTPHandler.post. The flagged sites now pass the right shape or carry a rule-scoped pyright ignore with the reason.
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
|
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| # LiteLLM still supports 3.10, where they are distinct exception classes. | ||
| if isinstance(exc, (TimeoutError, asyncio.TimeoutError)): | ||
| return True | ||
| return type(exc).__name__.endswith("TimeoutError") |
There was a problem hiding this comment.
Jev timeouts skip the circuit breaker
High Severity
Jev classifier HTTP timeouts do not open the circuit breaker. HttpJevClassifierClient.evaluate uses the shared HTTP client's timeout, so a timed-out call raises httpx.TimeoutException (or ReadTimeout) through asyncio.wait_for instead of TimeoutError. _is_classifier_timeout only treats TimeoutError and names ending in TimeoutError as timeouts, so a closed breaker ignores the failure. Every later request then waits the full Jev timeout again, which is the outage the breaker is meant to stop.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit a392f7b. Configure here.
… to stable/1.100.x Bugbot on #42668 flagged that the backport's picks left the member auto-router management path unwired on this line. This ports the pieces that make it work, mirroring main: the member write slot in model_management_endpoints (FOR UPDATE lock, team reload with the model table include, identity and name-collision checks, post-commit config publish), StoredAutoRouterIdentity wiring, the license feature helpers, team tpd_limit, Router.config_deployments, the member_auto_router ModelInfo flag, the _TEAM_GRANT_RELATIONS include on team lookups, and the UserAPIKeyAuth fields the team_grants unpack needs. Test files were rebuilt as line content plus the picks' own additions, and ui_sso/test_team_grants carry the pick's grant assertions. Gate-clearing edits stay local to what the picks added: prisma TypedDict arguments replace mutable dict literals, remaining dict/mapping arguments carry reasoned mutable-ok comments, test-quality-ok comments mark the picks' internal-seam patches, and the regenerated dashboard api types are staged. The only remaining make check failure is pre-existing staging drift in untouched tests/test_litellm/test_router.py:2969.
…tpd_limit The merged TestTeamMemberAutoRouterWrites class dropped the parametrize decorator on test_admin_router_changes_release_member_scope, leaving the test with an unsatisfiable endpoint fixture; this restores main's patch/legacy x config/strategy/unrelated matrix. _full_team now sets tpd_limit so the every-team-field grant coverage assertion sees the new LiteLLM_VerificationTokenView field populated.
…stable/1.100.x Greptile flagged that the member create path committed the router row inside the write slot and then failed on the endpoint-level authorization in team_model_add, leaving the router orphaned. Port append_team_models from main into team_endpoints and call it directly like main does, and widen the grant/view plumbing and write payload calls so the LIT001 and LIT002 gates stay within their ceilings.
|
bugbot run |
Bugbot flagged that the member create path ran its name-collision check without any lock, so two concurrent creates of the same name could both pass. Take the same pg_advisory_xact_lock main takes at the top of the write transaction so member writes serialize before the checks run.
|
bugbot run |
|
|
||
|
|
||
| AUTO_ROUTER_WRITE_SLOT_LOCK_KEY: Final = 5_872_301 | ||
| _WRITE_SLOT_LOCK_SQL: Final = "SELECT 1 AS locked FROM pg_advisory_xact_lock($1)" |
There was a problem hiding this comment.
Invalid advisory lock SQL statement
High Severity
_WRITE_SLOT_LOCK_SQL calls pg_advisory_xact_lock in the FROM clause. That function returns void, so Postgres rejects it and every member auto-router write fails at lock acquisition. Other query_raw lock sites invoke the function in the select list instead.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit e39e1c8. Configure here.
There was a problem hiding this comment.
pg_advisory_xact_lock in FROM runs fine on Postgres (void functions yield one row); verified on the QA database. This statement is identical to main.
…e/1.100.x This reverts commits 54b42e3, 4864227, 77915d4 and e39e1c8. Jev does not use the member auto-router write path and no other stable line ships it, so dropping the port leaves 1.100.x matching stable/1.101.x and stable/1.102.x The classifier circuit breaker's litellm Timeout detection that 54b42e3 carried is kept, since main and the other lines have it This reverts commit e39e1c8. This reverts commit 77915d4. This reverts commit 4864227. This reverts commit 54b42e3.
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 55d1597. Configure here.
This branch has not been deployed
7 participants
TLDR
Problem this solves:
stable/1.100.xonly carries the TypeSafe Jev passthrough (feat(proxy): add TypeSafe AI Jev evaluate passthrough with registry-priced spend tracking #41607, merged via feat(typesafe): backport #41607 to stable/1.100.x for v1.100.2 #42597)How it solves it:
stable/1.100.x, one commit per PR, original author keptIncluded PRs:
Backport notes:
git cherry-pick -n -m 1from the main merge commit for cf42b60, 34718f0, 2edda5a, 1e161f5 and a83773c, plain pick for the #42301 squash 1106b16, each committed with the original author, subject and a two-line provenance body.4bfac88281(the stable dashboard adaptation from #42595) is picked with-xright after #41886Prerequisites, each its own commit with a provenance body naming the main sha:
7015bf37bb(apply team model aliases on the JWT auth path),109ca70f66(allow opted-in team members to manage their routers: theDatabaseClientprotocol,AUTO_ROUTER_MANAGE, theprisma_clientkwarg oncan_key_call_model/can_team_access_model, the lazyteam_membershipparams on_check_team_member_model_access, the member dry-run plumbing and matching test helpers) and510424c86c(classifier circuit breaker, which the Jev classifier plugs into)Conflicts resolved, always toward what main has after the pick:
litellm/types/utils.py,complexity_router.pyandconfig.pyin #41615 (this line lacks the capability, llm_v2 and fuse features, so the Jev hunks were applied on the line's code),StoredComplexityRouterConfigstill lives inedit_auto_router_modal.tsxhere so the jev fields went there,schema.d.tsand_lazy_openapi_snapshot.jsontook ours plus each pick's delta and were then regenerated wholesale (see below). Own follow-up commits, all Devin identity with a body:fix(ui): adapt the jev dashboard pieces to stable/1.100.x,fix: place lint suppressions on the flagged annotation lines,fix(typing): clear the basedpyright errors the picks introducedGenerated artifacts: #41615 and #41757 brought
litellm/proxy/_lazy_openapi_snapshot.jsonandui/litellm-dashboard/src/lib/http/schema.d.tsverbatim from main.chore(backport): regenerate the openapi snapshot and dashboard api types for stable/1.100.xregenerates both under Python 3.12 (uv sync --python 3.12 --inexact --frozen --extra proxy --group proxy-dev --group e2e-dev,scripts/prisma_generate_if_needed.py,python -m litellm.proxy._lazy_openapi_snapshot,npm run gen:api), so the snapshot dropsagent_365from theunreachable_fallbackdescription andschema.d.tsdrops the upstream-only classifier surface andJsonValueand gains/auto_router/manageReview-loop follow-ups after a392f7b, all Devin identity with a body:
54b42e3f05ports main's member auto-router write path and grant plumbing (the write slot inmodel_management_endpoints,StoredAutoRouterIdentitywiring, the license feature helpers, teamtpd_limit,Router.config_deployments, themember_auto_routerflag and the team grant relations), which Bugbot found unwired on this line after the picks;4864227716restores the member-scope test matrix and covers teamtpd_limit;77915d43b8portsappend_team_modelsso a member create no longer leaves an orphaned router row when the endpoint-level authorization fails (Greptile P1);e39e1c8deatakes main'spg_advisory_xact_lockbefore the member name-collision check (Bugbot race). The other three lines (1.99.x, 1.101.x, 1.102.x) do not carry this member write path, so55d1597645reverts all four of those commits in one revert commit. Jev never calls the member write path, and after the revert the branch is byte-identical to a392f7b except for one three-line change incomplexity_router.py:_is_classifier_timeoutalso matcheslitellm.exceptions.Timeout, which main and every other stable line already have, so a Jev classifier timeout trips the circuit breaker instead of counting as a generic failureChecks run on the tip a392f7b and repeated on e39e1c8; at 55d1597 the touched Python files were rerun (791 passed, 0 failed across
complexity_router,test_auth_checks.py,test_model_management_endpoints.py,test_team_endpoints.py), and the generated artifacts are unchanged from a392f7b, so the dashboard checks from that tip still apply. Python: everytests/test_litellmfile the picks touch was run in the worktree withLITELLM_LOCAL_MODEL_COST_MAP=True, 658 passed, 14 failed; the 14 are therequires_semantic_routertests intest_complexity_router.py, which fail on the line's tip before these picks too becausesemantic_routeris not installed. Dashboard, underui/litellm-dashboard:npx vitest runon every touched test file PASS,npx eslinton every changed file PASS with 0 errors and no newlocal/no-large-inline-object-argwarnings,npx tsc --noEmiton non-test sources PASS,npm run test:typesPASS,npm run buildPASS.make checkat the repo root: FAIL, on exactly one finding,TQ002: total 743 over limit 742 (this change added 1) - tests/test_litellm/test_router.py:2969. That file is byte-identical toorigin/stable/1.100.xand no commit on this branch touches it. The gate measures againstmerge-base(origin/litellm_internal_staging, HEAD), so it counts drift the stable line picked up since it forked. Running the same gate with the line as base,scripts/test_quality_gate.py --base origin/stable/1.100.x, printsOK: every TQ rule is within its test-suite ceiling. Every othermake checkgate passed. No budget JSON was editedUser Flow
Before: an admin on stable 1.100.x cannot use Jev anywhere except the raw passthrough
TYPESAFE_API_KEY, starts the proxy fromstable/1.100.xand opens http://localhost:4000/ui/?page=models to add an auto-routertypesafe_compactionguardrail toconfig.yamland the proxy refuses to start with an unknown guardrail error"classifier_type": "jev"and gets 422, jev is not an accepted classifierAfter: the same admin configures Jev routing, compaction and budgets from the UI and API
TYPESAFE_API_KEY, starts the proxy from this branch and opens http://localhost:4000/ui/?page=models to add an auto-routertypesafe_compactionguardrail and the proxy starts and trims low-relevance turns before forwarding"cause": "jev_classifier"and the chosen tiertypesafe/jev-1.13Relevant issues
Backport of #41615, #41723, #41757, #41879, #41886 and #42301. Follows #42597 on this line and mirrors #42595
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Two proxies from two worktrees, 2 uvicorn workers each, own Postgres each, real TypeSafe, OpenRouter and OpenAI calls, no mocks. Before is the merge base on port 18400, After is the PR tip on port 18401
Before (9c1216a)
POST /openrouter/api/alpha/decisions
POST /model/new with a jev auto-router
POST /auto_router/test_routing with a jev classifier
Over-budget member key on /auto_router/test_routing
Not reachable: the jev classifier config is rejected at validation before the budget check runs
After (55d1597)
POST /openrouter/api/alpha/decisions
POST /model/new with a jev auto-router
POST /auto_router/test_routing with a jev classifier
Over-budget member key on /auto_router/test_routing
Member key under a team with
team_member_permissions=["/auto_router/manage"],max_budget=0.000001, spend pushed over the cap by one real gpt-4o-mini completionAlso on the After side: GET /typesafe/v1/models returns the TypeSafe model list, and a
typesafe_compactionguardrail loaded from config shows up in GET /v2/guardrails/listBefore evidence rerun on the same rig: merge base
9c1216a427on port 18400 returns {"detail":"Not Found"} on the decisions POST, rejects the jev classifier_type on /model/new with the 'heuristic', 'llm', 'custom' or 'heuristic_first' literal error, and rejects classifier_type=jev on /auto_router/test_routing with the same literal errorType
🆕 New Feature
Caveats (if any)
Medium
make checkfails on this branch onTQ002intests/test_litellm/test_router.py, a file this branch does not touch; the same gate passes with--base origin/stable/1.100.xLow
fix: place lint suppressions on the flagged annotation lines(46f81ba) has no commit body; rewriting it would need a force pushrequires_semantic_routertests fail on the line's tip with or without this PRFinal Attestation
Note
Medium Risk
Adds request-mutating guardrails, new external classifier/spend paths, and expanded auto-router authorization; misconfiguration could block requests or allow unintended model spend on dry-run routes.
Overview
Backports the remaining TypeSafe Jev surface onto the stable line: complexity auto-routers can use
classifier_type: jev, with HTTP calls to TypeSafe/v1/systemone, spend logging, routing-decision probabilities/confidence, and a process-local classifier circuit breaker (also wired for LLM classifiers) so timeouts do not hammer every session.Adds a
typesafeguardrail that compacts chat history by asking Jev whether older tool exchanges are still relevant and blanking low-scoring results (configurable fail_open / fail_closed). OpenRouter gets a/openrouter/*passthrough, allowlist entries, and pricing foropenrouter/typesafe/jev-1.13; OpenRouter/alpha/decisionsreuses the TypeSafe passthrough spend logger.Auto-router management tightens who can validate or test routing: team members need premium plus
/auto_router/manage, dependency models are checked against team/key/org/project scope (including access groups viaprisma_client), JEV test routing respects virtual-key budgets, and tests can load a saved deployment bysaved_model_id. Supporting changes includeteam_grantsto keep JWT team fields aligned with virtual-key auth, richer TypeSafe passthrough HTTP methods, evaluation router deps excluded from health dependency sweeps, and Jev credential merge when patching stored complexity-router configs.Reviewed by Cursor Bugbot for commit 55d1597. Bugbot is set up for automated code reviews on this repo. Configure here.
Link to Devin session: https://app.devin.ai/sessions/dcedce8035d64a199713bae4b2b90b7f
Open in Devin Desktop: https://app.devin.ai/desktop/session/dcedce8035d64a199713bae4b2b90b7f?variant=devin
Requested by: @mateo-berri