chore(release): backport auth, CLI SSO and guardrail fixes to rc/1.94.0 and refresh flagged dependencies - #34640
Conversation
Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit 10d2a27)
temp_budget_increase was only applied on the DB-fetch path of _user_api_key_auth_builder, so a key served from the auth cache reverted to its original max_budget and was wrongly blocked with BudgetExceededError once spend crossed the original budget while staying under the effective budget. Move _update_key_budget_with_temp_budget_increase out of the DB-only branch so it runs for every resolved token regardless of source. The cache stores the original budget and each cache hit returns a fresh model_copy(), so this never double-applies. Fixes #25760 Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit 089de50)
#34121) * refactor(auth): derive temp budget bump without mutation, tz-aware auth datetimes _update_key_budget_with_temp_budget_increase mutated max_budget in place, so correctness depended on every resolution path handing it a fresh copy of the cached token; one future re-cache of a live token would compound the bump per request. Return a model_copy instead so no caller can leak an increased budget into shared state. Also fixes the three remaining DTZ005 naive datetime.now() calls in user_api_key_auth.py (auth span start, builder start_time, service-log end_time; all consumers convert to epoch or subtract same-pair datetimes) and ratchets the DTZ005 strict budget 244 -> 241. * test: pin non-mutation of the temp budget helper input Adversarial mutation-testing showed reverting the helper to in-place mutation still passed every test: the cache's copy-on-read layer masks the mutation in the integration test and the direct unit test only inspected the return value. Assert the input object is left untouched and the result is a distinct object so the purity guarantee itself is load-bearing. (cherry picked from commit 76c9eca)
…e_redis_auth_cache (#33261) * fix(proxy): share CLI SSO login sessions across workers without enable_redis_auth_cache * fix(proxy): make CLI SSO flow state redis-authoritative across workers The CLI SSO flow is stored in a DualCache whose get_cache is memory-first, so the worker that served /sso/cli/start keeps serving its stale in-memory flow and never observes the sso_complete/session_data update another worker writes during the OAuth callback. Attaching Redis alone is not enough; poll on the original worker returns pending forever. Read and write the flow directly through the attached Redis backend when present so every worker sees the same authoritative state, falling back to the in-memory DualCache only when no Redis is configured. * fix(proxy): serialize CLI SSO flow as JSON for the redis round trip RedisCache stores values via str(value) and parses reads with json.loads then ast.literal_eval. The completed flow contains a LitellmUserRoles enum in session_data.user_role, whose repr is not a parseable literal, so any worker reading the completed flow from redis raised SyntaxError and returned 400 "CLI login session not found". Writing the flow as json.dumps makes the round trip lossless (the enum is a str subclass) and fails loudly at write time if a non-serializable value is ever added to the flow. * fix(proxy): point CLI SSO session-not-found hint at configuring Redis The error message and warning still told users to set enable_redis_auth_cache, but the CLI SSO session cache now gets Redis unconditionally whenever one is configured, so that flag no longer affects CLI login --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> (cherry picked from commit 17a83aa)
…ss_response (#34390) (#34405) * fix(guardrails/model_armor): handle None metadata in post_call _process_response On batch routes data["metadata"] is normalized to None (present key, None value), so request_data.get("metadata", {}) returned None and _process_response raised 'NoneType' object has no attribute 'get', 500ing every /v1/batches create with a post_call Model Armor guardrail (regression from v1.93.0 activating the post_call hook). Coalesce a falsy metadata to {} Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Clean up test case documentation Remove regression comment from test_process_response_with_none_metadata_does_not_crash. --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit 7257d0f)
…thout org_id (#34577) * fix(proxy): attribute spend to org for team-linked keys minted without org_id Keys attached to an org-linked team but minted without an organization_id produced spend that was never credited to the org: the spend writer reads user_api_key_dict.org_id with no team fallback, while the org budget check resolves the org from the team. The check therefore ran against a counter fed by almost none of the org's traffic and never tripped. Backfill org_id from the freshly fetched team object in _run_centralized_common_checks, per request only, so the spend writer and the budget check read the same org. A key with an explicitly pinned org_id always wins, and the cached key row is never mutated, so moving a team to a different org takes effect on the next auth once the team cache refreshes. * test(proxy): cover CLI session-token org backfill from team CLI session tokens from /sso/cli/poll are minted with a real team_id but no org_id, and their auth path decrypts the blob without the combined_view team join that fills org for DB keys. Spend from these tokens reached the team but never the org, so org budgets never tripped. The regression test mints a real CLI token, runs it through the centralized checks, and asserts the credential leaves auth with the team's org. (cherry picked from commit 579f41d)
|
Too many files changed for review. ( |
|
|
|
Moved to draft while one verification finding gets a decision. Everything else in the deep-verification pass came back clean, so this is a single open question, not a broken branch. The finding#33840 changes how a naive (offset-less)
Reproduced directly against the shipped functions on both trees, with Naive input is a first-class case on the public contract, not a hypothetical: Why this is not a defect introduced by the backportThe picked commit is byte-identical to its source on The right sequencing is to fix it on staging first and backport that fix, rather than hand-writing a divergent correction onto the release line. Everything else came back clean
|
|
Out of draft and ready for review. The naive Executed on a staging checkout at Staging reproduces both directions identically to this branch, so the behavior originates upstream and this backport only carries it. Staging has nothing that normalizes the value earlier either: the only validator on That makes holding this PR the worse option. The exposure is identical whether it merges or not, since the GA inherits the same behavior from staging, while holding it keeps three fixes off the line that were demonstrated broken on the current tip against live traffic. The follow-up belongs on staging, and the more durable shape is probably to normalize at the write path so the stored value is unambiguous, leaving the auth-side |
3 participants
Relevant issues
Backports six merged
litellm_internal_stagingfixes ontorc/1.94.0and brings the line's dependency set up to the versions staging already resolves. The rc branch was cut on 2026-07-21, so it missed both the auth and CLI SSO fixes that landed that week and every dependency refresh staging has taken since. Everything here rides out as the next-rc.Nbuild;pyprojectstays at1.94.0per this line's convention, andlitellm-proxy-extrasstays at0.4.79.post1because no pick touches a schema or migrationLinear ticket
What is included
Picks, in the order they merged to staging:
fix(proxy/auth): handle tz-awaretemp_budget_expiry, so a temp budget increase written with a timezone no longer raises when compared againstdatetime.now()fix(auth): applytemp_budget_increasefor cache-hit keys, so a key served from the auth cache keeps its increased budget instead of reverting to the original and being blockedrefactor(auth): derive the temp budget increase without mutating the token. Pulled in as a required companion to fix(auth): apply temp_budget_increase for cache-hit keys #33841, see belowfix(proxy): share CLI SSO login sessions across workers withoutenable_redis_auth_cache, solite logincompletes on multi-replica deploymentsfix(guardrails/model_armor): handleNonemetadata in the post_call_process_response, which was 500ing every/v1/batchescreate that had a post_call Model Armor guardrailfix(proxy): attribute org spend for team-linked credentials minted without anorg_id, so org budgets see the traffic they are meant to governThen the dependency commits, one per package, each confined to the lockfile it owns:
chore(deps): bump gitpython to 3.1.54chore(deps): bump pyasn1 to 0.6.4chore(deps): bump pypdf to 6.14.2chore(deps): bump next, postcss, sharp, js-yaml and brace-expansion in the dashboardchore: update Next.js build artifacts, the rebuilt dashboard bundle, see belowWhy #34121 came along
#33841 moves
_update_key_budget_with_temp_budget_increaseout of the DB-only branch so it runs for every resolved token, including cache hits. Its correctness argument is that each cache hit hands the helper a freshmodel_copy(), so the increase never compounds. Onrc/1.94.0that helper still mutatedmax_budgetin place, which left the whole guarantee resting on every current and future resolution path remembering to copy first. #34121 makes the helper return amodel_copyinstead, so the increase cannot leak into shared cache state regardless of the caller. Taking #33841 without it would have backported the fragile version of the exact invariant #33841 depends on. It also carries three naivedatetime.now()fixes in the same file and the matchingruff-strict-budget.jsonDTZ005 ratchet from 244 to 241Why the dependency set moved further than one refresh
rc/1.94.0predates several dependency refreshes on staging, so scanning the line found 30 advisories across 8 packages where staging carries 2. Every target below is a version staging already resolves at or above, and each was chosen as the lowest release that clears the finding rather than as a catch-up to staging, except where notedpostcssandsharpare the two deliberate exceptions to picking the strict minimum: the lowest clearing releases are 8.5.18 and 0.35.0, and staging's 8.5.22 and^0.35.0were used instead so this line and staging resolve identically. This branch becomes the GA, so any delta here is a delta customers meet on their very next upgradeEach Python bump was regenerated with the uv version this line's Dockerfile pins (0.11.7) and gated on its moved set: every one moved exactly its own package and nothing else, with the lock
revisionfield unchanged. The dashboard bump moved 42 lockfile entries, all accounted for: the six targets,nanoid3.3.12 to 3.3.16 as a floor postcss requires, and 35 platform binaries that version in lockstep with next and sharpTwo findings deliberately remain, both blocked on staging by the same three-day dependency cooldown documented in #34634.
gitpythonneeds 3.1.55 andbrace-expansionneeds 5.0.8; both were published 2026-07-23 and clear the cooldown on 2026-07-26, and staging has not taken them either. This line must not move ahead of staging on a dependency, or the next upgrade would walk straight back into the finding, so they belong in the same follow-up bump staging takes.GHSA-w8v5-vhqr-4h9v(diskcache) also remains and is unchanged from staging; it is already suppressed in this line'sosv-scanner.tomlwith no fixed release publishedWhy the dashboard bundle was rebuilt
No pick touches a production
.tsx, so nothing here changes dashboard source. Butnext,postcssandsharpare the dashboard's build toolchain, and the wheel ships the committed bundle inlitellm/proxy/_experimental/out/. Leaving the bundle alone would have shipped one built by Next 16.2.6 while the lockfile claimed 16.2.11. It was rebuilt on this branch with the repo's ownbuild_release_ui.shon Node v20.20.2, and the artifacts commit is confined tolitellm/proxy/_experimental/out/with no other path in it. The rebuild doubles as proof that the Next bump builds cleanAdaptation notes
Four of the six picks are byte-identical to their staging commits. Two diverge, both only in context:
_complete_cli_sso_callback_sessionhas nosso_assertionparameter, which appears in the staging diff only as unchanged context, so it drops out of the resolution. Every line the pick itself adds is identicalOne adaptation goes beyond context, folded into #33261's own commit:
test_redis_auth_cache_flag.py— the_patched_init_cachehelper restores every proxy global it patches but left three it mutates through_init_cache:litellm.cache,proxy_server.redis_usage_cache, andlitellm_config_cache.redis_cache. The last one is the load-bearing case:_attach_redis_usage_cacheassigns a Redis client tolitellm_config_cache, so after this file ran, a_FakeRedisCachestayed attached to a module-level cache and later files in the same pytest session tripped over it. The helper now restores all three in afinally. No assertion was touched and the new CLI SSO test is unchanged; this only makes the fixture put back what it borrowedThat change is worth a reviewer's eye because it diverges from staging, where the same latent leak still exists. It is a strict improvement here: it removes six pre-existing failures from this line's own baseline, detailed below
Known noise on this line
Before any pick, the targeted suites on
rc/1.94.0reported 4 failures and 2 errors, all intests/test_litellm/proxy/test_proxy_server.pyand all traceable to the fixture leak described above rather than to product code. After this branch they are gone: the same suites now report 882 passed, 0 failed. The comparison was run on the line tip and on this branch with the same command and the same collection orderScreenshots / Proof of Fix
Live reproducer, before and after
The temp-budget picks were replayed against two live proxies hitting real Anthropic APIs: one serving this branch, one serving the untouched line tip at
13fef68b12. Same key, same request body, sametemp_budget_increaseof 0.01 with a timezone-aware expiry, applied through/key/updateafter the key had already exceeded its budget.Untouched line tip (
13fef68b12), port 4010:This branch, port 4001:
Attempt 1 on the tip is #33840 failing; attempts 2 and 3 are #33841 failing, since the key is served from the auth cache by then and the increase never reached the cache path. Both are fixed here, and the repeated calls confirm the increase does not compound.
#34405 was exercised the same way, in process, because a live replay needs a configured Model Armor guardrail this environment does not have. Driving the real
_process_responsewith the batch-route shape{"model": ..., "metadata": None}:Two picks have no live reproducer and should be read as covered by their own regression tests only. #33261 needs a multi-worker deployment with Redis and a real identity provider. For #34577, an attempted reproducer using an ordinary database key passed on both trees and therefore proves nothing: a database key already receives its org through the combined view team join, and the gap the pick closes affects CLI session tokens, which need a full SSO flow to mint.
Behavioral review
A deep adversarial review ran five independent investigators against this tree. Four supported the hypothesis outright. The fifth raised the naive
temp_budget_expirytimezone question discussed in the comment below; it reproduces, it is byte-identical tolitellm_internal_staging, it was confirmed by execution against a staging checkout, and it has already shipped inv1.95.0-dev.1andv1.95.0-dev.2. It is being tracked and fixed on staging separately rather than corrected divergently on a release line.Points the review confirmed by execution:
_update_key_budget_with_temp_budget_increasehas exactly one production caller and that caller rebinds the return, so themodel_copychange cannot drop or leak a budget; theorg_idbackfill runs after team resolution and cannot poison the auth cache, since cache hits are served through a copy on read; and every symbol the picked diffs reference resolves on this line, including the fullcli_sso_session_cachewiring.Test delta
Targeted suites, same command and collection order on both trees:
Full mirrored suite showed no repeatable new failure. Nine candidates surfaced by the parallel run were each re-run in isolation and passed on both this branch and the line tip, which is the known behaviour of this suite under
-n 6.Scanner
Using osv-scanner 2.3.8, the same version and invocation the repo's
osv-scanjob runs.At the branch point,
13fef68b12:Dependency resolution on this branch, read out of the built environment rather than the lockfile:
Dashboard build against the bumped toolchain:
Re-scanned on this branch after the bumps:
30 down to 2, matching what staging carries; the two survivors are the cooldown deferrals described above.
Type
🐛 Bug Fix
🚄 Infrastructure
Changes
Six cherry-picks carrying their
-xprovenance footers, four dependency commits, and one build-artifacts commit. No product code was hand-written on this branch; every source change traces to a commit reachable fromlitellm_internal_stagingPre-Submission checklist
@greptileaito re-request a review after pushing changes)Final Attestation