Litellm token verification query optimization

[harish-berri]

Relevant issues

Problem

In Kubernetes or any multi-replica deployment, per-process memory cache is not shared. Requests can hit different pods, so in-memory-only user_api_key_cache does not deduplicate work across nodes and can yield inconsistent views of the same key.

Using Redis with DualCache shares cached key metadata across replicas and reduces redundant database lookups—if serialized values are JSON-safe on write and reads rehydrate to the types the proxy expects (e.g. UserAPIKeyAuth).

This can be amplified if each pod has multiple workers and since each worker is a subprocess , they dont share memory and the number of db calls in this path is num_pods X num_workers_in_each_pod X cache_miss_ratio.

Without a single encode/decode rule, Redis often returns a dict after JSON round-trip while memory may hold BaseModel instances. Code that assumes attributes (e.g. .spend) then fails with AttributeError, and spend/cache updates become fragile.

Approach

CacheCodec (litellm/proxy/common_utils/cache_pydantic_utils.py) centralizes the boundary:

• serialize — validate + model_dump(mode="json", exclude_none=True) when writing (aligned with Redis / json.dumps).
• deserialize — model_validate on dict hits; ValidationError → treat as miss (None) + warning log (no bad data served).

Scope note: CacheCodec is for Pydantic BaseModel + dict only; stdlib dataclasses are not supported—convert at the call site (e.g. dataclasses.asdict) or use a Pydantic model.

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

Code changes (summary)

Area	Change
cache_pydantic_utils.py	CacheCodec.serialize / deserialize, validation-on-read behavior, dataclass disclaimer in docstrings
proxy_server.py	_update_key_cache: deserialize before attribute access; serialize(..., UserAPIKeyAuth) for pipeline writes
auth_checks.py (and related)	Use CacheCodec when reading/writing cached key/team/project objects
Caching / auth guards	Avoid None Redis keys and noisy errors where payloads are missing (e.g. team id / team object)
Tests	tests/test_litellm/proxy/common_utils/test_cache_codec.py

Testing

Status	Item
Done	Unit tests for CacheCodec in tests/test_litellm/
DONE	End-to-end: Proxy with Redis-backed user_api_key_cache, multiple replicas (or workers), verify auth + spend paths, no dict vs model errors in logs
*DONE	One efficacy check: Prove Redis round-trip—write JSON-safe payload, read from another process / cold memory, assert behavior matches single-node baseline (e.g. key spend update or get_key_object)

Load Test (4 Pods)

Without Redis — combined view only (SELECT v.* … FROM "LiteLLM_VerificationToken" AS v …)

user_api_key_cache in-memory TTL: 60 seconds (UserAPIKeyCacheTTLEnum; no general_settings.user_api_key_cache_ttl override unless you set one).

Window	calls	total_ms	avg_ms	rows
5 min	1249	269.35	0.22	1249
10 min	1858	394.99	0.21	1858

With Redis — combined view only (same query)

With Redis enabled, the same default 60s TTL is applied to both in-memory and Redis when user_api_key_cache_ttl is not set (proxy_server mirrors TTL on both layers).

Window	calls	total_ms	avg_ms	rows
5 min	52	35.36	0.68	51
10 min	93	35.36	0.68	51

Locust Results

• 15 different virtual keys
• 200 concurrent requests, with a 10 second ramp up
• The left is without the redis cache and the right is with the cache
• 8 instances / pods of litellm run

[greptile-apps]

Greptile Summary

This PR introduces CacheCodec and UserApiKeyCache to enforce a consistent JSON-safe serialization boundary between in-memory and Redis cache layers, addressing the dict-vs-model format drift that caused AttributeError in multi-replica deployments. It also gates Redis attachment for user_api_key_cache behind a new litellm_settings.enable_redis_auth_cache flag and adds a Prometheus counter (litellm_auth_combined_view_queries_total) for measuring DB load reduction.

The core abstractions are sound. A few cache call sites were not fully migrated to the new model_type pattern (get_team_member_default_budget and the _update_user_cache/_update_end_user_cache helpers), leaving dead-code isinstance branches and inconsistent logging granularity, but these do not affect correctness.

Confidence Score: 5/5

Safe to merge — core serialization logic is correct and the behavioral change is properly gated behind a feature flag.

Only P2 style/consistency findings: two cache sites not fully migrated to the model_type pattern (dead-code branch and slightly weaker logging). No correctness bugs found; the CacheCodec design, UserApiKeyCache implementation, enable_redis_auth_cache flag gating, TTL propagation, and Prometheus counter wiring are all sound.

litellm/proxy/auth/auth_checks.py (get_team_member_default_budget) and litellm/proxy/proxy_server.py (_update_user_cache/_update_end_user_cache) for the consistency gap; no blocking issues.

Important Files Changed

Filename	Overview
litellm/proxy/common_utils/cache_pydantic_utils.py	New CacheCodec class centralizing serialize/deserialize at the DualCache boundary; clean design with proper ValidationError handling.
litellm/proxy/common_utils/user_api_key_cache.py	New UserApiKeyCache wrapping DualCache with typed model_type overloads and codec integration; overload ordering is unusual but valid.
litellm/proxy/auth/auth_checks.py	Most cache read/write sites migrated to model_type pattern; get_team_member_default_budget is a missed site with dead-code isinstance branch.
litellm/proxy/proxy_server.py	Spend-tracking cache updated to CacheCodec; _update_user_cache/_update_end_user_cache use inconsistent two-step deserialization vs rest of codebase.
litellm/caching/dual_cache.py	Adds attach_redis_cache no-op guard method for lazy Redis wiring; clean implementation.
litellm/proxy/auth/auth_metrics.py	New AuthMetrics helper correctly uses _inc_labeled_counter + prometheus_label_factory to increment litellm_auth_combined_view_queries_total.
tests/test_litellm/proxy/test_redis_auth_cache_flag.py	Flag-behavior tests are correct; _patched_init_cache properly extracts and forwards enable_redis_auth_cache; uses plain DualCache instead of UserApiKeyCache for the patched object but sufficient for flag verification.
tests/test_litellm/proxy/common_utils/test_cache_codec.py	Thorough unit tests for CacheCodec; _IncompatibleModel now correctly uses foo: int (missing name) ensuring ValidationError is raised.
litellm/proxy/auth/user_api_key_auth.py	Type hints updated to UserApiKeyCache; team-member cache reads/writes now use model_type; team cache write guards against None team_id.

_{Reviews (23): Last reviewed commit: "fix unused imports" | Re-trigger Greptile}

[greptile-apps]

Early return silently drops spend updates when key absent from cache

Before this PR, if async_get_cache returned None (cache miss), the function fell through to existing_spend = existing_spend_obj.spend. Now, cached_key is None triggers an early return, silently discarding the spend update. In a fresh-start or post-eviction scenario where the key hasn't been populated yet, all spend accrued during that window is lost. The introduction of a second return on deserialization failure (lines 1954–1960) compounds the silent-loss surface area.

[harish-berri]

On cached_key is None, the previous code path couldn’t safely continue to existing_spend_obj.spend unless existing_spend_obj was non-None — typically that would have been an AttributeError, not a successful spend update. The early return makes that cache-miss case explicit and matches other helpers in update_cache (e.g. user spend: no cache value → return). This block only updates an existing cached key object; it isn’t the sole source of truth for spend — DB/async spend logging still records usage — so the risk is stale cached spend until the key is rehydrated, not “lost” totals. The deserialize None path is the same: skip unsafe cache writes, with a warning log. Happy to track a follow-up to DB-load-then-update-cache on miss if we want tighter cache freshness.

[harish-berri]

@greptile review again

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.

[veria-ai]

Low: No security issues found

This PR centralizes Pydantic model serialization/deserialization for the DualCache (in-memory + Redis) layer via a new CacheCodec utility, attaches a shared Redis backend to user_api_key_cache, and replaces assert statements with proper HTTPException responses in auth code.

The exclude_none=True serialization strategy was reviewed against all security-critical fields (blocked, max_budget, models, guardrails, allowed_routes, etc.) and none lose their security semantics through the None-exclusion round-trip. The assert-to-HTTPException refactors are improvements that prevent stack trace leakage.

---

Status: 0 open
Risk: 2/10

Posted by Veria AI · 2026-04-23T23:45:09.599Z

[veria-ai]

Low: No security issues found

This PR introduces a CacheCodec utility for consistent Pydantic model serialization/deserialization in the dual-cache layer (in-memory + Redis), and adds an opt-in enable_redis_auth_cache flag to share virtual-key auth lookups across proxy workers via Redis. The serialization uses exclude_none=True which preserves all security-critical fields with non-None defaults (e.g., blocked: bool = False, models: list = []). Fields that default to None (like max_budget, guardrails) retain their "no restriction" semantics through the serialize/deserialize round-trip. Deserialization failures are treated as cache misses, falling through to the database as the source of truth. API keys stored in cache are already hashed. The .vscode/launch.json being committed to the repo is a developer tooling concern but contains no secrets.

---

Status: 0 open
Risk: 2/10

Posted by Veria AI · 2026-04-24T01:33:22.474Z