[Fix] Cache LiteLLM_Config param reads in DualCache and batch

[Michael-RZ-Berri]

Relevant issues

Caches config parameter reads in dual cache and batches reads to reduce reading in multi-pod setups.

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Ran end to end on docker and confirmed lower read count through writing / reading, 2.8 queries per second down to 0.7 queries per second. The improvement is larger across higher pod counts.

Type

🐛 Bug Fix
✅ Test

Changes

proxy_server.py and utils.py

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Michael Riad Zaky seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[veria-ai]

Caching layer for LiteLLM_Config database reads

This PR introduces a DualCache (in-memory + optional Redis) to reduce repetitive per-param database queries for internal configuration values. All cache keys are derived from hardcoded param names, invalidation is called after every write path, and the affected endpoints remain behind admin-only auth. No new user-controlled inputs flow into cache keys or values, and no security boundaries are modified.

---

Status: 0 open
Risk: 1/10

[greptile-apps]

Greptile Summary

This PR reduces DB read pressure in multi-pod deployments by introducing a DualCache-backed layer (litellm_config_cache) for LiteLLM_Config table reads, batching startup reads with a single find_many prefetch, and invalidating cache entries after every config write path.

Confidence Score: 5/5

Safe to merge; only P2 style issues found, core caching and invalidation logic is sound.

No P0 or P1 issues found. Previously flagged decorator-placement P1 is resolved in this version. Two P2 observations: return-type inconsistency and DEBUG-level logging for prefetch failures.

litellm/proxy/utils.py — review the get_config_param return type contract

Important Files Changed

Filename	Overview
litellm/proxy/utils.py	Adds module-level DualCache, get_config_param, prefetch_config_params, and invalidate_config_param; two P2 issues found.
litellm/proxy/proxy_server.py	Wires Redis layer, replaces direct Prisma calls with get_config_param, adds prefetch warm-up and invalidation after writes.
tests/test_litellm/proxy/test_proxy_server.py	Adds autouse cache-flush fixture and adapts mocks to new get_generic_data code path; test intent preserved.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[load_router_config] --> B[prefetch_config_params find_many batch query]
    B --> C[litellm_config_cache DualCache in-memory + Redis TTL=60s]
    D[get_config_param] --> E{Cache hit?}
    E -- yes --> F[_unpack_config_row returns _ConfigRow shim]
    E -- no --> G[get_generic_data Prisma find_first]
    G --> H[store in cache]
    H --> I[return real Prisma row]
    J[Config write endpoint] --> K[invalidate_config_param async_delete_cache]
    K --> C
    L[PrismaClient.insert_data config table] --> K

Loading

_{Reviews (10): Last reviewed commit: "cache LiteLLM_Config param reads in Dual..." | Re-trigger Greptile}

[greptile-apps]

Redis attachment is skipped if litellm.cache initialises before prisma_client

prisma_client._config_param_cache.redis_cache is only wired up here, inside the if litellm.cache is not None and isinstance(litellm.cache.cache, (RedisCache, RedisClusterCache)) block. In deployments where the cache is configured after this code path runs (or where prisma_client is initialised after load_router_config completes), the Redis layer will never be attached and all cross-pod cache sharing silently degrades to in-memory-only. Consider making the attachment explicit during PrismaClient.__init__ or adding a warning log when redis_cache remains None after startup.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!