chore(caching): isolate semantic cache entries - #26990
Conversation
Greptile SummaryThis PR scopes semantic cache entries to their LiteLLM-generated cache key for both Redis (via Confidence Score: 5/5Safe to merge; both findings are P2 with no immediate data-corruption risk on current deployments. No P0 or P1 issues found. Both findings are speculative or edge-case scenarios (stale isolated-index schema requires a second future upgrade; legacy Qdrant false-miss requires a near-identical cross-scope vector). The core isolation logic is correct and well-tested with mock-only tests. litellm/caching/redis_semantic_cache.py — stale isolated-index overwrite path; litellm/caching/qdrant_semantic_cache.py — legacy mode limit=1 search
|
| Filename | Overview |
|---|---|
| litellm/caching/redis_semantic_cache.py | Adds key-scoped isolation via filterable_fields and a graceful fallback to _isolated index on schema mismatch; the stale-isolated-index overwrite path silently destroys cached data without an opt-out flag |
| litellm/caching/qdrant_semantic_cache.py | Adds cache-key isolation via payload filters and best-effort index creation; legacy mode fetches only 1 result so wrong-scope hits block valid same-key hits from being found |
| tests/test_litellm/caching/test_redis_semantic_cache.py | Comprehensive new unit tests for isolation logic, legacy-mode fallback, bytes decoding, and filter-expression construction; all mock-based with no network calls |
| tests/test_litellm/caching/test_qdrant_semantic_cache.py | Adds tests for scoped cache hits/misses, unscoped rejection, legacy-mode opt-in, payload-index failure resilience, and metadata propagation; all mock-based |
Reviews (9): Last reviewed commit: "Add semantic cache legacy migration flag" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
@greptileai please re-review after ae9b63c. The Qdrant path now best-effort creates the payload index used for scoped lookups. Existing unscoped Qdrant points do not contain the generated LiteLLM cache key or request metadata needed to assign them to the correct scope, so an unfiltered fallback would keep the isolation issue. They are intentionally treated as misses and repopulated by scoped writes, which is the secure cold-cache upgrade path. |
|
@greptileai please re-review current head 1c19bdd after the coverage-only follow-up. |
|
@greptileai please re-review current head 74e9344. This follow-up addresses the remaining Qdrant sync scoped-miss metadata thread. |
|
@greptileai please re-review current head d8c11f9. This follow-up addresses the remaining Redis sync semantic-similarity metadata note. |
|
please resolve merge conflicts @stuxf |
|
Merge conflicts are resolved; the PR diff is back to semantic-cache isolation files only. Re-triggering review on the current head. |
|
Addressed this Redis isolated-index fallback edge case in Validation:
|
|
Addressed the Qdrant sync metadata finding in a05d3b5. Validation:
@greptileai please rerun on the latest commit. |
|
Cleaned up the Redis semantic-cache leftovers in 9f1feaadebb432b695017cb46a4c8f6c3458078c:
Validation:
@greptileai please rerun on the latest commit. |
|
Added an explicit semantic-cache legacy migration flag in af7794272b25b8f971025a0ee27fb23e0c1918449:
Validation:
@greptileai please rerun on the latest commit. |
Mypy infers the dict's value type from the first branch (Dict[str, bool]) which clashes with the scalar branch's mixed-type inner dict. Explicit Dict[str, Any] annotation lifts the inference.
The flag was an opt-in escape hatch for the cross-tenant leak the rest
of the patch closes — flipping it on (env var or constructor param)
re-enables exactly the VERIA-54 primitive on either backend. There is
no operational need that the secure path doesn't already meet:
- Qdrant: legacy points without ``litellm_cache_key`` payload are
excluded by the must-clause filter and treated as misses; new sets
populate the cache key, so cold-start lasts only as long as the
natural cache rebuild.
- Redis: existing unscoped index can't carry the new schema; the init
path falls back to ``{name}_isolated`` (and recreates it on stale
schema), leaving the legacy index untouched.
Drop the constructor param, env-var fallback, ``_using_legacy_unscoped_index``
flag, the legacy-reuse branch in ``_init_semantic_cache``, and the
matching guards in set/get paths. Update tests to drop the legacy-mode
cases and assert the secure-only behaviour.
07824b5
into
BerriAI:litellm_internal_staging
…nt-isolation chore(caching): isolate semantic cache entries
3 participants
Relevant issues
Veria: VERIA-54
What changed
Tests
uv run pytest tests/test_litellm/caching/test_redis_semantic_cache.py tests/test_litellm/caching/test_qdrant_semantic_cache.py tests/test_litellm/test_lazy_openapi_snapshot.py -quv run ruff check litellm/caching/redis_semantic_cache.py litellm/caching/qdrant_semantic_cache.py litellm/proxy/_lazy_openapi_snapshot.py tests/test_litellm/caching/test_redis_semantic_cache.py tests/test_litellm/caching/test_qdrant_semantic_cache.py tests/test_litellm/test_lazy_openapi_snapshot.pyuv run --with redisvl==0.4.1 mypy --no-incremental litellm/caching/redis_semantic_cache.py litellm/caching/qdrant_semantic_cache.pyuv run pytest tests/test_litellm/caching/test_qdrant_semantic_cache.py tests/test_litellm/caching/test_redis_semantic_cache.py tests/test_litellm/test_lazy_openapi_snapshot.py --cov=litellm.caching.qdrant_semantic_cache --cov=litellm.caching.redis_semantic_cache --cov=litellm.proxy._lazy_openapi_snapshot --cov-report=term-missing -quv run black .