fix(caching): bound the semantic cache embedding lookup so a dead embedding endpoint can't block requests - #37742
Conversation
A semantic cache lookup embeds the prompt before the request reaches the LLM, and that embedding call carried no deadline of its own. It inherited the 6000s request timeout and the Router's num_retries, so an embedding endpoint that is down or unroutable parked every proxied request for minutes and gave back nothing but x-litellm-semantic-similarity 0.0 once it finally gave up. The lookup now runs on its own short deadline, 5s by default, with retries off so failures cannot stack. Redis, Valkey and qdrant all pick it up, and the deadline is settable per cache with semantic_cache_embedding_timeout or globally with SEMANTIC_CACHE_EMBEDDING_TIMEOUT_SECONDS.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…/litellm into litellm_lit5879_semantic_cache_embedding_timeout
Merging this PR will degrade performance by 10.15%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | test_token_counter_multi_turn |
531.3 µs | 591.3 µs | -10.15% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing litellm_lit5879_semantic_cache_embedding_timeout (d57715b) with litellm_internal_staging (e07a712)1
Footnotes
Greptile SummaryThe PR bounds semantic-cache embedding lookups so an unavailable embedding endpoint degrades to a cache miss rather than indefinitely delaying an LLM request.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| litellm/caching/_embedding_router.py | Centralizes resolution of explicit and default semantic-cache embedding timeouts. |
| litellm/caching/caching.py | Accepts and forwards the configurable embedding timeout to all semantic-cache backends. |
| litellm/caching/redis_semantic_cache.py | Applies embedding timeouts and an async hard deadline while preserving fail-open cache behavior. |
| litellm/caching/qdrant_semantic_cache.py | Bounds synchronous provider attempts and wraps asynchronous embedding lookup in a hard deadline. |
| litellm/caching/valkey_semantic_cache.py | Initializes the inherited semantic-cache embedding timeout consistently. |
| litellm/constants.py | Defines the environment-configurable five-second default timeout. |
| tests/test_litellm/caching/test_redis_semantic_cache.py | Covers timeout propagation, unresponsive endpoints, defaults, and fail-open behavior. |
| tests/test_litellm/caching/test_qdrant_semantic_cache.py | Covers timeout propagation, default resolution, and asynchronous hard-deadline behavior. |
Reviews (2): Last reviewed commit: "chore(constants): drop the redundant com..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Qdrant timeout skips similarity signal
- Wrapped the embedding lookup in
async_get_cacheto catchasyncio.TimeoutError, setsemantic-similarityto 0.0 on the metadata, and return None so the cache-skip signal matches the Redis/Valkey backends, plus added a regression test mirroring the Redis fail-open coverage.
- Wrapped the embedding lookup in
Or push these changes by commenting:
@cursor push e5785e0c78
Preview (e5785e0c78)
diff --git a/litellm/caching/qdrant_semantic_cache.py b/litellm/caching/qdrant_semantic_cache.py
--- a/litellm/caching/qdrant_semantic_cache.py
+++ b/litellm/caching/qdrant_semantic_cache.py
@@ -425,7 +425,11 @@
messages: Final = kwargs["messages"]
prompt: Final = get_str_from_messages(messages)
- embedding_response: Final = await self._get_async_embedding(prompt, metadata=kwargs.get("metadata"))
+ try:
+ embedding_response: Final = await self._get_async_embedding(prompt, metadata=kwargs.get("metadata"))
+ except asyncio.TimeoutError:
+ kwargs.setdefault("metadata", {})["semantic-similarity"] = 0.0
+ return None
# get the embedding
embedding: Final = embedding_response["data"][0]["embedding"]
diff --git a/tests/test_litellm/caching/test_qdrant_semantic_cache.py b/tests/test_litellm/caching/test_qdrant_semantic_cache.py
--- a/tests/test_litellm/caching/test_qdrant_semantic_cache.py
+++ b/tests/test_litellm/caching/test_qdrant_semantic_cache.py
@@ -1023,6 +1023,52 @@
assert time.monotonic() - started < 1.0
+@pytest.mark.asyncio
+async def test_qdrant_async_get_cache_fails_open_when_embedding_hangs(monkeypatch):
+ import asyncio
+ import time
+
+ from litellm.caching.qdrant_semantic_cache import QdrantSemanticCache
+
+ cache = QdrantSemanticCache.__new__(QdrantSemanticCache)
+ cache.embedding_model = "sem-embed"
+ cache.embedding_max_input_tokens = None
+ cache.embedding_timeout = 0.05
+ cache.similarity_threshold = 0.8
+ cache.qdrant_api_base = "http://test.qdrant.local"
+ cache.collection_name = "test_collection"
+ cache.headers = {"Content-Type": "application/json", "api-key": "test_key"}
+ cache.async_client = MagicMock()
+ cache.async_client.post = AsyncMock()
+
+ async def never_responds(**kwargs):
+ await asyncio.sleep(3)
+ return {"data": [{"embedding": [0.1, 0.2]}]}
+
+ router = MagicMock()
+ router.get_configured_token_limits.return_value = (None, None)
+ router.aembedding = never_responds
+ monkeypatch.setitem(
+ sys.modules,
+ "litellm.proxy.proxy_server",
+ _router_proxy_module(router, "sem-embed"),
+ )
+
+ metadata: dict = {}
+ started = time.monotonic()
+ result = await cache.async_get_cache(
+ key="test_key",
+ messages=[{"role": "user", "content": "What is the capital of France?"}],
+ metadata=metadata,
+ )
+ elapsed = time.monotonic() - started
+
+ assert result is None
+ assert metadata["semantic-similarity"] == 0.0
+ assert elapsed < 1.0
+ cache.async_client.post.assert_not_called()
+
+
def test_qdrant_semantic_cache_defaults_embedding_timeout():
from litellm.caching.qdrant_semantic_cache import QdrantSemanticCache
from litellm.constants import SEMANTIC_CACHE_EMBEDDING_TIMEOUT_SECONDSYou can send follow-ups to the cloud agent here.
| input=embedding_input, | ||
| cache={"no-store": True, "no-cache": True}, | ||
| ) | ||
| return await asyncio.wait_for(embedding_call, self.embedding_timeout) |
There was a problem hiding this comment.
Qdrant timeout skips similarity signal
Low Severity
The new asyncio.wait_for deadline raises TimeoutError when the embedding endpoint is dead, but async_get_cache does not catch it. Redis and Valkey set semantic-similarity to 0.0 on that path; Qdrant leaves it unset, so callers lose the cache-skip signal this change documents.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit d57715b. Configure here.
There was a problem hiding this comment.
Qdrant already propagates embedding failures instead of failing open, so wait_for only bounds the wait. Making it fail open is a separate change
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
1 issue from previous review remains unresolved.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit d57715b. Configure here.
2 participants
TLDR
The semantic cache embedding lookup runs inline before the LLM call and had no deadline of its own, so it inherited the 6000s request timeout and the router's retries. When the configured embedding endpoint stops answering, every chat request parks in that lookup for minutes and the client often times out first. The failure is also silent: the only outward signs are the latency and
x-litellm-semantic-similarity: 0.0This gives the lookup its own 5s deadline, and a single attempt on the routed path, so an unreachable embedding endpoint costs a few seconds and a cache miss instead of the whole request. Redis, Valkey and qdrant semantic caches all get the bound, and it is tunable per cache config or with
SEMANTIC_CACHE_EMBEDDING_TIMEOUT_SECONDSMeasured on a live proxy against a real Anthropic model: 685.7s before, 7.4s after, same config.
User Flow
Before
Every chat completion through the gateway hangs for minutes, and most clients time out before an answer arrives
GET http://litellm-domain/health/livelinessreturns"I'm alive!"GET http://litellm-domain/cache/pingreturns{"status": "healthy", "cache_type": "redis-semantic"}redis_semantic_cache_embedding_modelstops answering. Nothing in the proxy says so, andGET /cache/pingstill answers"status": "healthy"because for a semantic cache it does not test the embedding endpoint at allPOST http://litellm-domain/v1/chat/completionswith{"model": "claude", "messages": [{"role": "user", "content": "What is the capital of France? Answer in one word."}]}. Nothing comes backx-litellm-response-duration-ms: 685519.593,x-litellm-overhead-duration-ms: 683421.74andx-litellm-semantic-similarity: 0.0. The answer itself is a normal "Paris", so the 11 minutes bought nothingcurl: (28) Operation timed out after 700002 milliseconds with 0 bytes receivedcacheblock and restarting the proxy. The same request then returns in 5.2 secondsAfter
The same request comes back in about seven seconds, with the cache skipped instead of waited on
GET http://litellm-domain/health/livelinessreturning"I'm alive!"GET http://litellm-domain/cache/pingreturning{"status": "healthy", "cache_type": "redis-semantic"}POST http://litellm-domain/v1/chat/completionsx-litellm-response-duration-ms: 7204.558,x-litellm-overhead-duration-ms: 5021.482andx-litellm-semantic-similarity: 0.0. The five seconds of overhead is the cache giving up, and the similarity of 0.0 is how the caller can tell the cache was skippedsemantic_cache_embedding_timeoutundercache_paramsor setsSEMANTIC_CACHE_EMBEDDING_TIMEOUT_SECONDSRelevant issues
Linear ticket
Resolves LIT-5879
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Both runs use the same config file, the same redis, the same real Anthropic model, and the same prompt. The only difference is the commit the proxy runs from.
Shared setup, run once:
10.255.255.1:9is an unroutable address, so every connection to the embedding endpoint blackholes. The request below is the same one in both runs:Before (e07a712)
litellm.aembedding(... api_base='http://10.255.255.1:9/v1', ... timeout=6000.0)andRouter._aembedding ..., num_retries - 2. The lookup inherits the request timeout and gets retriedcurl: (28)at a 700s cap with no response at all, and the proxy log had zero hits forapi.anthropic.comAfter (7d23d41)
'semantic-cache-embedding': True, timeout=5.0andRouter._aembedding ..., num_retries - 0, where the run above loggedtimeout=6000.0andnum_retries - 2grep -c 'api.anthropic.com'over the proxy log returns 2, one dispatch per request, against 0 for the hanging-endpoint case before the fixOverhead drops from 683421.74ms to 5021.482ms, a factor of 136, and the response is byte-identical apart from ids and timing.
Type
🐛 Bug Fix
Caveats (if any)
The default deadline is 5s. Raise it with
semantic_cache_embedding_timeoutundercache_params, or with theSEMANTIC_CACHE_EMBEDDING_TIMEOUT_SECONDSenv var, if your embedding endpoint is legitimately slower than that.An embedding endpoint that answers slowly but correctly will now miss the cache instead of making the caller wait for it.
When the embedding model is not a Router deployment, the lookup goes straight through
litellm.embedding, where the provider client still applies its own retries. A dead endpoint costs about 16s on that path rather than 5s. Each attempt is still bounded by the deadline, and forcing zero retries there would change retry behavior for every embedding call, not just cache lookupsFinal Attestation
Note
Medium Risk
Changes the hot path in front of LLM calls: embedding timeouts and retries now fail open to a cache miss. Slow-but-valid embed endpoints will miss cache unless the new timeout is raised.
Overview
Semantic-cache lookups no longer inherit the long request timeout and router retries. A hung embedding endpoint now fails the cache lookup quickly (default 5s, one attempt) and the request continues to the LLM as a miss.
semantic_cache_embedding_timeout(orSEMANTIC_CACHE_EMBEDDING_TIMEOUT_SECONDS) is applied on Redis, Valkey, and Qdrant backends. Embedding calls passtimeoutandnum_retries=0; async lookups are also wrapped inasyncio.wait_for. Tests cover the bound, hang fail-open, and config forwarding.Reviewed by Cursor Bugbot for commit d57715b. Bugbot is set up for automated code reviews on this repo. Configure here.