fix(redis): cache GCP IAM token to prevent async event loop blocking - #26441
Conversation
## Problem GCPIAMCredentialProvider.get_credentials() calls _generate_gcp_iam_access_token on every Redis connection establishment. This function performs synchronous HTTP and gRPC calls (google-auth + google-cloud-iam) which block Python's asyncio event loop while running. Under concurrent load (e.g. connection pool warm-up, parallel health checks), multiple connections are established simultaneously, each triggering an independent blocking IAM token refresh. These refreshes serialise behind each other inside the single-threaded event loop, causing individual Redis spans to take 20-25 seconds instead of milliseconds. Observed in production via Datadog APM: a single INCRBYFLOAT Redis span took 25.6 seconds (90% of a 28.4s trace), with GCP metadata + GenerateAccessToken gRPC calls visible inside the span. This cascaded into aiohttp SocketTimeoutError on upstream LLM API calls — not because the upstream was slow, but because the event loop was frozen and the 30-second sock_read timer fired on a connection that was never given CPU time. ## Fix Add a module-level token cache (dict keyed by service account, value is (token, expiry_monotonic)). _get_cached_gcp_iam_token() returns the cached token on cache hit (no I/O), and refreshes only when expired using double-checked locking so only one thread performs the network round-trip. GCP IAM tokens are valid for 1 hour; the cache TTL is set to 55 minutes (_GCP_IAM_TOKEN_TTL_SECONDS = 3300) to refresh safely before expiry. The cache is shared across all GCPIAMCredentialProvider instances for the same service account, so N concurrent Redis connections on the same pod share a single token and avoid N concurrent blocking refreshes. get_credentials_async() already used asyncio.to_thread (non-blocking), and is updated to call _get_cached_gcp_iam_token so it also benefits from caching. ## Tests - Updated existing test that expected a fresh token on every call to reflect the new caching behaviour. - Added tests for: cache hit (no redundant I/O), cache expiry and refresh, and cache sharing across multiple provider instances. - Added autouse fixture to clear the module-level cache between tests. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Low: No security issues foundThis PR adds module-level caching for GCP IAM tokens used for Redis authentication, replacing per-connection token generation. The cache key ( Status: 0 open Posted by Veria AI · 2026-04-24T17:27:27.254Z |
Greptile SummaryThis PR fixes a production latency issue where each Redis connection establishment independently called the GCP IAM API, serialising blocking gRPC calls on Python's async event loop and causing 20–25 second Redis spans. The fix introduces a module-level token cache with double-checked locking (55-minute TTL), so N concurrent connections share a single cached token and only one thread ever makes a network round-trip at refresh time. Confidence Score: 5/5Safe to merge — implementation is correct, well-tested, and strictly reduces blocking behavior with no regressions. No P0 or P1 issues found. The double-checked locking pattern is correct for CPython, the async path correctly offloads work via asyncio.to_thread, the autouse fixture properly isolates tests, and the test modifications reflect a legitimate intentional behavior change rather than a regression mask. No files require special attention.
|
| Filename | Overview |
|---|---|
| litellm/_redis_credential_provider.py | Adds module-level token cache with double-checked locking; implementation is correct and significantly reduces blocking IAM calls. |
| tests/test_litellm/test_redis.py | Replaces the old "fresh token per call" test with cache-hit, expiry, and cross-instance sharing tests; autouse fixture ensures test isolation. |
Sequence Diagram
sequenceDiagram
participant EL as Async Event Loop
participant T1 as Thread 1 (to_thread)
participant T2 as Thread 2 (to_thread)
participant Cache as _token_cache (dict)
participant Lock as _token_cache_lock
participant GCP as GCP IAM API
Note over EL: Connection pool warm-up (N connections)
EL->>T1: asyncio.to_thread(_get_cached_gcp_iam_token)
EL->>T2: asyncio.to_thread(_get_cached_gcp_iam_token)
T1->>Cache: get(service_account) → miss
T2->>Cache: get(service_account) → miss
T1->>Lock: acquire (wins)
T2->>Lock: acquire (waits)
T1->>Cache: double-check → still miss
T1->>GCP: GenerateAccessToken (blocking, ~1s)
GCP-->>T1: token
T1->>Cache: set(service_account, (token, now+3300s))
T1->>Lock: release
T1-->>EL: return token
T2->>Lock: acquire (now available)
T2->>Cache: double-check → HIT (token valid)
T2->>Lock: release
T2-->>EL: return cached token
Note over EL: Subsequent calls (cache warm)
EL->>T1: asyncio.to_thread(_get_cached_gcp_iam_token)
T1->>Cache: get(service_account) → HIT, valid
T1-->>EL: return cached token (no lock, no I/O)
Reviews (4): Last reviewed commit: "Merge branch 'litellm_gcp-iam-redis-toke..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| """ | ||
| redis.credentials.CredentialProvider implementation that generates a fresh GCP IAM | ||
| token on every new connection. This fixes the 1-hour token expiry issue for async | ||
| Redis cluster clients, which previously generated the token once at startup and | ||
| cached it as a static password. |
There was a problem hiding this comment.
The opening sentence still says "generates a fresh GCP IAM token on every new connection," which is exactly the behaviour this PR is fixing. After the change, tokens are cached for 55 minutes, so the first sentence contradicts the implementation and the second paragraph that was added.
| async def get_credentials_async(self) -> Tuple[str]: | ||
| token = await asyncio.to_thread( | ||
| _generate_gcp_iam_access_token, self._gcp_service_account | ||
| _get_cached_gcp_iam_token, self._gcp_service_account | ||
| ) | ||
| return (token,) |
There was a problem hiding this comment.
asyncio.to_thread spawns a thread-pool task for every get_credentials_async call. On a cache hit (the overwhelmingly common case after warm-up), _get_cached_gcp_iam_token performs only a dict lookup and a monotonic clock read — no I/O at all. The thread-pool round-trip (~50–200 µs of scheduling overhead) is wasted on each connection establishment.
Consider checking the cache without to_thread first, and only offloading to a thread when a network refresh is actually needed:
async def get_credentials_async(self) -> Tuple[str]:
cached = _token_cache.get(self._gcp_service_account)
if cached is not None:
token, expiry = cached
if time.monotonic() < expiry:
return (token,)
token = await asyncio.to_thread(
_get_cached_gcp_iam_token, self._gcp_service_account
)
return (token,)This keeps the non-blocking fast path for hits and still offloads the blocking gRPC call to a thread on misses.
…lass Updated the docstring for the GCPIAMCredentialProvider class to clarify its purpose and the caching mechanism for GCP IAM tokens. The changes enhance readability and maintainability by providing a more concise explanation of the token caching strategy and its benefits for Redis authentication.
…lass Updated the docstring for the GCPIAMCredentialProvider class to clarify its purpose and the caching mechanism for GCP IAM tokens. The changes enhance readability and maintainability by providing a more concise explanation of the token caching strategy and its benefits for Redis authentication.
…com/BerriAI/litellm into litellm_gcp-iam-redis-token-caching
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…erriAI#26441) * fix(redis): cache GCP IAM token to prevent async event loop blocking ## Problem GCPIAMCredentialProvider.get_credentials() calls _generate_gcp_iam_access_token on every Redis connection establishment. This function performs synchronous HTTP and gRPC calls (google-auth + google-cloud-iam) which block Python's asyncio event loop while running. Under concurrent load (e.g. connection pool warm-up, parallel health checks), multiple connections are established simultaneously, each triggering an independent blocking IAM token refresh. These refreshes serialise behind each other inside the single-threaded event loop, causing individual Redis spans to take 20-25 seconds instead of milliseconds. Observed in production via Datadog APM: a single INCRBYFLOAT Redis span took 25.6 seconds (90% of a 28.4s trace), with GCP metadata + GenerateAccessToken gRPC calls visible inside the span. This cascaded into aiohttp SocketTimeoutError on upstream LLM API calls — not because the upstream was slow, but because the event loop was frozen and the 30-second sock_read timer fired on a connection that was never given CPU time. ## Fix Add a module-level token cache (dict keyed by service account, value is (token, expiry_monotonic)). _get_cached_gcp_iam_token() returns the cached token on cache hit (no I/O), and refreshes only when expired using double-checked locking so only one thread performs the network round-trip. GCP IAM tokens are valid for 1 hour; the cache TTL is set to 55 minutes (_GCP_IAM_TOKEN_TTL_SECONDS = 3300) to refresh safely before expiry. The cache is shared across all GCPIAMCredentialProvider instances for the same service account, so N concurrent Redis connections on the same pod share a single token and avoid N concurrent blocking refreshes. get_credentials_async() already used asyncio.to_thread (non-blocking), and is updated to call _get_cached_gcp_iam_token so it also benefits from caching. ## Tests - Updated existing test that expected a fresh token on every call to reflect the new caching behaviour. - Added tests for: cache hit (no redundant I/O), cache expiry and refresh, and cache sharing across multiple provider instances. - Added autouse fixture to clear the module-level cache between tests. * refactor(redis): remove unused Optional import from _redis_credential_provider.py * refactor(redis): improve documentation for GCPIAMCredentialProvider class Updated the docstring for the GCPIAMCredentialProvider class to clarify its purpose and the caching mechanism for GCP IAM tokens. The changes enhance readability and maintainability by providing a more concise explanation of the token caching strategy and its benefits for Redis authentication. * refactor(redis): improve documentation for GCPIAMCredentialProvider class Updated the docstring for the GCPIAMCredentialProvider class to clarify its purpose and the caching mechanism for GCP IAM tokens. The changes enhance readability and maintainability by providing a more concise explanation of the token caching strategy and its benefits for Redis authentication. ---------
2 participants
Problem
Piping this PR's work https://github.com/BerriAI/litellm/pull/26317
GCPIAMCredentialProvider.get_credentials() calls _generate_gcp_iam_access_token on every Redis connection establishment. This function performs synchronous HTTP and gRPC calls (google-auth + google-cloud-iam) which block Python's asyncio event loop while running.
Under concurrent load (e.g. connection pool warm-up, parallel health checks), multiple connections are established simultaneously, each triggering an independent blocking IAM token refresh. These refreshes serialise behind each other inside the single-threaded event loop, causing individual Redis spans to take 20-25 seconds instead of milliseconds.
Observed in production via Datadog APM: a single INCRBYFLOAT Redis span took 25.6 seconds (90% of a 28.4s trace), with GCP metadata + GenerateAccessToken gRPC calls visible inside the span. This cascaded into aiohttp SocketTimeoutError on upstream LLM API calls — not because the upstream was slow, but because the event loop was frozen and the 30-second sock_read timer fired on a connection that was never given CPU time.
Fix
Add a module-level token cache (dict keyed by service account, value is (token, expiry_monotonic)). _get_cached_gcp_iam_token() returns the cached token on cache hit (no I/O), and refreshes only when expired using double-checked locking so only one thread performs the network round-trip.
GCP IAM tokens are valid for 1 hour; the cache TTL is set to 55 minutes (_GCP_IAM_TOKEN_TTL_SECONDS = 3300) to refresh safely before expiry.
The cache is shared across all GCPIAMCredentialProvider instances for the same service account, so N concurrent Redis connections on the same pod share a single token and avoid N concurrent blocking refreshes.
get_credentials_async() already used asyncio.to_thread (non-blocking), and is updated to call _get_cached_gcp_iam_token so it also benefits from caching.
Tests
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes