fix(proxy): recover from cached-plan errors by reconnecting the Prisma client

[yassin-berriai]

Relevant issues

Relates to #19424

Linear ticket

N/A

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Baseline for all before/after comparisons below is litellm_internal_staging, the base branch of this PR.

During a rolling deployment a schema change invalidates the prepared-statement plans that pooled connections still hold, and the next auth lookup fails with PostgreSQL's "cached plan must not change result type". On staging, _query_first_with_cached_plan_fallback recovers by retrying with a unique millisecond-timestamp comment injected into the SQL. The request succeeds, but because the retry text is unique every time, PostgreSQL can never cache a plan for it, and because nothing fixes the poisoned connection, the first attempt keeps failing on every subsequent request. Every auth lookup pays a failed attempt plus a full re-plan of the 9-table join, forever, until connections recycle. At fleet scale this pegs the database writer CPU (the production incident this PR addresses).

This PR recovers by recreating the Prisma client through the existing singleflight attempt_db_reconnect. Recreating the client kills the engine subprocess, which drops the server-side plans and the engine's client-side prepared-statement name cache together, so the retried query is prepared fresh. The retry reuses the SQL byte-for-byte, so PostgreSQL's plan cache keeps working and the error clears after a single reconnect instead of recurring on every request.

Live repro setup

Postgres 17 in Docker, proxy on port 4101 with database_connection_pool_limit: 1 for determinism. Auth results cache per key, so every DB-hitting request uses a fresh key. Keys are generated before the ALTER so key-generation inserts are not entangled in the repro.

docker run -d --name pg-29983 -e POSTGRES_PASSWORD=pw -e POSTGRES_DB=litellm -p 5501:5432 postgres:17

# config.yaml: one openai/gpt-4o fake-key model; general_settings.database_connection_pool_limit: 1
DATABASE_URL="postgresql://postgres:pw@localhost:5501/litellm" LITELLM_MASTER_KEY=sk-1234 \
  STORE_MODEL_IN_DB=False python litellm/proxy/proxy_cli.py --config config.yaml --port 4101

# 1. generate 14 keys (models=["gpt-4o"]) via POST /key/generate
# 2. warm 8 fresh keys on GET /v1/models (prime the prepared-statement plans)
# 3. ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN repro_col TEXT;
# 4. trigger 6 fresh keys on GET /v1/models; then 4 more for recovery

Before (litellm_internal_staging): requests succeed but every post-ALTER lookup re-plans

All 6 trigger requests return HTTP 200, but the proxy log shows the cached-plan warning firing on every single one (6 warnings for 6 requests), because the timestamp-comment retry never repairs the poisoned plan and never lets PostgreSQL cache the retry:

WARNING utils.py:3281 - PostgreSQL cached plan error detected for token lookup, retrying with fresh plan. This may occur during rolling deployments when schema changes are applied.   # x6, one per request

This is the per-request failed-attempt-plus-re-plan loop that pegs the DB CPU at fleet scale.

After (this PR, reconnect): all requests succeed and the connection recovers

### Trigger: 6 fresh keys on GET /v1/models ###
trigger req 1: HTTP 200
trigger req 2: HTTP 200
trigger req 3: HTTP 200
trigger req 4: HTTP 200
trigger req 5: HTTP 200
trigger req 6: HTTP 200
TRIGGER_PASS=6/6

### Recovery probe: 4 more fresh keys ###
recovery req 1: HTTP 200
recovery req 2: HTTP 200
recovery req 3: HTTP 200
recovery req 4: HTTP 200
RECOVERY_PASS=4/4

Exactly one cached-plan warning, one reconnect, zero 26000 errors, and no recurring warnings afterward:

WARNING utils.py:3290 - PostgreSQL cached plan error detected for token lookup; recreating the database connection and retrying with the same query. ...
WARNING utils.py:4730 - Attempting Prisma DB reconnect. reason=postgres_cached_plan_error
WARNING prisma_client.py:155 - Sent SIGTERM to prisma-query-engine PID 24005 during reconnect.
query-engine ac9d7041ed77bcc8a8dbd2ab6616b39013829574
INFO:     127.0.0.1 - "GET /v1/models HTTP/1.1" 200 OK   # retried lookup succeeds on the fresh engine

Unit tests

python -m pytest tests/test_litellm/proxy/utils/prisma_and_spend/test_prisma_client_get_data.py -k cached_plan -q
# 6 passed

The cached-plan tests fail on the old DEALLOCATE ALL code (they assert attempt_db_reconnect is awaited and that DEALLOCATE ALL is never sent) and pass on this commit.

Type

🐛 Bug Fix

Changes

On litellm_internal_staging, _query_first_with_cached_plan_fallback retries the cached-plan error with a unique millisecond-timestamp comment injected into the SQL, which defeats PostgreSQL's plan cache on every retry and never repairs the poisoned connection (see Proof of Fix above). This PR removes the SQL mutation entirely. On the cached-plan error it logs a warning, reconnects via the existing singleflight attempt_db_reconnect(reason="postgres_cached_plan_error"), and retries the identical query once. Using the shared reconnect path means a schema change that poisons every pooled connection at once triggers a single client recreate that concurrent waiters reuse, rather than each request racing to kill the others' fresh engine. The retry SQL is unchanged byte-for-byte so the plan cache keeps working after recovery.

[yassin-berriai]

@greptileai

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR fixes a Prisma-client poisoning bug triggered during PostgreSQL rolling deployments: instead of issuing DEALLOCATE ALL (or injecting a unique SQL comment) to recover from a "cached plan must not change result type" error, it now recreates the Prisma client via the existing singleflight attempt_db_reconnect, which kills the engine subprocess and clears both the server-side plans and Prisma's client-side prepared-statement name cache together, then retries the identical SQL.

• litellm/proxy/utils.py – _query_first_with_cached_plan_fallback drops the SQL-mutation approach and calls attempt_db_reconnect(reason="postgres_cached_plan_error"); the docstring explains the singleflight guarantee, the plan-cache preservation, and the cooldown edge-case.
• tests/test_prisma_client_get_data.py – Tests are updated to assert call order (reconnect before retry), identical SQL on retry, no execute_raw call, failure propagation, and correct behaviour when reconnect returns False (cooldown).

Confidence Score: 5/5

Safe to merge — the change is tightly scoped to the cached-plan recovery path, builds entirely on the existing and already-tested attempt_db_reconnect infrastructure, and is backed by a live repro showing 6/6 and 4/4 success after the fix.

The core logic is correct: recreating the Prisma client atomically clears both the server-side prepared statements and the engine's client-side name cache, avoiding the connection-poisoning that DEALLOCATE ALL caused. The singleflight lock in attempt_db_reconnect prevents a thundering-herd of concurrent recreations. The cooldown edge-case is explicitly documented and covered by the backoff on get_data. Tests verify call order, SQL identity on retry, no DEALLOCATE ALL, failure propagation, and the cooldown path — no prior assertions were weakened.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/utils.py	Replaces the flawed DEALLOCATE-ALL / unique-comment retry strategy with a Prisma client recreation via the existing singleflight attempt_db_reconnect, then retries the identical SQL. Docstring fully documents cooldown edge-case and plan-cache reasoning.
tests/test_litellm/proxy/utils/prisma_and_spend/test_prisma_client_get_data.py	Tests updated and expanded to pin reconnect-then-retry semantics: identical SQL on retry, call-order verification, no DEALLOCATE ALL, failure propagation, and cooldown-skipped reconnect path.

_{Reviews (4): Last reviewed commit: "fix(proxy): reconnect Prisma client on c..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

This PR fixes a writer-CPU-peg bug during rolling deployments by replacing the mutation-based cached-plan retry (injecting a unique millisecond-timestamp comment) with a DEALLOCATE ALL followed by a byte-identical retry.

• litellm/proxy/utils.py: _query_first_with_cached_plan_fallback now issues a best-effort DEALLOCATE ALL to clear stale prepared statements on a PostgreSQL cached-plan error, then retries the original query unchanged exactly once; any failure on the retry propagates instead of looping.
• tests/…/test_prisma_client_get_data.py: Existing retry test strengthened to assert byte-for-byte query identity and correct call ordering (query_first → execute_raw → query_first); three new tests added covering retry propagation, DEALLOCATE ALL failure resilience, and non-plan-error pass-through.

Confidence Score: 4/5

Safe to merge; the new retry path terminates cleanly in all cases and cannot reproduce the CPU-peg that motivated the fix.

The core logic change is correct and well-tested. The one limitation — that DEALLOCATE ALL and the retry query_first are dispatched independently into Prisma's connection pool and may land on different connections — means the stale plan might not always be cleared on the first attempt. The backoff decorator on get_data will then trigger up to three additional rounds, each of which also terminates cleanly, so there is no CPU peg or infinite loop. The PR description acknowledges this as best-effort, but the docstring does not, which could surprise future maintainers.

The implementation in litellm/proxy/utils.py (lines 3278–3285) is the only area worth a second look, specifically the connection-affinity assumption between DEALLOCATE ALL and the retry.

Important Files Changed

Filename	Overview
litellm/proxy/utils.py	Replaces the mutation-based cached-plan retry (unique timestamp comment) with a best-effort DEALLOCATE ALL followed by a byte-identical retry; correctly eliminates the CPU-pinning loop while introducing a best-effort connection-affinity limitation.
tests/test_litellm/proxy/utils/prisma_and_spend/test_prisma_client_get_data.py	Tests strengthened: renamed and rewrote the retry test to assert byte-identical query reuse; added coverage for retry propagation, DEALLOCATE ALL failure resilience, and call ordering via MagicMock manager. No tests weakened or real network calls added.

_{Reviews (2): Last reviewed commit: "fix(proxy): don't mutate SQL on cached-p..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 80.00000% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/utils.py	80.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yassin-berriai]

Thanks for the review. Addressed the one nuance you flagged: the docstring now states that DEALLOCATE ALL is best-effort, that Prisma may dispatch it and the retry onto different pooled connections, and that the get_data backoff envelope is what guarantees clean termination if the connection-affinity race happens. The behavior is unchanged; this is documentation so a future maintainer isn't surprised by the pooling caveat.

@greptileai

[greptile-apps]

The updated docstring covers all the right nuances: DEALLOCATE ALL is best-effort, Prisma can route it and the retry to different pooled connections, and the get_data backoff envelope is what guarantees termination if the connection-affinity race occurs. That's exactly the context a future maintainer needs to understand the design without being surprised by the pooling caveat.

LGTM.

[yassin-berriai]

@greptileai please re-review the latest commit (863b622). The only item behind the 4/5 was that the docstring did not mention the best-effort connection-affinity caveat; that is now documented, and you confirmed LGTM. Could you refresh the confidence score on the current head

[yassin-berriai]

@greptileai