fix(proxy): recover Prisma DB reconnect loop when client is disconnected by yassin-berriai · Pull Request #32323 · BerriAI/litellm

yassin-berriai · 2026-07-07T06:40:14Z

Relevant issues

Fixes #28322

Linear ticket

Resolves LIT-4161

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

The wedged state from issue #28322 is a Prisma client that is disconnected from its query engine; production pods reach it through rare shutdown/cancellation races after days of uptime (the issue logs show 1486+ consecutive reconnect failures over 30 days with zero recoveries). To reproduce it on demand against a live proxy, I used a custom callback as a fault injector that calls the real disconnect() on the active client when a magic message arrives, which puts the proxy into byte-for-byte the same state (every DB call raises ClientNotConnectedError)

Fault injector loaded via litellm_settings.callbacks:

from litellm.integrations.custom_logger import CustomLogger

class DBWedgeInjector(CustomLogger):
    async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
        messages = data.get("messages") or []
        if any("INJECT_DB_DISCONNECT" in str(m.get("content", "")) for m in messages):
            from litellm.proxy.proxy_server import prisma_client
            await prisma_client.db._original_prisma.disconnect()
        return data

db_wedge_injector = DBWedgeInjector()

Live proxy on localhost:4161 with a real Postgres container and real OpenAI calls; watchdog tuned for a fast demo loop (PRISMA_HEALTH_WATCHDOG_INTERVAL_SECONDS=5, PRISMA_RECONNECT_COOLDOWN_SECONDS=2)

Before (unfixed)

# baseline: virtual key works end to end
$ curl -s http://localhost:4161/v1/chat/completions -H "Authorization: Bearer $K1" \
    -H "Content-Type: application/json" \
    -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say OK"}]}'
OK!

# wedge the proxy (same trigger for before and after runs)
$ curl -s http://localhost:4161/v1/chat/completions -H "Authorization: Bearer sk-...master" \
    -H "Content-Type: application/json" \
    -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"INJECT_DB_DISCONNECT"}]}'

# a fresh (uncached) virtual key is now dead
$ curl -s -w "\nHTTP %{http_code}\n" http://localhost:4161/v1/chat/completions -H "Authorization: Bearer $K2" \
    -H "Content-Type: application/json" \
    -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say OK"}]}'
{"error":{"message":"Service Unavailable, the authentication database is temporarily unreachable. Please retry shortly.","type":"no_db_connection","param":"None","code":"503"}}
HTTP 503

Proxy log: the reconnect loop fails forever with the exact error signature from the issue, and never recovers (25 consecutive failures and climbing when I stopped the run; reconnect succeeded count stays 0)

09:25:36 WARNING utils.py - Escalating to heavy reconnect after 8 consecutive failures. reason=db_health_watchdog_connection_error
09:25:36 ERROR utils.py - Prisma DB reconnect failed (9 consecutive). reason=db_health_watchdog_connection_error error=Client is not connected to the query engine, you must call `connect()` before attempting to query data.
09:25:43 WARNING utils.py - Escalating to heavy reconnect after 10 consecutive failures. reason=auth_get_key_object_lookup_failure
09:25:43 ERROR utils.py - Prisma DB reconnect failed (11 consecutive). reason=auth_get_key_object_lookup_failure error=Client is not connected to the query engine, you must call `connect()` before attempting to query data.
...
09:26:51 ERROR utils.py - Prisma DB reconnect failed (25 consecutive). reason=db_health_watchdog_connection_error error=Client is not connected to the query engine, you must call `connect()` before attempting to query data.

$ grep -c "reconnect failed" before.log ; grep -c "reconnect succeeded" before.log
25
0

After (this fix)

Same trigger, same environment. The engine watcher detects the death from the disconnect and the very first reconnect attempt recovers:

09:27:32 INFO utils.py - Watching engine PID 94464 via waitpid thread.
09:27:38 WARNING utils.py - Attempting Prisma DB reconnect. reason=engine_process_death
09:27:38 INFO utils.py - Watching engine PID 94485 via waitpid thread.
09:27:38 INFO utils.py - Prisma DB reconnect succeeded. reason=engine_process_death

$ grep -c "reconnect failed" after.log ; grep -c "reconnect succeeded" after.log
0
1

# the fresh virtual key works again without restarting the pod
$ curl -s -w "\nHTTP %{http_code}\n" http://localhost:4161/v1/chat/completions -H "Authorization: Bearer $K2" \
    -H "Content-Type: application/json" \
    -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say OK"}]}'
{"id":"chatcmpl-Dyta0nJQYD6iUZxr5gnKPNBDl8QqZ", ... "content":"OK!" ...}
HTTP 200

I also verified on a live proxy that the two adjacent scenarios still self-heal with this change: killing the query engine with the DB up (recovers in one cycle) and killing it during a docker pause DB outage (recovers on the first cycle after docker unpause)

Independent e2e verification (Devin)

A separate Devin session independently reproduced and verified the fix end to end on its own environment (fresh clone, real Postgres 16, same fault injector, same config for both runs; only the two guarded files differ). On the fixed branch the wedge trigger leads to Prisma DB reconnect succeeded within one watchdog cycle (~10 seconds) with zero reconnect failed lines, and GET /key/info with a virtual key returns HTTP 200 afterwards. On the unfixed merge-base (5b93ba0ada) the same trigger loops Prisma DB reconnect failed (N consecutive) ... Client is not connected to the query engine climbing to 27 on camera with zero recoveries, GET /key/info returns HTTP 500 and the user-facing chat completion returns HTTP 503 no_db_connection. Screen recording of both runs:

(video attached below)

Type

🐛 Bug Fix

Changes

Once the active Prisma client is in the disconnected state, every DB call raises prisma.errors.ClientNotConnectedError. The reconnect machinery is the only way out of that state, but both PrismaWrapper._get_engine_pid (litellm/proxy/db/prisma_client.py) and PrismaClient._get_engine_pid (litellm/proxy/utils.py) inspected the broken client through prisma's _engine property, which re-raises that same ClientNotConnectedError, and they only caught (AttributeError, TypeError). So recreate_prisma_client failed at the very first line, before it could construct a replacement client, and every subsequent attempt (health watchdog, auth-triggered reconnect, IAM token refresh) died the same way; the pod never recovered without a restart

The fix guards both implementations with is_connected() so a disconnected client reads as "no engine subprocess" (pid 0); the recreate path then skips the kill step and proceeds to construct and connect a fresh client, which is exactly what recovery requires. All reconnect entry points funnel through these two call sites, so the guard covers the watchdog, the auth path, the IAM refresh path and the read replica recreate for free. The guard is safe against a torn read because prisma's _engine property only raises when the internal engine is None, which is exactly when is_connected() returns False, and there is no await between the check and the access

Provenance: the narrow except (AttributeError, TypeError) dates back to the engine zombie-recovery work (#21899 and commit 1f04fa2). The reconnect path that most easily produced a disconnected client in production (a disconnect() then connect() sequence, where a failed connect left the client permanently disconnected) was later replaced by the kill-then-construct flow in commit fbcdacc, which is why recent builds rarely enter the state; the recovery gap itself stayed latent until now. Reports on issue #28322 span builds that still had the old trigger

Regression tests (all four fail on the unfixed code and pass with the fix):

test_get_engine_pid_returns_zero_for_disconnected_client and test_recreate_prisma_client_recovers_from_disconnected_client in tests/test_litellm/proxy/db/test_prisma_client.py
test_heavy_reconnect_recovers_from_disconnected_prisma_client in tests/test_litellm/proxy/db/test_prisma_self_heal.py, which drives the full real path (attempt_db_reconnect -> _run_reconnect_cycle -> recreate_prisma_client) from the wedged state
test_get_engine_pid_returns_zero_when_client_disconnected in tests/test_litellm/proxy/utils/prisma_and_spend/test_prisma_client_engine_watcher.py

Two existing tests were updated to declare is_connected() -> True on their mocked clients, matching the state they were already simulating

CLAassistant · 2026-07-07T06:40:21Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

yassin-berriai · 2026-07-07T06:40:28Z

@greptileai

greptile-apps · 2026-07-07T06:44:02Z

Greptile Summary

This PR fixes a production reconnect deadlock where a Prisma client wedged in the disconnected state caused every reconnect attempt to fail. Both PrismaWrapper._get_engine_pid() and PrismaClient._get_engine_pid() previously inspected the _engine property directly, which itself raises ClientNotConnectedError on a disconnected client, so recreate_prisma_client could never build a replacement.

Guards both _get_engine_pid() implementations with an is_connected() check so a disconnected client reads as "no engine" (pid 0) and the kill step is skipped, allowing the recreate path to proceed unobstructed.
Switches PrismaClient._get_engine_pid() from self.db._original_prisma to self.writer_db._original_prisma, correctly routing through the underlying PrismaWrapper when read-replica routing is active.
Centralises the DisconnectedPrisma test stub into the proxy conftest and adds four regression tests (three unit, one end-to-end) that fail on unfixed code and pass with the fix.

Confidence Score: 5/5

Safe to merge; the change is narrowly scoped to the engine-PID inspection path and every reconnect entry point is covered by the guard.

The fix is correct and minimal: a two-line is_connected() guard in both _get_engine_pid() implementations prevents an exception from blocking the only recovery path. The writer_db change is an independent improvement that fixes PID lookup when read-replica routing is active. All existing tests were correctly updated to declare is_connected() → True, four new regression tests exercise the disconnected scenario, and the PR description provides a clear before/after reproduction trace.

The pre-existing test_get_engine_pid_returns_zero_when_engine_attr_missing test in test_prisma_client_engine_watcher.py now passes via the is_connected() exception handler rather than via the missing-_engine path its name describes; a minor fixture tweak would restore accurate coverage.

Important Files Changed

Filename	Overview
litellm/proxy/db/prisma_client.py	Adds `is_connected()` guard in `PrismaWrapper._get_engine_pid()` to prevent `ClientNotConnectedError` from escaping into the reconnect path when the client is disconnected.
litellm/proxy/utils.py	Adds `is_connected()` guard in `PrismaClient._get_engine_pid()` and switches from `self.db._original_prisma` to `self.writer_db._original_prisma`, correctly routing through the writer wrapper even when read-replica routing is active.
tests/test_litellm/proxy/conftest.py	Centralises `StubClientNotConnectedError`, `DisconnectedPrisma`, and the `disconnected_prisma` fixture in the shared proxy conftest, replacing the previous copy-paste across individual test files.
tests/test_litellm/proxy/db/test_prisma_client.py	Adds `is_connected = MagicMock(return_value=True)` to three existing tests that simulate connected clients, and adds two new regression tests covering `_get_engine_pid()` and `recreate_prisma_client()` when the client is in the disconnected state.
tests/test_litellm/proxy/db/test_prisma_planned_engine_restart.py	Adds `is_connected = MagicMock(return_value=True)` to the `_make_wrapper` helper to keep existing tests consistent with the new guard.
tests/test_litellm/proxy/db/test_prisma_self_heal.py	Fixes `mock_prisma_binary` to yield the module (enabling new test to configure `Prisma.return_value`), and adds an end-to-end regression test driving the full `attempt_db_reconnect → _run_reconnect_cycle → recreate_prisma_client` path from a wedged disconnected state.
tests/test_litellm/proxy/utils/prisma_and_spend/test_prisma_client_engine_watcher.py	Adds `is_connected = MagicMock(return_value=True)` to the PID-extraction test and a new regression test for the disconnected case; the pre-existing `test_get_engine_pid_returns_zero_when_engine_attr_missing` test now passes via a different code path than its name suggests.

_{Reviews (5): Last reviewed commit: "fix(proxy): recover Prisma DB reconnect ..." | Re-trigger Greptile}

yassin-berriai · 2026-07-07T06:46:02Z

@greptileai please review the current head 05935a4. It addresses the P2 (both remaining tests now declare is_connected() -> True) and removes issue references from docstrings

greptile-apps · 2026-07-07T06:47:47Z

Greptile Summary

This PR fixes a persistent reconnect-loop deadlock where Prisma pods could never self-heal after the active client entered the disconnected state. Both PrismaWrapper._get_engine_pid and PrismaClient._get_engine_pid now short-circuit to 0 via an is_connected() pre-check before touching the _engine property, which would otherwise raise ClientNotConnectedError and abort every reconnect attempt before a replacement client could be built.

Core fix (prisma_client.py, utils.py): is_connected() is not True guard added before _engine access in both _get_engine_pid implementations; utils.py also corrects self.db._original_prisma → self.writer_db._original_prisma so read-replica deployments (RoutingPrismaWrapper) are handled correctly.
Tests: Four new regression tests cover the wedged state at each layer — PrismaWrapper, PrismaClient, and the full attempt_db_reconnect path — all of which fail on the unfixed code and pass with the fix.
Existing test updates: Two mocks gain an explicit is_connected = MagicMock(return_value=True) to accurately reflect the connected state they were already simulating; this does not weaken any existing assertion.

Confidence Score: 4/5

Safe to merge — the fix is minimal, targets exactly the broken call sites, and is backed by four regression tests that reproduce the wedged state end-to-end.

The production change is a single defensive early-return in two symmetrical _get_engine_pid methods, with no new async paths or state mutations. The self.db → self.writer_db correction in utils.py properly handles read-replica deployments. All four regression tests fail on the unfixed code and pass after. The only non-critical observation is _StubClientNotConnectedError/_DisconnectedPrisma duplicated verbatim across three test files, which is purely a maintainability concern.

No files require special attention; litellm/proxy/utils.py is worth a quick second look to confirm that self.writer_db._original_prisma is the right access point in all reconnect scenarios, including read-replica deployments.

Important Files Changed

Filename	Overview
litellm/proxy/db/prisma_client.py	Guards `_get_engine_pid` with an `is_connected()` pre-check so a disconnected client returns 0 instead of raising `ClientNotConnectedError`, unblocking the reconnect path.
litellm/proxy/utils.py	Mirrors the `is_connected()` guard from `PrismaWrapper` and also corrects `self.db._original_prisma` to `self.writer_db._original_prisma`, which properly unwraps through `RoutingPrismaWrapper` in read-replica deployments.
tests/test_litellm/proxy/db/test_prisma_client.py	Adds regression test for disconnected client returning 0 from `_get_engine_pid`, and an end-to-end test that `recreate_prisma_client` succeeds from the wedged state. Existing mock updated to declare `is_connected() → True` accurately.
tests/test_litellm/proxy/db/test_prisma_planned_engine_restart.py	Adds `is_connected = MagicMock(return_value=True)` to existing mock to satisfy the new `is_connected()` pre-check; does not weaken existing assertions.
tests/test_litellm/proxy/db/test_prisma_self_heal.py	Fixture updated to yield the mock module (needed for the new test), and adds a full reconnect-path regression test from the wedged disconnected state (`attempt_db_reconnect → recreate_prisma_client`).
tests/test_litellm/proxy/utils/prisma_and_spend/test_prisma_client_engine_watcher.py	Adds regression test for `PrismaClient._get_engine_pid` returning 0 on a disconnected client; existing mock updated to declare `is_connected() → True` accurately.

_{Reviews (2): Last reviewed commit: "fix(proxy): recover Prisma DB reconnect ..." | Re-trigger Greptile}

greptile-apps · 2026-07-07T06:47:50Z

    mock_kill.assert_not_called()  # PID was 0, kill skipped
    mock_new_prisma.connect.assert_awaited_once()
+
+
+class _StubClientNotConnectedError(Exception):
+    pass
+
+
+class _DisconnectedPrisma:
+    """Mimics prisma-client-py after disconnect(): ``is_connected()`` is False
+    and the ``_engine`` property raises ``ClientNotConnectedError``."""
+
+    def is_connected(self) -> bool:
+        return False
+
+    @property
+    def _engine(self) -> None:
+        raise _StubClientNotConnectedError(
+            "Client is not connected to the query engine, you must call `connect()` "


Duplicated test stub across three files

_StubClientNotConnectedError and _DisconnectedPrisma are copy-pasted verbatim into test_prisma_client.py, test_prisma_self_heal.py, and test_prisma_client_engine_watcher.py. Consider extracting them into a shared tests/test_litellm/proxy/db/conftest.py or a _test_helpers.py module so a future change to the stub only needs to happen in one place.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Once the active Prisma client is in the disconnected state, every DB call raises ClientNotConnectedError. The reconnect machinery was supposed to recover from this, but _get_engine_pid() inspected the broken client via prisma's _engine property, which re-raises that same error, so recreate_prisma_client failed before it could build a replacement client and the proxy looped on failed reconnects forever (issue #28322 showed 1486+ consecutive failures over 30 days with zero recoveries) Guard both _get_engine_pid implementations with is_connected() so a disconnected client reads as "no engine" (pid 0) and the recreate path proceeds to construct and connect a fresh client

yassin-berriai · 2026-07-07T06:51:22Z

@greptileai please review the current head 8417b96. It deduplicates the test stub into tests/test_litellm/proxy/conftest.py as a shared disconnected_prisma fixture, addressing the maintainability observation

codecov · 2026-07-07T06:56:36Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

yassin-berriai · 2026-07-07T07:42:27Z

@greptileai review head 8417b96

yassin-berriai force-pushed the litellm_fix_prisma_reconnect_disconnected_client branch 2 times, most recently from 8d15e6f to 05935a4 Compare July 7, 2026 06:45

greptile-apps Bot reviewed Jul 7, 2026

View reviewed changes

yassin-berriai force-pushed the litellm_fix_prisma_reconnect_disconnected_client branch from 05935a4 to 8417b96 Compare July 7, 2026 06:50

yuneng-berri approved these changes Jul 7, 2026

View reviewed changes

yuneng-berri merged commit db60ce9 into litellm_internal_staging Jul 7, 2026
125 checks passed

yuneng-berri deleted the litellm_fix_prisma_reconnect_disconnected_client branch July 7, 2026 16:11

Mattchewone mentioned this pull request Jul 20, 2026

chore: sync upstream v1.93.0 bitovi/litellm#14

Merged

Navigation Menu

Uh oh!

Uh oh!

fix(proxy): recover Prisma DB reconnect loop when client is disconnected - #32323

fix(proxy): recover Prisma DB reconnect loop when client is disconnected#32323
yuneng-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_fix_prisma_reconnect_disconnected_client

yassin-berriai commented Jul 7, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Jul 7, 2026

Uh oh!

yassin-berriai commented Jul 7, 2026

Uh oh!

greptile-apps Bot commented Jul 7, 2026 •

edited

Loading

Important Files Changed

Uh oh!

yassin-berriai commented Jul 7, 2026

Uh oh!

greptile-apps Bot commented Jul 7, 2026

Important Files Changed

Uh oh!

greptile-apps Bot Jul 7, 2026

Uh oh!

yassin-berriai commented Jul 7, 2026

Uh oh!

codecov Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

yassin-berriai commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

yassin-berriai commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (unfixed)

After (this fix)

Independent e2e verification (Devin)

Type

Changes

Uh oh!

CLAassistant commented Jul 7, 2026

Uh oh!

yassin-berriai commented Jul 7, 2026

Uh oh!

greptile-apps Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

yassin-berriai commented Jul 7, 2026

Uh oh!

greptile-apps Bot commented Jul 7, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Uh oh!

greptile-apps Bot Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

yassin-berriai commented Jul 7, 2026

Uh oh!

codecov Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yassin-berriai commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yassin-berriai commented Jul 7, 2026 •

edited

Loading

greptile-apps Bot commented Jul 7, 2026 •

edited

Loading

codecov Bot commented Jul 7, 2026 •

edited

Loading