fix(proxy): prevent Prisma engine orphan leak on shutdown disconnect failure (#26619)

[pike00]

Relevant issues

Fixes #26619

Problem

When a proxy worker recycles — Gunicorn --max_requests_before_restart, uvicorn worker crash, deploy churn — the Prisma disconnect() is awaited inside the uvicorn lifespan. If it fails, the exception propagates up the lifespan, uvicorn skips the Python-level atexit/SIGTERM cleanup, and the Prisma query-engine subprocess is SIGKILL'd without closing its Postgres connections. Without TCP keepalives on DATABASE_URL, those dead sessions sit on the DB side for the OS keepalive default (~7200s) and accumulate toward max_connections until the proxy can no longer write to the DB.

I hit this in a self-hosted deployment twice in 7 days (47 orphan Prisma engine processes spawned in a single worker-recycle burst, ~188 leaked TCP sessions, FATAL: sorry, too many clients already). The traced root cause in the upstream issue matched exactly.

Approach

Three surgical patches scoped to #26619 specifically — not the broader "rip out Prisma" effort in #28366 / #28404, which are the right long-term fix but months out.

1. litellm/proxy/proxy_server.py — wrap the await proxy_shutdown_event() call inside proxy_startup_event (the uvicorn lifespan) in try/except. A failed shutdown event must not prevent normal lifespan exit; subprocess termination depends on reaching atexit/SIGTERM cleanly.

2. litellm/proxy/utils.py — stop re-raising from PrismaClient.disconnect(). The existing logging + failure_handler task already records the failure; disconnect() is called only from shutdown paths, no caller benefits from the re-raise, and the lifespan is actively harmed by it.

3. litellm/proxy/proxy_cli.py — emit libpq TCP keepalive defaults (keepalives=1, keepalives_idle=60, keepalives_interval=10, keepalives_count=3) from _build_db_connection_url_params. Kernel reaps dead Postgres sessions in ~90s instead of ~7200s, so even when an engine subprocess is killed before its session closes cleanly, the leak self-bounds. Operators who don't want the defaults can override via extra_params since extra_params.update runs last.

This is symmetric: the library no longer creates orphans (Patches 1+2), and the deployment no longer holds dead connections for hours when an orphan does slip through (Patch 3). Operators get protection without changes to their Compose/Helm files.

I verified the libpq keepalive params are accepted by Prisma's PostgreSQL connector in practice — I've been running them via DATABASE_URL appendage in my homelab deployment as a workaround for the last day, with no driver-level rejection. They flow through as query-string params to the underlying Rust postgres driver.

Tests

Added in tests/test_litellm/ and tests/proxy_unit_tests/:

• test_proxy_lifespan_swallows_shutdown_event_exception (tests/proxy_unit_tests/test_aproxy_startup.py, alongside the existing related test) — drives the full lifespan via proxy_startup_event(app=None) with proxy_shutdown_event monkeypatched to raise RuntimeError. Pins that the exception does not propagate out of the __aexit__ and that the failure is logged. Verified the test fails without Patch 1.

• test_prisma_disconnect_does_not_reraise_on_failure (tests/test_litellm/proxy/test_proxy_server.py) — constructs a stub PrismaClient (via __new__ to bypass the prisma-binary import) with self.db.disconnect raising. Pins that await client.disconnect() returns normally and that failure_handler is scheduled with call_type="disconnect".

• test_build_db_connection_url_params_includes_keepalives and test_build_db_connection_url_params_keepalives_overridable_via_extras (tests/test_litellm/proxy/test_proxy_cli.py) — pin the new keepalive defaults and the extras-wins override path.

The existing test_build_db_connection_url_params_defaults equality assertion was updated to include the new keepalive keys (the only existing test broken by Patch 3).

Local run:

uv run pytest tests/proxy_unit_tests/test_aproxy_startup.py tests/test_litellm/proxy/test_proxy_server.py tests/test_litellm/proxy/test_proxy_cli.py
# 223 passed

Pre-Submission checklist

• I have Added testing in the tests/test_litellm/ directory (test_prisma_disconnect_does_not_reraise_on_failure, test_build_db_connection_url_params_includes_keepalives, test_build_db_connection_url_params_keepalives_overridable_via_extras)
• My PR passes all unit tests on make test-unit for the touched files (223 passed in test_litellm/proxy/test_proxy_server.py + test_litellm/proxy/test_proxy_cli.py + proxy_unit_tests/test_aproxy_startup.py)
• My PR's scope is as isolated as possible, it only solves 1 specific problem (the orphan-engine leak in [Bug]: Gunicorn worker recycling via --max_requests_before_restart leaks Postgres connections — no keepalive configured, orphaned connections held for ~2 hours by Cloud SQL #26619 — no rip-and-replace, no feature flag, no neighbor cleanups)
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 — will request immediately after opening

Type

🐛 Bug Fix

Changes

• litellm/proxy/proxy_server.py — wrap await proxy_shutdown_event() in try/except
• litellm/proxy/utils.py — PrismaClient.disconnect() no longer re-raises
• litellm/proxy/proxy_cli.py — libpq keepalive defaults in _build_db_connection_url_params
• tests/proxy_unit_tests/test_aproxy_startup.py — lifespan-shutdown regression test
• tests/test_litellm/proxy/test_proxy_server.py — PrismaClient.disconnect no-reraise regression test
• tests/test_litellm/proxy/test_proxy_cli.py — keepalive defaults + override tests

[CLAassistant]

All committers have signed the CLA.

[pike00]

@greptileai

[codspeed-hq]

Congrats! CodSpeed is installed 🎉

🆕 16 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

• test_cost_per_token_anthropic: 558.9 µs
• test_cost_per_token_openai: 558.3 µs
• test_get_llm_provider_anthropic: 142.4 µs
• test_get_llm_provider_azure: 144.5 µs
• test_get_llm_provider_openai: 138.8 µs
• test_token_counter_long_content: 1.7 ms
• test_get_llm_provider_with_prefix: 137.4 µs
• test_token_counter_multi_turn: 565.6 µs
• test_get_model_cost_key_case_insensitive: 82.4 µs
• test_token_counter_raw_text: 184.3 µs
• test_get_model_cost_key_exact_match: 80 µs
• test_token_counter_simple_message: 235.6 µs
• test_get_model_info_anthropic: 79.2 µs
• test_token_counter_with_tools: 412.2 µs
• test_get_model_info_openai: 82.7 µs
• test_get_model_info_with_provider: 80.3 µs

---

[greptile-apps]

Greptile Summary

Three targeted patches that prevent Prisma engine subprocess orphans on proxy worker shutdown: the uvicorn lifespan now catches proxy_shutdown_event failures instead of propagating them, PrismaClient.disconnect() no longer re-raises (and its now-dead @backoff decorator is removed), and libpq TCP keepalive defaults are injected into DATABASE_URL/DIRECT_URL at startup so the kernel reaps dead Postgres sessions in ~90 s instead of ~7200 s.

• proxy_server.py + utils.py: Shutdown failures are logged and swallowed rather than allowed to abort the lifespan, ensuring uvicorn's normal atexit/SIGTERM path is always reached and the Prisma engine process exits cleanly.
• proxy_cli.py: Keepalive defaults are applied only when not already present in the existing URL query string, and extra_params still wins last — operator-supplied values are fully preserved at all precedence layers.
• Tests: Six focused regression tests cover the lifespan swallow, the disconnect() no-reraise path, keepalive default emission, extra_params override, and existing-URL preservation; the modified test_build_db_connection_url_params_defaults assertion was correctly updated to strict-equality-match the new output.

Confidence Score: 5/5

Safe to merge — changes are confined to shutdown paths and startup URL construction, with no impact on the hot request path.

All three patches are scoped to shutdown/startup code that runs outside the hot request path. The previously raised concerns about the dead @backoff decorator and keepalive defaults overwriting existing URL values have both been fully addressed in this revision: the decorator is removed, and existing_url parsing ensures defaults are skipped for keys already present in the URL. Test coverage is concrete and well-targeted.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/utils.py	Removes @backoff decorator (now dead code) and swallows disconnect() exceptions instead of re-raising; well-commented with issue reference
litellm/proxy/proxy_server.py	Wraps proxy_shutdown_event() in try/except inside lifespan so a disconnect failure cannot orphan the Prisma engine subprocess
litellm/proxy/proxy_cli.py	Adds libpq keepalive defaults; correctly skips defaults that are already present in the existing DATABASE_URL/DIRECT_URL query string
tests/proxy_unit_tests/test_aproxy_startup.py	New regression test drives the full lifespan with a raising proxy_shutdown_event and pins that the exception is caught and logged
tests/test_litellm/proxy/test_proxy_server.py	New regression test verifies PrismaClient.disconnect() swallows errors and schedules failure_handler via create_task
tests/test_litellm/proxy/test_proxy_cli.py	Existing defaults test updated (strict equality now includes keepalive keys); three new tests cover default emission, extra_params override, and existing-URL preservation

_{Reviews (4): Last reviewed commit: "fix(proxy): prevent Prisma engine orphan..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 70.58824% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/proxy_cli.py	76.92%	3 Missing ⚠️
litellm/proxy/proxy_server.py	50.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

Three surgical patches address Prisma engine orphan leaks under worker churn (#26619): the uvicorn lifespan now wraps proxy_shutdown_event in a try/except so a failed disconnect cannot abort lifespan exit, PrismaClient.disconnect() stops re-raising its exception, and libpq TCP keepalive defaults are injected into the DATABASE_URL so dead Postgres sessions self-bound in ~90 s instead of ~2 h.

• proxy_server.py + utils.py: Two complementary guards ensure a Prisma disconnect failure never propagates into the lifespan — preventing orphaned engine subprocesses and the too many clients cascade.
• proxy_cli.py: Keepalive defaults (keepalives=1, idle=60s, interval=10s, count=3) are appended to both DATABASE_URL and DIRECT_URL; operators can override via database_extra_connection_params in general settings.
• Tests: Dedicated regression tests cover all three patches — lifespan swallowing, no-reraise, and keepalive defaults/override — and the one existing test that changed was correctly updated to reflect the new expected output.

Confidence Score: 4/5

Safe to merge — the core fixes correctly break the exception-propagation chain that caused orphan Prisma engine subprocesses, and all three patches are backed by targeted regression tests.

The proxy_server and utils changes are minimal and well-guarded. The keepalive injection in proxy_cli correctly wires up the defaults, but the merge order in append_query_params means any keepalive settings already present in a DATABASE_URL query string are silently overwritten by the new defaults rather than respected — operators relying on URL-embedded params would need to move those settings to database_extra_connection_params without being told to do so.

litellm/proxy/proxy_cli.py — the keepalive default merge order relative to DATABASE_URL-embedded params deserves a second look.

Important Files Changed

Filename	Overview
litellm/proxy/proxy_server.py	Wraps proxy_shutdown_event() in a try/except inside the uvicorn lifespan so a failed shutdown does not abort lifespan exit and orphan the Prisma engine subprocess.
litellm/proxy/utils.py	Removes raise e from PrismaClient.disconnect() so a disconnect failure is logged and reported via failure_handler without propagating up to abort the lifespan.
litellm/proxy/proxy_cli.py	Adds libpq keepalive defaults to _build_db_connection_url_params; these win over any keepalive settings already embedded in DATABASE_URL because append_query_params calls parsed_query.update(params).
tests/proxy_unit_tests/test_aproxy_startup.py	Adds regression test that drives the full lifespan with proxy_shutdown_event monkeypatched to raise, verifying the exception does not propagate.
tests/test_litellm/proxy/test_proxy_server.py	Adds test_prisma_disconnect_does_not_reraise_on_failure using __new__ to bypass the prisma binary import; correctly pins swallow-and-log behavior and verifies failure_handler is scheduled.
tests/test_litellm/proxy/test_proxy_cli.py	Updates the existing defaults equality test to include keepalive keys; adds two new tests for keepalive presence and override-via-extras. Test modification is appropriate — it reflects the new expected behavior.

Comments Outside Diff (2)

1. litellm/proxy/proxy_cli.py, line 66-72 (link)

   Keepalive defaults silently override DATABASE_URL keepalive params

   In append_query_params, parsed_query.update(params) causes the newly-added defaults to overwrite keepalive values already embedded in the DATABASE_URL query string. An operator who set ?keepalives_idle=30 directly in DATABASE_URL would see it silently replaced with 60. The documented override path requires moving the setting into database_extra_connection_params in the config YAML — which operators relying on URL-embedded params may not know to do. Reversing the merge so DATABASE_URL params win over the defaults (while still letting extra_params take highest precedence) would make the precedence order consistent with the "operators can opt out" guarantee.

2. litellm/proxy/proxy_server.py, line 978-980 (link)

   Redundant {e} in .exception() call

   verbose_proxy_logger.exception(...) already appends the exception message and traceback automatically. The {e} in the f-string duplicates the message in the log output. Consider using a plain string to avoid the duplication.

_{Reviews (1): Last reviewed commit: "fix(proxy): prevent Prisma engine orphan..." | Re-trigger Greptile}

[pike00]

@greptileai re-review please — addressed both P1s:

1. Removed the now-inert @backoff.on_exception decorator on PrismaClient.disconnect(). Since the exception is swallowed in-body, backoff would never see a raise; the decorator was misleading future readers about retry behavior.
2. Keepalive defaults are now skipped for any key already present in the existing DATABASE_URL — operators who set keepalives=0 (or any other explicit value) in their connection string keep it. Added existing_url parameter to _build_db_connection_url_params and threaded the DATABASE_URL / DIRECT_URL into the per-URL build. New test test_build_db_connection_url_params_respects_existing_url_keepalives pins this behavior.

[pike00]

Also dropped the redundant {e} in the verbose_proxy_logger.exception(...) call — .exception() already attaches the exception/traceback. Force-pushed; ready for re-review.

[pike00]

@greptileai

[pike00]

A note on the codecov/patch failure (70.59%):

This appears to be a codecov false negative driven by flag_management.default_rules.carryforward: true in codecov.yaml. The lines codecov flags as MISS are all on code paths that the tests in this PR directly execute, and CI confirms the tests ran and passed:

litellm/proxy/proxy_server.py:974-975 — try: and await proxy_shutdown_event(). These are physically impossible to skip if except on L976 is reached, and L976+977 ARE marked HIT by codecov. The happy-path test test_proxy_lifespan_runs_shutdown_event_on_clean_exit runs proxy_startup_event to completion with shutdown mocked to a no-op, which exercises L974+L975 exactly. CI run 26369065917 proxy-infra / Run tests log:

tests/test_litellm/proxy/test_proxy_server.py::test_proxy_lifespan_runs_shutdown_event_on_clean_exit
[gw0] [ 5%] PASSED tests/test_litellm/proxy/test_proxy_server.py::test_proxy_lifespan_runs_shutdown_event_on_clean_exit

Local coverage run on these two tests confirms zero missing lines in the 970-979 range.

litellm/proxy/proxy_cli.py:71-89 — the body of _build_db_connection_url_params. Every passing test in TestProxyInitializationHelpers::test_build_db_connection_url_params_* (8 of them) calls this function directly and asserts on its output. Same CI run log:

tests/test_litellm/proxy/test_proxy_cli.py::TestProxyInitializationHelpers::test_build_db_connection_url_params_defaults
[gw0] [ 28%] PASSED ...
tests/test_litellm/proxy/test_proxy_cli.py::TestProxyInitializationHelpers::test_build_db_connection_url_params_includes_keepalives
[gw0] [ 28%] PASSED ...
tests/test_litellm/proxy/test_proxy_cli.py::TestProxyInitializationHelpers::test_build_db_connection_url_params_respects_existing_url_keepalives
[gw0] [ 28%] PASSED ...

The codecov proxy-infra flag upload for this commit completed successfully (proxy-infra / Upload coverage to Codecov PASSED), but the codecov compare API is returning per-line coverage that doesn't match the actual uploaded coverage.xml. The carryforward: true config means a stale flag from a sibling test partition (which never touched these lines) wins over the fresh proxy-infra upload that does. Greptile gave 5/5 on this PR and the codecov-uploaded sessions show coverage went UP at the totals level (hits: 139195 → 139536) — the per-file patch breakdown just doesn't reflect that.

Happy to retry codecov, push a no-op commit to refresh, or scope the carryforward in codecov.yaml if maintainers prefer — let me know.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.