fix(proxy): recover from prisma-query-engine zombie process

[hcavarsan]

Relevant issues

Related to the All connection attempts failed issues reported after upgrading to v1.81.x (related: #15536 #15585 #20427).

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🐛 Bug Fix

Changes

prisma-query-engine can silently die after heavy queries (like /tag/list on large datasets hitting OOM), and when it does every request that touches the DB returns All connection attempts failed and 401 for anything that queries the db.

This PR extends the connection self-heal watchdog from #21706 with engine-process death detection. At startup the engine PID is discovered and watched via pidfd_open (Linux 5.3+, event-driven) or /proc/<pid>/stat polling every 2s as fallback. On death, _run_reconnect_cycle calls recreate_prisma_client to spawn a fresh engine subprocess and re-arms the watcher. If the engine is alive or unknown the existing lightweight disconnect → connect → SELECT 1 path from #21706 is used instead.

Unit tests cover engine liveness detection, process disappearance and zombie state triggering reconnect, pidfd event while lock is held, heavy vs lightweight reconnect branching, missing DATABASE_URL, and watcher lifecycle.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Feb 21, 2026 4:37am

[CLAassistant]

All committers have signed the CLA.

[hcavarsan]

@greptileai

[greptile-apps]

Greptile Summary

Adds a background watchdog to PrismaClient that detects when the prisma-query-engine subprocess dies or becomes a zombie, and automatically reconnects to the database. This addresses All connection attempts failed errors reported after the engine silently exits (e.g., OOM during heavy queries like /tag/list on large datasets).

• Adds _get_engine_pid() to locate the engine PID via Prisma internals with /proc scan fallback
• Uses pidfd_open (Linux 5.3+) for event-driven death detection, falling back to /proc/<pid>/stat polling every 2 seconds
• On engine death: cleans up watcher resources, reaps zombie, reconnects with exponential-backoff retry (5 attempts, 60s max), and re-arms the watcher
• Integrates into proxy lifecycle: starts after DB health check on startup, stops during shutdown
• Guards against duplicate watchers and serializes concurrent reconnection via asyncio.Lock
• Includes 8 mock-only unit tests covering process disappearance, zombie detection, retry/give-up behavior, clean shutdown, and pidfd scheduling

Confidence Score: 4/5

• This PR is safe to merge — it adds a self-healing background watchdog with clean lifecycle management, proper guards, and thorough test coverage.
• The implementation is well-structured, addresses all feedback from the previous review round (exponential backoff, scoped zombie reaping, duplicate watcher guard, pidfd leak fix, IndexError handling), and includes 8 unit tests. The feature gracefully degrades on non-Linux platforms. The only minor concern is the Linux-specific nature of the /proc and pidfd mechanisms, but this is documented and handled.
• Pay attention to litellm/proxy/utils.py — the core watchdog logic. All other files are minimal integration or test updates.

Important Files Changed

Filename	Overview
litellm/proxy/utils.py	Adds ~190 lines implementing a prisma-query-engine watchdog: PID detection (via Prisma internals + /proc scan), pidfd-based event-driven monitoring, /proc polling fallback, death handling with exponential-backoff reconnect. Clean implementation with proper resource cleanup and guards against duplicate watchers.
litellm/proxy/proxy_server.py	Minimal integration: starts the watchdog after DB health check on startup, stops it during proxy shutdown. Two-line change, well-placed in the lifecycle.
tests/litellm/proxy/test_prisma_engine_watchdog.py	New test file covering 8 key scenarios: process disappearance, zombie detection, successful reconnect + re-arm, missing DATABASE_URL, clean shutdown, transient failure retry, permanent failure graceful degradation, pidfd handler task scheduling. All mock-based, no real network calls.
tests/proxy_unit_tests/test_proxy_server.py	Adds _start_db_health_check_loop mock to existing Prisma client test fixture to accommodate the new startup call.
tests/proxy_unit_tests/test_proxy_utils.py	Adds _start_db_health_check_loop mock to existing health-check-disabled test to prevent test failure from the new startup integration.
tests/test_litellm/proxy/test_proxy_server.py	Adds _start_db_health_check_loop and _stop_db_health_check_loop stub methods to MockPrisma class for compatibility with the new lifecycle hooks.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Proxy Startup] --> B[_start_db_health_check_loop]
    B --> C{Find engine PID}
    C -->|Not found| D[Log warning, no watchdog]
    C -->|Found| E{pidfd_open available?}
    E -->|Yes| F[Register pidfd reader]
    E -->|No| G[Start /proc polling task, 2s interval]
    
    F -->|Engine exits| H[_on_pidfd_readable]
    G -->|Process gone/zombie| H2[Detect death in poll loop]
    
    H --> I[_cleanup_watcher]
    H2 --> I
    I --> J[_handle_engine_death]
    
    J --> K[Acquire reconnect lock]
    K --> L[_reap_zombies for engine PID]
    L --> M{DATABASE_URL set?}
    M -->|No| N[Log error, stop]
    M -->|Yes| O[Reconnect with backoff, max 5 retries]
    O -->|Success| P[_start_db_health_check_loop re-arm]
    O -->|All retries failed| Q[Log critical, watchdog inactive]
    
    R[Proxy Shutdown] --> S[_stop_db_health_check_loop]
    S --> I2[_cleanup_watcher]

Loading

_{Last reviewed commit: 866c609}

[greptile-apps]

_{3 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[hcavarsan]

@greptileai

[greptile-apps]

_{3 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[hcavarsan]

@greptileai

[greptile-apps]

_{6 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[hcavarsan]

@greptileai

[greptile-apps]

_{6 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[hcavarsan]

@greptileai

[greptile-apps]

_{6 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[ishaan-jaff]

LGTM