fix(proxy): recover from prisma-query-engine zombie process

[hcavarsan]

Relevant issues

re-do of #21707 (reverted in #21827) — fixes the infinite reconnect loop on macOS

related to the All connection attempts failed issues after upgrading to v1.81.x (#15536 #15585 #20427)

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🐛 Bug Fix

Changes

re-submission of #21707 which was reverted in #21827. @ishaan-jaff caught an infinite reconnect loop when running the proxy locally on macOS, the watcher found the engine PID, then instantly concluded it disappeared, looping nonstop. this was my fault, i only tested inside docker (linux containers) and never ran it on macOS before merging

after digging into the revert logs i found two root causes. the first was that the original code depended on /proc for everything — checking engine liveness via /proc/<pid>/stat, scanning /proc/*/cmdline for PID discovery, and polling /proc as the fallback watcher. macOS doesn't have /proc, so all of that threw FileNotFoundError which _is_engine_alive() read as "process is gone", triggering a false-positive reconnect that restarted the watcher and looped forever. that's exactly what the revert logs showed

the second issue was that the "primary" SIGCHLD signal handler was silently dead the whole time. litellm always runs with uvloop on unix, and uvloop uses libuv internally which takes ownership of SIGCHLD and hard-blocks any user handler with a RuntimeError (uvloop#409, uvloop#582). pidfd_open + add_reader() also gets rejected under libuv's fd restrictions. so both event-driven paths were dead code and everything fell through to the /proc fallback — which was the broken path on macOS. in docker on linux /proc works fine, so the fallback happened to work and i never noticed

this version completely removes /proc and SIGCHLD logic.

the new primary detection is os.waitpid(pid, 0) in a dedicated daemon thread that blocks until the engine exits, then notifies the asyncio loop via call_soon_threadsafe(). this is the same pattern CPython itself uses in _ThreadedChildWatcher and this works with any event loop. it handles the race where uvloop reaps the child first by catching ChildProcessError

pidfd_open is still there as a secondary path for linux 5.3+, though the waitpid thread already covers it. the last-resort fallback now uses os.kill(pid, 0) polling instead of /proc — the POSIX-standard signal-zero existence check, same approach psutil uses for cross-platform process liveness. works on linux, macOS, and any POSIX system

_get_engine_pid() now only uses prisma internals (no /proc scan), _is_engine_alive() uses os.kill(pid, 0) instead of parsing /proc/<pid>/stat, and _attempt_reconnect_with_lock_timeout() was inlined into its only caller. zombie reaping still uses os.waitpid(-1, WNOHANG) for PID 1 responsibility in containers

the rest of the watchdog is the same as #21707 — engine death sets _engine_confirmed_dead, _run_reconnect_cycle uses the flag to pick heavy reconnect (recreate_prisma_client + re-arm watcher) vs lightweight (disconnect → connect → SELECT 1). the existing DB health watchdog from #21706 is untouched

this is behavior when testing locally with a stress test script:

prismalitellm.mp4

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Feb 22, 2026 10:03pm

[hcavarsan]

@greptileai

[greptile-apps]

Greptile Summary

This PR re-introduces the prisma-query-engine zombie process recovery feature (originally #21707, reverted in #21827) with fixes for the infinite reconnect loop observed on macOS. The core issue was that the original implementation relied on /proc filesystem for engine liveness checks and PID discovery, which doesn't exist on macOS, causing false-positive death detection that looped forever. Additionally, the SIGCHLD signal handler was silently broken under uvloop.

The new implementation replaces all /proc-based logic with POSIX-portable alternatives:

• Primary: os.waitpid(pid, 0) in a dedicated daemon thread (same pattern as CPython's _ThreadedChildWatcher), with call_soon_threadsafe() for event loop notification
• Secondary: pidfd_open for Linux 5.3+ (only if waitpid is unavailable)
• Fallback: os.kill(pid, 0) polling at 1s intervals (POSIX-standard signal-zero check)

The _run_reconnect_cycle now branches between heavy reconnect (engine dead — recreate Prisma client + re-arm watcher) and lightweight reconnect (network blip — disconnect/connect/SELECT 1), using the _engine_confirmed_dead flag set by the detection handlers.

• Removes unused Callable/Coroutine imports and inlines _attempt_reconnect_with_lock_timeout into its only caller
• Race conditions are well-handled: _engine_confirmed_dead is set before _cleanup_engine_watcher resets _engine_pid, stale PID notifications are ignored, and double-trigger is prevented
• Comprehensive test suite with 20 tests covering all detection paths, race conditions, and lifecycle management

Confidence Score: 4/5

• This PR is safe to merge — it fixes a real infinite-loop regression on macOS with well-tested, POSIX-portable process monitoring logic.
• The code correctly replaces /proc-dependent logic with portable POSIX alternatives (waitpid, os.kill signal-zero). Race conditions between detection handlers and cleanup are well-handled via the _engine_confirmed_dead flag. The three-tier fallback (waitpid thread → pidfd → os.kill polling) provides robust coverage across Linux and macOS. Comprehensive test coverage with 20 tests. The only minor concern is that the _start_engine_watcher log message after the "already dead" path is slightly misleading, but this is cosmetic.
• litellm/proxy/utils.py — the core engine watcher logic is the primary area of complexity; the heavy reconnect timeout/retry behavior should be validated in staging.

Important Files Changed

Filename	Overview
litellm/proxy/utils.py	Adds cross-platform engine process death detection (waitpid thread, pidfd, os.kill polling) replacing /proc-dependent logic. Introduces heavy vs lightweight reconnect branching, proper zombie reaping, and clean lifecycle management. Well-structured with appropriate race condition guards.
tests/litellm/proxy/test_prisma_engine_watchdog.py	Comprehensive test suite covering all detection paths (waitpid thread, pidfd, polling), race conditions (already-dead engine, stale PID, double-trigger), reconnect cycle branching (heavy vs lightweight), and lifecycle start/stop. Good use of mocking patterns.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[start_db_health_watchdog_task] --> B[_start_engine_watcher]
    B --> C{_get_engine_pid}
    C -->|PID found| D{_try_waitpid_watch}
    C -->|PID == 0| Z[No watcher - detection unavailable]
    D -->|Child process - thread started| E[waitpid thread blocks on os.waitpid]
    D -->|Already dead| F[Set _engine_confirmed_dead]
    D -->|Not child process| G{_try_pidfd_watch}
    G -->|pidfd_open success| H[Register fd with asyncio add_reader]
    G -->|Unavailable/failed| I[_poll_engine_proc - os.kill polling 1s]
    
    E -->|Engine exits| J[call_soon_threadsafe → _on_engine_death_from_thread]
    H -->|pidfd readable| K[_on_pidfd_readable]
    I -->|ProcessLookupError| L[Engine gone detected]
    
    J --> M{_engine_confirmed_dead?}
    K --> M
    F --> N[attempt_db_reconnect force=True]
    L --> N
    M -->|No - first detection| N
    M -->|Yes - already handled| O[Skip]
    
    N --> P[_run_reconnect_cycle]
    P --> Q{engine_is_dead?}
    Q -->|Yes| R[Heavy: recreate_prisma_client + re-arm watcher]
    Q -->|No| S[Lightweight: disconnect → connect → SELECT 1]

Loading

_{Last reviewed commit: a1a66c5}

[greptile-apps]

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[hcavarsan]

@greptileai

[greptile-apps]

_{2 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}