Skip to content

Revert "fix(proxy): recover from prisma-query-engine zombie process" - #21827

Merged
yuneng-jiang merged 1 commit into
mainfrom
revert-21707-add-watchdog-prisma
Feb 21, 2026
Merged

Revert "fix(proxy): recover from prisma-query-engine zombie process"#21827
yuneng-jiang merged 1 commit into
mainfrom
revert-21707-add-watchdog-prisma

Conversation

@yuneng-jiang

@yuneng-jiang yuneng-jiang commented Feb 21, 2026

Copy link
Copy Markdown
Contributor

Reverts #21707

Issue:
image

Loops infinitely, and does not allow me to access the DB locally

14:19:40 - LiteLLM Proxy:INFO: utils.py:3859 - Found prisma-query-engine at PID 45601.
2026-02-21 14:19:40 - LiteLLM Proxy - INFO - utils.py:3859 - _start_engine_watcher() - Found prisma-query-engine at PID 45601.
14:19:40 - LiteLLM Proxy:INFO: utils.py:3877 - Watching engine PID 45601 via /proc polling (1s fallback).
2026-02-21 14:19:40 - LiteLLM Proxy - INFO - utils.py:3877 - _start_engine_watcher() - Watching engine PID 45601 via /proc polling (1s fallback).
14:19:40 - LiteLLM Proxy:INFO: utils.py:3995 - Prisma DB reconnect succeeded. reason=engine_process_death
2026-02-21 14:19:40 - LiteLLM Proxy - INFO - utils.py:3995 - _attempt_reconnect_inside_lock() - Prisma DB reconnect succeeded. reason=engine_process_death
14:19:40 - LiteLLM Proxy:ERROR: utils.py:3802 - prisma-query-engine PID 45601 disappeared; triggering reconnect.
2026-02-21 14:19:40 - LiteLLM Proxy - ERROR - utils.py:3802 - _poll_engine_proc() - prisma-query-engine PID 45601 disappeared; triggering reconnect.
14:19:40 - LiteLLM Proxy:WARNING: utils.py:3987 - Attempting Prisma DB reconnect. reason=engine_process_death
2026-02-21 14:19:40 - LiteLLM Proxy - WARNING - utils.py:3987 - _attempt_reconnect_inside_lock() - Attempting Prisma DB reconnect. reason=engine_process_death
14:19:40 - LiteLLM Proxy:WARNING: utils.py:3915 - prisma-query-engine PID 0 is dead; performing heavy reconnect.
2026-02-21 14:19:40 - LiteLLM Proxy - WARNING - utils.py:3915 - _run_reconnect_cycle() - prisma-query-engine PID 0 is dead; performing heavy reconnect.
query-engine ac9d7041ed77bcc8a8dbd2ab6616b39013829574

@hcavarsan Please take a look when you get a chance

@vercel

vercel Bot commented Feb 21, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
litellm Error Error Feb 21, 2026 10:21pm

Request Review

@yuneng-jiang
yuneng-jiang merged commit f61dc66 into main Feb 21, 2026
9 of 35 checks passed
@greptile-apps

greptile-apps Bot commented Feb 21, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR reverts #21707, which added a prisma-query-engine process watchdog (SIGCHLD handler, pidfd monitoring, /proc polling, and heavy reconnect logic). The reverted feature was causing immediate false-positive engine death detection on startup — the watcher found the engine PID, then immediately concluded it had disappeared and triggered an unnecessary reconnect cycle (visible in the logs as "PID disappeared" right after "Found prisma-query-engine at PID").

The revert:

  • Removes ~350 lines of engine PID tracking, SIGCHLD signal handling, pidfd_open monitoring, /proc polling, zombie reaping, and heavy reconnect (via recreate_prisma_client) from PrismaClient in litellm/proxy/utils.py
  • Deletes the corresponding 431-line test file tests/litellm/proxy/test_prisma_engine_watchdog.py
  • Restores the simpler _run_reconnect_cycle that only does lightweight disconnect → connect → SELECT 1
  • Preserves the existing DB health watchdog (_db_health_watchdog_loop) and all existing self-heal tests in test_prisma_self_heal.py

The revert is clean: no dangling references to removed methods or import signal remain in the codebase.

Confidence Score: 5/5

  • This PR is safe to merge — it is a clean revert that removes a buggy feature and restores well-tested prior behavior.
  • This is a straightforward git revert of a single feature PR (fix(proxy): recover from prisma-query-engine zombie process #21707). The revert cleanly removes all added code and tests with no residual references. The reverted-to state has existing test coverage via test_prisma_self_heal.py. The PR includes clear evidence of the bug being fixed (log screenshots showing false-positive engine death detection).
  • No files require special attention.

Important Files Changed

Filename Overview
litellm/proxy/utils.py Reverts the engine watchdog feature: removes ~350 lines of engine PID tracking, SIGCHLD handling, pidfd monitoring, /proc polling, and heavy reconnect logic. Restores the simpler lightweight reconnect cycle (disconnect → connect → SELECT 1). No residual references to removed methods or imports.
tests/litellm/proxy/test_prisma_engine_watchdog.py Deletes the entire 431-line test file for the reverted engine watchdog feature. The remaining self-heal tests in tests/test_litellm/proxy/db/test_prisma_self_heal.py still cover the lightweight _run_reconnect_cycle and watchdog start/stop lifecycle.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["DB Health Watchdog Loop<br/>(periodic SELECT 1 probe)"] -->|"Probe fails"| B["attempt_db_reconnect()"]
    B --> C["_run_reconnect_cycle()"]
    
    subgraph BEFORE["Before Revert (PR #21707)"]
        D["_start_engine_watcher()"] --> E{"Engine PID alive?"}
        E -->|"Dead (SIGCHLD/pidfd/poll)"| F["Heavy reconnect:<br/>recreate_prisma_client()"]
        F --> D
        E -->|"Alive"| G["Lightweight reconnect:<br/>disconnect → connect → SELECT 1"]
    end
    
    subgraph AFTER["After Revert (This PR)"]
        H["_run_reconnect_cycle()"] --> I["Lightweight reconnect:<br/>disconnect → connect → SELECT 1"]
    end
    
    C -.->|"Removed"| BEFORE
    C -->|"Restored"| AFTER

    style BEFORE fill:#ffcccc,stroke:#cc0000
    style AFTER fill:#ccffcc,stroke:#00cc00
Loading

Last reviewed commit: 8c5be4c

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

@hcavarsan

Copy link
Copy Markdown
Contributor

@yuneng-jiang this was a bug only on macos , ive made the PR #21899 with details on how i fixed issues

@ishaan-berri
ishaan-berri deleted the revert-21707-add-watchdog-prisma branch March 26, 2026 22:30
fzowl pushed a commit to fzowl/litellm that referenced this pull request Jun 24, 2026
…og-prisma

Revert "fix(proxy): recover from prisma-query-engine zombie process"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants