litellm oss staging 02 23 2026

[ghost]

• fix(proxy): recover from prisma-query-engine zombie process

• fix(proxy): remove unused imports and extract helper to fix PLR0915 in utils.py

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Feb 24, 2026 2:04pm

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[greptile-apps]

Greptile Summary

This PR adds process-level death detection for the Prisma query engine subprocess, addressing issue #21899 where a zombie prisma-query-engine process could leave the proxy unable to reach the database.

Key changes:

• Introduces a 3-tier engine process watcher with automatic reconnect: os.waitpid() in a dedicated thread (preferred), pidfd_open (Linux 5.3+), and os.kill(pid, 0) polling (fallback)
• When engine death is detected, triggers a "heavy" reconnect path that recreates the entire Prisma client and re-arms the watcher, vs. the existing "lightweight" path for network blips
• Extracts _attempt_reconnect_inside_lock from a nested closure to a proper method, and removes the now-unnecessary _attempt_reconnect_with_lock_timeout wrapper, reducing complexity (fixes PLR0915)
• Removes unused Callable and Coroutine imports

The implementation carefully handles race conditions (e.g., _engine_confirmed_dead flag set before cleanup resets _engine_pid) and includes comprehensive tests covering all detection paths, edge cases, and lifecycle management.

Confidence Score: 4/5

• This PR is safe to merge with minimal risk; the new watchdog logic is well-isolated and has comprehensive tests.
• The implementation is thorough with 3-tier detection, proper race condition handling, and idempotent cleanup. Tests cover all major code paths including edge cases. The only concerns are minor: an unused import in the test file and a potential AttributeError on non-POSIX platforms (which the proxy doesn't target). The refactoring of _attempt_reconnect_inside_lock preserves existing behavior correctly.
• litellm/proxy/utils.py — the new engine watcher methods (3568-3828) are the core of this PR and contain cross-thread coordination that warrants careful review.

Important Files Changed

Filename	Overview
litellm/proxy/utils.py	Adds engine process death detection (waitpid thread, pidfd, polling fallback) with automatic reconnect. Removes unused imports and extracts _attempt_reconnect_inside_lock to a proper method. Well-structured with multi-tier detection and thorough cleanup paths.
tests/litellm/proxy/test_prisma_engine_watchdog.py	Comprehensive test suite covering all three watchdog detection methods, reconnect cycle branching (heavy vs lightweight), race condition guards, lifecycle integration, and edge cases. Minor unused import (time).

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[start_db_health_watchdog_task] --> B[_start_engine_watcher]
    B --> C{_get_engine_pid}
    C -->|PID found| D[_try_waitpid_watch]
    C -->|PID = 0| Z[No watcher - detection unavailable]
    D -->|Child process| E[Spawn waitpid thread]
    D -->|Already dead| F[Set _engine_confirmed_dead]
    D -->|Not child| G[_try_pidfd_watch]
    G -->|pidfd_open OK| H[Register pidfd reader on event loop]
    G -->|Unavailable| I[_poll_engine_proc - os.kill polling 1s]

    E -->|Engine exits| J[_on_engine_death_from_thread]
    H -->|pidfd readable| K[_on_pidfd_readable]
    I -->|ProcessLookupError| L[Engine death detected]
    F --> M[asyncio.create_task - attempt_db_reconnect]

    J --> N[Set _engine_confirmed_dead + _reap_all_zombies]
    K --> N
    L --> N

    N --> O[attempt_db_reconnect - force=True]
    M --> O
    O --> P[_run_reconnect_cycle]
    P -->|engine_is_dead| Q[Heavy: recreate_prisma_client + re-arm watcher]
    P -->|engine alive| R[Lightweight: disconnect + connect + SELECT 1]
    Q --> B

Loading

_{Last reviewed commit: 7ae1571}

[greptile-apps]

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_reap_all_zombies crashes on Windows

os.WNOHANG does not exist on Windows, so _reap_all_zombies() will raise AttributeError if reached on a non-POSIX platform. Since this method is called from multiple code paths (death handlers, _run_reconnect_cycle), it would propagate up and fail the reconnect cycle. The proxy typically runs on Linux/Docker, so this may be acceptable, but adding a platform guard or catching AttributeError would make this more robust.

Suggested change

	@staticmethod
	def _reap_all_zombies() -> set:
	"""Reap ALL zombie child processes via waitpid(-1, WNOHANG).
	Returns a set of reaped PIDs. As PID 1 in Docker (or any
	process that spawns children), we must reap ALL terminated
	children to prevent zombie accumulation.
	"""
	reaped: set = set()
	while True:
	try:
	pid, _ = os.waitpid(-1, os.WNOHANG)
	if pid == 0:
	break
	reaped.add(pid)
	except ChildProcessError:
	break
	return reaped
	@staticmethod
	def _reap_all_zombies() -> set:
	"""Reap ALL zombie child processes via waitpid(-1, WNOHANG).
	Returns a set of reaped PIDs. As PID 1 in Docker (or any
	process that spawns children), we must reap ALL terminated
	children to prevent zombie accumulation.
	"""
	if not hasattr(os, "WNOHANG"):
	return set()
	reaped: set = set()
	while True:
	try:
	pid, _ = os.waitpid(-1, os.WNOHANG)
	if pid == 0:
	break
	reaped.add(pid)
	except ChildProcessError:
	break
	return reaped

[greptile-apps]

Unused import time

time is imported but never used in this test file.

Suggested change

import time

[CLAassistant]

All committers have signed the CLA.