[WIP] Litellm client disconnect relay. Routing OSS PR into CI

[harish-berri]

Relevant issues

TODO:

• Regression profiling under high concurrent loads
• Check for memory leaks when requests are cancelled
• Profile for memory leaks when this is enabled

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

[greptile-apps]

Greptile Summary

This PR refactors client disconnect detection by moving _check_request_disconnection from proxy_server.py into the core ProxyBaseLLMRequestProcessing.make_call path in common_request_processing.py, where it is now unconditionally spawned as a background task for every routed request.

• P1 – post_call_failure_hook skipped for 499: _handle_llm_api_exception raises early for 499 HTTPException, bypassing post_call_failure_hook and any registered failure callbacks (usage logging, billing, alerting).
• P1 – No opt-out flag: The disconnect watcher is always-on with no user-controlled toggle, despite the PR description noting open TODOs for memory leak and regression profiling under load.

Confidence Score: 3/5

Not safe to merge as-is: failure hooks are silently skipped for all cancelled requests, and the always-on background task spawning has unresolved profiling TODOs.

Two P1 findings: post_call_failure_hook is bypassed for 499 (missing billing/logging callbacks), and the disconnect watcher is unconditionally enabled with no opt-out flag despite open profiling TODOs cited in the PR itself. Score is below the P1 ceiling of 4 due to the combination of both findings affecting the critical request path.

litellm/proxy/common_request_processing.py — both the _handle_llm_api_exception early-exit and the unconditional task spawning need attention before merge.

Important Files Changed

Filename	Overview
litellm/proxy/common_request_processing.py	Core change: adds _check_request_disconnection and integrates it unconditionally into make_call; _handle_llm_api_exception early-exits for 499, skipping post_call_failure_hook and failure callbacks.
litellm/proxy/proxy_server.py	Removes dead check_request_disconnection function; straightforward cleanup with no logic changes.
litellm/constants.py	Adds DEFAULT_CLIENT_DISCONNECT_CHECK_TIMEOUT_SECONDS (env-configurable, default 600s); straightforward constant addition.
tests/test_litellm/proxy/test_client_disconnection.py	New unit tests for _check_request_disconnection; uses AsyncMock(side_effect=[False, True]) to correctly simulate disconnect polling — mock setup is accurate and covers both disconnect and cancel paths.

_{Reviews (3): Last reviewed commit: "Refactor tests for client disconnection ..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 76.19048% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/common_request_processing.py	76.19%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

[harish-berri]

@greptile review again

[harish-berri]

@greptile the P1 on the test is handled, review again

[greptile-apps]

post_call_failure_hook bypassed for 499 responses

The early raise e at line 1784 exits the function before post_call_failure_hook is called (line 1790). Any registered failure callbacks — usage logging, billing hooks, alerting integrations — will silently not run for client-disconnected requests. If the proxy has hooks that track spend or emit telemetry on every request failure, those records will be missing for all 499s.

Consider calling post_call_failure_hook before re-raising, or at minimum documenting that failure hooks are intentionally skipped for client disconnects.

[greptile-apps]

Disconnect detection always-on with no opt-out flag

The disconnect watcher task is now unconditionally spawned for every request routed through ProxyBaseLLMRequestProcessing.make_call. The PR description itself lists "Regression profiling under high concurrent loads" and "Check for memory leaks when requests are cancelled" as open TODOs, indicating this path hasn't been load-tested yet. Per the team's backward-compatibility rule, risky additive behaviours on the critical request path should be guarded by a user-controlled flag (e.g. litellm.enable_client_disconnect_checking) so operators can disable it if regressions appear, rather than requiring a full version rollback.

Rule Used: What: avoid backwards-incompatible changes without... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[kursadlacin]

The two open TODOs here (load regression risk and the memory leak note) and the two Greptile P1s are exactly what #30223 addresses: the watcher is opt-in (default off, zero overhead and byte-identical default path) and 499 falls through to post_call_failure_hook so accounting and slot release stay intact. It also carries focused unit tests plus a live curl-and-kill transcript showing the backend aborts within ~1.5s of the client dropping. Feel free to fold any of it into this PR instead if that routes better through CI