fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in)

[kursadlacin]

Relevant issues

Addresses #13774. Re-fixes #22805, which regressed when #14295 was reverted in #14304. Builds on the receive-based design from #25776 (credited via Co-authored-by) and the maintainer port in #27146, and resolves the two P1 findings Greptile raised on both: the watcher is now opt-in instead of always-on, and the 499 path no longer skips post_call_failure_hook. Related accounting context: #14457, #30020

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Setup: a slow OpenAI-compatible mock backend on :9090 that takes 20s per completion and logs whether it observed the client (the proxy) abort; this stands in for vLLM, whose request slot stays occupied exactly as long as the proxy holds the connection open. Proxy config:

model_list:
  - model_name: mock-model
    litellm_params:
      model: openai/mock-model
      api_base: http://localhost:9090/v1
      api_key: fake-key

general_settings:
  master_key: sk-1234
  cancel_on_disconnect: true

Run the proxy and kill the client 2 seconds into the request:

$ python litellm/proxy/proxy_cli.py --config /tmp/cancel_on_disconnect_config.yaml --port 4000 --detailed_debug
$ curl --max-time 2 -sS http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d '{"model": "mock-model", "messages": [{"role": "user", "content": "hi"}]}'
curl: (28) Operation timed out after 2010 milliseconds with 0 bytes received

Backend log with cancel_on_disconnect: true; the upstream request is aborted the moment the client drops instead of running the full 20s:

[backend] request started at 20:34:34
[backend] request ABORTED by client after 1.4s (GPU slot freed)

Proxy log shows the 499 flowing through the normal failure path (so post_call_failure_hook fires: parallel-request slot release, spend logging, alerting):

20:34:35 - LiteLLM Proxy:INFO: common_request_processing.py:561 - litellm.proxy.proxy_server._handle_llm_api_exception(): client disconnected, upstream LLM request cancelled
20:34:35 - LiteLLM Proxy:DEBUG: common_request_processing.py:1898 - An error occurred: 499: Client disconnected the request

Control run with the flag omitted (default), same curl kill at 2s; behavior is unchanged from today, the backend keeps generating to completion:

[backend] request started at 20:35:13
[backend] request COMPLETED after 20.4s

Type

🐛 Bug Fix

Changes

On the non-streaming path, base_process_llm_request awaited the upstream LLM call with no disconnect monitoring. When the HTTP client went away, the upstream request kept running until completion or request_timeout (6000s default), holding a backend slot (for example a vLLM GPU slot) for output nobody would read. vLLM aborts correctly when its own client disconnects, so propagating the disconnect from LiteLLM is enough to free the backend

This adds an opt-in general_settings.cancel_on_disconnect flag (default off). When enabled, a watcher task awaits request.receive() and on http.disconnect cancels the asyncio.gather driving the upstream call. Receive-based monitoring is used instead of is_disconnected() polling because Starlette's BaseHTTPMiddleware breaks is_disconnected() (cf. vllm-project/vllm#10087, #11190), and it costs nothing while the client stays connected

Design notes. The resulting CancelledError is converted to HTTPException(499) only when the watcher set its disconnect event, so server-initiated cancellations (shutdown) still propagate as-is; the conversion also matters because CancelledError subclasses BaseException and would bypass the route handlers' except Exception accounting path entirely. The 499 then flows through _handle_llm_api_exception like any other failure, so post_call_failure_hook still releases max_parallel_requests slots and fires spend and alerting callbacks; the disconnect-specific 499 (matched on the shared detail message, so other 499s keep their tracebacks) is logged at info level instead of a full traceback since a client hangup is not a server error. If request.receive() itself fails (e.g. a transport reset) the watcher logs a warning and degrades to a no-op rather than crashing or cancelling the call. With the flag off (default) the awaited path is byte-identical to before, which is what killed #14295 (always-on watcher changed behavior for every request) and what blocks #25776/#27146. Scope is non-streaming only; the streaming path already releases slots on disconnect since #30020. Retries and fallbacks inside the cancelled gather are cancelled too, which is the desired outcome since the client is gone

Also removes the dead check_request_disconnection helper in proxy_server.py (zero call sites since at least the #14304 revert) together with its behavior-pin tests, and documents the flag in proxy_server_config.yaml

Tests in tests/test_litellm/proxy/test_common_request_processing.py (TestCancelOnDisconnect): the watcher cancels the call and sets the event on http.disconnect and is a no-op while the client stays connected; a CancelledError without a disconnect re-raises instead of masking as 499; driving base_process_llm_request end to end shows the default path ignores a disconnect and returns normally while the enabled path cancels the upstream task and raises 499; and a regression guard proves the 499 path still awaits post_call_failure_hook exactly once

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

kursad seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[kursadlacin]

@greptileai

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

Adds an opt-in general_settings.cancel_on_disconnect: true flag that cancels the in-flight upstream LLM request (non-streaming) when the HTTP client disconnects, freeing backend slots (e.g. vLLM GPU slots). The flag is off by default, leaving the default code path byte-identical to before. Previously-raised P1 issues from related PRs have been addressed: the watcher now logs receive() failures as a warning instead of silently degrading, and the quieter INFO-level logging for 499 is scoped to the specific _CLIENT_DISCONNECT_DETAIL message rather than all 499s.

• Introduces _cancel_llm_call_on_client_disconnect (receive-based watcher, avoids the broken is_disconnected() polling) and _await_llm_call_cancelling_on_disconnect (orchestrator that converts client-initiated CancelledError to HTTPException(499) while allowing server-initiated cancellations to propagate as-is).
• Removes the dead check_request_disconnection helper (zero call sites since PR Revert "[Feat]Cancel upstream on client disconnect" #14304) and its two behaviour-pin tests; replaces them with a TestCancelOnDisconnect suite covering the watcher, graceful degradation, the non-disconnect CancelledError passthrough, the default-off path, and a regression guard that post_call_failure_hook fires on the 499 path.

Confidence Score: 5/5

Safe to merge. The change is opt-in behind a new flag, leaves the default request path identical to before, and the two previously-raised issues (silent watcher degradation, over-broad 499 log suppression) are both resolved in this revision.

The cancellation logic is gated strictly behind general_settings.cancel_on_disconnect (default False), so no existing deployment is affected unless the operator sets the flag. The watcher correctly distinguishes client-initiated from server-initiated CancelledError, post_call_failure_hook still fires on the 499 path, and transport errors degrade to a no-op with a warning log. Dead code removal is clean. Test coverage is thorough and all edge cases identified in prior reviews are pinned by regression tests.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/common_request_processing.py	Core change: adds three helpers (_log_llm_api_exception, _cancel_llm_call_on_client_disconnect, _await_llm_call_cancelling_on_disconnect) and gates the cancellation path behind general_settings.cancel_on_disconnect; error and disconnect edge cases are well-handled.
litellm/proxy/proxy_server.py	Removes the dead check_request_disconnection helper (no call sites since #14304 revert); no functional change to any live code path.
tests/test_litellm/proxy/test_common_request_processing.py	Adds TestCancelOnDisconnect with 7 async tests covering the watcher, receive() failure degradation, server-initiated CancelledError passthrough, default-off path, cancellation path, and post_call_failure_hook regression guard; asyncio_mode=auto covers the missing @pytest.mark.asyncio decorators.
tests/test_litellm/proxy/proxy_server/test_lifecycle.py	Removes two tests for the now-deleted check_request_disconnection helper; no coverage is lost since the function itself is gone.
proxy_server_config.yaml	Adds a commented-out example for cancel_on_disconnect: true with an inline description of its effect.

_{Reviews (2): Last reviewed commit: "fix(proxy): scope 499 quiet logging to d..." | Re-trigger Greptile}

[justalittleadam]

Supporting data for this PR from an independent reproduction at Kubernetes scale — we hit the non-streaming variant of this in a load/DR test campaign on litellm 1.86.4 (proxy on GKE, 4 single-worker pods, in-cluster mock upstream).

Setup: k6 open-loop at 30 RPS against the proxy; 25% of requests routed to an OpenAI-compatible mock deployment whose api_base never responds (a blackholed provider); clients abandon at 70s; request_timeout: 3600.

Observed over 5 minutes of load + ~90 minutes of watching:

• Upstream sockets accumulated linearly for the whole run (≈7.5 hung req/s × elapsed → ~2,250 ESTABLISHED across 4 pods) — client disconnects at 70s never released them, consistent with the dead check_request_disconnection() path this PR re-wires.
• After load stopped, the count stayed flat for 75+ minutes — straight through the 3600s request_timeout expiry window, zero timeout entries in proxy logs. (Possibly a separate issue: the orphaned call doesn't appear to be bounded by request_timeout either — adjacent to [Bug]: timeout silently ignored for Bedrock and Vertex AI streaming requests #23375 / responses(): timeout parameter silently dropped on completion transformation path (Anthropic, Bedrock, Vertex) #28132?)
• Killing the upstream deployment didn't reclaim them either: workers kept ~500 ESTABLISHED sockets/pod to the deleted pod's IP (plus ~400 in LAST_ACK) — no FIN routes from a dead pod, and nothing app-side watches the socket. The only reclaim was restarting the proxy pods.
• Operational impact: healthy-traffic p99 degraded 112ms → ~2min while this accumulated, and a CPU-based HPA never scaled (held slots are idle awaits — no CPU signal).

So +1 to the opt-in watcher approach — in our environment a hung/blackholed provider is the one failure mode where nothing (disconnect, timeout, upstream death) reclaims the slot. Happy to re-run the same scenario against this branch if useful.

[Sameerlite]

@kursadlacin This is a good PR, but can you add the following things in the followup PR so that other contributors know about this:

1. Add to ConfigGeneralSettings with a short description.
2. Add to allowed_args in get_config_list.
3. A docs snippet under proxy config / general settings

[kursadlacin]

Thanks for the review and merge @Sameerlite. Items 1 and 2 are up in #30295 (ConfigGeneralSettings entry, allowed_args in get_config_list, plus a test that drives /config/list end to end and asserts the field is returned as Boolean). For item 3 the general_settings reference lives in the litellm-docs repo's docs/proxy/config_settings.md; opening a companion PR there and will link it from #30295