fix(proxy): deterministically close upstream streams for /v1/responses and /v1/messages

[vivekvar-dl]

TLDR

Problem this solves:

• Cancelled /v1/responses and /v1/messages streams leave the upstream connection open
• Backends like vLLM keep generating into a dead socket
• Whether the connection closes depends on garbage collection timing

How it solves it:

• Both routes' stream objects now expose an explicit close
• The /v1/messages passthrough closes its upstream response on stream end
• Same contract the chat completions route received in fix(proxy): close upstream LLM stream when client disconnects mid-stream #30245

User Flow

Before: a developer streaming against a self-hosted backend cancels a request, but the backend keeps generating until the response is complete

1. They send POST http://localhost:4000/v1/messages with "stream": true for a model served by their vLLM box
2. A few seconds in, they cancel (Ctrl-C in curl, or their app navigates away and drops the connection)
3. Watching the backend, the request keeps decoding to the very end; GPU stays busy and the serving slot stays occupied for the full response length

After: cancelling frees the backend within about a second

1. They send the same POST http://localhost:4000/v1/messages with "stream": true
2. A few seconds in, they cancel the same way
3. The backend sees its connection close right away, aborts the request, and the slot frees

The same before and after applies to POST http://localhost:4000/v1/responses with "stream": true. Chat completions was already covered by #30245 and is unchanged here

Relevant issues

Fixes #36123

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Setup: live proxy (python litellm/proxy/proxy_cli.py --config <config> --port 4000) in front of a local SSE upstream that emits one chunk per second for 60 seconds and logs the moment its client socket dies. The upstream stands in for a self-hosted backend because this run had no provider credentials to spend; the observed signal, the upstream socket closing, is exactly what #36123 reports vLLM never seeing. No LiteLLM code is mocked, the requests below are what a customer sends

After the fix (commit a4ce0f2), both endpoints, curl killed 3 seconds in:

curl -sN --max-time 3 http://localhost:4000/v1/responses \
  -H "Authorization: Bearer sk-litellm-test-1234" -H "Content-Type: application/json" \
  -d '{"model": "mock-openai", "input": "count forever", "stream": true}'

curl -sN --max-time 3 http://localhost:4000/v1/messages \
  -H "Authorization: Bearer sk-litellm-test-1234" -H "Content-Type: application/json" \
  -d '{"model": "mock-anthropic", "max_tokens": 500, "messages": [{"role": "user", "content": "count forever"}], "stream": true}'

Upstream log, close arrives about one second after each curl dies instead of 60 chunks later:

[08:12:17.189] STREAM START /v1/responses
[08:12:21.199] STREAM ABORTED /v1/responses after 4 chunks (ConnectionResetError) -- upstream connection was closed
[08:12:38.611] STREAM START /v1/messages
[08:12:41.618] STREAM ABORTED /v1/messages after 3 chunks (ConnectionResetError) -- upstream connection was closed

Honest caveat on the before run (commit e24a914): in this small local repro the connection also closed within about a second, because with nothing else holding references CPython immediately collects the abandoned stream objects and the transport closes on finalization. The leak in #36123 shows up when that collection does not happen promptly, for example when the response object is retained for the request's lifetime under load. What this PR changes is that the close no longer depends on collection timing at all: cleanup now invokes an explicit close on both routes, which is the same guarantee #30245 gave chat completions. With the fix reverted, the new regression tests fail with AttributeError: 'LiteLLMCompletionStreamingIterator' object has no attribute 'aclose' and with the upstream response left open in the passthrough generator

For an operator with a vLLM or sglang backend, the two curl commands above (pointed at a real model, cancelled mid-stream) plus the backend's abort log or nvidia-smi are the full verification

Type

🐛 Bug Fix

Changes

BaseResponsesAPIStreamingIterator gains a null-safe aclose that closes the httpx response it holds; null-safe because two subclasses bypass its constructor and carry no response. The bridge iterator used when /v1/responses is served by a chat-completions model overrides it to close its stream wrapper, which releases the provider connection through the #30245 machinery. The /v1/messages passthrough generator now closes its upstream response in its finally, after spend logging is scheduled, so partial-usage billing on disconnect is untouched. The MCP-enhanced wrapper keeps its upstream-owning iterator in a separate field, so it overrides the shared close to delegate there (flagged by Greptile). The proxy's existing streaming cleanup and the router's fallback cleanup both already probe for aclose; they now find one on these routes

Tests: regression coverage in tests/test_litellm/proxy/pass_through_endpoints/test_streaming_handler.py (new, mirrors its module), tests/test_litellm/responses/test_streaming_iterator.py (extended), and tests/test_litellm/responses/litellm_completion_transformation/test_litellm_completion_streaming_iterator.py (new; descriptive name because the directory is not a package and the mirrored name collides with an existing file). Disconnect and natural-completion paths are both pinned, plus the no-response subclass case and the MCP wrapper delegation in tests/test_litellm/responses/mcp/test_mcp_streaming_iterator.py

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

All committers have signed the CLA.

[greptile-apps]

Greptile Summary

This PR adds deterministic asynchronous cleanup for Responses API and Anthropic passthrough streams, including the previously reported MCP wrapper path. One MCP lifecycle gap remains:

• Upstream HTTP responses and chat-completion wrappers now expose explicit close delegation.
• Passthrough streaming responses close in generator teardown while preserving partial-stream logging.
• MCP follow-up creation can still discard the previous stream without closing it.

Confidence Score: 4/5

The PR is not yet safe to merge because MCP follow-up rounds can discard an upstream stream without explicitly closing it.

MCP auto-execution replaces base_iterator immediately after a completed event, while aclose only reaches the replacement, so an earlier round's upstream response can remain open until garbage collection.

Files Needing Attention: litellm/responses/mcp/mcp_streaming_iterator.py

Important Files Changed

Filename	Overview
litellm/responses/mcp/mcp_streaming_iterator.py	Adds current-iterator cleanup, but follow-up rounds overwrite the previous iterator without closing it.
litellm/responses/streaming_iterator.py	Adds null-safe closure of the underlying httpx response.
litellm/responses/litellm_completion_transformation/streaming_iterator.py	Delegates cleanup to the chat-completion stream wrapper.
litellm/proxy/pass_through_endpoints/streaming_handler.py	Closes passthrough upstream responses in generator teardown without masking cleanup.
tests/test_litellm/responses/mcp/test_mcp_streaming_iterator.py	Covers closing the currently stored MCP iterator but not replacement across follow-up rounds.
tests/test_litellm/proxy/pass_through_endpoints/test_streaming_handler.py	Covers disconnect, natural completion, and Anthropic wrapper cleanup.
tests/test_litellm/responses/litellm_completion_transformation/test_litellm_completion_streaming_iterator.py	Verifies closure reaches the wrapped completion stream.
tests/test_litellm/responses/test_streaming_iterator.py	Verifies underlying httpx response closure and the response-less no-op path.

Comments Outside Diff (1)

1. litellm/responses/mcp/mcp_streaming_iterator.py, line 815 (link)

   Previous MCP stream remains open

   When MCP auto-execution starts a follow-up request after response.completed, this assignment replaces the previous base_iterator without closing it. A later disconnect only closes the replacement iterator, leaving the previous upstream connection open until garbage collection.

_{Reviews (2): Last reviewed commit: "fix(responses): propagate aclose through..." | Re-trigger Greptile}

[vivekvar-dl]

@greptileai

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../proxy/pass_through_endpoints/streaming_handler.py	50.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing vivekvar-dl:litellm_fix_stream_close_responses_messages (62ca98a) with litellm_internal_staging (c28cbb8)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (e24a914) during the generation of this report, so c28cbb8 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[vivekvar-dl]

On the follow-up round note: base_iterator is only replaced after StopAsyncIteration, so the discarded stream is already exhausted and its connection released