fix(proxy): close common streaming responses on exit

[WZStephen]

Relevant issues

Related to #25776.

Linear ticket

N/A

Pre-Submission checklist

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Targeted tests pass locally:

uv run --no-sync pytest tests/test_litellm/test_common_request_processing_streaming_cleanup.py
# 3 passed

Lint on changed files passes locally:

uv run --no-sync ruff check litellm/proxy/common_request_processing.py tests/test_litellm/test_common_request_processing_streaming_cleanup.py
# All checks passed

Type

🐛 Bug Fix
✅ Test

Changes

• Close response.aclose() from ProxyBaseLLMRequestProcessing.async_streaming_data_generator in a shielded finally block.
• Prevent leaked upstream streaming responses when the downstream generator is closed early, such as on client disconnect.
• Add regression coverage for early exit, normal completion, and mid-stream error paths.

[WZStephen]

@greptileai

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR adds a finally block to ProxyBaseLLMRequestProcessing.async_streaming_data_generator that calls response.aclose() inside a shielded anyio.CancelScope, ensuring upstream streaming connections are closed whether the generator exits normally, mid-stream on error, or early due to client disconnect.

• The cleanup path is guarded with hasattr(response, \"aclose\") and anyio.CancelScope(shield=True), so it runs even when the enclosing task is being cancelled, and any errors during close are caught and logged at debug level without disrupting the caller.
• Three new mock-based regression tests cover early exit, full completion, and error propagation scenarios, confirming aclose is called exactly once in each case.

Confidence Score: 5/5

Safe to merge — the change is tightly scoped to cleanup logic with no effect on the happy-path data flow.

The fix is minimal and correct: a shielded finally block that closes a resource after the generator exits. anyio is already a project dependency used elsewhere in the proxy. The inner try/except BaseException ensures any double-close or already-closed errors are absorbed without surfacing to callers. The three new tests exercise every exit path and all pass locally per the PR description.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/common_request_processing.py	Adds a shielded finally block to async_streaming_data_generator to call response.aclose() on exit, preventing upstream connection leaks on early client disconnect.
tests/test_litellm/test_common_request_processing_streaming_cleanup.py	New test file covering early exit, normal completion, and mid-stream error paths — all mock-based, no real network calls.

_{Reviews (1): Last reviewed commit: "fix(proxy): close common streaming respo..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

This PR fixes a resource leak in ProxyBaseLLMRequestProcessing.async_streaming_data_generator where upstream HTTP response streams were not closed when the downstream generator exited early (e.g., client disconnect). The fix adds a shielded finally block that calls response.aclose(), with anyio.CancelScope(shield=True) ensuring cleanup completes even when the enclosing task is being cancelled.

• Adds import anyio and a finally block to async_streaming_data_generator that calls response.aclose() if the response object supports it, swallowing any errors from double-close.
• async_sse_data_generator delegates to async_streaming_data_generator and benefits from the fix transitively when it is closed early.
• Three new mock-only tests cover early exit, normal completion, and mid-stream error paths.

Confidence Score: 4/5

The change is a targeted cleanup addition with no effect on the data path; safe to merge once the double-close consideration is acknowledged.

The fix is logically correct and the three new tests validate the key paths. The one open question is whether any async_post_call_streaming_iterator_hook implementation itself closes the response before returning, which would make the finally block call aclose() a second time. The except BaseException swallows any error silently, so it won't crash — but it is worth confirming that hook implementations do not close the stream themselves.

No files require special attention beyond the finally block in common_request_processing.py.

Important Files Changed

Filename	Overview
litellm/proxy/common_request_processing.py	Adds a shielded finally block to async_streaming_data_generator that calls response.aclose() on exit, preventing leaked upstream streaming connections on early client disconnect.
tests/test_litellm/test_common_request_processing_streaming_cleanup.py	New test file covering three paths for the cleanup fix: early exit via aclose(), normal completion, and mid-stream exception. All use mocks with no real network calls.

_{Reviews (2): Last reviewed commit: "fix(proxy): close common streaming respo..." | Re-trigger Greptile}

[greptile-apps]

Potential silent double-close if hooks drain the stream

async_post_call_streaming_iterator_hook receives the raw response object and iterates it. If any registered hook implementation calls response.aclose() internally after draining (e.g. a logging or billing hook), the finally block will call aclose() a second time. The except BaseException swallows that silently, so there is no crash, but the debug log will fire on every normal request through such hooks. It is worth confirming that no hook implementation closes the stream, or alternatively documenting that response must not be closed by callers of the hook.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[oss-pr-review-agent-shin]

🤖 litellm-agent: This PR is currently BLOCKED from merge.

Score: 4/5 ❌

Why blocked:

• 1 unresolved reviewer concern (greptile) (unresolved_concern, -1 pts)

Details: Score docked for: 1 unresolved reviewer concern (greptile).

Fix the issues above and push an update — the bot will re-review automatically.

Note: This bot is still in beta and might not always work as expected. Please share any feedback via Slack.

[WZStephen]

Closing this PR to reopen it from nhcf/litellm-nanhu so future updates can be pushed from the organization fork.