fix: recover streamed responses completed output

[emsi]

Relevant issues

Related to #25429 and #26179

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Verified against a live local proxy started with:

uv run --extra proxy litellm --config /tmp/litellm-local-auth/litellm.config.yaml --debug

Reproduction command:

curl -sS -N http://localhost:4000/v1/responses \
  -H "Authorization: Bearer test-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.5",
    "input": [
      {
        "role": "user",
        "content": [
          {
            "type": "input_text",
            "text": "Reply exactly: OK my lord"
          }
        ]
      }
    ],
    "stream": true
  }'

Before this change, the final response.completed event had response.output: [] even though response.output_item.done contained the completed assistant message

After this change, the final response.completed event includes the recovered output item with OK my lord

Local checks run:

uv run --no-sync python scripts/ruff_strict_gate.py --base origin/litellm_internal_staging
uv run --extra proxy pytest tests/test_litellm/llms/chatgpt/responses/test_chatgpt_responses_transformation.py tests/test_litellm/test_responses_streaming_container_ownership.py -q
uv run --extra proxy pytest tests/test_litellm/completion_extras/litellm_responses_transformation/test_completion_extras_litellm_responses_transformation_transformation.py -q
uv run --extra proxy ruff check litellm/responses/streaming_iterator.py tests/test_litellm/llms/chatgpt/responses/test_chatgpt_responses_transformation.py
git diff --check

Type

Bug Fix
Test

Changes

This updates the per-request Responses streaming iterator to remember completed output items seen earlier in the same SSE stream. When a terminal response.completed event has an empty response.output, the iterator backfills it from the previously streamed response.output_item.done or response.output_text.done events. If the provider already sends a non-empty completed output, that output remains authoritative

The regression tests cover the reported ChatGPT streaming shape, verify that existing completed output is preserved, and confirm recovered output items are copied before being attached to the completed response

[CLAassistant]

All committers have signed the CLA.

[emsi]

@greptileai

[greptile-apps]

Greptile Summary

This PR fixes a bug where the terminal response.completed SSE event emitted an empty output array even though the assistant message had been fully streamed via earlier response.output_item.done / response.output_text.done events. The fix introduces per-request state tracking in BaseResponsesAPIStreamingIterator and a new shared module (sse_output_recovery.py) that records completed output items as they stream and backfills them into the terminal event when the provider omits them.

• streaming_iterator.py: Two new dict attributes track full and text-only streamed output items; _record_streamed_output_chunk and _backfill_completed_response_output are called before transform_streaming_response so the corrected payload flows through the normal transformation path.
• sse_output_recovery.py: Shared helpers centralise the recording logic; record_output_text_chunk correctly skips indices already covered by a full OUTPUT_ITEM_DONE event, and a merge in _recovered_streamed_output_items ensures the richer item always wins.
• README / images: A large Terraform deployment guide and two badge images are bundled into this PR despite being unrelated to the streaming fix.

Confidence Score: 4/5

The streaming fix itself is safe to merge; all findings are non-blocking style concerns.

The core streaming recovery logic is correct and well-tested with mock-only unit tests. The backfill only activates when the provider sends an empty output, leaving existing behaviour unchanged for well-behaved providers. The README additions are out-of-scope for this PR but do not affect runtime behaviour. The inline _MAX_CONTENT_INDEX and the returned-by-reference items from _recovered_streamed_output_items are minor quality concerns with no realistic failure path given the sequential, single-use nature of the iterator.

README.md bundles unrelated Terraform deployment documentation; consider splitting into a separate PR. litellm/responses/sse_output_recovery.py defines _MAX_CONTENT_INDEX inline rather than in constants.py.

Important Files Changed

Filename	Overview
litellm/responses/streaming_iterator.py	Adds output-item tracking state and backfill logic to recover empty response.completed output from previously streamed output_item.done / output_text.done events; logic is correct and sequential.
litellm/responses/sse_output_recovery.py	New shared module implementing record_output_item_chunk and record_output_text_chunk helpers; _MAX_CONTENT_INDEX guard prevents large allocations, but the constant is defined inline rather than in constants.py.
tests/test_litellm/llms/chatgpt/responses/test_chatgpt_responses_transformation.py	Adds two focused mock-only tests covering the recovery and authoritative-output-preservation paths; no real network calls.
README.md	Adds a large "Deploy on AWS or GCP with Terraform" documentation section and two deployment badge images, unrelated to the streaming bug fix; team rule requires docs to live in the litellm-docs repo.

Comments Outside Diff (1)

1. README.md, line 6-145 (link)

   Unrelated documentation added to README

   This PR's stated scope is a streaming iterator bug fix, but it bundles a large "Deploy on AWS or GCP with Terraform" section (140+ lines) plus two new deployment badge images. The team rule requires documentation additions to live in the litellm-docs repo rather than this repository. These changes should be split into a separate PR targeting the docs repo.

   Rule Used: Prevent documentation from being added - needs to ... (source)

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (1): Last reviewed commit: "fix: recover streamed responses complete..." | Re-trigger Greptile}

[greptile-apps]

Recovered items are returned by reference, not deep-copied

_recovered_streamed_output_items returns the same dict objects stored in _streamed_output_items / _streamed_text_only_output_items. If transform_streaming_response mutates any item in completed_chunk["response"]["output"] in-place, those mutations will silently persist in the iterator's state dicts. A shallow copy per item (e.g. [dict(item) for ...]) would prevent accidental aliasing without meaningful overhead.

[codecov]

Codecov Report

❌ Patch coverage is 86.11111% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/responses/streaming_iterator.py	86.11%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

[emsi]

Closing in favor of #30934. The replacement PR was opened from a fresh branch to avoid stale automated feedback from the earlier branch history