fix(streaming): backfill response.completed output from output_item.done events

[crognlie]

Relevant issues

Fixes #25429. Related to #26179. Supersedes #30934

Linear ticket

N/A

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all unit tests on make test-unit (201 pre-existing failures in prisma/database, vertex AI, and MCP semantic filter tests; none in touched files — 324 tests across tests/test_litellm/responses/ and tests/test_litellm/llms/chatgpt/ pass cleanly)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

The chatgpt/ provider routes through chatgpt.com's Codex backend, which sends response.completed with output: []. The actual assistant content arrives via preceding response.output_item.done SSE events. Without accumulation, the chat-completions bridge receives an empty output list and raises:

ChatgptException - Unknown items in responses API response: []

The fix teaches BaseResponsesAPIStreamingIterator._process_chunk to accumulate response.output_item.done payloads as they stream in and backfill them into the response.completed chunk before it is stored as completed_response. Items are serialized to plain dicts via model_dump() so the downstream _handle_raw_dict_response_item callback in the transformation layer can process them. The existing _recover_output_items_from_raw_sse fallback in LiteLLMResponsesTransformationHandler.transform_response is preserved as a second layer if the streaming path is ever bypassed.

Before:

$ curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-test" \
  -d '{"model":"gpt-5.4","messages":[{"role":"user","content":"Reply with exactly: hello"}],"stream":false}'

{"error":{"message":"litellm.APIConnectionError: ChatgptException - Unknown items in responses API response: []. Received Model Group=gpt-5.4\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"500"}}

After:

$ curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-test" \
  -d '{"model":"gpt-5.4","messages":[{"role":"user","content":"Reply with exactly: hello"}],"stream":false}'

{"id":"chatcmpl-764df91c-940a-47ea-be8c-d2aff173b8f4","created":1782409329,"model":"gpt-5.4","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"hello","role":"assistant"}}],"usage":{"completion_tokens":5,"prompt_tokens":1635,"total_tokens":1640,"completion_tokens_details":{"reasoning_tokens":0},"prompt_tokens_details":{"cached_tokens":0}}}

Type

Bug Fix

Changes

BaseResponsesAPIStreamingIterator accumulates response.output_item.done items into _streamed_output_items during streaming. A secondary _streamed_text_only_items dict catches providers that emit response.output_text.done without a preceding response.output_item.done. At response.completed or response.incomplete, if output is empty and either dict is non-empty, the merged items are sorted by output_index and backfilled onto the response object. Items are serialized via model_dump() when they are Pydantic instances and passed through unchanged when they are already plain dicts; both the serialization and the assignment sit inside a try/except so any failure degrades to a logged warning rather than crashing the stream.

FallbackResponsesStreamWrapper (router.py) and MCPEnhancedStreamingIterator (mcp_streaming_iterator.py) both bypass super().__init__() and are updated to mirror the two new instance attributes.

Twelve regression tests are added to tests/test_litellm/responses/test_streaming_iterator_output_recovery.py, covering: the core backfill, multi-item index ordering, authoritative output preservation, response.incomplete backfill, response.failed no-backfill, output_text.done fallback, output_item.done precedence, replace-in-place content slots, gap padding, absent output_index sequential fallback, exception swallowing, and the dict-type contract required by the downstream transformation layer.

The bulk of the line-count change in streaming_iterator.py is ruff reformatting of pre-existing code required to pass ruff format; the logical additions are confined to the accumulation block in _process_chunk and the two new instance attributes in __init__.

---

Note

^{Cursor Bugbot is generating a summary for commit 755a8b1. Configure here.}

[crognlie]

@greptileai

[CLAassistant]

All committers have signed the CLA.

[greptile-apps]

Greptile Summary

This PR fixes a streaming failure where certain providers (e.g. chatgpt.com's Codex backend) send response.completed with an empty output array while delivering actual assistant content via preceding response.output_item.done SSE events. Without accumulation, the downstream transformation receives an empty output list and raises an error.

• BaseResponsesAPIStreamingIterator now accumulates response.output_item.done items into _streamed_output_items and response.output_text.done items into _streamed_text_only_items during streaming. When a terminal response.completed or response.incomplete event arrives with no output, it backfills from the accumulated items before storing completed_response.
• FallbackResponsesStreamWrapper and MCPEnhancedStreamingIterator, which bypass super().__init__(), are updated to initialize the two new instance attributes so attribute lookups on those objects remain safe.
• Twelve new unit tests cover the core backfill, item ordering, authoritative output preservation, response.incomplete, response.failed no-backfill, text-only fallback, precedence, replace-in-place, gap padding, missing-index fallback, exception swallowing, and the dict-type contract required by downstream transformation.

Confidence Score: 5/5

The change is safe to merge: it adds opt-in backfill that only fires when output is empty and accumulated items exist, so existing providers that send authoritative output in response.completed are completely unaffected.

The backfill path is tightly guarded (empty-output check + non-empty accumulator check), the list comprehension is inside the try/except so serialization failures degrade to a logged warning rather than crashing the stream, Pydantic and dict items are both handled, and all new code is thoroughly tested with 12 focused regression tests.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
litellm/responses/streaming_iterator.py	Core fix: adds _streamed_output_items/_streamed_text_only_items accumulation in init, backfill logic in _process_chunk, and _accumulate_streamed_output_item method. Backfill list comprehension is correctly inside the try/except. Both Pydantic and dict items are handled via hasattr(item, "model_dump") check.
litellm/responses/mcp/mcp_streaming_iterator.py	Defensive initialization of _streamed_output_items and _streamed_text_only_items for MCPEnhancedStreamingIterator, which bypasses super().init(). These are never populated by the MCP class itself (it delegates to an inner base_iterator), but prevent AttributeError if something inspects them.
litellm/router.py	Defensive initialization of the two new dicts on FallbackResponsesStreamWrapper, which also bypasses super().init(). The wrapper proxies already-processed chunks from inner iterators (where backfill already occurred), so these dicts are never populated.
tests/test_litellm/responses/test_streaming_iterator_output_recovery.py	New test file with 12 regression tests covering all significant code paths; uses only mocks and locally constructed httpx.Response objects — no real network calls. Well-structured helpers and comprehensive edge case coverage.
tests/test_litellm/llms/chatgpt/responses/test_chatgpt_responses_transformation.py	Only ruff reformatting changes (line wrapping in function arguments); no logical changes to assertions or test behavior.

_{Reviews (5): Last reviewed commit: "fix(streaming): backfill response.comple..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

This PR fixes a bug where chatgpt.com's Codex backend sends response.completed with an empty output array while delivering the actual content via preceding response.output_item.done SSE events. BaseResponsesAPIStreamingIterator._process_chunk now accumulates those events and backfills them into the completed response before it is stored.

• _streamed_output_items collects OUTPUT_ITEM_DONE payloads; _streamed_text_only_items provides a secondary fallback for providers that emit OUTPUT_TEXT_DONE without a matching OUTPUT_ITEM_DONE. At response.completed / response.incomplete, both dicts are merged, sorted by output_index, and assigned to response.output when it is empty.
• FallbackResponsesStreamWrapper (router.py) and MCPEnhancedStreamingIterator (mcp_streaming_iterator.py) — which bypass super().__init__() — are updated to initialize the two new dicts directly.
• Seven regression tests cover the core backfill, ordering, fallback precedence, response.incomplete handling, and the dict-type contract required by the downstream transformation layer.

Confidence Score: 4/5

Safe to merge once the model_dump() call is moved inside its guarding try-except; all other changes are formatting or straightforward attribute additions.

The backfill logic in _process_chunk guards only the final output assignment inside a try-except, but the model_dump() list comprehension that builds the backfill payload sits just outside it. If any accumulated item lacks model_dump() — for example, a plain dict returned by a provider's transformation layer — the exception bypasses the warning handler and reaches the outer except block, which calls _handle_failure() and re-raises. That turns a best-effort recovery step into a hard stream failure. In the current chatgpt.com provider path the items are always BaseLiteLLMOpenAIResponseObject instances, so the happy path works, but the error boundary is incorrectly drawn for other providers.

litellm/responses/streaming_iterator.py — specifically the backfill block inside _process_chunk around the model_dump() call

Important Files Changed

Filename	Overview
litellm/responses/streaming_iterator.py	Adds OUTPUT_ITEM_DONE and OUTPUT_TEXT_DONE accumulation + response.completed backfill; the model_dump() call is incorrectly placed outside its guarding try-except, which can cause stream failure instead of graceful degradation
litellm/responses/mcp/mcp_streaming_iterator.py	Mirrors the two new accumulator dict attributes in MCPEnhancedStreamingIterator, which bypasses super().init(); straightforward and correct
litellm/router.py	Adds the two new accumulator dicts to FallbackResponsesStreamWrapper; uses bare dict type annotation instead of dict[int, BaseLiteLLMOpenAIResponseObject] — minor inconsistency, no runtime impact
litellm/llms/chatgpt/responses/transformation.py	Ruff-only reformatting; no logical changes
tests/test_litellm/responses/test_streaming_iterator_output_recovery.py	Seven new unit tests covering backfill, ordering, fallback precedence, incomplete handling, and dict-type contract; all mock-only, no real network calls
tests/test_litellm/llms/chatgpt/responses/test_chatgpt_responses_transformation.py	Ruff-only reformatting of existing tests; no assertion changes, no weakened coverage

_{Reviews (1): Last reviewed commit: "fix(streaming): backfill response.comple..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[crognlie]

@greptileai

[niStee]

Validated this approach locally against ghcr.io/berriai/litellm:main-latest with the ChatGPT provider / mode: responses path.

Before the patch, /v1/chat/completions for gpt-5.5 failed with:

{"message":"ChatgptException - Unknown items in responses API response: []"}

After applying the streaming iterator accumulation/backfill approach from this PR locally:

• non-streaming /v1/chat/completions returned assistant content successfully
• streaming /v1/chat/completions returned content chunks and [DONE]
• the fix specifically covered the case where response.completed.response.output is empty but prior response.output_item.done events contain the assistant message

So this PR matches the failure mode I hit and fixes it in practice.

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[niStee]

Following up on my June 26 validation — confirmed still reproducible on main-latest with an important nuance:

stream: true → works fine, delta chunks arrive correctly
stream: false → consistently fails with ChatgptException: Unknown items in responses API response: [] → 500

The non-streaming path hits the response.completed parser which chokes on the empty output: []. Streaming bypasses that codepath entirely, which is why it's easy to miss in testing.

In agent frameworks like oh-my-openagent this is further masked: non-streaming sub-calls (structured tool calls, validation probes, finalizations) fail with 500, the client-side runtime-fallback catches it and silently routes to an alternate model. The agent loop succeeds, the broken path goes unnoticed.

Manually applied the patch to main-latest — there's a conflict in _process_chunk caused by the _get_openai_response_types() changes that landed in main after this PR was opened. Resolved it by rebasing the accumulation block after the updated type-guard. After that:

• stream: false to chatgpt/gpt-5.5 returns 200 OK with backfilled "content": "hello"
• End-to-end via OpenCode TUI agent loop also succeeds

Config: chatgpt/gpt-5.5 with mode: responses, LiteLLM proxy main-latest. The conflict resolution should be straightforward for a rebase — happy to share the diff if helpful.

[niStee]

Conflict resolution detail (for the rebase):

The merge conflict in _process_chunk is caused by main-latest introducing two new local _get_openai_response_types() calls inside _process_chunk that didn't exist when the PR was opened — one for the encrypted_content_affinity_enabled block, one for the RESPONSE_COMPLETED/INCOMPLETE/FAILED type check — so the patch hunks have no matching context.

Resolution that worked against current main-latest:

1. Chunk accumulation — insert the OUTPUT_ITEM_DONE accumulation block directly after _chunk_type = getattr(openai_responses_api_chunk, "type", None), before the existing openai_types = _get_openai_response_types() call on the next line
2. Backfill on completed/incomplete — nest the backfill logic inside the existing RESPONSE_COMPLETED/RESPONSE_INCOMPLETE/RESPONSE_FAILED block (reusing the openai_types already resolved there), guarded by not getattr(_response_obj, "output", None) and (self._streamed_output_items or self._streamed_text_only_items)

The model_dump() call sits inside the try/except block with a hasattr guard for non-Pydantic items — addressing the Greptile suggestion from the original review.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing crognlie:fix_chatgpt_streaming_empty_output (755a8b1) with litellm_internal_staging (559310f)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (c696fdf) during the generation of this report, so 559310f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[crognlie]

@greptileai

[crognlie]

@ryan-crabbe-berri @mateo-berri would appreciate a review when you get a chance. Greptile is 5/5 and veria-ai shows no open security concerns. The three failing CI checks are all pre-existing (repo-wide lint, a flaky agentcore mock test, and the SSO test broken by #33261).

[crognlie]

@greptileai

[crognlie]

All CI checks are now green. @ryan-crabbe-berri @mateo-berri this is ready for a final review and merge.

[Rootax]

This will fix chatgpt suscriptions not working, right ? What is blocking this important fix at this point ?
Thx for the hard work.

[rh7]

Reproduced on released v1.97.0 (chatgpt/gpt-5.4, mode: responses) via Goose Desktop POST /v1/chat/completions.

Non-stream: HTTP 500 ChatgptException - Unknown items in responses API response: [].
OAuth token is valid. Streaming still works as the bypass.

This is the same empty response.completed.output as #25429. The iterator backfill in this PR is the fix we need — please review/merge rather than opening a third parallel PR.

[safrano9999]

As soon as this is fixed one wrapper help container to pipe through the chatgpt llms will not be needed anymore in kubernetes.

[dkrisman]

Verified this also fixes the #25429 bridge failure: non-streaming /v1/chat/completions on chatgpt/gpt-5.4 returned 500 on staging and 200 on this head.

[mubashir1osmani]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 755a8b1. Configure here.}

[safrano9999]

Please also consider the option to

1. use multiple accounts with ChatGPT OAuth
2. and for litellm-database an option to use postgres.
   Just for the oauth code an extra volume is needed. Normally the cool thing is litellm-database runs 100% percent ephemeral with SQL. And then, using 1+ chatGPT would be good ( next level would be an auto rotate when one account hits ratelimits)
   We use PAYG in my company but 3rd party uses sometimes ChatGPT subs...

[safrano9999]

If you do not want to wait for this PR to merge, you can try the fix now with an ephemeral litellm-database deployment:

https://github.com/safrano9999/litellm-database-chatgpt-reasoning#try-it-now

For an existing Podman Quadlet setup, it is a drop-in image replacement; the only additional mount is one named volume to persist the ChatGPT authentication directory. The existing database configuration and environment stay unchanged.

Immutable image:

ghcr.io/safrano9999/litellm-database-chatgpt-reasoning@sha256:b83d7037a3b10f6f75067ae0a8bd164318f12aeefc5f465481f72dcd70dd5bc7

The Containerfile is deliberately only two lines. Its SHA-256 is:

5011c41afcd4e54692f2136d3ce7a4b6f0867da718872bbfb5f9ca03ad8de13f

The build also publishes OCI source/base/patch hashes, BuildKit provenance, an SPDX SBOM, and a GitHub artifact attestation bound to the image digest.

This is now running reliably in my live setup: ChatGPT Responses output is recovered correctly and the models work end-to-end without an additional wrapper. Everything finally works cleanly. Based on that result, I recommend merging this fix into main.

[niStee]

Please also consider the option to

1. use multiple accounts with ChatGPT OAuth
2. and for litellm-database an option to use postgres.
   Just for the oauth code an extra volume is needed. Normally the cool thing is litellm-database runs 100% percent ephemeral with SQL. And then, using 1+ chatGPT would be good ( next level would be an auto rotate when one account hits ratelimits)
   We use PAYG in my company but 3rd party uses sometimes ChatGPT subs...

Out of scope for this PR i believe