fix(responses): synthesize missing streaming lifecycle events for native providers

[alikhan126]

Relevant issues

Fixes #20975

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Reproduce against a live proxy pointed at a native provider that truncates the streaming lifecycle (the issue reproduces on github_copilot/*, Azure gpt-5, ollama cloud, and vLLM). Standard OpenAI already emits the full sequence, so it will not show a difference; that is exactly the idempotency guarantee.

1. Add a truncating native model to litellm/proxy/dev_config.yaml, for example a github_copilot/gpt-5 or Azure gpt-5 entry
2. Start the proxy on this branch: python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log
3. Stream a request and watch the raw SSE:

curl -sN http://localhost:4000/v1/responses \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"gpt-5","input":"Hello","stream":true}'

On litellm_internal_staging the stream is only response.output_text.delta frames followed by response.completed. On this branch the same request now emits response.created, response.in_progress, response.output_item.added, and response.content_part.added before the first output_text.delta, and response.output_text.done, response.content_part.done, response.output_item.done before response.completed, anchored to the same item_id / output_index / content_index as the deltas.

4. End to end, point the OpenAI Codex CLI at the proxy (base_url = http://localhost:4000/v1) and confirm it renders output with no OutputTextDelta without active item in ~/.codex/log/codex-tui.log

Type

🐛 Bug Fix

Changes

Streaming a Responses API request through the proxy with stream: true was producing only response.output_text.delta frames and a trailing response.completed, dropping the required lifecycle wrapper events. Strict clients reject that shape; the OpenAI Codex CLI fails hard with OutputTextDelta without active item because no active item is ever opened

There are three /responses streaming code paths. The chat-completions bridge and the fake-stream / Mock / Cached paths already synthesize the full spec sequence. The gap was the native live passthrough in litellm/responses/streaming_iterator.py, which transformed one upstream SSE line into exactly one event and emitted it with no memory of which lifecycle events had been seen, so when the upstream omitted the openers and teardown they were never produced. The exclude_none / exclude_unset serialization in the proxy is not involved; it can only strip unset fields, never drop a whole event, since the type discriminator is always set

This adds an idempotent, seen-tracking gap filler that the live async and sync iterators drain before pulling the next SSE frame. Given one transformed upstream event it prepends only the openers and teardown that have not already been seen, tracks state per output_index so multi-item responses stay correct, and backfills the output_text.done text from the accumulated deltas. Because every injection is guarded on "not already seen", providers that already emit the full sequence pass through byte for byte, so compliant OpenAI, Azure, and vLLM streams are unchanged. The synthesized openers carry a generated resp_{uuid} id since the real id only arrives at response.completed, and the real completed id is left untouched so clients can still use it for follow-up GETs. Message items get a content_part.added / content_part.done pair; function-call items get an output_item.added / function_call_arguments.done / output_item.done triple with no content part. Mock and Cached iterators override the loop and drain their own prebuilt events, so they are untouched and never double-synthesize

The post-call streaming deployment hook runs on each real provider chunk before the gap filler accumulates it, so the synthesized done events carry post-hook text (for example a guardrail-redacted delta) rather than the raw provider content; a hook that redacts response.output_text.delta is therefore not bypassed on the teardown

WebSocket mode and the cursor_data_generator chat-chunk re-transform are out of scope and left as follow-ups

Tests drive the real ResponsesAPIStreamingIterator and SyncResponsesAPIStreamingIterator with the real OpenAIResponsesAPIConfig, feeding a dependency-injected fake SSE byte stream rather than monkeypatching the code under test. They cover the truncated-text case (full ordered sequence, opener anchoring, accumulated done text), idempotency on an already-complete stream (identical types and counts, no duplicate openers), a truncated function-call stream, round-tripping every synthesized event through the exact model_dump_json(exclude_none=True, exclude_unset=True) call the proxy uses so no required field is stripped off the wire, and a redacting streaming hook proving the synthesized *.done text reflects the post-hook (redacted) content

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[greptile-apps]

Greptile Summary

This PR adds a _ResponsesLifecycleGapFiller class to litellm/responses/streaming_iterator.py that synthesizes missing streaming lifecycle events for native providers (e.g. github_copilot/*, Azure gpt-5, ollama cloud, vLLM) whose upstream SSE streams omit the required opener/teardown wrapper events, causing strict clients like the OpenAI Codex CLI to fail with OutputTextDelta without active item.

• Gap filler logic: An idempotent, seen-tracking state machine keyed per output_index prepends missing response.created, response.in_progress, response.output_item.added, and response.content_part.added openers before the first delta and synthesizes *.done teardown events before response.completed. Providers that already emit the full spec sequence pass through byte-for-byte with no double-wrapping.
• Hook ordering: The post-call streaming deployment hook runs on the raw provider chunk before the gap filler accumulates it, so guardrail-redacted delta text is correctly reflected in synthesized output_text.done / content_part.done / output_item.done payloads.
• Test coverage: Comprehensive new tests cover the truncated-text case, idempotency on a complete stream, function-call lifecycle synthesis, proxy serialization (model_dump_json(exclude_none=True, exclude_unset=True)), and a redacting streaming hook verifying the done events carry post-hook content.

Confidence Score: 5/5

Safe to merge. The gap filler is idempotent and only injects events not already emitted by the upstream, so compliant providers are unaffected and the fix is purely additive for truncating providers.

The core algorithm correctly tracks per-output_index lifecycle state, the hook fires before accumulation so guardrail hooks govern the synthesized done-event text, and all edge cases (function-call items, refusals, multi-item responses, already-complete streams) are tested with real iterator classes. No functional regressions were identified.

No files require special attention. The synthesized output_item.added/output_item.done events bypass _encode_container_id_on_output_item (noted in previous review and already documented), but this only affects container routing metadata on synthesized events for truncating providers.

Important Files Changed

Filename	Overview
litellm/responses/streaming_iterator.py	Adds ~390 lines: the _ResponsesLifecycleGapFiller class, helpers (_obj_get, _safe_int, _safe_str, _build_bag, _ResponsesStreamItemState), and a small integration into the async/sync __anext__ loops via _pending_events. Logic is correct, idempotent, and hook-ordered. No blocking issues found.
tests/test_litellm/responses/test_streaming_iterator.py	Adds comprehensive mock-only lifecycle synthesis tests: truncated text, idempotency, function-call, proxy serialization, and redacting hook. All tests use real iterator classes with a dependency-injected fake SSE stream. No real network calls.
tests/llm_responses_api_testing/test_base_responses_api_streaming_iterator.py	Two existing assertions relaxed from len == 1 to delta in chunks and is last to reflect the new synthesized opener events. Change is justified and explained in comments; new tests in test_streaming_iterator.py provide stronger coverage of the exact event order.

_{Reviews (3): Last reviewed commit: "fix(responses): synthesize missing strea..." | Re-trigger Greptile}

[alikhan126]

@greptileai

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[alikhan126]

@greptileai

[reitowo]

This seems breaking Codex to use GPT-5.6.