fix(responses_bridge): keep one chat completion id per stream and always stream completed responses

[devin-ai-integration]

TLDR

Problem this solves:

• Bridged streams gave every chunk a fresh chatcmpl id
• SDKs that accumulate by id silently drop the response
• A completed bridge result crashed streaming chat requests

How it solves it:

• The bridge stream iterator pins the first chunk's id
• Completed responses are wrapped in a real chunk stream
• Live e2e tests pin the bridged streaming contract

Relevant issues

Fixes #32854
Fixes #33154

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxies serving real GPT-5.6 on AWS Bedrock Mantle (real API calls, real $). Config used for both runs:

model_list:
  - model_name: gpt-5.6-mantle
    litellm_params:
      model: bedrock_mantle/openai.gpt-5.6-sol
      aws_region_name: us-east-1

general_settings:
  master_key: sk-1234

The streaming request sent to each proxy:

curl -sN http://localhost:<port>/v1/chat/completions \
  -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
  -d '{"model":"gpt-5.6-mantle","messages":[{"role":"user","content":"Count from 1 to 5, one number per line"}],"stream":true,"max_tokens":64}' \
  | grep '^data: ' | grep -v DONE | sed 's/^data: //' \
  | python3 -c "
import json,sys
chunks=[json.loads(l) for l in sys.stdin if l.strip()]
ids=[c['id'] for c in chunks]
print('chunks:', len(ids), '| distinct ids:', len(set(ids)))
"

Before the fix, at litellm_internal_staging head 77ed122981, every chunk carries its own id (#32854):

chunks: 10 | distinct ids: 10
first 3 ids: ['chatcmpl-1ba6bd44-7c0f-4391-86be-1dcca5f18acc', 'chatcmpl-7147a94d-9ca2-4135-87f1-dc0254ae29a4', 'chatcmpl-0d5841d1-99b3-4245-9a66-fae2ce685537']

After the fix, at this PR's head 4299c6d191, the same request streams identical content ("1\n2\n3\n4\n5", finish_reason stop, terminated by [DONE]) under one id:

chunks: 10 | distinct ids: 1

Streamed tool calls over the bridge, same after-proxy at 4299c6d191, reassemble into one well-formed call:

chunks: 8 | distinct ids: 1 | tool: get_weather | args: {"location":"San Francisco"}

Exact customer-client repro: a Go program using the openai-go SDK pinned to v1.12.0 (the reporter's version) whose ChatCompletionAccumulator drops any chunk whose id differs from the first (streamaccumulator.go:107), streaming the same tool-call request through each proxy. Command, run from the Go module directory against each proxy in turn:

LITELLM_BASE_URL=http://localhost:<port> LITELLM_KEY=sk-1234 LITELLM_MODEL=gpt-5.6-mantle go run .

Before the fix, at litellm_internal_staging head 77ed122981 (identical output across 2 runs, exit code 1):

chunks received: 8
chunks rejected by accumulator (id mismatch): 7
JustFinishedToolCall fired: false (name="" args="")
accumulated tool calls: 1
VERDICT: FAIL (accumulator dropped chunks or tool call never assembled)

After the fix, at this PR's head 4299c6d191 (identical output across 2 runs, exit code 0):

chunks received: 8
chunks rejected by accumulator (id mismatch): 0
JustFinishedToolCall fired: true (name="get_weather" args="{\"location\":\"San Francisco\"}")
accumulated tool calls: 1
VERDICT: PASS

Checkpoint	Commit	Route	Result
before fix	77ed122981	stream, basic	FAIL: 10 ids
after fix	4299c6d191	stream, basic	PASS: 1 id
before fix	77ed122981	stream, tool call	FAIL: per-chunk ids
after fix	4299c6d191	stream, tool call	PASS: 1 id, parseable args
before fix	77ed122981	openai-go accumulator	FAIL: 7 of 8 dropped
after fix	4299c6d191	openai-go accumulator	PASS: tool call assembled

The #33154 path (bridge hands back an already-completed ModelResponse for a streaming request) cannot be forced on demand against a healthy provider; it is pinned by the new unit regressions test_acompletion_streams_completed_model_response and test_completion_streams_completed_model_response, verified to fail against litellm_internal_staging source ("streaming request got ModelResponse", the exact condition behind the customer-reported "'async for' requires an object with aiter method" crash) and pass on this branch

Type

🐛 Bug Fix
✅ Test

Changes

OpenAiResponsesToChatCompletionStreamIterator now records the first chunk's id and stamps it on every later chunk, so one logical stream serves one chat completion id as the OpenAI spec requires. ResponsesToCompletionBridgeHandler.completion and acompletion no longer return a bare ModelResponse when the caller asked for a stream; the completed response is replayed through MockResponseIterator inside a CustomStreamWrapper, so the proxy's SSE generator always gets an iterable. New live e2e coverage (tests/e2e/llm_translation/test_responses_bridge_streaming_e2e.py) pins the bridged streaming contract end to end, with stream_done tracking added to the e2e HTTP helper so tests can assert the [DONE] terminator

QA runbook

• tests/e2e/llm_translation/test_responses_bridge_streaming_e2e.py::TestResponsesBridgeChatCompletionsStreaming::test_bridged_stream_shares_one_chunk_id - a responses-only model streamed over /chat/completions serves every chunk under one chat completion id
  • Create a bridged model: curl -X POST http://localhost:4000/model/new -H "Authorization: Bearer sk-1234" -d '{"model_name": "bridge-qa", "litellm_params": {"model": "openai/gpt-5.3-codex", "api_key": "os.environ/OPENAI_API_KEY"}}' (needs OPENAI_API_KEY and STORE_MODEL_IN_DB=True)
  • Stream a chat completion: curl -sN http://localhost:4000/v1/chat/completions -H "Authorization: Bearer sk-1234" -d '{"model": "bridge-qa", "messages": [{"role": "user", "content": "Count from 1 to 5"}], "stream": true, "max_tokens": 64}'
  • Expect every data: chunk to carry the same chatcmpl-prefixed id
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/llm_translation/test_responses_bridge_streaming_e2e.py::TestResponsesBridgeChatCompletionsStreaming::test_bridged_stream_delivers_content_finish_reason_and_done - the bridge answers a streaming request with real SSE, never a completed object the SSE generator cannot iterate
  • Reuse the bridged model and stream: curl -sN http://localhost:4000/v1/chat/completions -H "Authorization: Bearer sk-1234" -d '{"model": "bridge-qa", "messages": [{"role": "user", "content": "Reply with the single word pong"}], "stream": true, "max_tokens": 32}'
  • Expect content deltas, a finish_reason chunk, and a terminating data: [DONE] line, with no "'async for' requires an object with aiter method" error in the proxy log
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/llm_translation/test_responses_bridge_streaming_e2e.py::TestResponsesBridgeChatCompletionsStreaming::test_bridged_stream_reassembles_tool_call - tool calls translated from Responses events reassemble into one named call with parseable JSON arguments
  • Reuse the bridged model and stream a tool-forced request: curl -sN http://localhost:4000/v1/chat/completions -H "Authorization: Bearer sk-1234" -d '{"model": "bridge-qa", "messages": [{"role": "user", "content": "What is the weather in San Francisco? Use the get_weather tool."}], "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}}], "tool_choice": "required", "stream": true, "max_tokens": 256}'
  • Expect the streamed tool_calls deltas to concatenate to name get_weather and arguments that parse as JSON with a location
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

The PR corrects Responses-to-Chat streaming behavior.

• Reuses one chat-completion ID across every translated stream chunk.
• Replays completed model responses through a stream wrapper when streaming was requested.
• Extends unit and live end-to-end coverage for stable IDs, SSE completion, and streamed tool calls.
• Adds explicit [DONE] tracking to the end-to-end HTTP helper and registers the new coverage scenarios.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains within the scope of this follow-up review.

Important Files Changed

Filename	Overview
litellm/completion_extras/litellm_responses_transformation/handler.py	Wraps completed bridge responses in a compatible stream while preserving existing stream post-processing.
litellm/completion_extras/litellm_responses_transformation/transformation.py	Stores the first generated chat-completion ID and applies it consistently to subsequent translated chunks.
tests/e2e/e2e_http.py	Records whether a consumed SSE response contains the terminal [DONE] marker.
tests/e2e/llm_translation/test_responses_bridge_streaming_e2e.py	Adds live bridge coverage for stable chunk IDs, complete SSE delivery, and reconstructable tool calls.
tests/test_litellm/completion_extras/litellm_responses_transformation/test_completion_extras_litellm_responses_transformation_handler.py	Adds regression coverage for completed responses returned to synchronous and asynchronous streaming callers.
tests/test_litellm/completion_extras/litellm_responses_transformation/test_completion_extras_litellm_responses_transformation_transformation.py	Verifies that IDs remain stable within a stream and remain distinct across independent streams.

_{Reviews (2): Last reviewed commit: "fix(responses-bridge): return CustomStre..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_responses_bridge_streaming_contract (4299c6d) with litellm_internal_staging (77ed122)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (a10365e) during the generation of this report, so 77ed122 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[mateo-berri]

@greptileai

[mateo-berri]

LGTM