fix(responses): honor caller stream flag when provider forces SSE (internal copy of #34095)

[devin-ai-integration]

TLDR

Problem this solves:

• chatgpt/* ignores a caller's stream: false since v1.90.0
• /v1/responses hands back raw SSE instead of JSON
• /v1/chat/completions and /v1/messages intermittently 500 with Unknown items in responses API response: []

How it solves it:

• Transport streaming and response shape are decided separately
• Only the caller's own stream value (top level or in extra_body) picks the shape
• Providers that force SSE are still read as streams
• Handler tests inject the ChatGPT authenticator, so CI never logs in

User Flow

Before: a developer calling a ChatGPT subscription model without streaming gets back a body their client cannot parse

1. They send POST https://litellm-domain/v1/responses with a ChatGPT subscription model and "stream": false, or with no stream at all
2. A 200 comes back with content-type: text/event-stream and ten SSE frames instead of one JSON object
3. Their OpenAI SDK fails on it, the Node one with Cannot use 'in' operator to search for 'object' in data: followed by the whole stream
4. They send the same prompt to POST https://litellm-domain/v1/chat/completions and POST https://litellm-domain/v1/messages without streaming
5. About half of those calls return 500 with Unknown items in responses API response: [], so only "stream": true works reliably

After: the same requests come back in the shape the caller asked for

1. They send the same POST https://litellm-domain/v1/responses with "stream": false, or with no stream at all
2. A 200 comes back with content-type: application/json and one response object whose status is completed
3. Their client parses it normally
4. They send the same prompt to POST https://litellm-domain/v1/chat/completions and POST https://litellm-domain/v1/messages without streaming
5. Both return 200 with one JSON object carrying the assistant's reply, and "stream": true still streams on all three routes

Relevant issues

Fixes #34094

Supersedes #34095, the community PR these commits come from

Affected release

Regression since v1.90.0, introduced by #30189

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Last updated: f1cdfae. QA and /live-pr-risk ran on 78b0bea, the last commit with a product diff; f1cdfae adds the audit's integration tests only

Both legs boot a no-DB proxy from the named commit with --num_workers 2 on a random free port ($PORT below), the master key as $KEY, and this config.yaml, which points m-terra at the real ChatGPT subscription backend (the provider reads the stored subscription token from CHATGPT_TOKEN_DIR, so there is no api_key):

model_list:
  - model_name: m-terra
    litellm_params:
      model: chatgpt/gpt-5.6-terra
      reasoning_effort: low

general_settings:
  master_key: sk-1234
  dangerously_permit_weak_or_unset_master_key: true

The clients are the ones the issue names: the Node openai SDK (7.28.0, what n8n wraps), the Python openai SDK (2.33.0, sync and async), the Python anthropic SDK (0.84.0) for /v1/messages, and curl replays of the same bodies. Every call sends Reply with exactly the word: pong. The SDK scripts print one line per call. Response ids are account-scoped and left out

sdk_cases.py (run as python sdk_cases.py $PORT):

import asyncio, sys, anthropic, openai

PORT = sys.argv[1]
BASE = f"http://127.0.0.1:{PORT}"
INPUT = [{"role": "user", "content": "Reply with exactly the word: pong"}]

def responses_sync(stream):
    client = openai.OpenAI(base_url=f"{BASE}/v1", api_key="sk-1234")
    kwargs = {} if stream is None else {"stream": stream}
    r = client.responses.create(model="m-terra", input=INPUT, **kwargs)
    return f"{len(list(r))} events" if stream else f"object={r.object} status={r.status} text={r.output_text!r}"

async def responses_async(stream):
    client = openai.AsyncOpenAI(base_url=f"{BASE}/v1", api_key="sk-1234")
    kwargs = {} if stream is None else {"stream": stream}
    r = await client.responses.create(model="m-terra", input=INPUT, **kwargs)
    return f"{len([e async for e in r])} events" if stream else f"object={r.object} status={r.status} text={r.output_text!r}"

def chat_sync(stream):
    client = openai.OpenAI(base_url=f"{BASE}/v1", api_key="sk-1234")
    kwargs = {} if stream is None else {"stream": stream}
    r = client.chat.completions.create(model="m-terra", messages=INPUT, **kwargs)
    return f"{len(list(r))} chunks" if stream else f"object={r.object} finish={r.choices[0].finish_reason} text={r.choices[0].message.content!r}"

def messages_sync(stream):
    client = anthropic.Anthropic(base_url=BASE, api_key="sk-1234")
    if stream:
        with client.messages.stream(model="m-terra", max_tokens=64, messages=INPUT) as s:
            return f"stop_reason={s.get_final_message().stop_reason} text={''.join(s.text_stream)!r}"
    r = client.messages.create(model="m-terra", max_tokens=64, messages=INPUT)
    return f"type={r.type} stop_reason={r.stop_reason} text={r.content[0].text!r}"

node_case.mjs (run as node node_case.mjs $PORT):

import OpenAI from "openai";
const client = new OpenAI({ baseURL: `http://127.0.0.1:${process.argv[2]}/v1`, apiKey: "sk-1234" });
const r = await client.responses.create({ model: "m-terra", input: [{ role: "user", content: "Reply with exactly the word: pong" }] });
console.log(`object=${r.object} status=${r.status} text=${JSON.stringify(r.output_text)}`);

Before (a3a5246)

/v1/responses

1. Run the Node client without stream

   node openai responses.create stream omitted: FAIL TypeError: Cannot use 'in' operator to search for 'object' in data: {"type":"response.created","response":{"id":"resp_...
   

2. Run the Python SDK cases

   openai.responses.create stream omitted: FAIL AttributeError: 'str' object has no attribute 'object'
   openai.responses.create stream=False: FAIL AttributeError: 'str' object has no attribute 'object'
   openai.responses.create stream=True: OK 9 events, last=response.completed
   openai.AsyncOpenAI responses.create stream omitted: FAIL AttributeError: 'str' object has no attribute 'object'
   openai.AsyncOpenAI responses.create stream=True: OK 9 events, last=response.completed
   

3. Replay the non-streaming body with curl

   curl -sS -i http://localhost:$PORT/v1/responses -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
     -d '{"model":"m-terra","input":[{"role":"user","content":"Reply with exactly the word: pong"}],"stream":false}'

4. Observe an SSE body on a 200 (the same with stream left out)

   HTTP/1.1 200 OK
   content-type: text/event-stream; charset=utf-8
   10 SSE data frames, last two: ['response.completed', '[DONE]']
   

/v1/chat/completions

1. Run the Python SDK cases

   openai.chat.completions.create stream omitted: OK object=chat.completion finish=stop text='pong'
   openai.chat.completions.create stream=True: OK 2 chunks, text='pong'
   

2. Replay the non-streaming body with curl

   curl -sS -i http://localhost:$PORT/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
     -d '{"model":"m-terra","messages":[{"role":"user","content":"Reply with exactly the word: pong"}],"stream":false}'

3. Observe a 500 (the backend's final event sometimes carries the text and sometimes not, so this fails on some calls and not others)

   HTTP/1.1 500 Internal Server Error
   content-type: application/json
   {"error":{"message":"litellm.APIConnectionError: APIConnectionError: ChatgptException - Unknown items in responses API response: []\n\nLiteLLM: model group 'm-terra' failed with the error above. No fallback was attempted.","type":"internal_server_error","param":null,"code":"500"}}
   

4. Send the same curl six times in a row and read the status codes

   before /v1/chat/completions no stream x6: 200 500 500 200 200 500
   

5. Send it with "stream": true and observe it stream

   HTTP/1.1 200 OK
   content-type: text/event-stream; charset=utf-8
   3 SSE data frames, last two: ['chat.completion.chunk', '[DONE]']
   

/v1/messages

1. Run the Anthropic SDK cases

   anthropic.messages.create stream omitted: OK type=message stop_reason=end_turn text='pong'
   anthropic.messages.stream: OK stop_reason=end_turn text='pong'
   

2. Replay the non-streaming body with curl

   curl -sS -i http://localhost:$PORT/v1/messages -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
     -d '{"model":"m-terra","max_tokens":64,"messages":[{"role":"user","content":"Reply with exactly the word: pong"}],"stream":false}'

3. Observe the same intermittent 500

   HTTP/1.1 500 Internal Server Error
   content-type: application/json
   {"type":"error","error":{"type":"api_error","message":"litellm.APIConnectionError: APIConnectionError: ChatgptException - Unknown items in responses API response: []\n\nLiteLLM: model group 'm-terra' failed with the error above. No fallback was attempted."}}
   

4. Send the same curl six times in a row and read the status codes

   before /v1/messages no stream x6: 500 200 500 500 500 200
   

5. Send it with "stream": true and observe it stream

   HTTP/1.1 200 OK
   content-type: text/event-stream; charset=utf-8
   6 SSE data frames, last two: ['message_delta', 'message_stop']
   

After (78b0bea)

/v1/responses

1. Run the Node client without stream

   node openai responses.create stream omitted: OK object=response status=completed text="pong"
   

2. Run the Python SDK cases

   openai.responses.create stream omitted: OK object=response status=completed text='pong'
   openai.responses.create stream=False: OK object=response status=completed text='pong'
   openai.responses.create stream=True: OK 9 events, last=response.completed
   openai.AsyncOpenAI responses.create stream omitted: OK object=response status=completed text='pong'
   openai.AsyncOpenAI responses.create stream=True: OK 9 events, last=response.completed
   

3. Replay the non-streaming body with curl

   curl -sS -i http://localhost:$PORT/v1/responses -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
     -d '{"model":"m-terra","input":[{"role":"user","content":"Reply with exactly the word: pong"}],"stream":false}'

4. Observe one JSON response object (the same with stream left out, and "stream": true still returns 10 SSE frames ending in response.completed and [DONE])

   HTTP/1.1 200 OK
   content-type: application/json
   10375 bytes, one JSON document: {'object': 'response', 'status': 'completed', 'text': 'pong'}
   

/v1/chat/completions

1. Run the Python SDK cases

   openai.chat.completions.create stream omitted: OK object=chat.completion finish=stop text='pong'
   openai.chat.completions.create stream=True: OK 2 chunks, text='pong'
   

2. Replay the non-streaming body with curl

   curl -sS -i http://localhost:$PORT/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
     -d '{"model":"m-terra","messages":[{"role":"user","content":"Reply with exactly the word: pong"}],"stream":false}'

3. Observe one chat completion

   HTTP/1.1 200 OK
   content-type: application/json
   1337 bytes, one JSON document: {'object': 'chat.completion', 'finish_reason': 'stop', 'text': 'pong'}
   

4. Send the same curl six times in a row and read the status codes

   after /v1/chat/completions no stream x6: 200 200 200 200 200 200
   

5. Send it with "stream": true and observe it stream

   HTTP/1.1 200 OK
   content-type: text/event-stream; charset=utf-8
   3 SSE data frames, last two: ['chat.completion.chunk', '[DONE]']
   

/v1/messages

1. Run the Anthropic SDK cases

   anthropic.messages.create stream omitted: OK type=message stop_reason=end_turn text='pong'
   anthropic.messages.stream: OK stop_reason=end_turn text='pong'
   

2. Replay the non-streaming body with curl

   curl -sS -i http://localhost:$PORT/v1/messages -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
     -d '{"model":"m-terra","max_tokens":64,"messages":[{"role":"user","content":"Reply with exactly the word: pong"}],"stream":false}'

3. Observe one message object

   HTTP/1.1 200 OK
   content-type: application/json
   274 bytes, one JSON document: {'type': 'message', 'stop_reason': 'end_turn', 'text': 'pong'}
   

4. Send the same curl six times in a row and read the status codes

   after /v1/messages no stream x6: 200 200 200 200 200 200
   

5. Send it with "stream": true and observe it stream

   HTTP/1.1 200 OK
   content-type: text/event-stream; charset=utf-8
   6 SSE data frames, last two: ['message_delta', 'message_stop']
   

Observations from the run:

• Chat and messages 500s on main are intermittent; the PR removes them
• A plain string input gets 400 Input must be a list from the backend; pre-existing, unchanged
• Non-stream bodies echo the provider's Codex instructions; pre-existing, unchanged
• An upstream 400 (a bad reasoning.effort) comes back as 400 on both legs
• Sync SDK litellm.responses fixed the same way; checked live

Audit round 1 of 3 (head 78b0bea, merge base a3a5246)

Rig: a two-worker proxy from each commit against an audit-owned Postgres and Redis in the CI environment (LITELLM_MODE=PRODUCTION, local cost map, router_settings.num_retries seeded to 0), the Codex backend scripted by a wire double that replays the frame shape captured live on 2026-10-05 (tests/integration/_support/codex_vendor.py), and an agentic-loop probe callback at the destination edge (tests/integration/_support/agentic_probe.py). Each cell boots its own two-worker proxy with CHATGPT_TOKEN_DIR and CHATGPT_API_BASE at the double, since the chatgpt chat path ignores a deployment's api_base. Cells: tests/integration/providers/test_chatgpt_responses_caller_stream_wire.py (30) and tests/integration/providers/test_chatgpt_responses_caller_stream_chaos.py (3), run as uv run --no-sync python tests/integration/run.py providers <node ids>. Head runs h5 and h6: 33 collected, 33 passed each, identical selections, seeds 4106601 and order 0, no skips or retries. Base run b1: 33 collected, 13 passed, 20 red, every red the SSE body or the bridge 500 the cell exists to catch. Controls c1, once on the head: 3 collected, 3 passed (test_bedrock_gpt_responses_native_wire.py::test_openai_sdk_responses_request_is_served_by_the_native_responses_route, test_responses_minted_reasoning_replay_chaos.py::test_login_backed_and_foundry_deployments_forward_the_minted_item_unchanged[chatgpt-sync] and [chatgpt-stream])

Row	Cell	Base a3a5246 (b1)	Head 78b0bea (h5)	Head (h6)	Verdict
1	test_openai_sdk_request_without_a_stream_flag_gets_the_aggregated_json_response	red (expected red)	green	green	PASS
2	test_async_openai_sdk_request_with_stream_false_gets_the_aggregated_json_response	red (expected red)	green	green	PASS
3	test_raw_request_without_a_stream_key_gets_json_not_sse	red (expected red)	green	green	PASS
4	test_openai_sdk_stream_request_still_streams	green (expected green)	green	green	PASS
5	test_async_openai_sdk_stream_request_still_streams	green (expected green)	green	green	PASS
6	test_openai_sdk_chat_completion_is_bridged_to_a_json_answer	red (expected red)	green	green	PASS
7	test_async_openai_sdk_chat_completion_is_bridged_to_a_json_answer	red (expected red)	green	green	PASS
8	test_openai_sdk_chat_completion_stream_still_streams	green (expected green)	green	green	PASS
9	test_anthropic_sdk_message_is_bridged_to_a_json_answer	red (expected red)	green	green	PASS
10	test_anthropic_sdk_message_stream_still_streams	green (expected green)	green	green	PASS
11	test_identical_request_is_served_from_the_response_cache_as_json	red (expected red)	green	green	PASS
12	test_agentic_hook_sees_the_aggregated_response_once	red (expected red)	green	green	PASS
12	test_agentic_hook_stays_out_of_a_stream_request	green (expected green)	green	green	PASS
13	test_odd_stream_values_decide_the_shape_by_their_truth[null]	red (expected red)	green	green	PASS
13	test_odd_stream_values_decide_the_shape_by_their_truth[empty-string]	red (expected red)	green	green	PASS
13	test_odd_stream_values_decide_the_shape_by_their_truth[empty-list]	red (expected red)	green	green	PASS
13	test_odd_stream_values_decide_the_shape_by_their_truth[zero]	red (expected red)	green	green	PASS
13	test_odd_stream_values_decide_the_shape_by_their_truth[false-twice]	red (expected red)	green	green	PASS
13	test_odd_stream_values_decide_the_shape_by_their_truth[true-then-false]	red (expected red)	green	green	PASS
13	test_odd_stream_values_decide_the_shape_by_their_truth[one]	green (expected green)	green	green	PASS
13	test_odd_stream_values_decide_the_shape_by_their_truth[string-false]	green (expected green)	green	green	PASS
13	test_odd_stream_values_decide_the_shape_by_their_truth[five-kb-string]	green (expected green)	green	green	PASS
13	test_odd_stream_values_decide_the_shape_by_their_truth[true-twice]	green (expected green)	green	green	PASS
14	test_extra_body_stream_true_streams	green (expected green)	green	green	PASS
14	test_extra_body_stream_false_gets_json	green (expected green)	green	green	PASS
15	test_string_input_is_refused_by_the_vendor_as_a_400	green (expected green)	green	green	PASS
16	test_vendor_401_reaches_the_caller_as_401	green (expected green)	green	green	PASS
17	test_vendor_response_failed_event_reaches_the_caller_as_a_json_error	red (expected red)	green	green	PASS
18	test_vendor_stream_dying_mid_transfer_answers_a_json_error_and_leaves_the_proxy_serving	red (expected red)	green	green	PASS
19	test_repeated_identical_cache_busted_requests_each_land_once	red (expected red)	green	green	PASS
20	test_mixed_burst_across_the_three_endpoints_answers_each_in_its_own_shape	red (expected red)	green	green	PASS
21	test_vendor_outage_fails_each_call_cleanly_and_the_restarted_vendor_serves_the_next_burst	red (expected red)	green	green	PASS
22	test_worker_sigkill_mid_burst_leaves_the_sibling_answering_json	red (expected red)	green	green	PASS

Audit observations:

• extra_body.stream is merged into the vendor body after the transform on both legs, so {"extra_body": {"stream": false}} reaches the backend as stream: false; the wire double tolerates it and the real backend was not probed with it. On the base that override was the one way a caller got JSON; on the head the caller's own flag decides the shape either way
• The Anthropic streaming bridge mints its own msg_<uuid> unrelated to the upstream id (both legs), so that cell lands the spend row by x-litellm-call-id
• A response.failed frame for a non-stream caller becomes a JSON error status on the head (an SSE 200 pass-through on the base); an upstream abort mid-stream becomes an error status with a JSON body while /health/liveliness keeps answering and the next call succeeds
• Junk stream values decide the shape by truthiness on the head (1, "false", a 5 KB string and a duplicated true stream; null, "", [], 0 and a duplicated false get JSON), no 400, proxy up
• A worker killed mid-burst leaves the sibling answering, and the follow-up non-stream call answers JSON; the vendor stopping mid-burst fails every in-flight call with a body and the restarted vendor serves the next burst, each call id landing once in LiteLLM_SpendLogs

Type

🐛 Bug Fix

Changes

The two Responses handlers in litellm/llms/custom_httpx/llm_http_handler.py decided whether to hand the caller a streaming iterator by reading stream off the provider payload:

stream = bool(stream or data.get("stream"))
...
is_stream_request = bool(stream)

data there is what goes out to the provider, not what the caller sent. ChatGPTResponsesAPIConfig.transform_responses_api_request sets request["stream"] = True unconditionally because the Codex backend only serves SSE, so data.get("stream") is always true for this provider and a caller that sent stream: false was silently moved onto the streaming path. Other Responses providers are unaffected; chatgpt is the only one whose transform touches stream at all

On /v1/responses that meant a raw SSE body was returned to a caller expecting one JSON object. On /v1/chat/completions and /v1/messages the Responses bridge fell into _collect_response_from_stream, which keeps only the response from the response.completed event; when the Codex backend emits that event with an empty output the request failed with Unknown items in responses API response: []. The recovery path for that case in _recover_output_items_from_raw_sse re-parses the raw SSE from logging_obj.model_call_details["original_response"], and the only place that populates it for this provider is transform_response_api_response, which lives on the non-streaming path and was no longer being reached

That merge line came from #30189, which set out to stop provider transforms that force stream: true from being sent through the non-streaming branch, ChatGPT Codex models included. The transport half of that is right and this PR keeps it. The problem is that a single stream variable was carrying two decisions at once: what to pass to httpx, and whether to return an iterator or an aggregated body. Merging the provider payload into it fixed the transport half and silently changed the response shape along with it

This keeps transport streaming exactly as it was, so a provider that only serves SSE is still read as a stream, and narrows the response-shape decision to the caller's own stream value, captured before the transform runs. When the provider forces SSE for a non-streaming caller the body is read and aggregated through the existing path. The bridge half of #30189, which preserves a top-level stream=True into the Responses request, is untouched

Tests drive both handlers through the real ChatGPTResponsesAPIConfig and a real SSE body, so the aggregation that turns response.output_item.done frames into one completed response actually runs and the returned text is asserted. A caller sending stream: false, and one omitting it, gets an aggregated ResponsesAPIResponse; a caller asking for streaming still gets an iterator that yields through response.completed; and fake_stream produces a mock iterator only for a streaming caller. A stream: true sent through extra_body counts as the caller's own ask too, so an SDK caller on that path keeps getting the iterator main gave them (a test per handler covers it). The two existing ..._streams_when_provider_transform_adds_stream tests from #30189 keep their transport-level assertions. ChatGPTResponsesAPIConfig now takes its Authenticator through the constructor (defaulting to a real one), so the handler tests pass a mock and never start the device-code login that hung the All Other Providers CI job

Caveats (if any)

Low

• Non-stream callers still cost a full SSE read server-side; provider only serves SSE
• Non-stream bodies carry the provider's Codex instructions text; pre-existing
• Assumes no other Responses provider transform sets stream; one that did would now aggregate for non-stream callers, which is the intended shape
• A plain string input to chatgpt/* gets 400 Input must be a list from the backend; pre-existing, out of scope
• An agentic-loop callback (any async_should_run_agentic_loop override) now sees chatgpt non-stream responses: once per /v1/responses call and twice per bridged chat or messages call (responses layer, then chat layer), which is how the chat-to-responses bridge already fires for openai/responses/* on main; left alone, bridge behavior
• Unit Tests run 37404081584 at f1cdfae is red on the non-required misc and proxy-infra-root jobs (tests/unit/test_seed_tracing_fixtures.py, two tests, and tests/unit/proxy/test_component_allowlists.py::test_gateway_plus_backend_covers_full_app), inherited from a merge ref built against main between fix(lens): reconstruct native coding agent conversations #44711 (2190003) and its fix fix(ci): namespace claude session ids in tracing seeds and allowlist /v1/logs on backend #44761 (ff50846, merged 2026-10-06T02:39Z); main is green since and the next build on this PR picks the fixed main up, so nothing to fix here

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• 78b0bea passes /live-pr-risk

• f1cdfae passes /audit

---

Note

Medium Risk
Changes core Responses API routing for ChatGPT subscription models—a high-traffic path—but behavior is narrowly scoped to separating transport streaming from caller-facing shape, with extensive regression tests.

Overview
Fixes a regression where ChatGPT/Codex always sets upstream stream: true, so the Responses handler treated every call as streaming and returned raw SSE (or broke chat/messages bridges) even when the caller omitted stream or sent stream: false.

Response shape is now driven only by the caller’s stream (including extra_body.stream), while httpx still streams when the provider requires SSE. Non-streaming callers get a buffered, aggregated JSON ResponsesAPIResponse; streaming callers still get the existing iterators (including fake_stream mocks only when they asked to stream).

ChatGPTResponsesAPIConfig accepts an optional Authenticator for unit tests so CI does not run device login.

Adds unit coverage for sync/async aggregation vs streaming and integration rigs (Codex wire double, agentic probe, burst/outage/worker-kill chaos) across /v1/responses, chat, and messages.

^{Reviewed by Cursor Bugbot for commit f1cdfae. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_responses_honor_caller_stream_flag (f1cdfae) with main (061d60b)¹}

Footnotes

1. No successful run was found on main (2156d6c) during the generation of this report, so 061d60b was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Fixes streaming behavior when provider forces server-sent events.

The PR appears safe to merge; no outstanding finding or new blocking issue was established.

Findings

1.  Manus streaming skips fake stream ▶
2.  Interrupted-stream test blocks ▶
3.  Redundant source comment ▶

Summary

The PR separates upstream SSE transport from the response shape requested by the caller, so non-streaming ChatGPT Responses calls can be aggregated into JSON. It also injects an authenticator for handler tests and adds wire and chaos integration coverage.

_{Reviews (6) · Last reviewed commit: "test(integration): audit the chatgpt cal..."}

[greptile-apps]

Redundant source comment

This comment only restates the return. Repository rules require source comments to explain necessary complex logic or serve tooling before merging

Suggested change

# Return the streaming iterator

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[mateo-berri]

@greptileai

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Manus streaming skips fake stream

When a Manus caller sets extra_body={"stream": true} without setting the top-level stream flag, this line selects a real streaming iterator. But should_fake_stream receives only the top-level flag, so Manus does not enable the fake-stream path it needs because it does not support real-time streaming. The request is sent through the SSE iterator instead and fails to produce the expected stream. The async handler has the same mismatch.

[mateo-berri]

Pre-existing: fake_stream is decided in responses/main.py from the top-level flag before this handler runs, the same on main, and extra_body.stream never fed it

[greptile-apps]

You're right — this mismatch is pre-existing and is not introduced by this PR. fake_stream is determined earlier from the top-level stream flag, and this handler has never propagated extra_body.stream into that decision (on main or in this change). The new caller_requested_stream only selects the response shape after that existing decision, so this comment does not identify a regression here. I’m withdrawing it.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit f1cdfae. Configure here.}

[mateo-berri]

@greptileai

[greptile-apps]

Interrupted-stream test blocks

The assertion message calls pending.result() while the vendor is waiting for _INCOMPLETE_GATE. The test does not set that gate until the next line, so it blocks instead of checking that the request is pending and can fail when the vendor’s five-second wait expires.

[mateo-berri]

Python evaluates an assert's message only when the condition fails, so pending.result() runs only once the future is already done and returns immediately

[greptile-apps]

You're right — the assertion message is evaluated only when not pending.done() is false. In that case, pending.done() guarantees the future has completed, so pending.result() returns immediately (assuming it was not cancelled). This finding is invalid; no change is needed.

[mateo-berri]

@greptileai

[mateo-berri]

LGTM