fix(agentic loop): return a real stream when code-interpreter interception converts the request

[vineethsaivs]

Title

fix(agentic loop): return a real stream when code-interpreter interception converts the request

Relevant issues

Fixes #37652

Pre-Submission checklist

• I have added testing in the tests/ directory: two cases in tests/test_litellm/litellm_core_utils/test_chat_completion_agentic_loop.py
• I have added a screenshot of my new test passing locally (output pasted below)
• My PR passes all unit tests
• My PR's scope is as isolated as possible

Type

🐛 Bug Fix

Root cause

code_interpreter_interception converts a stream: true chat request into a non-streaming call, marks it with _code_interpreter_interception_converted_stream, and then asks the agentic-loop dispatcher to hand the result back in streamed form.

_wrap_response_as_fake_stream (litellm/litellm_core_utils/chat_completion_agentic_loop.py) did that with convert_model_response_to_streaming, which returns a single ModelResponseStream object. That is one chunk, not a stream: it has no __aiter__. The caller iterates it, so:

TypeError: 'async for' requires an object with __aiter__ method, got ModelResponseStream

Two consequences, both of which the issue reports:

1. litellm/main.py guards the result with if isinstance(response, CustomStreamWrapper), and a ModelResponseStream fails that check, so the object is returned to the proxy untouched and blows up on iteration. This fires on both call sites, which is why a plain assistant reply with no tool call is enough: the no-follow-up path at the end of maybe_run_chat_completion_agentic_loop converts the original response the same way.
2. The chunk-shaped object is what reaches the response cache. A later request that reads that entry has delta where message is expected, hence the reported KeyError: 'message' inside convert_to_model_response_object.

The existing cast("ModelResponse | CustomStreamWrapper", ...) on the return value was simply not true.

Fix

Wrap the response in a CustomStreamWrapper over MockResponseIterator, which is the pattern already used elsewhere in the repo for exactly this (replaying an assembled response as a stream): see bedrock_guardrails.py, tool_permission.py, model_armor.py and six other guardrail hooks. MockResponseIterator calls the same convert_model_response_to_streaming internally, so the chunk content is unchanged; it is now delivered through an object that is actually iterable, and main.py's isinstance branch recognises it.

_wrap_response_as_fake_stream needs model, custom_llm_provider and logging_obj to build the wrapper. All three are already in scope at both call sites. If logging_obj is not a real logging object the helper returns the response unchanged, so this can never turn a working call into a new exception.

Not folded in: the interception path still cannot deliver token-by-token output, since the underlying call really is non-streaming. This makes the response a well-formed single-chunk SSE stream rather than a 500, which is option (a) in the issue at the granularity the current design allows.

Testing

Two new tests in the existing tests/test_litellm/litellm_core_utils/test_chat_completion_agentic_loop.py, one per affected path (after a tool-call follow-up, and with no tool call at all). Both assert the result is a CustomStreamWrapper and then actually async for over it and check the reassembled text.

$ pytest tests/test_litellm/litellm_core_utils/test_chat_completion_agentic_loop.py -q
.......                                                                  [100%]
7 passed in 0.77s

The same file against unpatched chat_completion_agentic_loop.py:

E       AssertionError: assert False
E        +  where False = isinstance(ModelResponseStream(id='chatcmpl-9d7f8aff-...', object='chat.completion.chunk', ...), CustomStreamWrapper)

FAILED ... ::test_converted_stream_result_is_async_iterable_after_the_loop_runs
FAILED ... ::test_converted_stream_result_is_async_iterable_without_a_tool_call
2 failed, 5 passed in 0.64s

Removing the isinstance assertion and going straight to the iteration reproduces the reported error verbatim on unpatched source:

returned type: ModelResponseStream
has __aiter__: False
async for RAISED TypeError: 'async for' requires an object with __aiter__ method, got ModelResponseStream

Regression control, the whole surrounding area, branch versus a pristine tree:

$ pytest tests/test_litellm/integrations/code_interpreter_interception tests/test_litellm/litellm_core_utils/ -q
branch:   3 failed, 1694 passed, 1 skipped in 45.44s
pristine: 3 failed, 1692 passed, 1 skipped in 37.98s

Exactly the two new tests, no other movement. The 3 failures are pre-existing on the base branch (test_bedrock_converse_messages_pt_document_various_formats, TestIsBlockedIp::test_blocks_ietf_protocol_assignments_old_oracle_metadata, test_logfire_logger_accepts_env_vars_for_base_url) and unrelated.

ruff check reports the identical 31 pre-existing findings on both files before and after, none in the changed lines; ruff format --check reports both files already formatted.

To be explicit about what I did not do: I validated this through the dispatcher and the test suite, not against a live proxy with a real sandboxed code-interpreter run, so the end-to-end SSE delivery to a client is not something I can claim to have exercised. I also could not reproduce the cache half of the issue directly; the fix addresses it by never producing the chunk-shaped object in the first place, which is the object the reporter traced the KeyError: 'message' to.

[codecov]

Codecov Report

❌ Patch coverage is 77.77778% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...litellm_core_utils/chat_completion_agentic_loop.py	77.77%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing vineethsaivs:fix-code-interpreter-converted-stream (edcd78b) with litellm_internal_staging (b091158)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (8672cd4) during the generation of this report, so b091158 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩