fix: preserve role='assistant' in Azure streaming with include_usage

[ishaan-berri]

Relevant issues

Fixes #24221

Pre-Submission checklist

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting `@greptileai` and received a Confidence Score of at least 4/5 before requesting a maintainer review

Type

🐛 Bug Fix

Changes

When stream_options.include_usage=True, Azure sends an initial chunk with choices=[] (prompt_filter_results) before the first content chunk. LiteLLM inflated this empty-choices chunk with a default StreamingChoices, which:

1. Consumed the sent_first_chunk flag
2. Caused strip_role_from_delta to strip role from the real first chunk
3. The real first chunk with role='assistant' and content='' was also discarded by is_chunk_non_empty as "empty"

Net result: no chunk ever contained role='assistant'.

Fix (4 files):

• streaming_handler.py - chunk_creator: Set model_response.choices = [] for chunks without choices, forwarding them faithfully instead of inflating with a default StreamingChoices
• streaming_handler.py - is_chunk_non_empty: Treat chunks with role in delta as non-empty (the first chunk with role='assistant' and content='' is a valid OpenAI chunk)
• streaming_handler.py - __next__/__anext__: Guard choices[0] access and only mark sent_first_chunk for chunks with real choices
• main.py + streaming_chunk_builder_utils.py: Guard choices[0] access in stream_chunk_builder for chunks with choices=[]

Originally filed as #24354 (merged to a staging branch, not main).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Apr 13, 2026 4:02pm

[codspeed-hq]

Merging this PR will not alter performance

✅ 16 untouched benchmarks

---

_{Comparing fix/azure-streaming-role-assistant (3794fd2) with main (0eae9f1)}

[codecov]

Codecov Report

❌ Patch coverage is 84.61538% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/litellm_core_utils/streaming_handler.py	84.61%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

Fixes a bug where Azure streaming with stream_options.include_usage=True omitted role='assistant' from all chunks: the initial choices=[] prompt-filter chunk was being inflated with a default StreamingChoices, prematurely consuming sent_first_chunk and then discarding the real first-content chunk as empty. The fix forwards empty-choices chunks faithfully, guards choices[0] access in both iterator paths, and adds len(choices) == 0 skips in stream_chunk_builder.

• P1: __anext__ asyncio.to-thread path (line 2133) is missing the same if processed_chunk.choices: guard added to the async for path — would crash with IndexError for any sync-only iterator provider emitting choices=[].
• P2: self.stream_options[\"include_usage\"] at line 1570 uses a direct key lookup instead of .get(), risking KeyError if stream_options is set without the include_usage key.

Confidence Score: 4/5

Safe to merge for the reported Azure scenario; one P1 inconsistency in the asyncio.to-thread path should be fixed to prevent future regressions.

The primary fix (path 1 of next/anext, stream_chunk_builder, chunk_creator) is correct and covered by a well-written regression test. The P1 issue — missing choices guard in anext path 2 (asyncio.to_thread for sync-only iterators) — doesn't affect Azure today since Azure uses async iterables, but the stated goal of the PR was to guard all choices[0] accesses and this one was missed.

litellm/litellm_core_utils/streaming_handler.py — specifically the asyncio.to_thread branch of anext (line 2133) and the stream_options dict access (line 1570).

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/streaming_handler.py	Core fix: chunk_creator now returns model_response with choices=[] instead of None for Azure include_usage empty chunks; next and anext async-for path gain choices guards; but the asyncio.to_thread path (line 2133) still accesses choices[0] without a guard.
litellm/litellm_core_utils/streaming_chunk_builder_utils.py	build_base_response correctly uses next(c for c in chunks if c.get('choices'), chunk) to skip empty-choices chunks when extracting role/finish_reason; iteration loop guards with len(chunk['choices']) > 0 — looks safe.
litellm/main.py	stream_chunk_builder fast-path and slow-path both guard with len(chunk['choices']) == 0 before accessing choices[0]; the empty-choices chunks are correctly skipped during assembly.
tests/test_litellm/litellm_core_utils/test_streaming_handler.py	Adds a dedicated regression test (test_azure_streaming_role_preserved_with_include_usage) that exercises both sync and async modes with real Azure-shaped chunks (choices=[], then role='assistant', then content); existing tests unchanged.

Sequence Diagram

sequenceDiagram
    participant Azure as Azure API
    participant CSW as CustomStreamWrapper
    participant CC as chunk_creator
    participant RPC as return_processed_chunk_logic

    Note over Azure,RPC: include_usage=True stream with prompt_filter_results

    Azure->>CSW: Chunk 1 choices=[] prompt_filter_results
    CSW->>CC: chunk_creator(chunk)
    CC->>CC: original_chunk.choices is empty else branch
    CC-->>CSW: model_response choices=[] NEW was None before fix
    CSW->>CSW: if response.choices False skip sent_first_chunk update GUARDED
    CSW-->>Azure: yield chunk with choices=[]

    Azure->>CSW: Chunk 2 choices=[delta(role=assistant, content='')]
    CSW->>CC: chunk_creator(chunk)
    CC->>RPC: return_processed_chunk_logic
    RPC->>RPC: is_chunk_non_empty True role present sent_first_chunk=False
    RPC->>RPC: strip_role_from_delta sets role=assistant sent_first_chunk=True
    CC-->>CSW: model_response role=assistant
    CSW-->>Azure: yield chunk with role=assistant

    Azure->>CSW: Chunk 3 choices=[delta(content=Hello!)]
    CSW->>CC: chunk_creator(chunk)
    CC-->>CSW: model_response content=Hello!
    CSW-->>Azure: yield content chunk

Loading

Comments Outside Diff (2)

1. litellm/litellm_core_utils/streaming_handler.py, line 2133-2138 (link)

   Missing choices guard in __anext__ asyncio.to_thread path

   Path 1 (the async for branch, line 2053) correctly guards choices[0] with if processed_chunk.choices:, but path 2 (the asyncio.to_thread branch for sync-only iterators) accesses processed_chunk.choices[0] directly. If a provider using a sync-only completion stream (one without __aiter__) emits a choices=[] chunk with include_usage=True, this raises IndexError.

2. litellm/litellm_core_utils/streaming_handler.py, line 1568-1573 (link)

   Direct dict key access can raise KeyError

   self.stream_options["include_usage"] throws KeyError if stream_options is a non-empty dict that doesn't contain the include_usage key (e.g. {"show_all": True}). The surrounding is not None guard doesn't protect against this. Prefer .get():

_{Reviews (1): Last reviewed commit: "fix: preserve role='assistant' in Azure ..." | Re-trigger Greptile}