fix(anthropic adapter): stop indexing choices[0] on choiceless streaming chunks

[devin-ai-integration]

TLDR

Problem this solves:

• choices: [] chunks crash /v1/messages streaming
• Client gets IndexError: list index out of range mid-stream
• Empty choices is valid OpenAI-compatible streaming (vLLM usage chunk)

How it solves it:

• Skip choiceless chunks before the content-block state machine
• Fold a choiceless usage chunk into the held message_delta

Relevant issues

Linear ticket

Resolves LIT-5034

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

An OpenAI-compatible upstream that emits a chunk with "choices": [] is required to hit this, and no hosted provider I have credentials for emits one on demand, so the upstream here is a small OpenAI-compatible server that streams the exact chunk shape vLLM sends (vllm/entrypoints/openai/serving_chat.py emits choices=[], usage=final_usage). Everything else in the path is the real proxy, the real router, and the real /v1/messages adapter

Upstream server (mock_vllm.py), streaming a choiceless chunk plus three content chunks, a finish chunk, and a final usage chunk:

def gen():
    base = {"id": "chatcmpl-mock", "object": "chat.completion.chunk", "created": created, "model": model}
    yield "data: " + json.dumps({**base, "choices": []}) + "\n\n"
    yield "data: " + json.dumps({**base, "choices": [{"index": 0, "delta": {"role": "assistant", "content": ""}, "finish_reason": None}]}) + "\n\n"
    for tok in ["Hello", " there", "!"]:
        yield "data: " + json.dumps({**base, "choices": [{"index": 0, "delta": {"content": tok}, "finish_reason": None}]}) + "\n\n"
    yield "data: " + json.dumps({**base, "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}) + "\n\n"
    yield "data: " + json.dumps({**base, "choices": [], "usage": {"prompt_tokens": 10, "completion_tokens": 3, "total_tokens": 13}}) + "\n\n"
    yield "data: [DONE]\n\n"

Config:

model_list:
  - model_name: mock-vllm
    litellm_params:
      model: hosted_vllm/mock-model
      api_base: http://localhost:8123/v1
      api_key: fake-key
      stream_options:
        include_usage: true

Command (identical for both runs):

python -m uvicorn mock_vllm:app --port 8123 &
python litellm/proxy/proxy_cli.py --config config.yaml --detailed_debug &

curl -s -N localhost:4000/v1/messages \
  -H 'content-type: application/json' -H 'x-api-key: sk-1234' \
  -d '{"model":"mock-vllm","max_tokens":100,"stream":true,"messages":[{"role":"user","content":"Say hello."}]}'

Before, at 71b825a7f0; the stream dies right after message_start, no content ever reaches the client:

event: message_start
data: {"type": "message_start", "message": {"id": "msg_8feb8dd9-...", "type": "message", "role": "assistant", "content": [], "model": "mock-model", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 0, "output_tokens": 0, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 0}}}

data: {"error": {"message": "list index out of range\n\nTraceback (most recent call last):\n ...
  File \"litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py\", line 623, in __anext__\n    should_start_new_block = self._should_start_new_content_block(chunk)\n
  File \"litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py\", line 909, in _should_start_new_content_block\n    if chunk.choices[0].finish_reason is not None:\n       ~~~~~~~~~~~~~^^^\nIndexError: list index out of range\n", "type": "None", "param": "None", "code": "500"}}

After, at 0b809cf7d6; full Anthropic event sequence, content intact, upstream usage carried onto message_delta:

event: message_start
data: {"type": "message_start", "message": {"id": "msg_95e8a488-...", "type": "message", "role": "assistant", "content": [], "model": "mock-model", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 0, "output_tokens": 0, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 0}}}

event: content_block_start
data: {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "Hello"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": " there"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "!"}}

event: content_block_stop
data: {"type": "content_block_stop", "index": 0}

event: message_delta
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn"}, "usage": {"input_tokens": 10, "output_tokens": 3}}

event: message_stop
data: {"type": "message_stop"}

Same repro without stream_options.include_usage in the config crashes identically before the fix and passes after, so this is not gated on the usage option

Type

🐛 Bug Fix

Changes

AnthropicStreamWrapper.__next__ and __anext__ ran _should_start_new_content_block(chunk) on every chunk, and that helper (along with _is_blank_delta and the is_final_chunk computation) reads chunk.choices[0] unconditionally. A chunk with no choices carries no content-block information at all, so both loops now consume it before the state machine runs:

if not getattr(chunk, "choices", None):
    if self._handle_choiceless_chunk(chunk):
        return self.chunk_queue.popleft()
    continue

_handle_choiceless_chunk keeps the existing hold-and-merge semantics: when a message_delta is being held for its stop_reason and the choiceless chunk carries usage, the usage is merged into that held chunk and the merged message_delta is queued, exactly as the choice-carrying usage chunk path already did. Otherwise the chunk is dropped, since Anthropic's SSE has no event to represent it

Regression coverage lives in tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_streaming_iterator_empty_choices.py: one test streams a metadata-only chunk ahead of the content and asserts the full text still arrives, the other puts the choiceless usage chunk last and asserts its tokens land on message_delta. Both fail on the parent commit (the sync path swallows the IndexError into a truncated stream, the async path propagates it)

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ devin-ai-integration[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

Fixes Anthropic-compatible streaming for OpenAI chunks with empty choices.

• Skips metadata-only chunks before entering the content-block state machine.
• Merges final usage-only chunks into the held message_delta.
• Adds synchronous and asynchronous regression coverage for empty-choice chunks.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py	Adds equivalent sync and async handling for choiceless metadata and final usage chunks while preserving the existing message-delta merge lifecycle.
tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_streaming_iterator_empty_choices.py	Adds focused regression tests covering a leading metadata-only chunk and a final choiceless usage chunk.

_{Reviews (2): Last reviewed commit: "fix(anthropic adapter): type _handle_cho..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 84.21053% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...mental_pass_through/adapters/streaming_iterator.py	84.21%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing devin_ai_fix_lit5034_empty_choices (166a97e) with litellm_internal_staging (e46721a)}

[devin-ai-integration]

Merge conflict with litellm_internal_staging resolved in 4068b4a; the only clash was _ensure_context_management_attached being retyped from Dict[str, Any] to MessageBlockDelta upstream, so this branch takes the new signature and keeps _handle_choiceless_chunk next to it

Proof of fix, re-run on the merged head. Upstream is a local OpenAI-compatible server on :8123 that streams a metadata chunk with "choices": [], three content deltas, a finish chunk, then a final "choices": [] chunk carrying usage, which is what vLLM sends with stream_options.include_usage. Two proxies ran against the same config, one on :4001 at 81ff7cb (the parent of this branch) and one on :4000 at 4068b4a (this branch), and the same request went to both; only the port differs

curl -s -N localhost:4000/v1/messages -H 'content-type: application/json' -H 'x-api-key: sk-1234' \
  -d '{"model":"mock-vllm","max_tokens":100,"stream":true,"messages":[{"role":"user","content":"Say hello."}]}'

After, on :4000 at 4068b4a; the choiceless usage chunk lands as usage on message_delta and the stream finishes normally:

event: message_start
data: {"type": "message_start", "message": {"id": "msg_ee5f782d-8e4b-4ef0-888c-5b75ce7a3b40", "type": "message", "role": "assistant", "content": [], "model": "mock-model", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 0, "output_tokens": 0, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 0}}}

event: content_block_start
data: {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "Hello"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": " there"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "!"}}

event: content_block_stop
data: {"type": "content_block_stop", "index": 0}

event: message_delta
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn"}, "usage": {"input_tokens": 10, "output_tokens": 3}}

event: message_stop
data: {"type": "message_stop"}

Before, on :4001 at 81ff7cb; the stream dies right after message_start and the client never sees any content:

event: message_start
data: {"type": "message_start", "message": {"id": "msg_59b5c4a9-d88d-4056-a627-0bc886915796", "type": "message", "role": "assistant", "content": [], "model": "mock-model", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 0, "output_tokens": 0, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 0}}}

data: {"error": {"message": "list index out of range\n\nTraceback (most recent call last):\n ...
  File \"litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py\", line 623, in __anext__\n    should_start_new_block = self._should_start_new_content_block(chunk)\n
  File \"litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py\", line 909, in _should_start_new_content_block\n    if chunk.choices[0].finish_reason is not None:\n       ~~~~~~~~~~~~~^^^\nIndexError: list index out of range\n", "type": "None", "param": "None", "code": "500"}}

Screenshots of the same two runs, captured on the pre-merge commit 0b809cf:

After the fix

Before the fix

Upstream SSE and proxy config used

[mateo-berri]

@greptileai

[mateo-berri]

LGTM