[Feat] Add streaming support for using bedrock invoke models with /v1/messages

[ishaan-jaff]

[Feat] Add streaming support for using bedrock invoke models with /v1/messages

This PR adds streaming support for using Bedrock invoke models via the /v1/messages endpoint by updating the asynchronous handlers and streaming iterators. Key changes include adding a new test for Bedrock invoke streaming, refactoring the async handler in llm_http_handler.py to utilize get_async_streaming_response_iterator, and updating various transformation and invoke handler modules to properly parse streaming chunks.

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• I have added a screenshot of my new test passing locally
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

🆕 New Feature
✅ Test

Changes

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 10, 2025 1:03am

[Copilot]

Pull Request Overview

This PR adds streaming support for using Bedrock invoke models via the /v1/messages endpoint by updating the asynchronous handlers and streaming iterators. Key changes include adding a new test for Bedrock invoke streaming, refactoring the async handler in llm_http_handler.py to utilize get_async_streaming_response_iterator, and updating various transformation and invoke handler modules to properly parse streaming chunks.

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py	Added a new test to verify streaming support for Bedrock invoke models.
litellm/llms/custom_httpx/llm_http_handler.py	Updated async handler to use get_async_streaming_response_iterator for streaming responses.
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py	Introduced asynchronous streaming iterator support for Bedrock invoke responses.
litellm/llms/bedrock/chat/invoke_handler.py	Modified _chunk_parser to include dict in its union return type for unparsed cases.
litellm/llms/base_llm/anthropic_messages/transformation.py	Extended the base transformation with an abstract get_async_streaming_response_iterator.
litellm/llms/anthropic/experimental_pass_through/messages/transformation.py	Implemented get_async_streaming_response_iterator for handling Anthropic streaming responses.
litellm/llms/anthropic/experimental_pass_through/messages/handler.py	Removed the deprecated static streaming method in favor of the new implementation.

Comments suppressed due to low confidence (2)

tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py:161

• [nitpick] Consider adding assertions to verify that the streaming chunks are processed as expected, rather than solely printing them.

async def test_anthropic_messages_streaming_bedrock_invoke():

litellm/llms/bedrock/chat/invoke_handler.py:1416

• It would be beneficial to include a docstring or inline comment explaining the conditions under which this method returns a dict, to improve clarity and maintainability.

def _chunk_parser(self, chunk_data: dict) -> Union[GChunk, ModelResponseStream, dict]: