feat: ✨ AWS Bedrock Provider tool streaming #7657
Open
+67
−77
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Enable tool streaming for the AWS Bedrock provider (note we had some confusion with the openai-adapter code previously which has already been updated).
Some time back the tool code was revised to support streaming tool call output and this takes advantage of this change.
When the model is claude, we are also enabling the fine grained tool call beta header to speed up tool output to not wait for matching JSON end tags for streaming. Beta headers are ignored when no longer applicable and are only enabled for claude models.
AI Code Review
@continue-general-review
or@continue-detailed-review
Checklist
Screen recording or screenshot
[ When applicable, please include a short screen recording or screenshot - this makes it much easier for us as contributors to review and understand your changes. See this PR as a good example. ]
Tests
Manual testing performed to validate changes. Visually noting that on file creation and edit activities for example the user receives feedback/output much faster. Before this change the provider is building the entire tool call payload before sending any data.
Summary by cubic
Enable streaming tool calls for the AWS Bedrock provider so tool output arrives incrementally instead of after full JSON completion. For Claude models, enable fine-grained tool streaming to reduce latency.
New Features
Refactors