Skip to content

Conversation

chezsmithy
Copy link
Contributor

@chezsmithy chezsmithy commented Sep 9, 2025

Description

Enable tool streaming for the AWS Bedrock provider (note we had some confusion with the openai-adapter code previously which has already been updated).

Some time back the tool code was revised to support streaming tool call output and this takes advantage of this change.

When the model is claude, we are also enabling the fine grained tool call beta header to speed up tool output to not wait for matching JSON end tags for streaming. Beta headers are ignored when no longer applicable and are only enabled for claude models.

AI Code Review

  • Team members only: AI review runs automatically when PR is opened or marked ready for review
  • Team members can also trigger a review by commenting @continue-general-review or @continue-detailed-review

Checklist

  • I've read the contributing guide
  • The relevant docs, if any, have been updated or created
  • The relevant tests, if any, have been updated or created

Screen recording or screenshot

[ When applicable, please include a short screen recording or screenshot - this makes it much easier for us as contributors to review and understand your changes. See this PR as a good example. ]

Tests

Manual testing performed to validate changes. Visually noting that on file creation and edit activities for example the user receives feedback/output much faster. Before this change the provider is building the entire tool call payload before sending any data.


Summary by cubic

Enable streaming tool calls for the AWS Bedrock provider so tool output arrives incrementally instead of after full JSON completion. For Claude models, enable fine-grained tool streaming to reduce latency.

  • New Features

    • Stream tool calls via ConverseStream: emit toolCalls on start and as input deltas arrive.
    • Enable Claude’s fine-grained tool streaming (anthropic_beta: fine-grained-tool-streaming-2025-05-14); ignored for non-Claude models.
    • Stream reasoning content, including text, signatures, and redacted thinking.
  • Refactors

    • Remove internal tool input accumulation; rely on Bedrock content block start/delta events.
    • Consolidate event parsing for ContentBlockStart/Delta and ToolUseBlock/Delta to simplify streaming.

@chezsmithy chezsmithy requested a review from a team as a code owner September 9, 2025 16:50
@chezsmithy chezsmithy requested review from sestinj and removed request for a team September 9, 2025 16:50
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Sep 9, 2025
@chezsmithy chezsmithy changed the title Feat-bedrock-tool-streaming feat: ✨ AWS Bedrock Provider tool streaming Sep 9, 2025
@chezsmithy chezsmithy force-pushed the feat-bedrock-tool-streaming branch from 6cb1723 to a4ffff1 Compare September 11, 2025 15:19
@chezsmithy chezsmithy force-pushed the feat-bedrock-tool-streaming branch from a4ffff1 to 7d390ee Compare September 11, 2025 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:L This PR changes 100-499 lines, ignoring generated files.
Projects
Status: Todo
Development

Successfully merging this pull request may close these issues.

1 participant