retain tool cache_control for bedrock converse - #5811
Merged
akshaydeo merged 1 commit intoAug 4, 2026
Merged
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughSummary by CodeRabbit
WalkthroughChangesBedrock invoke cache and usage
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant AnthropicInvoke
participant BedrockConverse
participant BifrostResponse
AnthropicInvoke->>BedrockConverse: send system and tool CachePoint entries
BedrockConverse->>BifrostResponse: return input, output, cache creation, and cache read usage
BifrostResponse->>AnthropicInvoke: expose mapped usage fields
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.12.2)level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies" Comment |
akshaydeo
marked this pull request as ready for review
August 3, 2026 19:03
Contributor
Author
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This was referenced Aug 3, 2026
akshaydeo
force-pushed
the
08-03-retain_tool_cache_control_for_bedrock_converse
branch
from
August 4, 2026 04:15
ffe8e62 to
d76cb55
Compare
TejasGhatte
reviewed
Aug 4, 2026
This was referenced Aug 4, 2026
Contributor
Author
Merge activity
|
akshaydeo
deleted the
08-03-retain_tool_cache_control_for_bedrock_converse
branch
August 4, 2026 09:13
This was referenced Aug 7, 2026
Closed
atharvamhaske
pushed a commit
to atharvamhaske/bifrost
that referenced
this pull request
Aug 13, 2026
## Summary Prompt caching markers (`cache_control`) on Anthropic-native invoke requests were silently dropped when routing through Bedrock's Converse API. This meant cache breakpoints set on system blocks and tools never reached the model, and cache token counts (`cache_creation_input_tokens`, `cache_read_input_tokens`) were never surfaced in either streaming or non-streaming `/invoke` responses. Closes maximhq#5629 ## Changes - **System block cache_control translation**: `parseSystemMessages` now detects a per-block `cache_control` field on Anthropic-dialect system entries and appends a positional `cachePoint` entry immediately after the marked block, matching the behavior `convertSystemMessages` already applies on the Bifrost→Bedrock egress path. - **Tool cache_control translation**: `convertAnthropicTools` now detects `cache_control` on each Anthropic-native tool map and appends a positional `cachePoint` entry after the marked tool. Nova-family exclusion is handled downstream by the shared `ToBifrostResponsesRequest` builder, so no additional gating is needed here. - **Cache token fields in non-streaming responses**: `toBedrockInvokeAnthropicResponse` now routes usage through `buildBedrockTokenUsage` and populates `cache_creation_input_tokens` and `cache_read_input_tokens` on `BedrockInvokeMessagesUsage`, which gains those two `omitempty` fields. - **Cache token fields in streaming responses**: `toAnthropicInvokeStreamBytes` now includes `input_tokens` and, when non-zero, `cache_read_input_tokens` and `cache_creation_input_tokens` in the `message_delta` usage object. Because Bedrock Converse only reports usage on the terminal stream event, all token counts are consolidated there rather than split across `message_start`/`message_delta` as native Anthropic does. - **`cacheControlTTL`** **helper**: Extracts the optional `"ttl"` string from an Anthropic-dialect `cache_control` map so it can be forwarded to `newBedrockCachePoint`, which already enforces Bedrock's supported TTL allow-list. - **New test file** **`invoke_test.go`**: Covers no-cache-control regression, cache_control on tools and system blocks, TTL forwarding and fallback, Nova-family exclusion, the full invoke→Converse→Bifrost pipeline, streaming usage fields, and non-streaming usage fields. ## Type of change - [x] Bug fix ## Affected areas - [x] Core (Go) - [x] Providers/Integrations ## How to test ```sh go test ./core/providers/bedrock/... -v -run "TestConvertAnthropicTools|TestParseSystemMessages|TestToBedrockConverseRequest_InvokeCache|TestToAnthropicInvokeStreamBytes|TestToBedrockInvokeAnthropicResponse" ``` Expected: all new tests pass. Existing Bedrock tests remain unaffected. To validate end-to-end, send an Anthropic-native `/invoke` request with `cache_control` on a system block or tool to a Claude model via Bedrock and confirm: - The model receives the cache breakpoint (observable via `cache_creation_input_tokens` > 0 on the first turn). - Subsequent identical requests show `cache_read_input_tokens` > 0 and reduced `input_tokens`. - Streaming responses include all three token fields in the `message_delta` event. ## Breaking changes - [x] No ## Related issues Closes maximhq#5629 ## Security considerations None. Changes are limited to token-count field mapping and cache marker translation within the Bedrock provider. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
akhsaul
pushed a commit
to akhsaul/bifrost
that referenced
this pull request
Aug 27, 2026
## Summary Prompt caching markers (`cache_control`) on Anthropic-native invoke requests were silently dropped when routing through Bedrock's Converse API. This meant cache breakpoints set on system blocks and tools never reached the model, and cache token counts (`cache_creation_input_tokens`, `cache_read_input_tokens`) were never surfaced in either streaming or non-streaming `/invoke` responses. Closes maximhq#5629 ## Changes - **System block cache_control translation**: `parseSystemMessages` now detects a per-block `cache_control` field on Anthropic-dialect system entries and appends a positional `cachePoint` entry immediately after the marked block, matching the behavior `convertSystemMessages` already applies on the Bifrost→Bedrock egress path. - **Tool cache_control translation**: `convertAnthropicTools` now detects `cache_control` on each Anthropic-native tool map and appends a positional `cachePoint` entry after the marked tool. Nova-family exclusion is handled downstream by the shared `ToBifrostResponsesRequest` builder, so no additional gating is needed here. - **Cache token fields in non-streaming responses**: `toBedrockInvokeAnthropicResponse` now routes usage through `buildBedrockTokenUsage` and populates `cache_creation_input_tokens` and `cache_read_input_tokens` on `BedrockInvokeMessagesUsage`, which gains those two `omitempty` fields. - **Cache token fields in streaming responses**: `toAnthropicInvokeStreamBytes` now includes `input_tokens` and, when non-zero, `cache_read_input_tokens` and `cache_creation_input_tokens` in the `message_delta` usage object. Because Bedrock Converse only reports usage on the terminal stream event, all token counts are consolidated there rather than split across `message_start`/`message_delta` as native Anthropic does. - **`cacheControlTTL`** **helper**: Extracts the optional `"ttl"` string from an Anthropic-dialect `cache_control` map so it can be forwarded to `newBedrockCachePoint`, which already enforces Bedrock's supported TTL allow-list. - **New test file** **`invoke_test.go`**: Covers no-cache-control regression, cache_control on tools and system blocks, TTL forwarding and fallback, Nova-family exclusion, the full invoke→Converse→Bifrost pipeline, streaming usage fields, and non-streaming usage fields. ## Type of change - [x] Bug fix ## Affected areas - [x] Core (Go) - [x] Providers/Integrations ## How to test ```sh go test ./core/providers/bedrock/... -v -run "TestConvertAnthropicTools|TestParseSystemMessages|TestToBedrockConverseRequest_InvokeCache|TestToAnthropicInvokeStreamBytes|TestToBedrockInvokeAnthropicResponse" ``` Expected: all new tests pass. Existing Bedrock tests remain unaffected. To validate end-to-end, send an Anthropic-native `/invoke` request with `cache_control` on a system block or tool to a Claude model via Bedrock and confirm: - The model receives the cache breakpoint (observable via `cache_creation_input_tokens` > 0 on the first turn). - Subsequent identical requests show `cache_read_input_tokens` > 0 and reduced `input_tokens`. - Streaming responses include all three token fields in the `message_delta` event. ## Breaking changes - [x] No ## Related issues Closes maximhq#5629 ## Security considerations None. Changes are limited to token-count field mapping and cache marker translation within the Bedrock provider. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
occcat
pushed a commit
to occcat/bifrost
that referenced
this pull request
Sep 2, 2026
## Summary Prompt caching markers (`cache_control`) on Anthropic-native invoke requests were silently dropped when routing through Bedrock's Converse API. This meant cache breakpoints set on system blocks and tools never reached the model, and cache token counts (`cache_creation_input_tokens`, `cache_read_input_tokens`) were never surfaced in either streaming or non-streaming `/invoke` responses. Closes maximhq#5629 ## Changes - **System block cache_control translation**: `parseSystemMessages` now detects a per-block `cache_control` field on Anthropic-dialect system entries and appends a positional `cachePoint` entry immediately after the marked block, matching the behavior `convertSystemMessages` already applies on the Bifrost→Bedrock egress path. - **Tool cache_control translation**: `convertAnthropicTools` now detects `cache_control` on each Anthropic-native tool map and appends a positional `cachePoint` entry after the marked tool. Nova-family exclusion is handled downstream by the shared `ToBifrostResponsesRequest` builder, so no additional gating is needed here. - **Cache token fields in non-streaming responses**: `toBedrockInvokeAnthropicResponse` now routes usage through `buildBedrockTokenUsage` and populates `cache_creation_input_tokens` and `cache_read_input_tokens` on `BedrockInvokeMessagesUsage`, which gains those two `omitempty` fields. - **Cache token fields in streaming responses**: `toAnthropicInvokeStreamBytes` now includes `input_tokens` and, when non-zero, `cache_read_input_tokens` and `cache_creation_input_tokens` in the `message_delta` usage object. Because Bedrock Converse only reports usage on the terminal stream event, all token counts are consolidated there rather than split across `message_start`/`message_delta` as native Anthropic does. - **`cacheControlTTL`** **helper**: Extracts the optional `"ttl"` string from an Anthropic-dialect `cache_control` map so it can be forwarded to `newBedrockCachePoint`, which already enforces Bedrock's supported TTL allow-list. - **New test file** **`invoke_test.go`**: Covers no-cache-control regression, cache_control on tools and system blocks, TTL forwarding and fallback, Nova-family exclusion, the full invoke→Converse→Bifrost pipeline, streaming usage fields, and non-streaming usage fields. ## Type of change - [x] Bug fix ## Affected areas - [x] Core (Go) - [x] Providers/Integrations ## How to test ```sh go test ./core/providers/bedrock/... -v -run "TestConvertAnthropicTools|TestParseSystemMessages|TestToBedrockConverseRequest_InvokeCache|TestToAnthropicInvokeStreamBytes|TestToBedrockInvokeAnthropicResponse" ``` Expected: all new tests pass. Existing Bedrock tests remain unaffected. To validate end-to-end, send an Anthropic-native `/invoke` request with `cache_control` on a system block or tool to a Claude model via Bedrock and confirm: - The model receives the cache breakpoint (observable via `cache_creation_input_tokens` > 0 on the first turn). - Subsequent identical requests show `cache_read_input_tokens` > 0 and reduced `input_tokens`. - Streaming responses include all three token fields in the `message_delta` event. ## Breaking changes - [x] No ## Related issues Closes maximhq#5629 ## Security considerations None. Changes are limited to token-count field mapping and cache marker translation within the Bedrock provider. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prompt caching markers (
cache_control) on Anthropic-native invoke requests were silently dropped when routing through Bedrock's Converse API. This meant cache breakpoints set on system blocks and tools never reached the model, and cache token counts (cache_creation_input_tokens,cache_read_input_tokens) were never surfaced in either streaming or non-streaming/invokeresponses.Closes #5629
Changes
parseSystemMessagesnow detects a per-blockcache_controlfield on Anthropic-dialect system entries and appends a positionalcachePointentry immediately after the marked block, matching the behaviorconvertSystemMessagesalready applies on the Bifrost→Bedrock egress path.convertAnthropicToolsnow detectscache_controlon each Anthropic-native tool map and appends a positionalcachePointentry after the marked tool. Nova-family exclusion is handled downstream by the sharedToBifrostResponsesRequestbuilder, so no additional gating is needed here.toBedrockInvokeAnthropicResponsenow routes usage throughbuildBedrockTokenUsageand populatescache_creation_input_tokensandcache_read_input_tokensonBedrockInvokeMessagesUsage, which gains those twoomitemptyfields.toAnthropicInvokeStreamBytesnow includesinput_tokensand, when non-zero,cache_read_input_tokensandcache_creation_input_tokensin themessage_deltausage object. Because Bedrock Converse only reports usage on the terminal stream event, all token counts are consolidated there rather than split acrossmessage_start/message_deltaas native Anthropic does.cacheControlTTLhelper: Extracts the optional"ttl"string from an Anthropic-dialectcache_controlmap so it can be forwarded tonewBedrockCachePoint, which already enforces Bedrock's supported TTL allow-list.invoke_test.go: Covers no-cache-control regression, cache_control on tools and system blocks, TTL forwarding and fallback, Nova-family exclusion, the full invoke→Converse→Bifrost pipeline, streaming usage fields, and non-streaming usage fields.Type of change
Affected areas
How to test
Expected: all new tests pass. Existing Bedrock tests remain unaffected.
To validate end-to-end, send an Anthropic-native
/invokerequest withcache_controlon a system block or tool to a Claude model via Bedrock and confirm:cache_creation_input_tokens> 0 on the first turn).cache_read_input_tokens> 0 and reducedinput_tokens.message_deltaevent.Breaking changes
Related issues
Closes #5629
Security considerations
None. Changes are limited to token-count field mapping and cache marker translation within the Bedrock provider.
Checklist
docs/contributing/README.mdand followed the guidelines