feat(models): add Bedrock mappings, fix cache - #1815
Conversation
Add aws-bedrock provider entries for Anthropic models that were missing Bedrock mappings but are available on AWS Bedrock. Prices verified against AWS Bedrock pricing. Models added: - claude-haiku-4-5 - claude-haiku-4-5-20251001 - claude-opus-4-6 - claude-3-haiku / claude-3-haiku-20240307 - claude-3-5-haiku-20241022 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
WalkthroughUpdates to Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
📝 Coding Plan for PR comments
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: bc646249-f6a8-4b11-866a-2400c573880f
📒 Files selected for processing (1)
packages/models/src/models/anthropic.ts
| { | ||
| providerId: "aws-bedrock", | ||
| modelName: "anthropic.claude-3-haiku-20240307-v1:0", | ||
| inputPrice: 0.25 / 1e6, | ||
| outputPrice: 1.25 / 1e6, | ||
| requestPrice: 0, | ||
| discount: 0.3, | ||
| contextSize: 200000, | ||
| maxOutput: 4096, | ||
| streaming: true, | ||
| vision: true, | ||
| tools: true, | ||
| }, |
There was a problem hiding this comment.
Don't rely on cachedInputPrice omission to disable Bedrock prompt caching.
Lines 734-746 and 771-783 intentionally drop cached pricing, but the Bedrock request path still falls back minCacheableTokens to 1024 when that field is missing in packages/actions/src/prepare-request-body.ts Lines 1139-1144, then emits cachePoint blocks from that threshold in Lines 1185-1190. For Claude 3 Haiku, cache-controlled requests can still be serialized as prompt-caching even though these mappings are supposed to represent a non-cacheable Bedrock model. Please gate cachePoint generation on an explicit caching capability instead of inferring it from price fields.
Also applies to: 771-783
Fix incorrect minCacheableTokens thresholds per official Anthropic docs: Haiku 4.5 → 4096, Sonnet 4.6 → 2048, Claude 3 Haiku → 1024. Remove Bedrock entries for models that don't support Global Cross-region inference. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rcogal
changed the title
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/models/src/models/anthropic.ts`:
- Line 709: Update the Claude 3 Haiku model entries to use a 2,048-token cache
threshold: find the model config(s) that set minCacheableTokens (the Claude 3
Haiku model identifier / config object) and change minCacheableTokens from 1024
to 2048 so the client won’t mark prompts as cacheable when Anthropic requires
≥2048 tokens; apply the same change to the other Claude 3 Haiku occurrence noted
in the review (the second minCacheableTokens instance).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 0ed7ee65-1461-4e18-b1b6-e792d33964f2
📒 Files selected for processing (1)
packages/models/src/models/anthropic.ts
| outputPrice: 1.25 / 1e6, | ||
| cachedInputPrice: 0.03 / 1e6, | ||
| minCacheableTokens: 2048, | ||
| minCacheableTokens: 1024, |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Find and examine the anthropic.ts file to understand the context around lines 709 and 733
cd packages/models/src/models && cat -n anthropic.ts | sed -n '700,740p'Repository: theopenco/llmgateway
Length of output: 1326
🏁 Script executed:
# Search for all minCacheableTokens occurrences in the file to see all affected instances
rg "minCacheableTokens" packages/models/src/models/anthropic.ts -nRepository: theopenco/llmgateway
Length of output: 1249
🌐 Web query:
Anthropic Claude 3 Haiku prompt caching minimum token threshold requirements 2025 2026
💡 Result:
For Anthropic prompt caching in the Anthropic API, the minimum cacheable prompt length for Claude 3 Haiku (and Claude 3.5 Haiku) is 2,048 tokens. Prompts shorter than that won’t be cached even if you set cache_control. [1], [2]
Use 2,048-token cache threshold for Claude 3 Haiku models.
Anthropic's prompt-caching documentation requires a minimum of 2,048 tokens for Claude 3 Haiku. Setting minCacheableTokens to 1,024 will cause the request body preparation to mark prompts below 2,048 tokens as cacheable, but Anthropic's API will not actually cache them.
🔧 Suggested fix
- minCacheableTokens: 1024,
+ minCacheableTokens: 2048,Also applies to: 733
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/models/src/models/anthropic.ts` at line 709, Update the Claude 3
Haiku model entries to use a 2,048-token cache threshold: find the model
config(s) that set minCacheableTokens (the Claude 3 Haiku model identifier /
config object) and change minCacheableTokens from 1024 to 2048 so the client
won’t mark prompts as cacheable when Anthropic requires ≥2048 tokens; apply the
same change to the other Claude 3 Haiku occurrence noted in the review (the
second minCacheableTokens instance).
2 participants
Summary
minCacheableTokensvalues across multiple Anthropic models per official docsBedrock models added:
claude-haiku-4-5anthropic.claude-haiku-4-5-20251001-v1:0claude-haiku-4-5-20251001anthropic.claude-haiku-4-5-20251001-v1:0claude-opus-4-6anthropic.claude-opus-4-6-v1Bedrock models removed (no Global Cross-region support):
claude-3-haiku/claude-3-haiku-20240307claude-3-5-haiku-20241022minCacheableTokensfixes:Test plan
pnpm buildpassespnpm formatpassesLOG_MODE=trueanthropic/claude-haiku-4-5(cached_tokens: 5843)🤖 Generated with Claude Code