Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions blog/bedrock_invoke_prompt_caching_incident/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,9 @@ Three PRs fixed it, all released July 10 in `v1.91.2` after extensive end-to-end
## What we are changing

- Our e2e suite will gain a scripted multi-turn Claude Code session growing to roughly 250k tokens of context against real Bedrock, asserting cache reads grow monotonically and never collapse (started in [#32963](https://github.com/BerriAI/litellm/pull/32963)).
- **Update (July 21):** [#32963](https://github.com/BerriAI/litellm/pull/32963) merged July 14 with live-Bedrock tests that fail whenever a mid-conversation system message costs a session its cached prefix; [#33807](https://github.com/BerriAI/litellm/pull/33807) extends the same coverage to Vertex AI and Azure. The full 250k-token session is still in progress.
- We will set up a weekly automated load test that will flag anomalies in spend, cache reads and writes, turn latency, error rates, etc., so cost and performance regressions are detected and fixed before a release.
- **Update (July 21):** done in [#34166](https://github.com/BerriAI/litellm/pull/34166). CI now runs concurrent Claude Code-shaped sessions weekly against real Anthropic and Bedrock Invoke deployments and fails if error rate, warm-turn cache reads, cache write volume, p95 turn latency, or recorded spend leaves its baseline.
- Daily automated diffs of Anthropic's SDKs and docs alert us to new features that need translation support before customer traffic finds them.
- We [dogfood](https://en.wikipedia.org/wiki/Eating_your_own_dog_food) LiteLLM internally and will set up monitoring for new request shapes, such as unknown `anthropic-beta` headers, and the same anomoly detction, which will alert us ahead of a release.
- Bug fixes now have a higher merge bar: validated means reproduced against the real client's traffic on their exact end-user application end-to-end and a complete understanding of the root cause; synthetic requests are not enough.
Expand All @@ -107,5 +109,6 @@ Three PRs fixed it, all released July 10 in `v1.91.2` after extensive end-to-end

1. Converse rejects system entries inside `messages` at any position, so on `bedrock_converse` we must still hoist, and Claude Code sessions routed through Converse still lose cached prefix on every mid-conversation system message. If you run Claude Code against Bedrock, route it through the Invoke path (`bedrock/invoke/<model>`). We are raising the API constraint with AWS.
2. We are testing whether the Vertex AI and Azure paths need equivalent hoisting and will update this post when we have more info.
- **Update (July 21):** they do, and [#33807](https://github.com/BerriAI/litellm/pull/33807) (merged July 20, in the next stable release) closes the gap. We verified live that both providers enforce the same per-model contract as Bedrock Invoke, so both paths now reuse the model-aware hoisting logic: older models return completions instead of 400s and newer models keep the cached prefix byte-identical. Live e2e tests cover both providers.

To every team whose bill went up because of this: we are sorry. The value of a gateway is that this class of provider change gets absorbed by us instead of reaching you, and the tests, monitoring, and process improvements above are how we intend to keep it that way.