From 0d9e0cef5ac5727681b6bb951cc9430354f984dc Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Tue, 21 Jul 2026 21:30:11 -0700 Subject: [PATCH 1/2] docs(blog): add July 21 progress updates to the Bedrock Invoke caching incident report --- blog/bedrock_invoke_prompt_caching_incident/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/blog/bedrock_invoke_prompt_caching_incident/index.md b/blog/bedrock_invoke_prompt_caching_incident/index.md index 3cfa36e11..2f8be7220 100644 --- a/blog/bedrock_invoke_prompt_caching_incident/index.md +++ b/blog/bedrock_invoke_prompt_caching_incident/index.md @@ -95,8 +95,8 @@ Three PRs fixed it, all released July 10 in `v1.91.2` after extensive end-to-end ## What we are changing -- Our e2e suite will gain a scripted multi-turn Claude Code session growing to roughly 250k tokens of context against real Bedrock, asserting cache reads grow monotonically and never collapse (started in [#32963](https://github.com/BerriAI/litellm/pull/32963)). -- We will set up a weekly automated load test that will flag anomalies in spend, cache reads and writes, turn latency, error rates, etc., so cost and performance regressions are detected and fixed before a release. +- Our e2e suite will gain a scripted multi-turn Claude Code session growing to roughly 250k tokens of context against real Bedrock, asserting cache reads grow monotonically and never collapse (started in [#32963](https://github.com/BerriAI/litellm/pull/32963)). **Update (July 21):** [#32963](https://github.com/BerriAI/litellm/pull/32963) merged July 14 with live-Bedrock regression tests that fail whenever a mid-conversation system message costs the session its cached prefix, [#33807](https://github.com/BerriAI/litellm/pull/33807) extended the same coverage to Vertex AI and Azure, and the weekly load test below now drives multi-turn sessions that fail on any cache-read collapse. The full 250k-token scripted Claude Code session is still in progress. +- We will set up a weekly automated load test that will flag anomalies in spend, cache reads and writes, turn latency, error rates, etc., so cost and performance regressions are detected and fixed before a release. **Update (July 21):** done in [#34166](https://github.com/BerriAI/litellm/pull/34166). CI now runs concurrent multi-turn Claude Code-shaped sessions every week against real Anthropic and Bedrock Invoke deployments and fails if the error rate, the share of warm-turn input read from the cache, cache write volume, p95 turn latency, or the spend the gateway records for the run leaves its baseline. - Daily automated diffs of Anthropic's SDKs and docs alert us to new features that need translation support before customer traffic finds them. - We [dogfood](https://en.wikipedia.org/wiki/Eating_your_own_dog_food) LiteLLM internally and will set up monitoring for new request shapes, such as unknown `anthropic-beta` headers, and the same anomoly detction, which will alert us ahead of a release. - Bug fixes now have a higher merge bar: validated means reproduced against the real client's traffic on their exact end-user application end-to-end and a complete understanding of the root cause; synthetic requests are not enough. @@ -106,6 +106,6 @@ Three PRs fixed it, all released July 10 in `v1.91.2` after extensive end-to-end ## Known limitations 1. Converse rejects system entries inside `messages` at any position, so on `bedrock_converse` we must still hoist, and Claude Code sessions routed through Converse still lose cached prefix on every mid-conversation system message. If you run Claude Code against Bedrock, route it through the Invoke path (`bedrock/invoke/`). We are raising the API constraint with AWS. -2. We are testing whether the Vertex AI and Azure paths need equivalent hoisting and will update this post when we have more info. +2. We are testing whether the Vertex AI and Azure paths need equivalent hoisting and will update this post when we have more info. **Update (July 21):** they do, and [#33807](https://github.com/BerriAI/litellm/pull/33807) (merged July 20, in the next stable release) closes the gap. We verified live that Vertex AI and Azure enforce the same contract as Bedrock Invoke: every Claude model rejects a leading `role: "system"` entry inside `messages`, models from Opus 4.8 onward accept mid-conversation entries in place, and older models reject them in any position. Both paths now reuse the model-aware Invoke logic, hoisting only what each model rejects, so older models return completions instead of 400s and newer models keep the cached prefix byte-identical. Live e2e tests cover both providers. To every team whose bill went up because of this: we are sorry. The value of a gateway is that this class of provider change gets absorbed by us instead of reaching you, and the tests, monitoring, and process improvements above are how we intend to keep it that way. From f99cc485168b2bea351b43b9a4b2801ee6e97964 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:44:07 -0700 Subject: [PATCH 2/2] docs(blog): tighten July 21 updates into nested bullets in the caching incident report --- blog/bedrock_invoke_prompt_caching_incident/index.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/blog/bedrock_invoke_prompt_caching_incident/index.md b/blog/bedrock_invoke_prompt_caching_incident/index.md index 2f8be7220..a50905f11 100644 --- a/blog/bedrock_invoke_prompt_caching_incident/index.md +++ b/blog/bedrock_invoke_prompt_caching_incident/index.md @@ -95,8 +95,10 @@ Three PRs fixed it, all released July 10 in `v1.91.2` after extensive end-to-end ## What we are changing -- Our e2e suite will gain a scripted multi-turn Claude Code session growing to roughly 250k tokens of context against real Bedrock, asserting cache reads grow monotonically and never collapse (started in [#32963](https://github.com/BerriAI/litellm/pull/32963)). **Update (July 21):** [#32963](https://github.com/BerriAI/litellm/pull/32963) merged July 14 with live-Bedrock regression tests that fail whenever a mid-conversation system message costs the session its cached prefix, [#33807](https://github.com/BerriAI/litellm/pull/33807) extended the same coverage to Vertex AI and Azure, and the weekly load test below now drives multi-turn sessions that fail on any cache-read collapse. The full 250k-token scripted Claude Code session is still in progress. -- We will set up a weekly automated load test that will flag anomalies in spend, cache reads and writes, turn latency, error rates, etc., so cost and performance regressions are detected and fixed before a release. **Update (July 21):** done in [#34166](https://github.com/BerriAI/litellm/pull/34166). CI now runs concurrent multi-turn Claude Code-shaped sessions every week against real Anthropic and Bedrock Invoke deployments and fails if the error rate, the share of warm-turn input read from the cache, cache write volume, p95 turn latency, or the spend the gateway records for the run leaves its baseline. +- Our e2e suite will gain a scripted multi-turn Claude Code session growing to roughly 250k tokens of context against real Bedrock, asserting cache reads grow monotonically and never collapse (started in [#32963](https://github.com/BerriAI/litellm/pull/32963)). + - **Update (July 21):** [#32963](https://github.com/BerriAI/litellm/pull/32963) merged July 14 with live-Bedrock tests that fail whenever a mid-conversation system message costs a session its cached prefix; [#33807](https://github.com/BerriAI/litellm/pull/33807) extends the same coverage to Vertex AI and Azure. The full 250k-token session is still in progress. +- We will set up a weekly automated load test that will flag anomalies in spend, cache reads and writes, turn latency, error rates, etc., so cost and performance regressions are detected and fixed before a release. + - **Update (July 21):** done in [#34166](https://github.com/BerriAI/litellm/pull/34166). CI now runs concurrent Claude Code-shaped sessions weekly against real Anthropic and Bedrock Invoke deployments and fails if error rate, warm-turn cache reads, cache write volume, p95 turn latency, or recorded spend leaves its baseline. - Daily automated diffs of Anthropic's SDKs and docs alert us to new features that need translation support before customer traffic finds them. - We [dogfood](https://en.wikipedia.org/wiki/Eating_your_own_dog_food) LiteLLM internally and will set up monitoring for new request shapes, such as unknown `anthropic-beta` headers, and the same anomoly detction, which will alert us ahead of a release. - Bug fixes now have a higher merge bar: validated means reproduced against the real client's traffic on their exact end-user application end-to-end and a complete understanding of the root cause; synthetic requests are not enough. @@ -106,6 +108,7 @@ Three PRs fixed it, all released July 10 in `v1.91.2` after extensive end-to-end ## Known limitations 1. Converse rejects system entries inside `messages` at any position, so on `bedrock_converse` we must still hoist, and Claude Code sessions routed through Converse still lose cached prefix on every mid-conversation system message. If you run Claude Code against Bedrock, route it through the Invoke path (`bedrock/invoke/`). We are raising the API constraint with AWS. -2. We are testing whether the Vertex AI and Azure paths need equivalent hoisting and will update this post when we have more info. **Update (July 21):** they do, and [#33807](https://github.com/BerriAI/litellm/pull/33807) (merged July 20, in the next stable release) closes the gap. We verified live that Vertex AI and Azure enforce the same contract as Bedrock Invoke: every Claude model rejects a leading `role: "system"` entry inside `messages`, models from Opus 4.8 onward accept mid-conversation entries in place, and older models reject them in any position. Both paths now reuse the model-aware Invoke logic, hoisting only what each model rejects, so older models return completions instead of 400s and newer models keep the cached prefix byte-identical. Live e2e tests cover both providers. +2. We are testing whether the Vertex AI and Azure paths need equivalent hoisting and will update this post when we have more info. + - **Update (July 21):** they do, and [#33807](https://github.com/BerriAI/litellm/pull/33807) (merged July 20, in the next stable release) closes the gap. We verified live that both providers enforce the same per-model contract as Bedrock Invoke, so both paths now reuse the model-aware hoisting logic: older models return completions instead of 400s and newer models keep the cached prefix byte-identical. Live e2e tests cover both providers. To every team whose bill went up because of this: we are sorry. The value of a gateway is that this class of provider change gets absorbed by us instead of reaching you, and the tests, monitoring, and process improvements above are how we intend to keep it that way.