Skip to content

fix(anthropic): preserve message cache control for list content - #43447

Draft
Oxygenesis wants to merge 2 commits into
BerriAI:mainfrom
Oxygenesis:codex/anthropic-cache-control
Draft

Oxygenesis wants to merge 2 commits into
BerriAI:mainfrom
Oxygenesis:codex/anthropic-cache-control

Conversation

@Oxygenesis

@Oxygenesis Oxygenesis commented Sep 27, 2026 •

Copy link
Copy Markdown

Summary

  • Preserve message-level cache_control for Anthropic list content by applying it to the last converted block when no block-level setting exists.
  • Preserve the same behavior for Anthropic system/assistant paths and Bedrock Converse sync/async user paths.
  • Keep block-level cache_control precedence.
  • Add mocked regression coverage.

Problem

Fixes #43324. When content is a list, the translators inspect each content element but ignore cache_control on the containing message, so prompt-cache breakpoints disappear silently for Anthropic and Bedrock Converse.

Testing

  • Targeted regression tests: 3 passed.
  • Affected suites: 912 passed with the macOS text/markdown MIME mapping supplied; the native run otherwise hit one unrelated existing MIME-type fixture failure.
  • Ruff format/check passed for the changed production modules.
  • py_compile and git diff --check passed.
  • No provider credentials or live model calls were used; the test socket guard blocked an attempted external connection.

@codspeed

codspeed Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing Oxygenesis:codex/anthropic-cache-control (ed55255) with main (d463049)

Open in CodSpeed

@codecov

codecov Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.96970% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...llm/litellm_core_utils/prompt_templates/factory.py 96.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@Oxygenesis

Copy link
Copy Markdown
Author

The latest head (ed55255) now passes the lint/type fix and coverage checks. The remaining misc shard is failing in four unrelated OpenAPI compliance tests: test_create_model_interaction_request_schema, test_input_types_match_spec, test_delete_endpoint_exists, and test_get_endpoint_exists in tests/unit/interactions/test_openapi_compliance.py (CreateModelInteractionParams and GET/DELETE interaction endpoints are absent from the loaded spec). This PR only changes Bedrock cache-control call typing and does not touch the interaction schema or tests, so I have left the unrelated baseline issue out of this PR.

@Oxygenesis

Copy link
Copy Markdown
Author

Follow-up on the shared CI failure: the four OpenAPI failures in the misc shard are now isolated in #43807, which updates the compliance tests to the current live Interactions schema; its OpenAPI tests pass in CI. I'm keeping this Bedrock cache-control PR focused and leaving those unrelated test changes out. After #43807 lands, this PR should only need a misc-shard rerun.

@Oxygenesis

Copy link
Copy Markdown
Author

I also opened #43812 for the separate ultrafast model-cost schema mismatch now present on main. This Bedrock cache-control PR remains unchanged; after the OpenAPI and cost-map maintenance fixes land, rerun the misc shard here.

@Oxygenesis

Copy link
Copy Markdown
Author

Update: #43764 brought the ultrafast schema fields into main; #43807 now covers the remaining OpenAPI test drift, with refreshed CI running

@Oxygenesis

Copy link
Copy Markdown
Author

Correction: #43532 now contains the current Interactions contract update and its misc shard passes; #43807 was a duplicate, so I closed it

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Message-level cache_control is dropped when the message's content is a list (Anthropic, Bedrock Converse)

1 participant