fix(caching): engage prompt caching for LiteLLM Claude on the OpenAI wire (envelope layout)

[kshitijk4poor]

Summary

LiteLLM-fronted Claude on the OpenAI-compatible wire now gets prompt caching, with the marker layout that wire actually honors. Salvage of #84982 (@justinbowes) onto current main, plus a follow-up commit that fixes the marker layout and three scoping gaps.

anthropic_prompt_cache_policy() only granted Anthropic cache_control to a LiteLLM endpoint reached over the native Anthropic wire. A LiteLLM deployment exposing /v1/chat/completions instead (with /v1/messages returning 404) matched no grant branch and fell through to (False, False) — no markers, system prompt on the wire as a plain string, zero cache hits, full prompt re-billed every turn, silently. Reproduced on current main:

anthropic_prompt_cache_policy(agent)  # provider=custom:litellm, api_mode=chat_completions, model=claude-opus-5
# -> (False, False)

Fixes #84506.

Commits

1. @justinbowes' original fix, cherry-picked — authorship preserved. Adds the grant branch plus the TestLiteLLMOpenAIWire matrix.
2. Envelope layout + three scoping fixes (below).
3. Token-wise provider matching — self-review caught that commit 2 fixed substring matching on the host but left the provider id as a bare substring, so custom:notlitellm still matched. Same bug class, half-fixed.
4. Perf narrowing — benchmarking commit 2 showed the widened capability-lookup gate pulled every litellm-ish chat_completions route into a config read, including non-Claude models the grant can never match. Narrowed to the exact grant condition and computed once.

Why the layout changed from (True, True) to (True, False)

This was the open question on #84982, and @Enough1122 raised it in #84550/#87039 on consistency grounds. @justinbowes' counter was fair — consistency isn't correctness, and he asked to settle it with measurements rather than a style argument. So we measured it.

use_native_layout=True makes _apply_cache_marker write a top-level msg["cache_control"] on role:tool and empty-content messages, relying on the Anthropic adapter to relocate it into the content block. That adapter only runs for api_mode == "anthropic_messages" (agent/transports/anthropic.py:251), and agent/transports/chat_completions.py performs no relocation at all. On this wire those markers are emitted as an unrecognized top-level field and lost.

Measured on a 3-tool-turn transcript, feeding Hermes's own planner output through the real litellm library:

	breakpoints honored	empty marked text blocks
(True, True)	2 of 4	1
(True, False)	4 of 4	0

The second column is the sharper problem. When LiteLLM itself relocates a top-level marker for an OpenRouter-backed Claude route (OpenrouterConfig._move_cache_control_to_content), the marker lands on an empty assistant turn (pure tool_calls) and produces a cache_control-marked empty text block — the HTTP 400 "text content blocks must contain" shape already guarded in agent/anthropic_adapter.py (#69512). That's the same failure class as the DeepSeek/OpenCode exclusion (#77217) the original PR cites to justify gating on Claude.

Two corroborating in-repo signals: the envelope layout is what every other OpenAI-wire grant in this function returns (OpenRouter Claude/Kimi, Portal Qwen, Alibaba/Qwen), and the function's own docstring says markers go on the envelope for "OpenRouter and OpenAI-wire proxies".

@justinbowes also flagged, correctly, that for a plain-string system prompt both layouts are byte-identical — so the headline 0%→96% result reproduces under either. The difference is entirely breakpoint placement on tool-result turns, which is what the table above measures.

Other fixes in commit 2

• Host matching — "litellm" in base_url_hostname(...) is the substring false-positive class base_url_hostname's own docstring warns against; it granted markers to notlitellm.attacker.example, foolitellmbar.example, etc. Replaced with a label-token match in a named helper (_is_litellm_route), so litellm must be a whole dot- or hyphen-delimited token. All three original test hosts still match; a litellm path segment on an unrelated host still doesn't. (fix(caching): combined LiteLLM OpenAI-wire prompt caching from #84550 and #84982 (#84506) #87039 made the same host-only call deliberately — kept.)
• Transport gate — not is_anthropic_wire also swept in codex_responses, bedrock_converse, codex_app_server. Now gated on api_mode == "chat_completions".
• Operator override — the grant is inferred from a provider/host name, but the custom-provider capability lookup was gated on is_anthropic_wire, so an explicit prompt_caching: false for the route+model was honored on /v1/messages and silently ignored on /v1/chat/completions. The lookup now also runs for a LiteLLM route, and its layout follows the transport rather than the declaration (an explicit true must not promote a chat_completions request to the native layout). prompt_caching.cache_ttl: false already won correctly and still does.

Validation

	Before	After
Grant on LiteLLM chat_completions + Claude	(False, False)	(True, False)
Breakpoints honored on that wire	0	4 of 4
Empty marked text blocks (400 shape)	—	0
Lookalike host notlitellm.attacker.example	—	(False, False)
Explicit prompt_caching: false on that wire	ignored	honored

• 82 tests pass across tests/run_agent/test_anthropic_prompt_cache_policy.py and tests/agent/test_prompt_cache_ttl_propagation.py; 165 pass across the wider cache/prompt selection.
• Mutation-checked: reverting each fix individually turns its guard red — no vacuous tests.
• Differential matrix, 9,900 configs vs origin/main (15 providers x 11 base URLs x 6 api_modes x 10 models): 147 behavior changes, all of them (False, False) -> (True, False). Zero non-chat_completions routes changed, zero non-Claude models changed, zero (True, True) anywhere. A lookalike host alone never grants — only an explicit litellm provider id does.
• Perf (realistic install, config.yaml present so the mtime cache engages), vs origin/main: live-agent policy 20.6us -> 61.7us; destination planning 219.3us -> 347.7us. Non-LiteLLM routes flat (openrouter Claude ~7.9us).
• Adds the wire-shape contract the original matrix was missing: it asserts no breakpoint sits on the message envelope, rather than only checking the returned tuple. That gap is why 15 green tests didn't catch the layout issue.
• ruff clean.

Prompt-caching invariants: markers stay request-local (build_prompt_cache_plan deep-copies), system-prompt bytes unchanged, _cache_disabled early return still precedes every grant. Single chokepoint — main loop, MoA, and auxiliary all resolve through this one function, so no sibling call path needs a separate fix.

Credit

Original diagnosis, reproduction, measurements, and proposed patch by @ottosulin in #84506. Implementation and test matrix by @justinbowes in #84982 (cherry-picked here with authorship preserved). The envelope-layout call was raised by @Enough1122 in #84550/#87039 — this PR confirms it with wire-level measurements, and adopts the host-only matching from #84550.

Depends on #87516 (attribution mapping for justin@bowes.org) — merge that first or check-attribution fails.

Closes #84982.

[alt-glitch]

This was generated by AI during triage.

Duplicate of #84550: both enable the same LiteLLM OpenAI-wire Claude envelope cache-policy grant. This revision adds hardening and coverage but no distinct runtime mechanism.