fix: forward Anthropic prompt-caching cache_control through the gateway

[sakibsadmanshajib]

Problem

Hive's Anthropic-compatible /v1/messages endpoint (apps/edge-api/internal/anthropic/) had no cache_control field anywhere on its request or response types. translate_request.go rebuilds the Anthropic request field by field into an internal OpenAI-shaped OAIRequest, and cache_control was never one of the fields carried across, so it was silently dropped before the request ever reached LiteLLM. Every agent client that relies on Anthropic prompt caching (Claude Code, Cline, Cursor, Aider) got zero caching through Hive and paid full input price on every turn.

What this PR does

Request side. Adds a CacheControl struct ({"type":"ephemeral","ttl":"5m"|"1h"}) and wires it through every documented placement:

• content block (text, image, tool_use, tool_result) — ContentBlock.CacheControl
• system prompt block — SystemField now retains the original Blocks (previously collapsed to one concatenated string, losing any block-level data)
• tool definition — Tool.CacheControl (caching one tool in the array caches the whole array up to it; Anthropic has no separate "group" field)
• request root — MessagesRequest.CacheControl

All four are mirrored onto the internal OAI-shaped types (OAIContentPart, OAIMessage, OAIToolCall, OAITool, OAIRequest) with identical JSON shape, since the marshaled OAIRequest body is forwarded essentially byte-for-byte to LiteLLM (litellm_client.go's rewriteModel only ever touches the "model" key).

The flattening bug. Two spots in translate_request.go collapsed a content-block array down to a plain string when it looked simple enough (a lone text block; a lone text block mixed with tool calls). A flat string cannot carry a per-block cache_control, so a request that DID set a breakpoint on that block would have had it silently destroyed even with the field wired in. Both collapse sites now check for a cache breakpoint first and keep the block-array form when one is present, otherwise reproducing the exact prior output. TestToOAIRequest_CacheControl_SingleTextBlock_NotFlattenedToString is a dedicated regression test for this; I confirmed it actually fails without the fix (reverted the guard locally, test failed with the flattened bare string, restored the fix, test passes).

Response side — exclusive-shape echo contract. OpenRouter (Hive's only real dispatch path) reports usage in the OpenAI-compatible INCLUSIVE convention: prompt_tokens already contains the cached and written tokens. Anthropic-native clients (Claude Code's cost display included) expect the EXCLUSIVE convention: input_tokens as the fresh/uncached remainder only, plus cache_creation_input_tokens and cache_read_input_tokens reported alongside it. freshInputTokens in translate_response.go does that conversion (fresh = prompt - cached - written, clamped to 0, never negative) for both the non-streaming response and the streaming path (message_delta.usage in stream.go, since that relay only knows the terminal numbers once the upstream's usage-bearing chunk arrives — message_start fires before that and stays at its pre-existing zero, consistent with how output_tokens already worked before this change).

session_id. Not part of Anthropic's own /v1/messages schema, but OpenRouter uses it for sticky provider routing, which is what keeps a cache warm across turns. Added as passthrough on MessagesRequest/OAIRequest (truncated at 256 chars per OpenRouter's documented ceiling) so a client/proxy that sends it is no longer silently dropped by the same field-by-field rebuild.

Merge coordination (feat/cache-aware-billing)

This PR's OAIUsage.PromptTokensDetails ({cached_tokens, cache_write_tokens}) is a capture-only struct in the anthropic package. The bytes it unmarshals come from the inference package's own re-marshal of inference.UsageResponse (normalizeChatCompletion in chat_completions.go, and the typed chunk round-trip in stream.go) — both of which do a typed unmarshal-then-remarshal, which silently drops any JSON field their own struct doesn't declare. Today inference.UsageResponse has PromptTokensDetails.CachedTokens but no CacheWriteTokens, and OpenRouter's cache_creation_input_tokens/cache_read_input_tokens fields (if reported in Anthropic-native shape at all through the OpenRouter hop) have no matching field on that struct either.

Net effect: the response-side cache token plumbing in this PR is real and tested against the anthropic package's own types, but it cannot carry live data end-to-end until feat/cache-aware-billing adds a matching CacheWriteTokens (and whatever else it needs) to inference.UsageResponse with the same JSON field name (cache_write_tokens) I used here. I stayed out of pricing.go, stream.go (inference package), orchestrator.go and types.go per the coordination boundary; this is the merge point to close by hand once both branches land — I'd suggest whichever branch merges second checks the other's final field names against this PR's OAIPromptTokensDetails.

Verified against the actual upstream path

• Request side is safe today, independent of the billing coordination above. chat_completions.go's handleChatCompletions forwards the original raw request body bytes downstream (not a re-marshal of its own typed ChatCompletionRequest, whose Messages/Tools fields are json.RawMessage anyway), and litellm_client.go's rewriteModel only replaces the "model" key via map[string]json.RawMessage. So every cache_control this PR adds survives byte-for-byte from /v1/messages through to LiteLLM's /chat/completions.
• Whether it survives LiteLLM → OpenRouter → the actual model is NOT live-verified from this sandbox (no live provider keys available to this task). deploy/litellm/config.yaml currently routes every chat alias to openrouter/dots-studio/dots-3-note-preview:free plus four paid non-Anthropic routes (deepseek-v4, doc-vlm, embedding fallback) — there is no Anthropic/Claude model in the live catalog today, so cache_control has nowhere to take effect in production right now regardless of this fix. This is a routing/catalog fact, not something this PR's scope covers or should change. Recommend a follow-up live smoke test once/if a Claude route is added: send a >2k-token prompt with a cache_control breakpoint to that alias and confirm cache_read_input_tokens/cache_creation_input_tokens come back non-zero on the second turn.

Other fields translate_request.go drops on the floor (audit, not fixed here)

Per the task brief's request to audit this file for every dropped field even if not all get fixed:

• tool_choice: {"type":"none"} silently becomes "auto". convertToolChoice's switch only handles auto/any/tool; anything else (including the documented none, which means "forbid tool use") falls to the default branch and returns auto. This is a behavior inversion, not just an omission — a caller that explicitly disabled tools would get tool use enabled. Worth its own follow-up issue; flagged here per audit but not fixed, out of this PR's cache_control scope.
• MessagesRequest.Metadata (metadata.user_id) is parsed but ToOAIRequest never reads it at all.
• ToolChoice.disable_parallel_tool_use has no field at all on the ToolChoice type.
• Anthropic's top_k sampling parameter has no field on MessagesRequest at all (dropped at the type layer, not just the translator).
• Anthropic's extended-thinking thinking request field, and thinking/redacted_thinking content blocks, have no representation at all. A message consisting solely of a thinking block would fall through convertMessage's content-block switch with no matching case and produce an OAI message with empty content — silently, no error.
• The pre-existing "tool calls plus text" branch in convertMessage still drops any image block mixed with tool calls when neither has a cache_control (documented in a code comment at the fix site rather than fixed, to keep the no-cache-control regression guard exact).

Tests

Table-driven and mutation-checked (I deliberately broke the flattening guard locally and confirmed the dedicated test failed, then restored the fix and confirmed it passed again — see cache_control_test.go):

• TestToOAIRequest_CacheControl_Placements — system block, message content block (+ 1h ttl variant), tool definition, request root
• TestToOAIRequest_CacheControl_SingleTextBlock_NotFlattenedToString — the flattening regression guard
• TestToOAIRequest_NoCacheControl_Unchanged — regression guard: a request with no cache_control anywhere serializes with zero cache_control keys anywhere in the output
• TestToOAIRequest_CacheControl_ToolResultBlock, TestToOAIRequest_CacheControl_ToolUseBlock
• TestToOAIRequest_SessionID_Passthrough, TestToOAIRequest_SessionID_TruncatedAt256
• TestFromOAIResponse_CacheTokens_NonStreaming, TestFromOAIResponse_CacheTokens_ClampedNeverNegative, TestFromOAIResponse_NoCacheDetails_InputTokensUnchanged
• TestSSETranslator_CacheTokensInMessageDelta

Full apps/edge-api suite (all packages, including inference) passes on the rebased branch.

Buglog entry

{"error_message":"cache_control silently dropped on /v1/messages, zero prompt caching for agent clients","root_cause":"translate_request.go rebuilds the Anthropic request field-by-field into OAIRequest; cache_control (content block, system block, tool, request root) was never one of the carried fields, and two collapse sites in convertMessage flattened a typed content-block array to a plain string, which cannot carry a per-block cache_control even once the field exists","fix":"added CacheControl to ContentBlock/Tool/MessagesRequest/SystemField and their OAI-shaped mirrors; guarded both flattening sites in translate_request.go to keep block-array form when a cache breakpoint is present; added the exclusive-shape cache_creation_input_tokens/cache_read_input_tokens echo to ResponseUsage and StreamUsage via freshInputTokens' inclusive-to-exclusive subtraction","tags":["anthropic","cache_control","prompt-caching","translate_request","billing-coordination"]}

Test plan

• go build ./apps/edge-api/...
• go vet ./apps/edge-api/...
• go test ./apps/edge-api/... -count=1 (all packages green, rebased on origin/main)
• Mutation check on the flattening-regression test (fails without the fix, passes with it)
• Live smoke test once a Claude/Anthropic route exists in the catalog (tracked as a follow-up, not blocking)

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[coderabbitai]

Warning

Review limit reached

Next included review available in 19 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 88740127-2ee1-4045-9271-ed00a70e522c

📥 Commits

Reviewing files that changed from the base of the PR and between bfea338 and a982ee1.

📒 Files selected for processing (6)

• apps/edge-api/internal/anthropic/cache_control_test.go
• apps/edge-api/internal/anthropic/handler.go
• apps/edge-api/internal/anthropic/stream.go
• apps/edge-api/internal/anthropic/translate_request.go
• apps/edge-api/internal/anthropic/translate_response.go
• apps/edge-api/internal/anthropic/types.go

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sakibsadmanshajib]

Independent adversarial review (security-reviewer pass). Read the pushed diff directly, ran the anthropic-package suite in the project's docker toolchain, and independently reproduced the author's mutation-check claim by reverting the flattening guard locally (test failed as claimed, then restored, confirmed clean). No blocking findings; two MEDIUM correctness/observability gaps and two LOW hardening nits, all inline below. I would not block this merge on them, but MEDIUM #1 should get a follow-up before this feature is called billing-complete.

Verified true (not just trusted):

• Request-side byte-for-byte forwarding claim: confirmed by reading chat_completions.go (raw body bytes, not a re-marshaled typed struct, flow into executeSync/executeStreaming) and litellm_client.go's rewriteModel (map[string]json.RawMessage passthrough, only the "model" key is touched). cache_control and session_id do survive this hop unchanged.
• Flattening regression guard: traced every touched branch in convertMessage/systemContent by hand for the no-cache_control case; all reproduce the prior output exactly. Also reproduced the author's mutation check myself (reverted the parts[0].CacheControl == nil guard, TestToOAIRequest_CacheControl_SingleTextBlock_NotFlattenedToString failed with the exact flattened-string message the test asserts against, restored and confirmed clean).
• No logging/persistence of raw cache_control or session_id values anywhere in the diff.
• Full apps/edge-api/internal/anthropic suite green in the docker toolchain, including all new cache_control tests.
• No panics/nil-deref risk: every CacheControl pointer field is either passed straight through or nil-checked before use (systemContent's bl.CacheControl != nil).

Would not block merge. Recommend addressing MEDIUM #1 (or explicitly tracking it) before calling cache billing observability done, and linking a tracking issue for MEDIUM #2's coordination gap so cache_creation_input_tokens isn't silently wrong in production between when this merges and when feat/cache-aware-billing lands.