fix(proxy): classify /v1/messages pass-through streams as Anthropic on any host

[devin-ai-integration]

TLDR

Problem this solves:

• Two integration jobs on main fail every scheduled CircleCI run
• Streamed /anthropic/v1/messages calls through a non-Anthropic host log 0 tokens and $0
• An Arize cache test still expects /v1/messages cache hits without x-litellm-cache-key

How it solves it:

• Treat any pass-through target ending in /v1/messages as Anthropic, via is_anthropic_messages_url in the Anthropic package
• Streams then reach the Anthropic logger, so spend rows get usage, cost and the provider message id
• Assert every cache surface, /v1/messages included, returns a 64-hex x-litellm-cache-key

User Flow

Before: a team that points ANTHROPIC_API_BASE at their own gateway sees every streamed call logged at $0

1. The proxy admin sets ANTHROPIC_API_BASE=https://gateway.internal and restarts the proxy
2. A developer sends POST https://litellm-domain/anthropic/v1/messages with "stream": true
3. The stream comes back correctly and their app renders the reply
4. https://litellm-domain/ui/?page=logs shows the request with 0 tokens, $0 spend and a random UUID as request id

After: the same streamed call is logged with real usage, cost and the provider's msg_... id

1. The proxy admin sets ANTHROPIC_API_BASE=https://gateway.internal and restarts the proxy
2. A developer sends POST https://litellm-domain/anthropic/v1/messages with "stream": true
3. The stream comes back correctly and their app renders the reply
4. https://litellm-domain/ui/?page=logs shows the request with real token counts, non-zero spend and the msg_... id

Relevant issues

Related #40117. Overlaps the Anthropic half of #40137 by @Sunny-Soni00 (idle since 2026-10-08, conflicting with main); credit is carried as a co-author on the squash commit

CircleCI failures on main

Red scheduled integration workflow on main: pipeline 91062, also red in pipeline 91070 (workflow)

• integration-providers job 2252899: tests/integration/messages_endpoint/providers/anthropic/test_passthrough_migration_wire.py::test_native_passthrough_spend_rows_record_usage_tags_and_spend
  • Culprit: none in recent history. get_endpoint_type has matched only the api.anthropic.com host since c107bae ((feat) add usage / cost tracking for Anthropic passthrough routes #6835, 2024). The test added by 0e26edf (test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline #45288) is the first to stream through a non-Anthropic host, and it fails at its own merge and base
  • Mechanism: the stream is classified GENERIC, skips the Anthropic chunk logger, response_obj.id stays empty and the spend row falls back to litellm_call_id with 0 tokens
• integration-extensions job 2252896: tests/integration/observability/test_arize_otel_v2_openinference_spans.py::test_arize_otel_v2_a_cache[messages]
  • Culprit: d364d5e (refactor: expose public hidden_params accessors #44668, refactor: expose public hidden_params accessors), whose "keep hidden params writes on duck-typed responses" change lets the cache-hit cache_key land on dict /v1/messages responses. Bisected: bce020f passes, d364d5e fails
  • 308c42a (fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans #43698) then added a test pinning the pre-refactor: expose public hidden_params accessors #44668 behavior (no header on a messages hit); its base already contained refactor: expose public hidden_params accessors #44668, so it was red from the merge
  • Fix: the test now asserts the same 64-hex header on every surface, which is what chat and responses already return

Deferred to a later run (covered by #45530 or not yet root-caused), all red on main too: llm_translation_testing xAI reasoning effort, integration-accounting (red on main in each of the last 13 scheduled runs; Vertex image regional uplift chaos), integration-providers Bedrock Mantle and OpenAI dialect prompt-cache breakpoint tests, test_decisions_wire, worker-kill chaos tests, Redis usage-based routing reads. Worker-kill tests test_daily_activity_key_owner_faults, test_daily_activity_key_metadata_query_timeout and test_responses_websocket_session_limit failed once on this pipeline and passed twice each locally at the tip; they never touch pass-through routing

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

CircleCI on this branch: pipeline 91073 at a975c14388 (supersedes 91072). integration-extensions job 2253346 passed with all four test_arize_otel_v2_a_cache surfaces green. integration-providers job 2253353 no longer fails test_native_passthrough_spend_rows_record_usage_tags_and_spend; the job stays red on the deferred tests below, as on main. Every build_and_test job passed

Live proxy against the real Anthropic API. ANTHROPIC_API_BASE points at a local relay on 127.0.0.1 that forwards every byte to https://api.anthropic.com, standing in for a corporate gateway. Spend rows read from LiteLLM_SpendLogs

Before (63a4f3f)

Streamed /anthropic/v1/messages

1. POST http://127.0.0.1:/anthropic/v1/messages with "stream": true returns 200, stream id msg_011Cfrt69zdLqgRc3jxwYG4v
2. Spend row: request_id 93515cee-158e-46f3-8fb2-de8b480527cf, 0 prompt / 0 completion tokens, spend 0, provider empty

Non-streamed /anthropic/v1/messages

1. Same route without stream returns 200, id msg_011Cfrt7SNQKm2MZSfefd2J7
2. Spend row: request_id msg_011Cfrt7SNQKm2MZSfefd2J7, 12 / 4 tokens, spend 3.2e-05

Streamed custom pass-through endpoint targeting /v1/messages

1. POST http://127.0.0.1:/my-anthropic with "stream": true returns 200, stream id msg_011CfrtAkYdRKusAPB55NcvU
2. Spend row: request_id 09b8d854-8edc-424f-9f3c-1edd1329a719, 0 / 0 tokens, spend 0

After (a975c14)

Streamed /anthropic/v1/messages

1. POST http://127.0.0.1:/anthropic/v1/messages with "stream": true returns 200, stream id msg_011CfruEVk7iZTBcWjddMRwb
2. Spend row: request_id msg_011CfruEVk7iZTBcWjddMRwb, 12 prompt / 4 completion tokens, spend 3.2e-05, provider anthropic

Non-streamed /anthropic/v1/messages

1. Same route without stream returns 200, id msg_011CfruF9cBhK4XZUzh4xHES
2. Spend row: request_id msg_011CfruF9cBhK4XZUzh4xHES, 12 / 4 tokens, spend 3.2e-05 (unchanged)

Streamed custom pass-through endpoint targeting /v1/messages

1. POST http://127.0.0.1:/my-anthropic with "stream": true returns 200, stream id msg_011CfruFbRChdst7Zrn3wqTw
2. Spend row: request_id msg_011CfruFbRChdst7Zrn3wqTw, 12 / 4 tokens, spend 3.2e-05

/anthropic/v1/messages/count_tokens returned 200 {"input_tokens":12} on both revisions. The Arize change is test-only; its proof is integration-extensions passing in the pipeline above

Type

Bug Fix
Test

Caveats (if any)

Low

• Custom pass-through endpoints whose target ends in /v1/messages now use the Anthropic stream logger
  • This matches how their non-streaming calls are already logged

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• a975c14 passes /live-pr-risk

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge, with both previous findings addressed and no new blocking issues

Summary

This PR sends pass-through streams targeting /v1/messages on alternate hosts to the Anthropic logger

• Messages API streams through gateways now use Anthropic logging.
• Arize cache tests check the same key shape for messages.

_{Reviews (2) · Last reviewed commit: "refactor(anthropic): move the messages r..." · Reviewed by Greptile}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_circleci_heal_anthropic_stream_spend (a975c14) with main (63a4f3f)}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a975c14. Configure here.}

[mateo-berri]

LGTM