fix(proxy): resolve pass-through endpoint type from the route, not just the host

[Sunny-Soni00]

TLDR

Problem this solves:

• Pass-through endpoint type was resolved from the hostname alone
• Anthropic-compatible providers on any other host fell through to generic
• Their streamed calls logged an unparseable blob, no tokens, no spend

How it solves it:

• Fall back to the request path when the host checks miss
• A route ending in /v1/messages is Anthropic, /v1/chat/completions is OpenAI
• Suffix match, so only those two canonical routes are claimed

User Flow

Before: a developer who puts the gateway in front of a provider that speaks the Anthropic Messages API sees every streamed call logged at zero spend

1. They add a pass-through endpoint on their proxy whose target is their provider's https://<their-host>/v1/messages
2. They send POST http://localhost:4000/my-provider/v1/messages with "stream": true and an ordinary Anthropic Messages body
3. The stream comes back correctly and their app renders the reply
4. They look up that request in the proxy logs and find its response stored as the literal text cannot parse chunks to standard response object followed by the raw stream
5. The same entry carries no prompt or completion token counts and no cost, so their per-customer cost reporting misses every streamed call to that provider

After: the same call is logged as a normal response with real token counts and real spend

1. They add a pass-through endpoint on their proxy whose target is their provider's https://<their-host>/v1/messages
2. They send POST http://localhost:4000/my-provider/v1/messages with "stream": true and an ordinary Anthropic Messages body
3. The stream comes back correctly and their app renders the reply
4. They look up that request in the proxy logs and find the assistant's reply stored as a normal response
5. The same entry carries 17 prompt tokens and 40 completion tokens and is priced at 0.000651, so cost reporting matches what the provider billed

Relevant issues

Fixes #40117

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Shared setup, used unchanged for both runs. An Anthropic-compatible upstream is served on 127.0.0.1:9911, so the only thing that differs from a supported setup is the hostname, which is the whole point of the defect. POST /v1/messages there streams a normal Anthropic SSE response whose usage block reports 17 input tokens and 40 output tokens.

config.yaml:

general_settings:
  master_key: sk-1234
  pass_through_endpoints:
    - path: "/my-provider/v1/messages"
      target: "http://127.0.0.1:9911/v1/messages"
      headers:
        content-type: application/json

The proxy is started the same way in both runs:

python litellm/proxy/proxy_cli.py --config config.yaml --port 4000 --detailed_debug 2>&1 | tee litellm.log

Before (6bb60f3)

1. Send the streamed request through the pass-through route

curl -sS -X POST http://127.0.0.1:4000/my-provider/v1/messages \
  -H "Authorization: Bearer sk-1234" -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-4-5-20250929","max_tokens":64,"stream":true,"messages":[{"role":"user","content":"hi"}]}'

event: message_start
data: {"type": "message_start", "message": {"id": "msg_01", ... "usage": {"input_tokens": 17, ...}}}
...
event: message_stop
data: {"type": "message_stop"}

2. Ask the log what that request cost

grep -c "response_cost" litellm.log

0

The stream reached the client, and the proxy priced none of it.

After (c929dfd)

1. Send the same streamed request through the same route

curl -sS -X POST http://127.0.0.1:4000/my-provider/v1/messages \
  -H "Authorization: Bearer sk-1234" -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-4-5-20250929","max_tokens":64,"stream":true,"messages":[{"role":"user","content":"hi"}]}'

event: message_start
data: {"type": "message_start", "message": {"id": "msg_01", ... "usage": {"input_tokens": 17, ...}}}
...
event: message_stop
data: {"type": "message_stop"}

2. Ask the log what that request cost

grep -oE "response_cost: [0-9.e-]+" litellm.log

response_cost: 0.0006510000000000001

Same bytes on the wire to the client, and the request is now priced the same as it would be coming from api.anthropic.com.

Type

🐛 Bug Fix

Caveats (if any)

Medium

• Cost injection now reaches these routes too
  • include_cost_in_streaming_usage keys off the same endpoint type, so with that setting on, streams from these hosts now get cost added to their usage block
  • That matches what api.anthropic.com already does, and it does change the bytes those callers receive, so it is worth a reviewer's eye

Low

• Only the two canonical routes are claimed, so a provider that serves the Anthropic API under some other path shape still logs as generic
• The proof above ran against a local Anthropic-compatible upstream rather than a paid provider, because the defect is defined by the host not being api.anthropic.com and no paid call can exhibit it

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

All committers have signed the CLA.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing Sunny-Soni00:fix_passthrough_endpoint_type_detection (c929dfd) with litellm_internal_staging (6bb60f3)}

[Copilot]

🟢 Approval recommended

The change is small, targeted, and includes regression tests covering both the fixed case and conservative fallback behavior.

Pull request overview

This PR fixes pass-through streaming accounting by resolving the endpoint type from the request path when hostname-based detection is insufficient, enabling correct Anthropic/OpenAI parsing and billing.

Changes:

• Extend get_endpoint_type to fall back to suffix-matching canonical routes (/v1/messages, /v1/chat/completions)
• Add regression coverage ensuring Anthropic-compatible streamed pass-through calls on non-Anthropic hosts still produce usage and cost
• Add route-shape tests to keep unknown paths classified as GENERIC

File summaries

File	Description
litellm/proxy/pass_through_endpoints/pass_through_endpoints.py	Adds path-suffix fallback so endpoint type is not derived solely from hostname
tests/test_litellm/proxy/pass_through_endpoints/test_pass_through_endpoints.py	Adds regression and route-shape tests validating correct endpoint typing and billing behavior

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[greptile-apps]

Greptile Summary

This PR extends pass-through endpoint classification to use canonical URL path suffixes when hostname checks do not identify the provider.

• Classifies /v1/messages routes as Anthropic-compatible.
• Classifies /v1/chat/completions routes as OpenAI-compatible.
• Adds regression coverage for Anthropic stream parsing, token accounting, cost calculation, OpenAI-compatible routes, and routes that must remain generic.
• Refines the test helper to return the resolved optional response cost directly.

Confidence Score: 5/5

The PR appears safe to merge; no actionable new defects or outstanding previous findings remain.

The route fallback is restricted to the two intended canonical suffixes, unrelated paths remain generic, and the tests verify successful Anthropic stream accounting and OpenAI classification. All previous review threads are resolved, including the coarse helper return type corrected in the latest revision.

Important Files Changed

Filename	Overview
litellm/proxy/pass_through_endpoints/pass_through_endpoints.py	Adds narrowly scoped route-suffix fallback classification after existing hostname-based checks.
tests/test_litellm/proxy/pass_through_endpoints/test_pass_through_endpoints.py	Adds focused regression coverage and precisely types the stream-logging helper’s returned cost.

_{Reviews (3): Last reviewed commit: "test(proxy): return the resolved cost in..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Sunny-Soni00]

@greptileai

[Sunny-Soni00]

@greptileai

[Troy-2015]

@Sunny-Soni00 resolve conflicts is needed

[Sunny-Soni00]

Conflict resolved by merging the latest litellm_internal_staging. Both sets of appended tests are kept, and the full pass-through suite passes locally.

[Troy-2015]

@Sunny-Soni00 PR review failed

[Sunny-Soni00]

The one failing check is the Terraform provider Go test, which is failing on most open PRs today. This PR touches no Go code.

[Troy-2015]

@Sunny-Soni00 resolve conflicts is needed