fix(proxy): return provider auth errors from /v1/messages/count_tokens instead of masking or 500

[mateo-berri]

TLDR

Problem this solves:

• /v1/messages/count_tokens answers 200 with a silent local count when Anthropic refuses the credential
• A provider refusal that raises inside the endpoint becomes a blanket 500
• /v1/messages on the same deployment correctly answers 401

How it solves it:

• A provider token-count 401 now surfaces instead of falling back to the local tokenizer
• The endpoint maps status-carrying exceptions to their status with the Anthropic error envelope
• Every other provider failure (403, 429, 5xx, timeouts) keeps the deliberate local fallback

User Flow

Before: a developer whose Anthropic-SDK app counts tokens through the proxy sees the provider's refusal hidden, as a silent 200 on a revoked Anthropic key and a bare 500 when the refusal raises

1. Their app sends POST https://litellm-domain/v1/messages/count_tokens with {"model": "claude-haiku-4-5", "messages": [{"role": "user", "content": "count these tokens please"}]}
2. Even though the deployment's Anthropic key was revoked and Anthropic answered the proxy with 401 "API key is invalid.", the response is HTTP 200 {"input_tokens": 11}: a locally estimated count with nothing marking it as such
3. The same app sends POST https://litellm-domain/v1/messages with the same model and gets HTTP 401 with {"type":"error","error":{"type":"authentication_error",...}}, so counting and sending disagree about whether the credential works
4. On a deployment that routes the same model name to Gemini with a bad Google key, the count_tokens request returns HTTP 500 {"detail":{"error":"Internal server error: litellm.APIError: Google Gen AI Studio API error: 400 - ... API key not valid ..."}}, and their SDK reports a proxy-side internal error rather than the provider's refusal

After: the same requests answer with the provider's real status and the Anthropic error envelope, matching /v1/messages, so the app surfaces the refusal

1. Their app sends POST https://litellm-domain/v1/messages/count_tokens with {"model": "claude-haiku-4-5", "messages": [{"role": "user", "content": "count these tokens please"}]}
2. The response is HTTP 401 {"detail":{"type":"error","error":{"type":"authentication_error","message":"API key is invalid."}}}, so the SDK raises its normal AuthenticationError
3. POST https://litellm-domain/v1/messages keeps answering HTTP 401 the same way: counting and sending now agree
4. On the Gemini deployment, count_tokens answers HTTP 400 with an invalid_request_error envelope carrying Google's "API key not valid" text instead of a 500
5. When Anthropic is merely rate limiting or down (429s, timeouts), or answers 403, count_tokens still answers HTTP 200 with a locally computed count, unchanged

Relevant issues

Linear ticket

Resolves LIT-6507

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

All runs below are live proxies hitting the real provider APIs with real spend, no mocks. Each side is its own git worktree booting python litellm/proxy/proxy_cli.py --config <config> --num_workers 2 (one process, two uvicorn workers, no database). The client is the Anthropic Python SDK 1.1.0 plus raw curl; SDK and curl agreed at every step, so each step shows whichever is clearer. Shared payload MSG = [{"role": "user", "content": "count these tokens please"}], curl body {"model":"<model>","messages":[{"role":"user","content":"count these tokens please"}]} with -H 'Authorization: Bearer <master key>' -H 'content-type: application/json'

The main config defines claude-bad-key (real model, revoked Anthropic key), claude-good (real key), and claude-typo-model (anthropic/claude-nonexistent-model-lit6507 on the real key). Before runs at the merge base 8c58b93 on port 58248, After at 2d96db3 on port 35458. The federation case uses a second proxy pair on the workload-identity-federation branch (PR #38818): Before is that branch at c0739de (port 54636), After is local merge ee3a23cedc of this PR's tip with that same sha (port 22204; only test files conflicted, every runtime file auto-merged). Its config defines claude-wif-fed, whose federation token exchange Anthropic rejects

Before (8c58b93)

count_tokens with a revoked Anthropic key

1. curl -s -i -X POST http://127.0.0.1:58248/v1/messages/count_tokens ... -d '{"model":"claude-bad-key",...}'
2. HTTP/1.1 200 OK {"input_tokens":11}: a silent local estimate, nothing marks the dead credential
3. SDK client.messages.count_tokens(model="claude-bad-key", messages=MSG) returns MessageTokensCount(input_tokens=11)

count_tokens controls: good key and typo model

1. SDK count_tokens(model="claude-good") returns MessageTokensCount(input_tokens=11) (real provider count)
2. curl with claude-typo-model (Anthropic answers 404): HTTP/1.1 200 OK {"input_tokens":11}

/utils/token_counter with call_endpoint=true

1. curl -s -i -X POST 'http://127.0.0.1:58248/utils/token_counter?call_endpoint=true' ... -d '{"model":"claude-bad-key",...}'
2. HTTP/1.1 200 OK {"total_tokens":11,"request_model":"claude-bad-key","model_used":"claude-haiku-4-5","tokenizer_type":"huggingface_tokenizer","original_response":null,"error":false,"error_message":null,"status_code":null}: auth failure reported as error: false
3. Same curl with claude-good: HTTP/1.1 200 OK, "tokenizer_type":"anthropic_api", "total_tokens":11

Google countTokens route

1. curl -s -i -X POST 'http://127.0.0.1:58248/v1beta/models/claude-bad-key:countTokens' ... -d '{"contents":[{"role":"user","parts":[{"text":"count these tokens please"}]}]}'
2. HTTP/1.1 200 OK {"totalTokens":11,"promptTokensDetails":[]}

/v1/messages agreement and recount after the 401

1. SDK client.messages.create(model="claude-bad-key", max_tokens=16, messages=MSG) raises AuthenticationError 401 (authentication_error, "API key is invalid."), so sending already surfaces the failure that counting hides
2. The count_tokens curl from the first case repeated right after, and again 15 s later: both HTTP/1.1 200 OK {"input_tokens":11}

Paid completion on the good key

1. SDK create(model="claude-good", max_tokens=16, messages=MSG) returns Message(id='msg_011CefaNfPnVDknNUwkPzuAj', ..., usage=Usage(input_tokens=11, output_tokens=16))

Workload identity federation deployment (#38818 at c0739de)

1. curl -s -i -X POST http://127.0.0.1:54636/v1/messages/count_tokens ... -d '{"model":"claude-wif-fed",...}'
2. HTTP/1.1 200 OK {"input_tokens":11} (SDK: MessageTokensCount(input_tokens=11)): that branch's counter skips the provider call when it finds no static credential and hands back the local estimate
3. SDK create(model="claude-wif-fed") raises AuthenticationError 401 ("x-api-key header is required"), so sending and counting disagree here too

After (2d96db3)

count_tokens with a revoked Anthropic key

1. curl -s -i -X POST http://127.0.0.1:35458/v1/messages/count_tokens ... -d '{"model":"claude-bad-key",...}'
2. HTTP/1.1 401 Unauthorized {"detail":{"type":"error","error":{"type":"authentication_error","message":"API key is invalid."},"request_id":null}}
3. SDK count_tokens(model="claude-bad-key") raises AuthenticationError 401 with the same envelope

count_tokens controls: good key and typo model

1. SDK count_tokens(model="claude-good") returns MessageTokensCount(input_tokens=11), unchanged
2. curl with claude-typo-model: HTTP/1.1 200 OK {"input_tokens":11}, unchanged: the non-auth local fallback survives

/utils/token_counter with call_endpoint=true

1. Same curl with claude-bad-key
2. HTTP/1.1 401 Unauthorized {"error":{"message":"{\"type\":\"error\",\"error\":{\"type\":\"authentication_error\",\"message\":\"API key is invalid.\"},\"request_id\":null}","type":"token_counting_error","param":"model","code":"401"}}
3. Same curl with claude-good: HTTP/1.1 200 OK, "tokenizer_type":"anthropic_api", "total_tokens":11, unchanged

Google countTokens route

1. Same curl with claude-bad-key
2. HTTP/1.1 401 Unauthorized {"error":{"message":"{\"type\":\"error\",\"error\":{\"type\":\"authentication_error\",\"message\":\"API key is invalid.\"},\"request_id\":null}","type":"token_counting_error","param":"model","code":"401"}}

/v1/messages agreement and recount after the 401

1. SDK create(model="claude-bad-key") raises AuthenticationError 401, unchanged
2. The count_tokens curl repeated verbatim right after, and again 15 s later: both HTTP/1.1 401 Unauthorized with the same envelope as the first case, so counting and sending agree

Paid completion on the good key

1. SDK create(model="claude-good", max_tokens=16, messages=MSG) returns Message(id='msg_011CefcPsK9tvF4EzntDfLgB', ..., usage=Usage(input_tokens=11, output_tokens=16))

Workload identity federation deployment (#38818 merged with this tip, ee3a23cedc)

1. curl -s -i -X POST http://127.0.0.1:22204/v1/messages/count_tokens ... -d '{"model":"claude-wif-fed",...}'
2. HTTP/1.1 200 OK {"input_tokens":11} (SDK: MessageTokensCount(input_tokens=11)), unchanged: the counter never consults the provider on that branch, so there is no 401 result for this PR to surface
3. SDK create(model="claude-wif-fed") still raises AuthenticationError 401

Blast radius on the sibling surfaces and other providers

Same two-worker no-database boot, base 8c58b93 on port 41917 vs head 2d96db3 on port 47263, curl body {"model":"<model>","messages":[{"role":"user","content":"Hello Claude, count me"}]} on the two Anthropic-shaped routes and {"contents":[{"role":"user","parts":[{"text":"Hello Claude, count me"}]}]} on the Google one. claude-no-key is anthropic/claude-haiku-4-5 with no key at all, bedrock-api-key is a real Bedrock API key (bearer token) that can invoke models but lacks bedrock:CountTokens, bedrock-bad-creds is bogus SigV4 keys, gemini-bad-key and gemini-good are gemini/gemini-3.8-flash with a bogus and a real Google key

Deployment	Route	Base	Head
claude-bad-key	count_tokens	200 local	401 envelope
claude-bad-key	token_counter	200 local	401
claude-bad-key	countTokens	200 local	401
claude-no-key	all three	200 local	200 local
bedrock-api-key	all three	200 local	200 local
bedrock-bad-creds	all three	200 local	200 local
gemini-bad-key	count_tokens	500	400 envelope
gemini-bad-key	token_counter	500	500
gemini-bad-key	countTokens	500	500
gemini-good	count_tokens	500	400 envelope
gemini-good	token_counter	500	500
gemini-good	countTokens	200 real	200 real

The rows that differ, verbatim:

1. claude-bad-key count_tokens, base: HTTP 200 {"input_tokens":12}; head: HTTP 401 {"detail":{"type":"error","error":{"type":"authentication_error","message":"API key is invalid."},"request_id":null}}
2. claude-bad-key token_counter and countTokens, head: HTTP 401 {"error":{"message":"{\"type\":\"error\",\"error\":{\"type\":\"authentication_error\",\"message\":\"API key is invalid.\"},\"request_id\":null}","type":"token_counting_error","param":"model","code":"401"}}
3. gemini-bad-key count_tokens, base: HTTP 500 {"detail":{"error":"Internal server error: litellm.APIError: Google Gen AI Studio API error: 400 - {... \"message\": \"API key not valid. Please pass a valid API key.\", \"status\": \"INVALID_ARGUMENT\" ...}"}}; head: HTTP 400 {"detail":{"type":"error","error":{"type":"invalid_request_error","message":"litellm.APIError: Google Gen AI Studio API error: 400 - {... \"API key not valid. Please pass a valid API key.\" ...}"}}}
4. gemini-good count_tokens, base: HTTP 500 and head: HTTP 400 with the same envelope shapes around Google's "CountTokens requires generate_content_request or contents to be set." (a pre-existing counter bug on Anthropic-format input, see below)

Bedrock answers a Bedrock API key without bedrock:CountTokens and a dead SigV4 credential with the same 403, so both keep the local fallback on both sides. An earlier head of this PR raised on 403 too and turned the first case into a 403 on every count surface while sending kept working; 2d96db3 narrows the gate to 401

Surprises observed while QA'ing, all pre-existing on both sides:

• Bedrock API keys lack bedrock:CountTokens by default, answer 403
• Dead Bedrock SigV4 credentials also answer 403, fallback stays
• Gemini count path 500s without google-genai installed
• Gemini counter sends empty contents on Anthropic-format messages
• Google countTokens with claude-good: 200 yet totalTokens 0
• Typo-model local estimate indistinguishable from a real count
• /v1/messages error bodies litellm-wrapped, count_tokens uses detail
• token_counter and Google routes stringify the Anthropic envelope
• feat(anthropic): workload identity federation, pluggable identity sources, and provider-level setup (internal copy of #38013) #38818 counts locally when no static credential is set

Type

🐛 Bug Fix

Caveats (if any)

Medium

• The auth surfacing also reaches the two sibling count-tokens surfaces
  • /utils/token_counter with call_endpoint=true now answers 401 on a refused credential
  • The Google countTokens routes do too, instead of a masked 200 local count
  • Non-auth failures keep the silent local fallback on all surfaces
• Only providers that answer a dead credential with 401 surface it
  • Bedrock signals dead credentials with 403, which keeps the pre-existing local fallback
  • 403 also means "credential alive, count action denied" (a Bedrock API key without bedrock:CountTokens), so it must not abort
• During the post-401 router cooldown, count_tokens can briefly mask again
  • Observed at an earlier head: 200 local count 2 s after the 401, back to 401 once the cooldown lapsed
  • Not reproduced at 2d96db3: the recount right after the 401 and 15 s later both answered 401

Low

• The envelope sits under "detail", this endpoint's existing convention for every error
• Only provider result status 401 counts as an auth failure
• Provider refusals that raise (Gemini) now carry their status and an envelope instead of a bare 500
  • The sibling token_counter and countTokens surfaces still answer 500 on those

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• 2d96db3 passes /live-pr-risk

---

Note

Medium Risk
Changes error semantics on count-tokens surfaces (including /utils/token_counter with call_endpoint=true); auth failures surface correctly but brief post-401 router cooldown can still yield a masked 200 local count.

Overview
Fixes LIT-6507 so Anthropic-style count_tokens behaves like /v1/messages when credentials fail: clients get a 401 with the Anthropic error envelope instead of a 200 with a silent local estimate or a generic 500.

In proxy_server, provider token-count failures that return status_code == 401 now raise ProxyException (same path as when disable_token_counter is on), so auth errors are not swallowed by the local tokenizer fallback. 429, 403, and other non-auth provider errors still fall back locally.

In anthropic_endpoints count_tokens, uncaught exceptions that carry an HTTP status_code (4xx–5xx) are mapped through AnthropicExceptionMapping instead of always returning 500—covering cases like AuthenticationError from federation token exchange.

Tests were added/updated for endpoint mapping, _try_provider_token_count, and Anthropic provider detection mocking.

^{Reviewed by Cursor Bugbot for commit 2d96db3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

This PR surfaces provider token-count authentication failures through the Anthropic count-tokens endpoint while retaining local fallback for other provider failures

• Raises token-count errors returned with status 401 instead of silently estimating locally
• Maps status-carrying endpoint exceptions into Anthropic-shaped HTTP errors
• Adds regression coverage for authentication errors, fallback behavior, and endpoint error mapping

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/proxy/proxy_server.py	Changes provider token-count handling so 401 results abort rather than entering the local tokenizer fallback
litellm/proxy/anthropic_endpoints/endpoints.py	Maps exceptions carrying valid HTTP statuses into Anthropic error envelopes
tests/test_litellm/proxy/anthropic_endpoints/test_endpoints.py	Adds endpoint regression coverage for returned and raised authentication failures
tests/test_litellm/proxy/test_proxy_server.py	Adds unit coverage distinguishing provider 401 failures from fallback-eligible statuses
tests/proxy_unit_tests/test_proxy_token_counter.py	Makes the existing provider-detection test deterministic by replacing its external Anthropic request

_{Reviews (5): Last reviewed commit: "fix(proxy): only a provider 401 aborts t..." | Re-trigger Greptile}

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_count_tokens_auth_500 (2d96db3) with litellm_internal_staging (ff1f21a)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

🚅 Hi, thanks for the PR! I'm Agent Shin, the automated triage bot for this repository. What's this and why am I getting it?

I read the description against our contribution rubric. Here's how it lined up:

What you got right:

• ✅ Clear problem description
• ✅ Expected vs. actual behavior

What's still missing:

• End-to-end QA proof: the description says the live-proxy before/after proof is being generated and will be posted shortly, and it hasn't been posted

Once the promised live-proxy 401 before/after output lands in the description this passes.

If the description isn't updated in the next 2 hours, I'll auto-close this PR. That's not us saying we don't care about the change; we want the open-PR list to mirror what a maintainer can act on right now, so contributors don't get lost in a backlog. A closed PR is a soft "park this for later," not a rejection. Take your time; everything below still works after the close.

During the grace period: just update the PR description with the missing pieces. No need to ping me; I'll re-check on the next sweep and skip the auto-close if it now passes. See what counts as QA proof for the full rubric (a linked issue alone isn't enough; it covers context, not proof).

If the PR does get auto-closed in 2 hours, you still have easy recovery paths:

• Comment @agent-shin reconsider after updating the description. I'll re-evaluate and reopen the PR if it now passes.
• Comment @greptileai to request a fresh Greptile review; that still works even after the PR is closed, and a stronger score is one of the signals that lifts the PR back into the queue. So a low Greptile score isn't a blocker either.

Internal BerriAI contributors: this rubric doesn't apply to you; ping a maintainer.

(I'm an LLM, so I'm not infallible. If you think I got this wrong, ping a maintainer; they'll override me.)

[devin-ai-integration]

🚅 Hi, thanks for the PR! I'm Agent Shin, the automated triage bot for this repository. What's this and why am I getting it?

I read the description against our contribution rubric. Here's how it lined up:

What you got right:

• ✅ Clear problem description

What's still missing:

• End-to-end QA proof: a recording, screenshot, or the exact commands run with their real output against a live proxy / real provider

The PR promises live-proxy proof that is still pending; only vitest/render tests are shown.

Closing this PR isn't a rejection of the change. We want the open-PR list to mirror what a maintainer can act on right now, so contributors don't get lost in a backlog. A closed PR is a soft "park this for later"; your work is still here, the diff is still here, and getting it reopened is one comment away. Take your time.

To bring this PR back:

• Update the description with the missing pieces, then comment @agent-shin reconsider on this PR. I'll re-evaluate and reopen if it now passes.
• Or Open a new PR with the same fix and the updated description. GitHub doesn't always let external contributors reopen a bot-closed PR, so a fresh PR is the most reliable path back into the review queue.
• If Greptile's most recent score on this PR was below 4/5, comment @greptileai to request a fresh review; that still works even after the PR is closed, and a stronger score is one of the signals that lifts the PR back into the queue. A low Greptile score isn't a blocker.

What "end-to-end QA proof" means, since it's the most common gap: at least one of a short before/after screen recording / video (the bug reproducing, then the fix working; for a brand-new feature, a recording of it working end-to-end), a screenshot (or before/after screenshots) of it working, or the exact commands you ran paired with their real output against the real system. Running pytest on the repo's unit tests doesn't count; those mock the LLM provider, DB, and network, so they aren't end-to-end. Output from a real, no-mocks integration run is what we look for. A linked issue alone isn't enough either: it covers context, not proof. See the full rubric.

Internal BerriAI contributors: this rubric doesn't apply to you; ping a maintainer.

(I'm an LLM, so I'm not infallible. If you think I got this wrong, comment @agent-shin reconsider or ping a maintainer; they'll override me.)

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c79776d. Configure here.}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 70cb373. Configure here.}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2d96db3. Configure here.}