feat(proxy)!: return Anthropic-shaped errors on /v1/messages - #30385
feat(proxy)!: return Anthropic-shaped errors on /v1/messages#30385songkuan-zheng wants to merge 9 commits into
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Greptile SummaryThis PR fixes the
Confidence Score: 5/5Safe to merge. All error exit paths on both The error-handling logic in No files require special attention.
|
| Filename | Overview |
|---|---|
| litellm/anthropic_interface/exceptions/exception_mapping_utils.py | Adds _strip_litellm_wrapper_prefixes (correctly anchored to a provider allowlist), a raw_decode fallback for Router-appended debug suffixes, and OpenAI-compat nested error.message extraction. Logic is well-tested and the provider list ordering (longer prefixes before shorter ones) is correct. |
| litellm/proxy/anthropic_endpoints/endpoints.py | Replaces raise ProxyException with return JSONResponse on the /v1/messages error path, and converts all count_tokens error exits to Anthropic-shaped JSONResponse. All previously flagged concerns (ProxyException.code extraction, None-message guard, generic-Exception shape) are addressed in the current HEAD. |
| tests/test_litellm/proxy/test_anthropic_error_passthrough.py | New integration test file covering all major error shapes via TestClient with monkeypatched auth and base_process_llm_request. No real network calls; all paths exercised with stubs. |
| tests/proxy_unit_tests/test_proxy_token_counter.py | Updated existing tests from pytest.raises(HTTPException) to JSONResponse assertions; new tests added for HTTPException conversion, generic exception, ProxyException-with-None-message, and each missing-field path. Assertions are at least as strict as the originals. |
| tests/test_litellm/anthropic_interface/exceptions/test_exception_mapping_utils.py | Adds unit tests for prefix stripping, trailing-garbage recovery via raw_decode, OpenAI-compat extraction, and upstream Anthropic passthrough. Coverage is thorough. |
| gateway/routes/allowlist.py | Adds /api/event_logging/batch to the gateway allowlist so Claude Code telemetry requests are routed rather than rejected with 404. Trivial, low-risk change. |
Reviews (9): Last reviewed commit: "chore: add Co-authored-by trailer for at..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Greptile SummaryThis PR fixes
Confidence Score: 3/5The core fix is correct, but the change is unconditionally applied to an existing endpoint without an opt-out path, and the count_tokens endpoint still has an incomplete fix. The /v1/messages endpoint now unconditionally returns a different error shape with no way for existing clients to retain the old behaviour. The count_tokens generic exception handler was missed, leaving that path still returning a FastAPI-wrapped non-Anthropic body. There is also a narrow but real crash path when e.message is None. litellm/proxy/anthropic_endpoints/endpoints.py — both the missing feature-flag and the incomplete count_tokens fix are here.
|
| Filename | Overview |
|---|---|
| litellm/proxy/anthropic_endpoints/endpoints.py | Switches /v1/messages and count_tokens error paths from ProxyException/HTTPException to JSONResponse with Anthropic-shaped bodies; generic Exception handler in count_tokens still returns non-Anthropic HTTPException, and message=None edge case can crash the new error handler. |
| litellm/anthropic_interface/exceptions/exception_mapping_utils.py | Adds prefix-stripping, raw_decode fallback, and nested OpenAI-compat message extraction to transform_to_anthropic_error; logic is sound and well-tested. |
| tests/test_litellm/proxy/test_anthropic_error_passthrough.py | New integration tests using TestClient validate Anthropic error shape, header preservation, and absence of OpenAI-only fields on the /v1/messages error path. |
| tests/test_litellm/anthropic_interface/exceptions/test_exception_mapping_utils.py | New unit tests added for prefix stripping, raw_decode fallback, OpenAI-compat message extraction, and passthrough; comprehensive coverage of new logic. |
| tests/proxy_unit_tests/test_proxy_token_counter.py | Tests updated to assert JSONResponse body instead of HTTPException.detail; assertions are strengthened (added 'detail' not in body check), not weakened. |
Comments Outside Diff (2)
-
litellm/proxy/anthropic_endpoints/endpoints.py, line 118-160 (link)Breaking change without user-controlled opt-out flag
The error response shape for
/v1/messagesis changed unconditionally. The team's rule requires a user-controlled flag for backwards-incompatible changes rather than a hard switch. Clients already deployed against this endpoint that parseerror.paramorerror.codefrom the previous OpenAI-shaped response will start receivingnullfor those fields with no way to opt back in. A feature flag (e.g.litellm.anthropic_endpoint_return_anthropic_errors = True) would let existing integrations continue working while new ones opt in.Rule Used: What: avoid backwards-incompatible changes without... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
-
litellm/proxy/anthropic_endpoints/endpoints.py, line 250-258 (link)except Exceptionhandler incount_tokensstill raisesHTTPExceptionwith{"detail": {"error": "..."}}, which is a FastAPI-wrapped non-Anthropic shape. The PR only fixed theProxyExceptionpath; unexpected internal errors oncount_tokenswill still return a different envelope than what the PR claims is fixed. This inconsistency means any non-ProxyException internal error (e.g. a token counter crash) bypasses the Anthropic formatting entirely.
Reviews (2): Last reviewed commit: "feat(proxy)!: return Anthropic-shaped er..." | Re-trigger Greptile
|
@Sameerlite @krrish-berri-2 — heads up that the failing EvidenceThe failing assertion is: ```
This PR's base SHA equals Could the failing check be marked as not required for this PR, or the merge proceed once the other concerns are resolved? Happy to file the allowlist follow-up either way. |
|
Thanks for the contribution! A couple of things to address before this is ready for merge:
Once those are in, we'll take another look! |
…hten regex Three greptile threads on BerriAI#30385: P1 — **ProxyException silently mapped to 500** ProxyException stores its HTTP code on `.code` (string), not `.status_code`. Reading only `.status_code` falls through to the default 500 for every auth/permission/billing rejection. Now reads `.code` first (preferred when it's a numeric string), falls back to `.status_code`, defaults to 500 — matching the `count_tokens` handler in the same module. P2 — **None .message crashes the strip-prefix regex** `getattr(e, "message", str(e))` returns None when `.message` is explicitly None (e.g. `litellm.BadRequestError(message=None, ...)`). That None then crashes `re.sub` in `_strip_litellm_wrapper_prefixes`, yielding a bare 500 with no Anthropic-shaped body. Coerce None -> str(e) before passing to the mapping helper. P2 — **`_PROVIDER_EXCEPTION_PREFIX` regex was overly broad** `\w+Exception` matched any word ending in `Exception`, including generic `TimeoutException - <real upstream body>`, `ConnectionException - <real upstream body>`, `RequestException - <real upstream body>` — silently swallowing the front of legitimate error strings. Anchored to a known provider-name allowlist (47 names, maintained from `litellm/llms/**` + common aliases). Generic Python/network exception names no longer get swallowed. Tests (`test_anthropic_error_passthrough.py` +3): - `test_proxy_exception_code_attribute_is_honored`: ProxyException(code="401") -> HTTP 401, not 500. - `test_none_message_falls_back_to_str`: exception with .message=None yields a well-formed Anthropic body, not an unhandled 500. - `test_provider_exception_prefix_does_not_strip_generic_timeout`: `TimeoutException - upstream took too long` survives intact. 42/42 pass locally (existing + 3 new).
|
@Sameerlite — done in P1 — ProxyException silently mapped to 500 ( P2 — None P2 — Tests added (
42/42 pass locally. |
|
@Sameerlite @krrish-berri-2 — re-flagging the The assertion: Evidence this is unrelated:
Same pattern as the HF Hub flake on #29748 — please dismiss / re-run. |
|
Thanks for your contribution! A few things to get this ready:
We're also triggering a Greptile code review: |
…hten regex Three greptile threads on BerriAI#30385: P1 — **ProxyException silently mapped to 500** ProxyException stores its HTTP code on `.code` (string), not `.status_code`. Reading only `.status_code` falls through to the default 500 for every auth/permission/billing rejection. Now reads `.code` first (preferred when it's a numeric string), falls back to `.status_code`, defaults to 500 — matching the `count_tokens` handler in the same module. P2 — **None .message crashes the strip-prefix regex** `getattr(e, "message", str(e))` returns None when `.message` is explicitly None (e.g. `litellm.BadRequestError(message=None, ...)`). That None then crashes `re.sub` in `_strip_litellm_wrapper_prefixes`, yielding a bare 500 with no Anthropic-shaped body. Coerce None -> str(e) before passing to the mapping helper. P2 — **`_PROVIDER_EXCEPTION_PREFIX` regex was overly broad** `\w+Exception` matched any word ending in `Exception`, including generic `TimeoutException - <real upstream body>`, `ConnectionException - <real upstream body>`, `RequestException - <real upstream body>` — silently swallowing the front of legitimate error strings. Anchored to a known provider-name allowlist (47 names, maintained from `litellm/llms/**` + common aliases). Generic Python/network exception names no longer get swallowed. Tests (`test_anthropic_error_passthrough.py` +3): - `test_proxy_exception_code_attribute_is_honored`: ProxyException(code="401") -> HTTP 401, not 500. - `test_none_message_falls_back_to_str`: exception with .message=None yields a well-formed Anthropic body, not an unhandled 500. - `test_provider_exception_prefix_does_not_strip_generic_timeout`: `TimeoutException - upstream took too long` survives intact. 42/42 pass locally (existing + 3 new).
45a34ff to
852ea9a
Compare
|
@Sameerlite rebased onto Short version of the fix: I'll send curl-against-localhost:4000 proof-of-fix in a follow-up. |
|
Pushed The failing assertion was Added This is unrelated to the Anthropic-shaped-errors core of this PR. I bundled it here because the failure surfaced on this CI run; happy to extract to a separate PR if you'd rather keep scope tight. |
|
Thanks for the PR! A couple of things to get this over the finish line:
Triggering Greptile for a code review in the meantime: |
|
@Sameerlite proof of fix below; both before and after curls against a local proxy on Setup (same proxy config in both runs): model_list:
- model_name: claude-haiku
litellm_params:
model: anthropic/claude-haiku-4-5
api_key: os.environ/ANTHROPIC_API_KEY
general_settings:
master_key: sk-1234python litellm/proxy/proxy_cli.py --config dev_config.yaml --port 4000Same curl in both runs, asks for a model the proxy does not have, which routes through curl -sS -X POST http://localhost:4000/v1/messages \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{"model":"this-model-does-not-exist","messages":[{"role":"user","content":"hi"}],"max_tokens":10}'BEFORE (upstream {
"error": {
"message": "400: {'error': 'anthropic_messages: Invalid model name passed in model=this-model-does-not-exist. Call `/v1/models` to view available models for your key.'}",
"type": "None",
"param": "None",
"code": "400"
}
}HTTP 400. Anthropic SDK clients reject this body because the top-level discriminator they expect is AFTER (PR head {
"type": "error",
"error": {
"type": "invalid_request_error",
"message": "400: {'error': 'anthropic_messages: Invalid model name passed in model=this-model-does-not-exist. Call `/v1/models` to view available models for your key.'}"
}
}Still HTTP 400, same status code mapping ( The |
|
Thanks for the detailed proof of fix — the before/after curl output really helps! One remaining item:
Triggering another Greptile pass in the meantime: Once that clears, we'll take another look! 🙏 |
…hten regex Three greptile threads on BerriAI#30385: P1 — **ProxyException silently mapped to 500** ProxyException stores its HTTP code on `.code` (string), not `.status_code`. Reading only `.status_code` falls through to the default 500 for every auth/permission/billing rejection. Now reads `.code` first (preferred when it's a numeric string), falls back to `.status_code`, defaults to 500 — matching the `count_tokens` handler in the same module. P2 — **None .message crashes the strip-prefix regex** `getattr(e, "message", str(e))` returns None when `.message` is explicitly None (e.g. `litellm.BadRequestError(message=None, ...)`). That None then crashes `re.sub` in `_strip_litellm_wrapper_prefixes`, yielding a bare 500 with no Anthropic-shaped body. Coerce None -> str(e) before passing to the mapping helper. P2 — **`_PROVIDER_EXCEPTION_PREFIX` regex was overly broad** `\w+Exception` matched any word ending in `Exception`, including generic `TimeoutException - <real upstream body>`, `ConnectionException - <real upstream body>`, `RequestException - <real upstream body>` — silently swallowing the front of legitimate error strings. Anchored to a known provider-name allowlist (47 names, maintained from `litellm/llms/**` + common aliases). Generic Python/network exception names no longer get swallowed. Tests (`test_anthropic_error_passthrough.py` +3): - `test_proxy_exception_code_attribute_is_honored`: ProxyException(code="401") -> HTTP 401, not 500. - `test_none_message_falls_back_to_str`: exception with .message=None yields a well-formed Anthropic body, not an unhandled 500. - `test_provider_exception_prefix_does_not_strip_generic_timeout`: `TimeoutException - upstream took too long` survives intact. 42/42 pass locally (existing + 3 new).
f5f5ea5 to
b9c17be
Compare
| except ProxyException as e: | ||
| status_code = int(e.code) if e.code and e.code.isdigit() else 500 | ||
| detail = AnthropicExceptionMapping.transform_to_anthropic_error( | ||
| status_code=status_code, | ||
| raw_message=e.message, | ||
| ) | ||
| raise HTTPException( | ||
| status_code=status_code, | ||
| detail=detail, | ||
| ) | ||
| return _anthropic_error_response(status_code, e.message) |
There was a problem hiding this comment.
except ProxyException handler in count_tokens passes e.message directly to _anthropic_error_response without a None guard. If a ProxyException is constructed with message=None, this reaches _strip_litellm_wrapper_prefixes(None) where re.sub raises a TypeError, crashing the error handler and producing an unhandled 500 with no Anthropic-shaped body. The same scenario was explicitly fixed in the anthropic_response() path at lines 263-265 of this PR.
| except ProxyException as e: | |
| status_code = int(e.code) if e.code and e.code.isdigit() else 500 | |
| detail = AnthropicExceptionMapping.transform_to_anthropic_error( | |
| status_code=status_code, | |
| raw_message=e.message, | |
| ) | |
| raise HTTPException( | |
| status_code=status_code, | |
| detail=detail, | |
| ) | |
| return _anthropic_error_response(status_code, e.message) | |
| except ProxyException as e: | |
| status_code = int(e.code) if e.code and e.code.isdigit() else 500 | |
| raw_message = e.message if e.message is not None else str(e) | |
| return _anthropic_error_response(status_code, raw_message) |
86b6bf4 to
13bb0bc
Compare
…hten regex Three greptile threads on BerriAI#30385: P1 — **ProxyException silently mapped to 500** ProxyException stores its HTTP code on `.code` (string), not `.status_code`. Reading only `.status_code` falls through to the default 500 for every auth/permission/billing rejection. Now reads `.code` first (preferred when it's a numeric string), falls back to `.status_code`, defaults to 500 — matching the `count_tokens` handler in the same module. P2 — **None .message crashes the strip-prefix regex** `getattr(e, "message", str(e))` returns None when `.message` is explicitly None (e.g. `litellm.BadRequestError(message=None, ...)`). That None then crashes `re.sub` in `_strip_litellm_wrapper_prefixes`, yielding a bare 500 with no Anthropic-shaped body. Coerce None -> str(e) before passing to the mapping helper. P2 — **`_PROVIDER_EXCEPTION_PREFIX` regex was overly broad** `\w+Exception` matched any word ending in `Exception`, including generic `TimeoutException - <real upstream body>`, `ConnectionException - <real upstream body>`, `RequestException - <real upstream body>` — silently swallowing the front of legitimate error strings. Anchored to a known provider-name allowlist (47 names, maintained from `litellm/llms/**` + common aliases). Generic Python/network exception names no longer get swallowed. Tests (`test_anthropic_error_passthrough.py` +3): - `test_proxy_exception_code_attribute_is_honored`: ProxyException(code="401") -> HTTP 401, not 500. - `test_none_message_falls_back_to_str`: exception with .message=None yields a well-formed Anthropic body, not an unhandled 500. - `test_provider_exception_prefix_does_not_strip_generic_timeout`: `TimeoutException - upstream took too long` survives intact. 42/42 pass locally (existing + 3 new).
|
Thanks for the contribution! Kicking off a Greptile code review on this one. |
f070116 to
21423bd
Compare
…hten regex Three greptile threads on BerriAI#30385: P1 — **ProxyException silently mapped to 500** ProxyException stores its HTTP code on `.code` (string), not `.status_code`. Reading only `.status_code` falls through to the default 500 for every auth/permission/billing rejection. Now reads `.code` first (preferred when it's a numeric string), falls back to `.status_code`, defaults to 500 — matching the `count_tokens` handler in the same module. P2 — **None .message crashes the strip-prefix regex** `getattr(e, "message", str(e))` returns None when `.message` is explicitly None (e.g. `litellm.BadRequestError(message=None, ...)`). That None then crashes `re.sub` in `_strip_litellm_wrapper_prefixes`, yielding a bare 500 with no Anthropic-shaped body. Coerce None -> str(e) before passing to the mapping helper. P2 — **`_PROVIDER_EXCEPTION_PREFIX` regex was overly broad** `\w+Exception` matched any word ending in `Exception`, including generic `TimeoutException - <real upstream body>`, `ConnectionException - <real upstream body>`, `RequestException - <real upstream body>` — silently swallowing the front of legitimate error strings. Anchored to a known provider-name allowlist (47 names, maintained from `litellm/llms/**` + common aliases). Generic Python/network exception names no longer get swallowed. Tests (`test_anthropic_error_passthrough.py` +3): - `test_proxy_exception_code_attribute_is_honored`: ProxyException(code="401") -> HTTP 401, not 500. - `test_none_message_falls_back_to_str`: exception with .message=None yields a well-formed Anthropic body, not an unhandled 500. - `test_provider_exception_prefix_does_not_strip_generic_timeout`: `TimeoutException - upstream took too long` survives intact. 42/42 pass locally (existing + 3 new).
|
Thanks for the contribution! Triggering a fresh Greptile review. One important item from the previous review: there's an unresolved P1 thread about a |
|
@Sameerlite — the P1 you flagged ( except ProxyException as e:
status_code = int(e.code) if e.code and e.code.isdigit() else 500
raw_message = e.message if e.message is not None else str(e)
return _anthropic_error_response(status_code, raw_message)Same The fresh Greptile pass should reflect this — pinging for re-score. |
The Anthropic-compatible /v1/messages endpoint returned OpenAI-shaped
error bodies ({"error": {message, type, param, code}}) instead of
Anthropic-shaped ({"type": "error", "error": {type, message}}),
breaking Anthropic SDK clients that switch on error.error.type. The
message also leaked stacked LiteLLM class-name prefixes and an escaped
upstream JSON body (litellm.RateLimitError: AnthropicException - {...}).
Changes
-------
1. AnthropicExceptionMapping:
- Add _strip_litellm_wrapper_prefixes() to peel stacked
`litellm.<Class>:` and `<Provider>Exception - ` prefixes, and call
it inside transform_to_anthropic_error() so an embedded upstream
Anthropic error body is detected and passed through unchanged
(preserving the real error.type enum instead of deriving it from
the HTTP status).
- Use json.JSONDecoder.raw_decode as a fallback when safe_json_loads
rejects the message. The Router appends debug suffixes after the
upstream Anthropic JSON (`{"type":"error",...}. Received Model
Group=...`) that break strict JSON parsing — raw_decode parses
the leading object and ignores the trailing garbage.
- Extract nested error.message from OpenAI-compat upstream bodies
({"error":{"code","message","type"}}, used by OpenAI, the
`new-api` proxy, and many OpenAI-compat gateways) so the result
is the inner human message instead of the full stringified JSON.
2. /v1/messages error path: return JSONResponse with the Anthropic
body instead of raising ProxyException. ProxyException routes
through the global OpenAI-shaped handler; JSONResponse is required
(not HTTPException, which wraps the dict in a spurious
{"detail": ...}).
3. count_tokens: same root-cause fix — its existing
HTTPException(detail=...) was double-wrapping the Anthropic body in
{"detail": ...}. Switch to JSONResponse. Framework-level 400s for
missing model/messages stay as HTTPException.
x-litellm-* response headers are preserved on the error path.
Why `!`: clients that parsed error.param / error.code on this
Anthropic-compat endpoint will no longer see those OpenAI-only fields.
Tests
-----
- Prefix-stripper + wrapped-passthrough unit tests
- /v1/messages TestClient integration (asserts top-level shape, no
detail wrapper, no param/code, headers present)
- count_tokens error-format tests updated to assert the JSONResponse
body
- Trailing-garbage JSON recovery via raw_decode
- Nested OpenAI-compat error.message extraction
39 tests pass across the changed files.
…hten regex Three greptile threads on BerriAI#30385: P1 — **ProxyException silently mapped to 500** ProxyException stores its HTTP code on `.code` (string), not `.status_code`. Reading only `.status_code` falls through to the default 500 for every auth/permission/billing rejection. Now reads `.code` first (preferred when it's a numeric string), falls back to `.status_code`, defaults to 500 — matching the `count_tokens` handler in the same module. P2 — **None .message crashes the strip-prefix regex** `getattr(e, "message", str(e))` returns None when `.message` is explicitly None (e.g. `litellm.BadRequestError(message=None, ...)`). That None then crashes `re.sub` in `_strip_litellm_wrapper_prefixes`, yielding a bare 500 with no Anthropic-shaped body. Coerce None -> str(e) before passing to the mapping helper. P2 — **`_PROVIDER_EXCEPTION_PREFIX` regex was overly broad** `\w+Exception` matched any word ending in `Exception`, including generic `TimeoutException - <real upstream body>`, `ConnectionException - <real upstream body>`, `RequestException - <real upstream body>` — silently swallowing the front of legitimate error strings. Anchored to a known provider-name allowlist (47 names, maintained from `litellm/llms/**` + common aliases). Generic Python/network exception names no longer get swallowed. Tests (`test_anthropic_error_passthrough.py` +3): - `test_proxy_exception_code_attribute_is_honored`: ProxyException(code="401") -> HTTP 401, not 500. - `test_none_message_falls_back_to_str`: exception with .message=None yields a well-formed Anthropic body, not an unhandled 500. - `test_provider_exception_prefix_does_not_strip_generic_timeout`: `TimeoutException - upstream took too long` survives intact. 42/42 pass locally (existing + 3 new).
The proxy-infra component-allowlist test on this PR's CI was failing with /api/event_logging/batch missing from both gateway and backend allowlists. The route comes from upstream commit d2bd029 (PR BerriAI#20504), not from this PR's diff, but the union-coverage test fails on every fork PR that rebases onto internal_staging. Add the route to GATEWAY_EXACT_PATHS: it's a stub Anthropic event-logging endpoint that Claude Code clients hit as part of the /v1/messages data path (the endpoint's own docstring describes it as preventing 404s from Claude Code telemetry), so it belongs on the gateway component, not the control-plane backend. Drive-by because this PR's CI surfaced it; happy to extract to a separate PR if the maintainer prefers.
Greptile (4/5 on commit 40ecda4) flagged the bare `except Exception` fallback in `count_tokens` still re-raising as `HTTPException(500, detail={"error": ...})`, which leaves FastAPI's `{"detail": ...}` wrapper around an OpenAI-shaped body. Closing that gap by itself would still leave the same bug in the two `raise HTTPException(400, ...)` branches (missing `model`, missing `messages`) and in the `except HTTPException: raise` re-raise that lets an HTTPException from the internal counter through unchanged. Same bug class as the one this PR's main commit fixed on `/v1/messages`; same surface (the Anthropic SDK rejects any non-Anthropic envelope). Fix: every error path out of `count_tokens` now returns `AnthropicExceptionMapping.transform_to_anthropic_error(...)` wrapped in a `JSONResponse`. Extracted a `_anthropic_error_response(status_code, raw_message)` helper at module scope so the four exit points (400 missing model, 400 missing messages, HTTPException pass-through with detail extraction, ProxyException with `.code`, generic Exception fallback) all share one shape. Tests: extend `tests/proxy_unit_tests/test_proxy_token_counter.py` with four regressions covering the previously-uncovered paths (missing model, missing messages, HTTPException from internal counter, generic Exception fallback). Updated the pre-existing `test_anthropic_endpoint_error_handling` which pinned the old buggy `HTTPException` contract.
…ens path Greptile P1 on commit b9c17be: the new `except ProxyException` branch in `count_tokens` passes `e.message` directly to `_anthropic_error_response`, which forwards it to `_strip_litellm_wrapper_prefixes`. When `ProxyException` is constructed with `message=None`, that None reaches `re.sub` and raises TypeError, crashing the error handler and producing an unhandled 500 with no Anthropic-shaped body. The same scenario already has an explicit guard on the `/v1/messages` `anthropic_response()` path (introduced earlier in this PR); the `count_tokens` sibling needs the same. Coerce `e.message is None -> str(e)` before handing off. Adds a regression test that constructs a ProxyException with `.message = None` and asserts the handler returns a well-formed Anthropic envelope at the expected status code.
Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com>
21423bd to
c2d95c0
Compare
|
Rebased onto current Conflict regions:
Tests passing locally:
State now MERGEABLE / no failing CI. |
`ruff format --check` (delta-vs-base lint gate landed on litellm_internal_staging while this PR was open) flagged 3 line-wrapped calls in this PR's diff. Apply the formatter's canonical layout to satisfy the gate. No behavior change. Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com> Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com>
|
Pushed |
|
@ishaan-jaff friendly bump — open 15 days, MERGEABLE with green CI (latest rebase + ruff format yesterday). This makes |
|
@Sameerlite friendly bump — open 17 days, MERGEABLE with green CI (rebased + ruff format 3 days ago). The P1 you flagged ( |
2 participants
Relevant issues
No existing issue. This PR fixes the
/v1/messageserror envelope so Anthropic SDK clients can switch onerror.error.typeas documented in the Anthropic API spec.Type
🐛 Bug Fix
Pre-Submission checklist
make testlocallyDescription
The Anthropic-compatible
/v1/messagesendpoint returned OpenAI-shaped error bodies ({"error": {message, type, param, code}}) instead of Anthropic-shaped ({"type": "error", "error": {type, message}}), breaking Anthropic SDK clients that switch onerror.error.type. Themessagealso leaked stacked LiteLLM class-name prefixes and an escaped upstream JSON body (litellm.RateLimitError: AnthropicException - {...}).Changes
1.
AnthropicExceptionMapping_strip_litellm_wrapper_prefixes()to peel stackedlitellm.<Class>:and<Provider>Exception -prefixes, and call it insidetransform_to_anthropic_error()so an embedded upstream Anthropic error body is detected and passed through unchanged (preserving the realerror.typeenum instead of deriving it from the HTTP status).json.JSONDecoder.raw_decodeas a fallback whensafe_json_loadsrejects the message. The Router appends debug suffixes after the upstream Anthropic JSON ({"type":"error",...}. Received Model Group=...) that break strict JSON parsing —raw_decodeparses the leading object and ignores the trailing garbage.error.messagefrom OpenAI-compat upstream bodies ({"error":{"code","message","type"}}, used by OpenAI, thenew-apiproxy, and many OpenAI-compat gateways) so the result is the inner human message instead of the full stringified JSON.2.
/v1/messageserror pathReturn
JSONResponsewith the Anthropic body instead of raisingProxyException.ProxyExceptionroutes through the global OpenAI-shaped handler;JSONResponseis required (notHTTPException, which wraps the dict in a spurious{"detail": ...}).3.
count_tokensSame root-cause fix — its existing
HTTPException(detail=...)was double-wrapping the Anthropic body in{"detail": ...}. Switch toJSONResponse. Framework-level 400s for missingmodel/messagesstay asHTTPException.x-litellm-*response headers are preserved on the error path.Breaking change marker (
!)Clients that parsed
error.param/error.codeon this Anthropic-compat endpoint will no longer see those OpenAI-only fields. Existing fields (error.type,error.message) remain.Test plan
/v1/messagesTestClientintegration: asserts top-level shape, nodetailwrapper, noparam/code, headers presentcount_tokenserror-format tests updated to assert theJSONResponsebodyraw_decodeerror.messageextraction39 tests pass across:
tests/test_litellm/anthropic_interface/exceptions/test_exception_mapping_utils.pytests/test_litellm/proxy/test_anthropic_error_passthrough.pytests/proxy_unit_tests/test_proxy_token_counter.pyCo-authored-by: songkuan-zheng songkuan-zheng@users.noreply.github.com