fix(logging): keep tracebacks for provider-originated 4xx errors

[mateo-berri]

TLDR

Problem this solves:

• Provider 4xx failures (bad deployment key) lost their traceback since fix(proxy): stop expected 4xx responses from saturating worker CPU on failure logging #38102
• OTel error spans dropped litellm.provider.error.stack_trace on /v1/chat/completions, /v1/responses, and /v1/messages
• Scheduled e2e suite red on test_failed_chat_completions_error_span_attributes (litellm-e2e builds 68 to 70)

How it solves it:

• A 4xx that came back from a provider is never an expected client error: a mapped litellm exception carrying llm_provider, or the raw provider exception /v1/messages logs as-is
• The proxy's own pre-call 4xx rejections still skip the traceback, keeping fix(proxy): stop expected 4xx responses from saturating worker CPU on failure logging #38102's CPU win, including the proxy's own rate limiters and the key-over-budget 429 even though the auth handler stamps an llm_provider on them
• Regression tests at the helper, standard logging payload, and OTel span levels

User Flow

Before: an operator tracing LiteLLM into their OTel backend sees a provider auth failure span with no stack trace, so they cannot tell where in the gateway the call died

1. They register a deployment whose upstream key is wrong: POST https://litellm-domain/model/new with {"model_name":"otel-err","litellm_params":{"model":"anthropic/claude-haiku-4-5","api_key":"<invalid>"}} and get 200 with a model id
2. Their app sends POST https://litellm-domain/v1/chat/completions with {"model":"otel-err","messages":[...]} and gets 401 litellm.AuthenticationError: AnthropicException - {"type":"error","error":{"type":"authentication_error","message":"API key is invalid."}} plus an x-litellm-call-id header (POST https://litellm-domain/v1/responses and POST https://litellm-domain/v1/messages fail the same way)
3. They open that call id's trace in their OTel backend (e.g. http://jaeger-domain/search, service litellm): the chat otel-err span carries error.type=AuthenticationError, litellm.provider.error.code=401, and litellm.provider.error.llm_provider=anthropic, but no litellm.provider.error.stack_trace tag, so the trace ends at "the provider said no" with nothing about where the gateway was when it failed

After: the same span carries the stack trace, so the operator sees the gateway frames the failing provider call went through

1. They register a deployment whose upstream key is wrong: POST https://litellm-domain/model/new with {"model_name":"otel-err","litellm_params":{"model":"anthropic/claude-haiku-4-5","api_key":"<invalid>"}} and get 200 with a model id
2. Their app sends POST https://litellm-domain/v1/chat/completions with {"model":"otel-err","messages":[...]} and gets 401 litellm.AuthenticationError: AnthropicException - {"type":"error","error":{"type":"authentication_error","message":"API key is invalid."}} plus an x-litellm-call-id header (POST https://litellm-domain/v1/responses and POST https://litellm-domain/v1/messages fail the same way)
3. They open that call id's trace in their OTel backend: the chat otel-err span carries the same error.type, litellm.provider.error.code, and litellm.provider.error.llm_provider tags and now also litellm.provider.error.stack_trace with the gateway frames of the failed call, on all three routes

Relevant issues

Regression from #38102

Linear ticket

Resolves LIT-6163

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup for both legs: a local Jaeger (docker run -d --name lit6163-jaeger -p 37398:16686 -p 49068:4318 jaegertracing/all-in-one), a local Postgres with the schema pushed, and the proxy started with --num_workers 2 and LITELLM_OTEL_V2=1 PHOENIX_COLLECTOR_HTTP_ENDPOINT=http://localhost:49068/v1/traces on this config (the deployment under test is registered through the API, not the config):

model_list:
  - model_name: claude-haiku-4-5
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
  callbacks: ["arize_phoenix"]

general_settings:
  master_key: sk-1234
  store_model_in_db: true

Each leg registers one deployment with an invalid Anthropic key, sends it one request on each of /v1/chat/completions, /v1/responses, and /v1/messages, and reads every error span of that call id back from Jaeger's query API. The last case runs tests/e2e/logging/test_otel_trace_e2e.py::TestOtelTraceCompleteness::test_failed_chat_completions_error_span_attributes against the same proxy with LITELLM_PROXY_URL=http://localhost:<port> E2E_OTEL_QUERY_URL=http://localhost:37398, exactly what the scheduled suite runs

Before (72b8b47)

Register a deployment with an invalid Anthropic key

1. Register it

   curl -s http://localhost:45575/model/new -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model_name":"otel-err-before-20295","litellm_params":{"model":"anthropic/claude-haiku-4-5","api_key":"sk-upstream-invalid-for-langfuse-e2e-only"}}'
   

   {"model_name": "otel-err-before-20295", "model_id": "3b7c2127-8ffe-46b1-97b2-25b654dc525a"}
   

Chat completion through it

1. Send the request

   curl -s -D - http://localhost:45575/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"otel-err-before-20295","messages":[{"role":"user","content":"trigger an upstream auth failure"}],"max_tokens":16}'
   

   HTTP/1.1 401 Unauthorized
   x-litellm-call-id: 1c27cdad-f558-42c2-9c16-0bc28bfceab9
   {"error":{"message":"litellm.AuthenticationError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"authentication_error\",\"message\":\"API key is invalid.\"},\"request_id\":null}. Received Model Group=otel
   

2. Read the error spans back from Jaeger

   curl -s 'http://localhost:37398/api/traces?service=litellm&tags=%7B%22litellm.call_id%22%3A%221c27cdad-f558-42c2-9c16-0bc28bfceab9%22%7D' | python3 -c '...print every litellm.provider.error.* and error.* tag of every span that carries error.type...'
   

   traces: 1 spans: ['auth /v1/chat/completions', 'chat otel-err-before-20295', 'postgres get_data', 'POST /v1/chat/completions']
     span: chat otel-err-before-20295
       error.type = 'AuthenticationError'
       litellm.provider.error.code = '401'
       litellm.provider.error.llm_provider = 'anthropic'
       litellm.provider.error.stack_trace present: False
     span: POST /v1/chat/completions
       error.type = 'ProxyException'
       litellm.provider.error.code = '401'
       litellm.provider.error.llm_provider = 'anthropic'
       litellm.provider.error.stack_trace present: False
   

Responses API through it

1. Send the request

   curl -s -D - http://localhost:45575/v1/responses -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"otel-err-before-20295","input":"trigger an upstream auth failure","max_output_tokens":16}'
   

   HTTP/1.1 401 Unauthorized
   x-litellm-call-id: 3c96d60c-96d6-4c0c-bbec-2f52340400e3
   {"error":{"message":"litellm.AuthenticationError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"authentication_error\",\"message\":\"API key is invalid.\"},\"request_id\":null}. Received Model Group=otel
   

2. Read the error spans back from Jaeger

   curl -s 'http://localhost:37398/api/traces?service=litellm&tags=%7B%22litellm.call_id%22%3A%223c96d60c-96d6-4c0c-bbec-2f52340400e3%22%7D' | python3 -c '...print every litellm.provider.error.* and error.* tag of every span that carries error.type...'
   

   traces: 1 spans: ['auth /v1/responses', 'chat otel-err-before-20295', 'postgres get_data', 'POST /v1/responses']
     span: chat otel-err-before-20295
       error.type = 'AuthenticationError'
       litellm.provider.error.code = '401'
       litellm.provider.error.llm_provider = 'anthropic'
       litellm.provider.error.stack_trace present: False
     span: POST /v1/responses
       error.type = 'ProxyException'
       litellm.provider.error.code = '401'
       litellm.provider.error.llm_provider = 'anthropic'
       litellm.provider.error.stack_trace present: False
   

Anthropic Messages API through it

1. Send the request

   curl -s -D - http://localhost:45575/v1/messages -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"otel-err-before-20295","messages":[{"role":"user","content":"trigger an upstream auth failure"}],"max_tokens":16}'
   

   HTTP/1.1 401 Unauthorized
   x-litellm-call-id: f1f91484-6082-4cb6-a7aa-1e30641462e2
   {"error":{"message":"{\"type\":\"error\",\"error\":{\"type\":\"authentication_error\",\"message\":\"API key is invalid.\"},\"request_id\":null}. Received Model Group=otel-err-before-20295\nAvailable Model Group Fallbacks
   

2. Read the error spans back from Jaeger

   curl -s 'http://localhost:37398/api/traces?service=litellm&tags=%7B%22litellm.call_id%22%3A%22f1f91484-6082-4cb6-a7aa-1e30641462e2%22%7D' | python3 -c '...print every litellm.provider.error.* and error.* tag of every span that carries error.type...'
   

   traces: 1 spans: ['postgres get_data', 'POST /v1/messages', 'auth /v1/messages', 'chat otel-err-before-20295']
     span: POST /v1/messages
       error.type = 'ProxyException'
       litellm.provider.error.code = '401'
       litellm.provider.error.stack_trace present: False
     span: chat otel-err-before-20295
       error.type = 'BaseLLMException'
       litellm.provider.error.code = '401'
       litellm.provider.error.stack_trace present: False
   

The scheduled e2e test against the same proxy

1. Run the test

   LITELLM_PROXY_URL=http://localhost:45575 LITELLM_MASTER_KEY=sk-1234 E2E_OTEL_QUERY_URL=http://localhost:37398 .venv/bin/python -m pytest "tests/e2e/logging/test_otel_trace_e2e.py::TestOtelTraceCompleteness::test_failed_chat_completions_error_span_attributes" -p no:cacheprovider -q
   

           stack = _tag(span, "litellm.provider.error.stack_trace")
   >       assert isinstance(stack, str) and stack, (
               "the error span must carry a non-empty litellm.provider.error.stack_trace"
           )
   E       AssertionError: the error span must carry a non-empty litellm.provider.error.stack_trace
   E       assert (False)
   E        +  where False = isinstance(None, str)
   tests/e2e/logging/test_otel_trace_e2e.py:283: AssertionError
   1 failed in 21.03s
   

After (514cae1)

Register a deployment with an invalid Anthropic key

1. Register it

   curl -s http://localhost:57641/model/new -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model_name":"otel-err-after5-12755","litellm_params":{"model":"anthropic/claude-haiku-4-5","api_key":"sk-upstream-invalid-for-langfuse-e2e-only"}}'
   

   {"model_name": "otel-err-after5-12755", "model_id": "d7d9404d-594a-43f8-ae6e-68dc62e29d8b"}
   

Chat completion through it

1. Send the request

   curl -s -D - http://localhost:57641/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"otel-err-after5-12755","messages":[{"role":"user","content":"trigger an upstream auth failure"}],"max_tokens":16}'
   

   HTTP/1.1 401 Unauthorized
   x-litellm-call-id: 02be2ff7-293f-4231-811b-cf2c53338d10
   {"error":{"message":"litellm.AuthenticationError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"authentication_error\",\"message\":\"API key is invalid.\"},\"request_id\":null}. Received Model Group=otel
   

2. Read the error spans back from Jaeger

   curl -s 'http://localhost:37398/api/traces?service=litellm&tags=%7B%22litellm.call_id%22%3A%2202be2ff7-293f-4231-811b-cf2c53338d10%22%7D' | python3 -c '...print every litellm.provider.error.* and error.* tag of every span that carries error.type...'
   

   traces: 1 spans: ['auth /v1/chat/completions', 'chat otel-err-after5-12755', 'postgres get_data', 'POST /v1/chat/completions']
     span: chat otel-err-after5-12755
       error.type = 'AuthenticationError'
       litellm.provider.error.code = '401'
       litellm.provider.error.llm_provider = 'anthropic'
       litellm.provider.error.stack_trace = '  File "/Users/mateo/Development/litellm_fix_otel_provider_error_stack_trace/litellm/utils.py", line 1856, in wrapper_async\n    result = awa' ... (1007 chars)
       litellm.provider.error.stack_trace present: True
     span: POST /v1/chat/completions
       error.type = 'ProxyException'
       litellm.provider.error.code = '401'
       litellm.provider.error.llm_provider = 'anthropic'
       litellm.provider.error.stack_trace = '  File "/Users/mateo/Development/litellm_fix_otel_provider_error_stack_trace/litellm/proxy/proxy_server.py", line 10107, in chat_completion\n' ... (4429 chars)
       litellm.provider.error.stack_trace present: True
   

Responses API through it

1. Send the request

   curl -s -D - http://localhost:57641/v1/responses -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"otel-err-after5-12755","input":"trigger an upstream auth failure","max_output_tokens":16}'
   

   HTTP/1.1 401 Unauthorized
   x-litellm-call-id: 8da5a7f5-69b1-47c5-8187-85af5c7cd0c1
   {"error":{"message":"litellm.AuthenticationError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"authentication_error\",\"message\":\"API key is invalid.\"},\"request_id\":null}. Received Model Group=otel
   

2. Read the error spans back from Jaeger

   curl -s 'http://localhost:37398/api/traces?service=litellm&tags=%7B%22litellm.call_id%22%3A%228da5a7f5-69b1-47c5-8187-85af5c7cd0c1%22%7D' | python3 -c '...print every litellm.provider.error.* and error.* tag of every span that carries error.type...'
   

   traces: 1 spans: ['auth /v1/responses', 'chat otel-err-after5-12755', 'postgres get_data', 'POST /v1/responses']
     span: chat otel-err-after5-12755
       error.type = 'AuthenticationError'
       litellm.provider.error.code = '401'
       litellm.provider.error.llm_provider = 'anthropic'
       litellm.provider.error.stack_trace = '  File "/Users/mateo/Development/litellm_fix_otel_provider_error_stack_trace/litellm/utils.py", line 1856, in wrapper_async\n    result = awa' ... (1007 chars)
       litellm.provider.error.stack_trace present: True
     span: POST /v1/responses
       error.type = 'ProxyException'
       litellm.provider.error.code = '401'
       litellm.provider.error.llm_provider = 'anthropic'
       litellm.provider.error.stack_trace = '  File "/Users/mateo/Development/litellm_fix_otel_provider_error_stack_trace/litellm/proxy/response_api_endpoints/endpoints.py", line 340, i' ... (6284 chars)
       litellm.provider.error.stack_trace present: True
   

Anthropic Messages API through it

1. Send the request

   curl -s -D - http://localhost:57641/v1/messages -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"otel-err-after5-12755","messages":[{"role":"user","content":"trigger an upstream auth failure"}],"max_tokens":16}'
   

   HTTP/1.1 401 Unauthorized
   x-litellm-call-id: 2c781610-d163-4562-901d-0a1c58408839
   {"error":{"message":"{\"type\":\"error\",\"error\":{\"type\":\"authentication_error\",\"message\":\"API key is invalid.\"},\"request_id\":null}. Received Model Group=otel-err-after5-12755\nAvailable Model Group Fallbacks=
   

2. Read the error spans back from Jaeger

   curl -s 'http://localhost:37398/api/traces?service=litellm&tags=%7B%22litellm.call_id%22%3A%222c781610-d163-4562-901d-0a1c58408839%22%7D' | python3 -c '...print every litellm.provider.error.* and error.* tag of every span that carries error.type...'
   

   traces: 1 spans: ['auth /v1/messages', 'chat otel-err-after5-12755', 'postgres get_data', 'POST /v1/messages']
     span: chat otel-err-after5-12755
       error.type = 'BaseLLMException'
       litellm.provider.error.code = '401'
       litellm.provider.error.stack_trace = '  File "/Users/mateo/Development/litellm_fix_otel_provider_error_stack_trace/litellm/utils.py", line 1856, in wrapper_async\n    result = awa' ... (1350 chars)
       litellm.provider.error.stack_trace present: True
     span: POST /v1/messages
       error.type = 'ProxyException'
       litellm.provider.error.code = '401'
       litellm.provider.error.stack_trace = '  File "/Users/mateo/Development/litellm_fix_otel_provider_error_stack_trace/litellm/proxy/anthropic_endpoints/endpoints.py", line 97, in an' ... (5754 chars)
       litellm.provider.error.stack_trace present: True
   

The scheduled e2e test against the same proxy

1. Run the test

   LITELLM_PROXY_URL=http://localhost:57641 LITELLM_MASTER_KEY=sk-1234 E2E_OTEL_QUERY_URL=http://localhost:37398 .venv/bin/python -m pytest "tests/e2e/logging/test_otel_trace_e2e.py::TestOtelTraceCompleteness::test_failed_chat_completions_error_span_attributes" -p no:cacheprovider -q
   

   .                                                                        [100%]
   1 passed in 20.37s
   

QA observations:

• /v1/messages error spans report error.type=BaseLLMException, no llm_provider tag; pre-existing, left alone
• /v1/messages 401 body lacks the litellm.AuthenticationError prefix; pre-existing, left alone

Neighbouring paths, live A/B (base 72b8b47 vs 514cae1, 2 workers each, spans read from Jaeger)

Scenario	Status	Base stack_trace	Tip stack_trace
Valid key, valid model	200	n/a	n/a
Invalid virtual key	401	absent	absent
Unknown model (#38102 flow)	400	absent	absent
Key over budget	429	absent	absent
Key rpm_limit=1 hit	429	absent	absent
Provider 401, non-streaming	401	absent	present
Provider 401, streaming	401	absent	present
Provider 400 (temperature 5)	400	absent	present

Response bodies and statuses were identical on both sides for every row. The key-over-budget row carried a stack trace at a6e1708 because the auth handler stamps the model's provider onto the budget error; 514cae1 closes that

Type

🐛 Bug Fix

Caveats (if any)

Low

• Provider 429s and 4xx validation errors regain their traceback too, as intended
  • Bounded by upstream round trips, so they cannot saturate a worker the way local rejections could
• Router "no deployments available" errors carry an empty llm_provider, so they still count as expected client errors
• The proxy's own rate limiters raise an HTTPException subclass that carries an llm_provider, and the auth handler stamps one onto BudgetExceededError, so the HTTPException check and litellm's own rate-limit category both run before the llm_provider carve-out and keep them expected

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
• 514cae1 passes /live-pr-risk

---

Note

Medium Risk
Changes classification of exceptions on the hot logging path; misclassification could either leak tracebacks for cheap local 4xx or drop them for operator-critical upstream failures, but behavior is heavily regression-tested.

Overview
Fixes a regression where upstream provider 4xx (bad deployment key, invalid upstream auth) were treated like local proxy rejections, so standard logging and OTel v2 dropped stack traces from error payloads and litellm.provider.error.stack_trace on LLM error spans.

is_expected_client_error in core_helpers.py is narrowed: only proxy-side rejections before a provider call stay “expected” (no traceback when log_client_error_tracebacks is off). New helpers distinguish proxy vs provider via HTTPException, LiteLLM rate-limit category, llm_provider, and BaseLLMException. Proxy budget and LiteLLM limiter 429s remain expected even when auth decorates them with a model provider.

Regression tests cover the helper, StandardLoggingPayloadSetup.get_error_information, and OTel v2 failure spans (mapped and unmapped provider errors on /v1/messages).

^{Reviewed by Cursor Bugbot for commit 514cae1. Bugbot is set up for automated code reviews on this repo. Configure here.}

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[greptile-apps]

Greptile Summary

This PR refines client-error classification so provider-originated 4xx failures retain tracebacks while proxy-originated rejections remain inexpensive.

• Distinguishes provider failures from proxy HTTP, budget, and rate-limit rejections.
• Adds helper, structured-logging, and OpenTelemetry regression coverage.
• Preserves traceback details on provider error spans across mapped and raw provider exceptions.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/core_helpers.py	Adds provider-versus-proxy error classification before deciding whether a 4xx traceback should be omitted.
tests/test_litellm/integrations/otel/test_otel_v2_logger.py	Verifies mapped and unmapped provider authentication failures retain stack traces in OpenTelemetry spans.
tests/test_litellm/litellm_core_utils/test_core_helpers.py	Covers provider errors, proxy rate limits, router rejections, and budget-limit classification.
tests/test_litellm/litellm_core_utils/test_litellm_logging.py	Verifies standard logging retains provider tracebacks while omitting tracebacks for local budget rejections.

_{Reviews (3): Last reviewed commit: "fix(logging): keep the proxy's own budge..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 90.47619% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/litellm_core_utils/core_helpers.py	90.47%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 514cae1. Configure here.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_otel_provider_error_stack_trace (514cae1) with litellm_internal_staging (273b01a)}