fix(otel): mark v2 server spans as failed for pre-call errors

[devin-ai-integration]

TLDR

Problem this solves:

• OTel v2: pre-call failures leave the server span UNSET
• Malformed body traces carry an error message but no error event
• v1 marked the same requests as errors; v2 regressed
• Malformed body traces are also anonymous: no key / team / user, no child span

How it solves it:

• record_error_attributes_on_span now sets the ERROR status
• It records the exception event when nothing stamped the span yet
• Attributes keep being restamped so error.code stays the response status
• A body that fails to parse is authenticated first, then the parse error is re-raised, so identity lands on the trace

Relevant issues

Linear ticket

Resolves LIT-4780

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Every image below is generated from a recorded run against live proxies; no numbers are typed by hand. The suite is 42 scenarios run in identical order against three trees on separate ports and separate Postgres instances, with real Gemini calls and the OTel v2 console exporter: the merge-base b6e3ff63, the previous head 626e1eeff8, and the current head 613ea891.

The bug this PR fixes

A malformed body used to leave the SERVER span UNSET with no exception event, on an anonymous trace. It now reads as a failure, carries the caller's identity, and has the auth child span.

The gate: authorization is skipped only when the body never parsed

Deferring the parse failure meant the full auth phase ran, and its last step reserves budget by incrementing the live spend counters. Those are released by the endpoint, which never runs for a malformed body, so the previous head leaked a reservation per request and locked the key out. Three malformed requests exhausted a 1.00 USD key while its recorded spend was 0.0000031.

Azure-style model-in-path route, six malformed requests against a 1.00 USD key, then one legitimate call:

Same on the Gemini-compatible route:

A malformed body sent with a rejected key returns the 400 it returned before the PR; the auth failure is still recorded on the auth child span:

The gate in the other direction

The risk of a skip-authorization gate is that it skips too much. For a well-formed body every authorization check still runs, and head matches base on all of them:

Concurrency: 15 malformed and 15 valid requests fired simultaneously on one budgeted key return 400 and 200 respectively, and the following legitimate call returns 200, so the gate does not race.

The dedup gate

The exception event is recorded only when nothing stamped the span yet, so a failure that already went through async_post_call_failure_hook is not double counted. Exactly one event on every path:

Happy paths and edge cases

The three 500s and the 422 in that matrix reproduce at the merge-base as well, so they are pre-existing and not introduced here.

Full matrix

42 of 42 scenarios on the current head are identical to the merge-base. Seven differed on the previous head; all seven are fixed.

Type

🐛 Bug Fix

Changes

The proxy's global exception handlers turn a failure into a JSONResponse, so the FastAPI instrumentor never sees the exception and never statuses the span it owns. For failures that reach async_post_call_failure_hook (auth rejection, unknown model, empty body) that hook stamps status plus the exception event, so the trace looks right. A body that fails to parse dies earlier, inside the user_api_key_auth dependency, so the only OTel hook that runs is record_error_attributes_on_span, which deliberately passed record_event=False, set_status=False; the span then carries error.message while reading as a non-error span with no event. v1 set the ERROR status for the same request from _close_dangling_otel_server_span, which is why this only shows up on v2.

record_error_attributes_on_span now sets the status, and records the exception event only when the span has no error.type on it yet, so the paths that already recorded one don't get a second. Attributes are restamped either way, keeping error.code pinned to the real response status.

Separately, user_api_key_auth read and parsed the body before authenticating, so a parse failure short-circuited auth entirely; identity was never resolved and seed_request_identity never ran, which is why the root span had no key / team / user on it and no children. It now catches the parse failure, authenticates the caller with an empty body, and re-raises the original ProxyException once identity is seeded, so the trace gets the auth child span plus identity attributes like every other failure. This part of the change is not conditional on OTel, it reorders the dependency for every deployment

A request whose body never parsed is authenticated but deliberately not authorized: the common_checks gate is skipped, since there is no model to check it against and its budget reservation increments live spend counters that only the endpoint's post-call path releases. The endpoint never runs here, so running it would leak a reservation per malformed request and lock a budgeted key out. The client-visible response is unchanged from base in every case, including a malformed body sent with a key that fails auth, which still gets the 400 rather than the auth status

Note that such a request still has no LLM child span; there was no provider call to represent, and inventing one would put a phantom CLIENT span in the trace.

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/a132d58d757c4b1395d3ddce219d8d6b
Requested by: @shivamrawat1

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

Updates malformed-request handling to authenticate before re-raising parse errors and marks affected OTel v2 server spans as failed without duplicating exception events

• Defers JSON parse failures until caller identity is resolved and seeded
• Keeps authorization and budget reservation disabled for requests that cannot be parsed
• Adds regression coverage for span status, exception-event deduplication, identity seeding, and error precedence

Confidence Score: 5/5

The PR appears safe to merge because no blocking failures remain

No blocking failures remain

Important Files Changed

Filename	Overview
litellm/integrations/otel/logger.py	Marks pre-call server-span failures as errors and conditionally records an exception event
litellm/proxy/auth/user_api_key_auth.py	Authenticates malformed requests for trace identity before re-raising the original parse failure
tests/test_litellm/integrations/otel/test_otel_v2_logger.py	Covers server-span error status and exception-event deduplication
tests/test_litellm/proxy/auth/test_user_api_key_auth.py	Covers malformed-body authentication, identity seeding, skipped authorization, and preserved parse-error responses

_{Reviews (4): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 97.22222% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/auth/user_api_key_auth.py	97.05%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_otel_v2_error_status_lit_4780 (613ea89) with litellm_internal_staging (f05d468)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (b6e3ff6) during the generation of this report, so f05d468 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[shivamrawat1]

@gerptile run again

[yucheng-berri]

@greptileai review latest head

[yucheng-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 626e1ee. Configure here.}

[cursor]

Recovered auth skips identity seed

Low Severity

When _handle_authentication_error recovers and body_parse_exception is set, the parse error is re-raised inside the phase_span before seed_request_identity runs. That leaves the rejected trace without identity and marks the auth span as failed for a parse error after auth already recovered.

Additional Locations (1)

• litellm/proxy/auth/user_api_key_auth.py#L2632-L2640

 

^{Reviewed by Cursor Bugbot for commit 626e1ee. Configure here.}

[yucheng-berri]

Ran a base vs head A/B on a live proxy (real Postgres, real provider calls, OTel v2 console exporter). The span fix works as described: a malformed-body 400 goes from status=UNSET with no events to status=ERROR with a single exception event, and the auth child span is present and correctly left UNSET. A real paid call is untouched, 200 with all spans UNSET on both sides.

Two things worth a look before merge.

Budget reservation is orphaned on model-in-path routes. Deferring the parse failure means the full auth phase now runs, and its last step reserves budget by incrementing the live spend counters. The re-raise at the end of user_api_key_auth means the endpoint never runs, so neither _ProxyDBLogger.async_post_call_failure_hook nor release_budget_reservation_on_cancel releases it. On a key with max_budget: 1.0:

base: 10 malformed -> 400 x10,                legit call after: 200
head: 10 malformed -> 400 400 400 429 429..., legit call after: 429
      "Budget has been exceeded! Current cost: 1.0, Max budget: 1.0"

Three malformed requests exhaust the budget and the key stays locked out, while recorded DB spend is 0.0000031. Repro:

curl -X POST localhost:4000/openai/deployments/gpt-5.1/chat/completions \
  -H "Authorization: Bearer $KEY" -H 'content-type: application/json' -d '{"model":'

It does not reproduce on /v1/chat/completions, where the model is unresolvable from an empty body and the reservation bails; it does reproduce on /v1beta/models/{model}:generateContent. Raising the parse error immediately after _user_api_key_auth_builder returns, before _run_centralized_common_checks, keeps identity and the auth span while skipping everything that writes.

The client-visible status code changes. Malformed body plus an invalid key returns 400 on base and 401 on head, since _handle_authentication_error raises before the deferred parse error. The description says "The client still gets the same 400 with the same message" and then says an auth failure takes precedence; only the second holds.

Scoping note: the deferral and both re-raise sites carry no OTel condition, so this applies with v1, with v2, and with OTel disabled. The PR reads as v2-scoped; the auth change underneath is global. That one I confirmed by reading the code, not by running with OTel off.

[devin-ai-integration]

Both fixed in cfc1866: authorization now skipped for unparsed bodies, so no reservation, and a rejected key still returns 400. Description corrected, budget A/B re-run clean

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ shivamrawat1
❌ devin-ai-integration[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[yucheng-berri]

@greptileai review latest head

[yucheng-berri]

Ran a dependency blast-radius pass on 613ea891 against merge-base b6e3ff63, two proxies, two Postgres instances, real Gemini calls. No breaking changes: chat completions, streaming, /v1/messages, responses, management, Gemini and Vertex passthrough, multipart audio and /metrics all answer identically to base, authorization still holds for well-formed bodies, and end-user attribution still lands (user in the body still writes end_user to spend logs, so moving that block into _authorize_authenticated_request did not drop it).

Two side effects of authenticating before rejecting that are not mentioned in the description or covered by a test. Both follow from the design rather than being defects, but they are worth recording so the trade is made on purpose.

An operator's custom_auth callable now runs for requests whose body never parsed. Same 400 to the caller, but their code executes:

base   5 malformed requests -> custom_auth invocations: 0
head   5 malformed requests -> custom_auth invocations: 5

Anyone whose custom auth does an IdP round trip, an audit write, or a per-caller counter picks up new volume on unparseable input.

Unauthenticated requests now do database work. Twenty requests carrying an unknown key and a malformed body:

base: 2 db transactions,  0.27s wall
head: 86 db transactions, 0.71s wall

That is roughly four transactions per request from a caller who is not authenticated, against essentially none before. It is a cheap amplifier, though a rate limiter in front covers the realistic case.

Which raises the design question worth answering before merge: how much of the identity goal needs the full resolution? The API key is already in the header, so hashing it would put litellm.api_key.hash on the trace with no database work and no custom-auth invocation. Only team and user IDs need the lookup. If attribution by key is enough to answer "which caller is sending these", the cheaper path gets most of the value without either side effect above.

Separately, and not a blocker for this PR: _mcp_oauth_user_api_key_auth in litellm/proxy/management_endpoints/mcp_management_endpoints.py is a parallel copy that reads the body the old way and calls _user_api_key_auth_builder directly, so it does not get this fix. Verified on the head build:

POST /v1/chat/completions                    status=ERROR  events=['exception']  identity=yes
POST /v1/mcp/server/oauth/{server_id}/token  status=UNSET  events=[]             identity=no

LIT-4780 is fixed on the 585 routes that go through the dependency and still live on that one. Worth a follow-up ticket so the inconsistency is tracked.

[devin-ai-integration]

Hashing the key alone misses what the ticket asked for, team name, id and metadata, which need the lookup. Filing a follow-up for the MCP OAuth copy

[devin-ai-integration]

Filed LIT-5363 for the MCP OAuth copy: https://linear.app/litellm-ai/issue/LIT-5363/bug-mcp-oauth-token-route-misses-the-otel-v2-error-span-fix