feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls

[yassin-berriai]

Relevant issues

Linear ticket

Resolves LIT-4308

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both runs use the same config and the same real OpenAI call; the request is deliberately invalid (max_tokens: -5) so OpenAI returns a real 400 that litellm maps to BadRequestError. A second, successful call confirms no event is emitted on success

# otel_events_config.yaml
model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  callbacks: ["otel"]

general_settings:
  master_key: sk-1234

export LITELLM_OTEL_V2=true
export LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS=true
export OTEL_EXPORTER=console
python litellm/proxy/proxy_cli.py --config otel_events_config.yaml --port 4118 2>&1 | tee proxy.log

Before (at 60729f733e)

curl -s -X POST http://127.0.0.1:4118/v1/chat/completions \
  -H 'Content-Type: application/json' -H 'Authorization: Bearer sk-1234' \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}],"max_tokens":-5}'

{"error":{"message":"litellm.BadRequestError: OpenAIException - Invalid 'max_tokens': integer below minimum value. Expected a value >= 1, but got -5 instead.. Received Model Group=gpt-4o-mini\nAvailable Model Group Fallbacks=None","type":"invalid_request_error","param":"max_tokens","code":"400"}}

grep -c "gen_ai.client.operation.exception" proxy.log   # -> 0
grep -c '"observed_timestamp"' proxy.log                # -> 0   (no logs-signal records at all)
grep -c '"exception.stacktrace"' proxy.log              # -> 0

The failed span carries only error.type, error.message, a generic exception span event with no stacktrace, and the stacktrace under the vendor key litellm.provider.error.stack_trace. Nothing named gen_ai.* describes the failure

After (at 5b436b57a8)

Same curl, plus a successful call:

curl -s -X POST http://127.0.0.1:4118/v1/chat/completions \
  -H 'Content-Type: application/json' -H 'Authorization: Bearer sk-1234' \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"say ok"}],"max_tokens":5}'

{"id":"chatcmpl-DznKgEbd8Oy1vibxtWfdO6OejqClx","model":"gpt-4o-mini","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"OK","role":"assistant"}}]}

grep -c "gen_ai.client.operation.exception" proxy.log   # -> 1   (the failure only; the success emits none)

The emitted log record, correlated to the failed LLM span:

{
  "event.name": "gen_ai.client.operation.exception",
  "severity_number": "<SeverityNumber.WARN: 13>",
  "exception.type": "BadRequestError",
  "exception.message": "litellm.BadRequestError: OpenAIException - Invalid 'max_tokens': integer below minimum val...",
  "exception.stacktrace": "  File \"/Users/.../litellm/main.py\", line ...",
  "trace_id": "0x21adc5aff4773c29b66e765c620625ae",
  "span_id": "0x40fc385c20eaf0a2"
}

The span_id above resolves to the failed chat gpt-4o-mini span in the same run, whose span-side surface is unchanged:

MATCHED span: chat gpt-4o-mini status: ERROR
  span error.type: BadRequestError
  span error.message present: True
  span litellm.provider.error.* keys: ['litellm.provider.error.code', 'litellm.provider.error.stack_trace', 'litellm.provider.error.llm_provider']
  span events (unchanged): ['exception']

Independent end-to-end verification (screen recording, at 5b436b57a8)

Verified independently on a fresh checkout of this branch, making a real call to api.openai.com with a deliberately-bogus OPENAI_API_KEY (sk-proj-invalid0000000000) so OpenAI returns a genuine 401 that litellm maps to AuthenticationError; nothing is mocked. Same config as above, same failing curl, run twice: once with LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS=true and once with it unset

With the flag on, the failure emits exactly one gen_ai.client.operation.exception log record at severity WARN (13) carrying exception.type (AuthenticationError), exception.message, and exception.stacktrace, and its trace_id / span_id match the failed chat gpt-4o-mini ERROR span in the same run (trace_id 0x0ff2b15a85add046d44aafc36cae3cae, span_id 0x66a99bfc87c78d2b). With the flag unset the identical 401 emits zero such records, while the span-side surface (error.type, error.message, the exception span event, and the litellm.provider.error.* keys) stays unchanged across both runs

The console log records reproduced above were captured from that independent run; a terminal screen recording of the whole flow (proxy launch, both curls, resulting log records) is archived internally and can be attached on request

Type

🆕 New Feature

Changes

The GenAI semantic conventions define failures of a GenAI client operation as a log-based event named gen_ai.client.operation.exception, recorded at severity WARN and carrying exception.type, exception.message, and exception.stacktrace, correlated to the failed span through the trace and span ids. OTel v2 never emitted it. A failed LLM call produced the error.* span attributes (whose error.message is deprecated upstream), a generic exception span event with no stacktrace, and the stacktrace only under the vendor key litellm.provider.error.stack_trace; nothing in the gen_ai.* namespace described the failure

This adds the logs pipeline the event needs, mirroring the existing metrics plumbing: build_log_exporter selects console, OTLP/HTTP, OTLP/gRPC, or in-memory from the same exporter kind the other signals use, build_logger_provider attaches a Simple processor for console and in-memory exporters and a Batch processor otherwise, and resolve_logger_provider reuses an operator-configured LoggerProvider global so the events land wherever the operator's other logs land. An explicit NoOpLoggerProvider global is treated as an opt-out and builds no recorder at all, so no event is ever constructed

Emission is gated on the enable_events config flag (LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS), which until now was defined on OpenTelemetryV2Config but consumed nowhere, and is scoped to LLM_CALL spans since the event describes a GenAI client operation; a failed guardrail or service span keeps its span-side error surface and records no event. exception.stacktrace is omitted when the payload carries no traceback

The existing span-side error surface stays exactly as it is, so consumers reading error.type, error.message, the exception span event, or the litellm.provider.error.* detail keys are unaffected. Teams that want to move off the deprecated error.message attribute now have a spec-compliant signal to read instead

Link to Devin session: https://app.devin.ai/sessions/a6eb6a9987ce403da9756be9dba004d6

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR implements the gen_ai.client.operation.exception log-based event from the GenAI semantic conventions, emitted at severity WARN with the exception.* attribute trio and correlated to the failed span via trace/span IDs. The feature is gated behind LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS (defaults off), and the existing span-side error surface is unchanged.

• plumbing/events.py introduces GenAIEventRecorder; plumbing/providers.py adds the matching log exporter/provider factory chain mirroring the existing metrics plumbing; emitter.py wires the recorder in, gated on SpanRole.LLM_CALL.
• resolve_logger_provider follows the same resolution order as resolve_meter_provider: injected provider wins, then an existing SDK global is reused, an explicit NoOpLoggerProvider global is treated as an opt-out, and only the default placeholder triggers building and publishing a new provider.
• Nine new component tests and two integration-level logger tests cover role gating, the required-pair guarantee, stacktrace conditionality, the opt-out path, and end-to-end correlation; all use in-memory exporters with no network calls.

Confidence Score: 5/5

Safe to merge — the new log-based event pipeline is additive only, gated off by default, and the existing span-side error surface is untouched.

All changes are additive and isolated to the OTel integration. The feature is opt-in via a flag that defaults to false, so deployments not setting LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS=true are completely unaffected. The provider resolution logic mirrors the battle-tested metrics path. Tests are comprehensive — role gating, the required-pair contract, stacktrace conditionality, the NoOp opt-out, operator-global reuse, and the full failure-callback path are all covered with in-memory exporters and no network calls. No backwards-incompatible changes to any existing signal.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/integrations/otel/plumbing/events.py	New GenAIEventRecorder class that emits the gen_ai.client.operation.exception log-based event; required pair always present, stacktrace conditional on truthiness
litellm/integrations/otel/plumbing/providers.py	Adds build_log_exporter, build_logger_provider, resolve_logger_provider, and get_event_logger, closely mirroring the existing metrics plumbing; OTLP gRPC exporter correctly omits path rewriting consistent with other signal providers
litellm/integrations/otel/emitter.py	SpanEmitter gains optional GenAIEventRecorder dependency; event emission is correctly gated on LLM_CALL role and recorder presence, preserving existing span-side error surface
litellm/integrations/otel/logger.py	OpenTelemetryV2 now accepts LoggerProvider and wires _init_events to build the recorder; type annotation tightened from Any to SDK LoggerProvider
litellm/integrations/otel/model/semconv.py	Adds ExceptionEvent.STACKTRACE constant and new GenAIEvent class with OPERATION_EXCEPTION; constants are pinned by a dedicated test
tests/test_litellm/integrations/otel/test_otel_v2_components.py	Adds 9 new unit tests covering event emission on failure, role gating, required-pair guarantee, no-op opt-out, operator global reuse, and semconv key pinning; all use in-memory exporters, no real network calls
tests/test_litellm/integrations/otel/test_otel_v2_logger.py	Adds integration-level tests exercising the full failure callback to log event path via the OpenTelemetryV2 logger, plus a default-off assertion; in-memory only, no network calls

_{Reviews (2): Last reviewed commit: "docs(otel): document the events plumbing..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 89.61039% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/otel/plumbing/providers.py	83.67%	8 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai please review the current head 3b9c11e

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_otel_genai_operation_exception_event (2038e77) with litellm_internal_staging (eb7e4a5)}