Skip to content

## Fix OpenTelemetry Logging functionality - #15645

Merged
2 commits merged into
BerriAI:mainfrom
ITV:fix-otel-logging
Oct 21, 2025
Merged

## Fix OpenTelemetry Logging functionality#15645
2 commits merged into
BerriAI:mainfrom
ITV:fix-otel-logging

Conversation

@tomhaynes

Copy link
Copy Markdown
Contributor

Bugfixes

  • select otel logging provider according to specified protocol
  • normalise otel paths correctly, for different telemetry types

Relevant issues

No ticket, but previously enabling otel logging caused an AttributeError because it was assuming the wrong logging provider.

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • I have added a screenshot of my new test passing locally
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

🐛 Bug Fix
✅ Test

Changes

@vercel

vercel Bot commented Oct 17, 2025

Copy link
Copy Markdown

@tomhaynes is attempting to deploy a commit to the CLERKIEAI Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Oct 17, 2025

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@ghost

ghost commented Oct 20, 2025

Copy link
Copy Markdown

hey @tomhaynes do you mind explaining the change you made here?

I don't follow what was causing the previous error - so it'll be helpful to understand the change

@tomhaynes

Copy link
Copy Markdown
Contributor Author

Hi sure @krrishdholakia, there were two issues I was bumping into. I'm looking to forward logs to an otel-collector, I initially followed these docs https://docs.litellm.ai/docs/proxy/logging#opentelemetry

Initially there were just no logs generated (metrics / traces come through fine). I found a change from last month that seemed to require LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS to be enabled - when I did this I got the following stacktrace:

Exception while exporting logs.
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/opentelemetry/sdk/_logs/_internal/export/__init__.py", line 311, in _export_batch
    self._exporter.export(self._log_records[:idx])  # type: ignore
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/grpc/_log_exporter/__init__.py", line 108, in export
    return self._export(batch)
           ~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/grpc/exporter.py", line 263, in _export
    request=self._translate_data(data),
            ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/grpc/_log_exporter/__init__.py", line 105, in _translate_data
    return encode_logs(data)
  File "/usr/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/common/_internal/_log_encoder/__init__.py", line 38, in encode_logs
    return ExportLogsServiceRequest(resource_logs=_encode_resource_logs(batch))
                                                  ~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/common/_internal/_log_encoder/__init__.py", line 60, in _encode_resource_logs
    sdk_resource = sdk_log.log_record.resource
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'LogRecord' object has no attribute 'resource'

This sounded similar to this issue open-telemetry/opentelemetry-python#3552 (comment) that this LogRecord object should not be instantiated in this way. I also noticed that the code hardcodes the grpc exporter rather than setting it dynamically based on protocol (although the attribute error manifests with either grpc/http). I tweaked the import to import LogRecord as SdkLogRecord to avoid conflict with native python LogRecord.

The second issue I saw was similar to this #13942 - regarding the endpoint configuration, specifically the path appended to it (/v1/traces or /v1/logs etc) - my changes to normalise the endpoint for each telemetry type should resolve this issue also, although they argue that we should be using the dedicated endpoint config for each OTEL_EXPORTER_OTLP_TRACES_ENDPOINT etc. - this arguably would be slightly better, but my normalisation logic works as long as all telemetry is expected to go to the same collector.

Thanks to @mubashir1osmani who was helping me investigate this, and has confirmed that other customers are also failing to get otel logging through.

@ghost
ghost merged commit 98f1d63 into BerriAI:main Oct 21, 2025
4 of 6 checks passed
@ghost

ghost commented Oct 21, 2025

Copy link
Copy Markdown

Thanks for explaining that!

@mubashir1osmani please feel free to submit a PR to update any relevant documentation on our end

fzowl pushed a commit to fzowl/litellm that referenced this pull request Jun 24, 2026
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants