perf(otel): memoize per-request lazy import of otel runtime hooks

[yassin-berriai]

Relevant issues

Linear ticket

LIT-4104

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

The proxy auth path calls phase_span() and seed_request_identity() (in litellm/integrations/otel/runtime.py) on every request. Each did a try: from litellm.integrations.otel.logger import ... except Exception lazy import. otel.logger imports the OpenTelemetry SDK at module scope, so when the SDK is not installed (the default) the import raises. CPython never caches a failed import, so every request re-ran a full find_spec scan of sys.path and contended on the global import lock

Benchmark is locust against a real litellm proxy, 750 concurrent users, spawn 100/s, 60s, 4 worker processes, /v1/chat/completions only, Postgres and Redis both connected. The upstream is a static mock OpenAI server on purpose so the proxy's own per-request overhead dominates the measurement rather than provider latency; real-provider latency would swamp the signal this PR is about. Same config for every version, proxy run with --num_workers 4

Launch and sanity check

DATABASE_URL=postgresql://.../litellm DISABLE_SCHEMA_UPDATE=True \
  litellm --config config.yaml --port 4111 --num_workers 4

curl -s -X POST http://127.0.0.1:4111/v1/chat/completions \
  -H 'Authorization: Bearer sk-...' -H 'Content-Type: application/json' \
  -d '{"model":"mock-gpt-4o","messages":[{"role":"user","content":"hi"}]}' -w "\nHTTP %{http_code}\n"
# -> HTTP 200

Load test

locust -f locustfile.py --headless --host http://127.0.0.1:4111/v1 \
  -u 750 -r 100 -t 60s --processes 4 --tags completion

Controlled interleaved A/B (185, 191-unpatched, 191-patched), 2 runs each, constant background load

Version	Avg RPS	Median	p95	Error rate	vs 1.85.0
1.85.0	724.7	~715ms	~2550ms	0%	baseline
1.91 unpatched	636.5	~820ms	~2850ms	0%	-12.2%
1.91 patched (this PR)	705.8	~755ms	~2500ms	0%	-2.6%

The fix closes about 79% of the gap. The single-worker cProfile run (cleanest apples-to-apples CPU comparison, no multi-worker scheduling noise) reads 116.2 rps patched vs 117.0 for 1.85.0 vs 105.1 unpatched, so per-request overhead is back to baseline. The residual ~2.6% under 4-worker load sits inside 1.85.0's own run-to-run variance and traces to genuinely new per-request features (per-token cost and cache breakdown, overhead-latency metric) plus a FastAPI/starlette bump, not this code path

Direct evidence of the cause and the fix, captured by instrumenting PathFinder.find_spec

BEFORE (over ~1977 requests, unpatched):       AFTER (over ~2325 requests, patched):
  4424  opentelemetry                             5  opentelemetry
  4420  litellm.integrations.otel.logger          (otel.logger no longer scanned per request)

Full load-test CSVs, the find_spec probe output, and the cProfile diff: https://gist.github.com/yassin-berriai/c4a5be8099c6467d7e8330e9e82ec0e2

Type

🐛 Bug Fix

Changes

Resolve the SDK-backed OTel hooks once and memoize the outcome, absence included, with functools.cache in litellm/integrations/otel/runtime.py, so the lazy import is attempted a single time instead of on every request. Public behavior is unchanged: the wrappers still no-op when the SDK is absent or V2 is not the active logger, and still nest spans when it is

Added tests/test_litellm/integrations/otel/test_runtime.py, which counts import attempts across 50 calls and asserts the resolution happens at most once, asserts the functools.cache is memoized, and checks the SDK-absent path still no-ops without raising. The test fails on the pre-fix code (one import attempt per call) and passes with the fix

---

Note

Low Risk
Localized performance fix in the OTel shim with unchanged public no-op/span behavior and targeted regression tests.

Overview
Fixes a proxy auth hot-path regression where phase_span and seed_request_identity in litellm/integrations/otel/runtime.py attempted a lazy otel.logger import on every request. When the OpenTelemetry SDK is not installed, that import fails and CPython does not cache failures, so each call re-scanned sys.path and hit the import lock.

The change adds a @cache-memoized _otel_runtime() that resolves phase_span and seed_request_identity once (including caching None when the SDK is missing). phase_span and seed_request_identity delegate to that tuple; behavior is unchanged—still no-op without the SDK/V2 logger, still nest spans when OTel is active.

Adds tests/test_litellm/integrations/otel/test_runtime.py to assert at most one logger import across many calls, functools.cache hit/miss behavior, and no-op wrappers when runtime is absent.

^{Reviewed by Cursor Bugbot for commit 12b48de. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[yassin-berriai]

@greptileai

[greptile-apps]

Greptile Summary

This PR memoizes the OpenTelemetry runtime hook lookup. The main changes are:

• Cached resolution of phase_span and seed_request_identity, including the SDK-absent path
• Kept the public wrappers as no-ops when the OpenTelemetry SDK or V2 logger is unavailable
• Added tests for repeated hook calls, cache hits, and SDK-absent behavior

Confidence Score: 5/5

The change is narrowly scoped to OpenTelemetry runtime hook resolution and preserves no-op behavior when hooks are unavailable.

The implementation is covered by focused tests for repeated calls, cache behavior, and SDK-absent handling, with no code issues identified in the changed files.

T-Rex Logs

What T-Rex did

• Ran the base commit to exercise 6 import attempts for 6 absent-wrapper calls, and observed no exceptions when the logger import failed.
• Ran the head commit to exercise 1 import attempt for the same 6 absent-wrapper calls, after which subsequent calls were served from cache and returned None without exceptions.
• Compared before and after runs and confirmed present logger delegation events (enter/exit for present-0 and present-1) and two seed calls appear in both.

_{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "perf(otel): memoize per-request lazy imp..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

This PR reduces repeated OTel lazy-import work on the proxy request path. The main changes are:

• Caches SDK-backed OTel hook resolution in litellm/integrations/otel/runtime.py
• Preserves the SDK-absent no-op behavior for phase_span() and seed_request_identity()
• Adds tests for import-attempt memoization and SDK-free no-op behavior

Confidence Score: 4/5

The change is narrowly scoped to memoizing optional OTel hook resolution while preserving no-op behavior when the SDK is unavailable.

The added tests cover repeated resolution, cache behavior, and SDK-absent execution, which are the main behavioral risks for this patch.

No specific files require follow-up attention.

T-Rex Logs

What T-Rex did

• I ran the memoization proof to compare the base and head phase_span runs, noting the import_attempts and cache_info differences (base shows import_attempts=5 with cache_info=null, head shows import_attempts=1 with cache_info containing hits/misses).
• I examined the public behavior proof showing that when logger import failed, both base and head paths returned span None and seed None without raising an exception.
• I validated that both base and head paths delegated to the fake logger, which yielded the result {'fake_span': 'present'}.
• I confirmed that seed_request_identity was recorded with {'api_key':'k2'} and model-b during the public behavior test.

_{Ran code and verified through T-Rex}

_{Reviews (2): Last reviewed commit: "perf(otel): memoize per-request lazy imp..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 86.66667% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/otel/runtime.py	86.66%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 12b48de. Configure here.}