fix(otel v2): a callback exports only to the destination it owns, so vendor spans stay off the generic OTEL_* collector by devin-ai-integration[bot] · Pull Request #40881 · BerriAI/litellm

devin-ai-integration · 2026-09-12T19:14:07Z

TLDR

Problem this solves:

every vendor OTel v2 preset also exports to the operator's OTEL_* collector
so vendor vocabulary (langfuse.*, OpenInference) lands on a collector that never asked for it
with ["langfuse_otel", "otel"] the otel entry reuses the Langfuse logger, so callback order decides what leaks
a preset alone with no OTEL_* env prints every span as JSON to proxy stdout
the Langfuse logger names the request root span even when another logger's provider owns it
there since OTel v2 landed (feat(otel): typed semconv-aligned OpenTelemetry instrumentation #28909); fix(otel v2): name Langfuse traces from the langfuse_trace_name header or metadata.trace_name #40793 gave langfuse_otel its own class, which surfaced it in the SDK path

How it solves it:

a preset lists only the exporter it owns, never the inherited OTEL_* one
otel alone serves OTEL_*; a generic collector plus a vendor now needs both in callbacks
otel reuses only a logger whose every exporter is the operator's collector (a plain otel logger, or a mapper-only preset like langtrace), and the published global provider prefers that one
when otel rides a mapper-only preset built before it, that logger still gets callback_settings.otel and the Phoenix auto-init a lone otel entry performs
the Langfuse logger stamps the request root only when its own provider created it
Weave v2 reads its config without rewriting the process-wide OTEL_* env
the interim langfuse.* prefix filter from the first revision of this PR is gone

User Flow

Before: an operator who ships traces to both a generic OTLP collector and Langfuse sees Langfuse-only attributes and duplicate spans in the generic collector

They set callbacks: ["otel", "langfuse_otel"], OTEL_EXPORTER_OTLP_ENDPOINT to their collector, and LANGFUSE_* to their Langfuse project
A developer sends POST https://litellm-domain/v1/chat/completions with header langfuse_trace_name: private-langfuse-only-name and gets HTTP 200
Langfuse shows a trace named private-langfuse-only-name with a chat proof-model generation, as expected
The generic collector shows the request root span with langfuse.trace.name=private-langfuse-only-name, the standard chat proof-model span, and a second chat proof-model span carrying langfuse.observation.input, langfuse.observation.output and six more langfuse.* keys
If they swap the callback order to ["langfuse_otel", "otel"], the collector instead gets every span with langfuse.* keys, and with callbacks: ["langfuse_otel"] and no OTEL_* env the same spans are printed to the proxy's stdout

After: the generic collector only sees standard spans, Langfuse is unchanged

They set callbacks: ["otel", "langfuse_otel"], OTEL_EXPORTER_OTLP_ENDPOINT to their collector, and LANGFUSE_* to their Langfuse project
A developer sends POST https://litellm-domain/v1/chat/completions with header langfuse_trace_name: private-langfuse-only-name and gets HTTP 200
Langfuse shows a trace named private-langfuse-only-name with a chat proof-model generation, as before
The generic collector shows the request root, the auth span and one chat proof-model span, all with zero langfuse.* keys
Swapping the callback order changes nothing, and with callbacks: ["langfuse_otel"] alone nothing is printed to stdout; an operator who wants the collector as well adds otel to callbacks

Relevant issues

Follow-up to #40793 and #28909

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live no-mock audit of every OTel v2 callback (otel, langfuse_otel, arize, arize_phoenix, weave_otel, newrelic, agentops, levo, langtrace). Before is the merge base 27f8d2a9baa1f81b37243878a17f7e55733a64e9, After is this tip b565a8d155f1dc2b19d208295c39bce0cce9f87a. Each leg runs from its own worktree with its own Postgres database and a 2-worker proxy (Before on 21081, After on 21080), started as python litellm/proxy/proxy_cli.py --config cfg/<K>.yaml --port $PORT --num_workers 2 --use_v2_migration_resolver with LITELLM_OTEL_V2=true and LITELLM_DISABLE_NO_REDIS_WARNING=true. Providers are real OpenAI (openai/gpt-5.4-mini as proof-model) and real Anthropic (anthropic/claude-sonnet-4-6 as proof-claude). Destinations are a real Jaeger all-in-one as the generic OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:21118 collector (UI on 21186), real Langfuse cloud, real Weave, real Arize and New Relic OTLP ingest endpoints, and a real Arize Phoenix on 21106. Two destinations are disclosed OTLP/HTTP emulators because no account exists: Levo (21119) and the team-level Langfuse callback_vars host (21120). Every request is checked at every destination by its gen_ai.response.id, never by count. Full per-cell tables (146 happy cells, sad, edge and 190 chaos requests) live in the audit report attached to the linked Devin session

Config for the all-callbacks cases (cfg/K2.yaml); the other configurations change only callbacks and the env, as named in each step

model_list:
  - model_name: proof-model
    litellm_params:
      model: openai/gpt-5.4-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: proof-claude
    litellm_params:
      model: anthropic/claude-sonnet-4-6
      api_key: os.environ/ANTHROPIC_API_KEY
general_settings:
  master_key: sk-1234
  database_url: os.environ/DATABASE_URL
litellm_settings:
  callbacks: ["otel", "langfuse_otel", "arize", "arize_phoenix", "weave_otel", "newrelic", "levo", "langtrace"]

Request and readback shapes used by every step below. $K is the configuration, $LEG is base or head, every stream is consumed to the end before the readback. gen=N[...] is the number of spans the generic collector holds for that response id and the vocabulary on each, root is the vendor keys on the request root span at the collector

H=(-H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -H "langfuse_trace_name: litaudit-$K-$LEG")
curl -s -D - localhost:$PORT/v1/chat/completions "${H[@]}" \
  -d '{"model":"proof-model","messages":[{"role":"user","content":"Reply with the single word ok"}],"max_tokens":20,"stream":false}'
curl -s -D - localhost:$PORT/v1/messages "${H[@]}" \
  -d '{"model":"proof-claude","messages":[{"role":"user","content":"Reply with the single word ok"}],"max_tokens":20,"stream":true}'
curl -s -D - localhost:$PORT/v1/responses "${H[@]}" \
  -d '{"model":"proof-model","input":"Reply with the single word ok","max_output_tokens":20}'
curl -s "localhost:21186/api/traces?service=litaudit-$LEG&tags=%7B%22gen_ai.response.id%22%3A%22$RID%22%7D"   # generic collector
curl -s -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" "$LANGFUSE_HOST/api/public/observations?name=chat%20proof-model&limit=50"
curl -s localhost:21119/dump; curl -s localhost:21120/dump                                                  # Levo and team emulators
psql "$DATABASE_URL" -c "select request_id from \"LiteLLM_SpendLogs\" where request_id = '$RID'"

Before (`27f8d2a`)

/v1/chat/completions, all eight v2 callbacks

K=K2 chat curl, stream off: HTTP 200, chatcmpl-EO7dw6wP6ogaPNQnoJYXyLVOZ3KL3. Collector: gen=5[generic, langfuse, openinference, openinference, openinference+weave], root langfuse.trace.name. Langfuse 1 observation named litaudit-K2-base, Weave 4 duplicate calls, Phoenix 1, Levo 1, spend log 1
chat curl, stream on: HTTP 200, chatcmpl-EO7dwRvaOrVL0u2BGavbxx5tgjZRV. Collector: gen=5 with the same four vendor duplicates, root leaked
Same request from the dashboard leg (langfuse_trace_name: audit-ui-base): HTTP 200, chatcmpl-EO9HBH6On35YrZDOB0BsN8xh0EGXR, x-litellm-call-id: eaf90459-63ca-4e74-ae03-5e411d53fdf9, Jaeger trace 15b5d6a1ca7992b1af81875fb2003d91 with 9 spans, four chat proof-model, langfuse.observation.*, llm.* plus openinference.span.kind, weave.*, and langfuse.trace.name=audit-ui-base on the root

/v1/messages, all eight v2 callbacks

messages curl, stream off: HTTP 200, msg_011Cf3y8wDRXYyhQFMfktRXe. Collector gen=5[langfuse, generic, openinference, openinference, openinference+weave], root leaked
messages curl, stream on: HTTP 200, msg_011Cf3y91yuUY5mw2p6Bnsec. Collector gen=5, root leaked. Langfuse 1 named, Phoenix 1, Levo 1, spend log 1
Dashboard leg: HTTP 200, msg_011Cf477E73eTY9bqL4N5zqE, x-litellm-call-id: 32828cc4-27b3-4f6b-8023-54646d5095da

/v1/responses, all eight v2 callbacks

responses curl, stream off: HTTP 200, resp_iyx7bFqbWthEBEhZxSQzzFFmqWyYNUf1P-OZNZi6_Vw. Collector gen=5, root leaked. Langfuse 1 named, Phoenix 1, Levo 1, spend log 1
responses curl, stream on: HTTP 200, resp_FfmUL5RPf2GlJnYuRfTkF1gsvoRvQBhxd9poyOhZFAj. The streamed client id differs from the logged id and the collector was restarted (chaos C1) before a re-read, so this cell is NOT RUN at the collector; the LiteLLM SDK stream cells below cover streamed responses
Dashboard leg: HTTP 200, x-litellm-call-id: d92d2d21-9025-4c9a-b623-d37eba372b34

OpenAI, Anthropic and LiteLLM SDK clients through the proxy, all eight v2 callbacks

openai sync and async chat.completions.create and responses.create, stream on and off, anthropic sync and async messages.create, stream on and off, base_url=http://localhost:21081: 12 cells, every one HTTP 200, every collector readback gen=5 with vendor duplicates and root leaked
litellm.completion / acompletion / responses / aresponses with model="openai/proof-model", api_base=http://localhost:21081, stream on and off: 8 cells, all HTTP 200, e.g. chatcmpl-EO7rHY8MVgjDTruzGHzG3YnfTY2tQ gen=5[openinference, ...], resp_8tN4ZskAq-zr5s-QUvhc2lILlK0awQW-gGEZNVCqgNf (stream) gen=5[langfuse, ...]

Other callback configurations, one 2-worker boot each, three endpoints each

K1 ["otel"], OTEL_* set: chatcmpl-EO7uuIthLSg0LTjvyqn7GbGojK0yF gen=1[generic], root clean
K3 ["langfuse_otel", "arize", "arize_phoenix", "weave_otel", "newrelic", "levo"] without otel, OTEL_* set: chatcmpl-EO7wOeRQ8tHBaHypZupeVtsWWPTza gen=4[langfuse, openinference, openinference, openinference+weave], root leaked; the collector receives vendor spans it never asked for
K4 ["langfuse_otel", "otel"]: chatcmpl-EO7yGZ7sKjWKSJeHuSKmHwDR0AjBX gen=1[langfuse], root langfuse.trace.name; the otel entry reused the Langfuse logger, Langfuse 1 named litaudit-K4-base
K5 ["weave_otel", "otel"]: chatcmpl-EO7zugaamhNxAkpNlwH0p7Obyt4ht gen=1[openinference+weave], Weave 1
K6 ["langtrace", "otel"] plus PHOENIX_COLLECTOR_HTTP_ENDPOINT: chatcmpl-EO81KwwoiVUjFi5dx395bUKyIH4UB Phoenix 0 spans for all three ids, the auto-init is skipped behind the mapper-only callback
K7 ["otel"] plus PHOENIX_*: chatcmpl-EO82ig93HSNfkJRoExxVwYi4qBAXb Phoenix 3 spans
K8 ["langtrace"]: chatcmpl-EO843K7ztZpHbxpdq5crqlTaeSeze gen=1[langtrace]
K9 ["langfuse_otel"], no OTEL_*: chatcmpl-EO87R7XY5rdcsk4TMwR4TOvpd88hx Langfuse 1 named litaudit-K9-base, and grep -c '"resource":' proxy.log = 12, every span printed as JSON to stdout
K10 ["otel"], OTEL_EXPORTER=console: chatcmpl-EO892WX75MxXwpo1SQiTWs6R5KV1s, 12 span objects on stdout, spend log 1
K11 ["otel", "langfuse_otel"], LANGFUSE_* unset: chatcmpl-EO8ATy5uM1fxVJTEFDfHCMYO4cSn5 HTTP 200, gen=1[generic], Langfuse 0
K12 ["otel", "langfuse_otel"] with a team whose metadata.logging names langfuse_otel and callback_vars.langfuse_host=http://127.0.0.1:21120, keys on that team, on a team without settings, and with no team, then /team/update and a 65 s wait for the metadata cache: Team A chatcmpl-EO8GNH0tzGQZwsx8w2xFBUNW0PJeQ gen=2[generic, langfuse], root leaked on the collector and on the tenant root, team sink 1 generic span; after the update chatcmpl-EO8Ki1obnYCh5FDjO4DzCXnKJfhV0 gen=2[langfuse, generic], root leaked, team sink 1
K13 ["otel", "agentops"], no AgentOps key: chatcmpl-EO8C1PnJZqfD20XAAhHfvvUsraTYE gen=2[generic, generic]
K14 K2 plus callback_settings.otel.message_logging: false: chatcmpl-EO8OIHuLpMYwM4sXV5YOwxDBz8sgN gen=5 vendor duplicates as K2
K15 callbacks: ["nope"] then callbacks: null: each worker exits at boot, ValueError: Empty module name and AttributeError: 'NoneType' object has no attribute 'startswith'
K16 callbacks: [], OTEL_* set: chatcmpl-EO8Tx3v9frNFsCg1seRndO6Lp5qhk gen=1[generic] from the env-built provider

Sad paths, K2 configuration

"model":"no-such-model" on all three routes: HTTP 400 Invalid model name passed in model=no-such-model
proof-model pointed at a bad OPENAI_API_KEY: HTTP 401 AuthenticationError ... Incorrect API key provided, then HTTP 429 No deployments available ... cooldown once the router cooled the deployment down
"max_tokens": 10000000: HTTP 400 on chat and messages, HTTP 200 on responses (provider accepted it)
POST /key/health on a key with Langfuse callback_vars: HTTP 200; GET /health/liveliness: HTTP 200 after every failure above; a valid key on proof-model right after: HTTP 200
A real vendor OTLP endpoint answered 403 to export batches during K2, K3 and K14: callers unaffected, logged as exporter failures only

Edge paths, K2 configuration

Three identical chat requests in a row: each id landed exactly five spans on the collector (one generic plus four vendor duplicates), exactly one spend row
Three concurrent POST /team/update on Team A while 12 Team A requests ran: all HTTP 200, zero new tracebacks in either worker log

Chaos, K2 configuration, 30 to 40 concurrent mixed requests (chat, messages, responses, stream on and off)

C1 docker stop Jaeger for 20 s mid-burst, restart within the exporter retry window: 30/30 callers HTTP 200, Levo got every id once, collector served post-restore traffic. Exactly-once landing of the outage-window spans is not provable, the in-memory Jaeger dropped its store on stop
C2 docker pause Jaeger for 20 s mid-burst, unpause: 30/30 HTTP 200, /health/liveliness 200 every 5 s, after unpause every id landed with gen=5 leaked, spend log 30/30
C3 SIGTERM the proxy master 1.2 s into a 40-request burst, restart: 40/40 accepted requests finished 200, proxy back with 2 workers, spend log 40/40, exporter queue drained for 22/40 ids
C4 kill -KILL one worker 4 s into a 30-request burst: 29/30 HTTP 200 plus 1 connection reset, master respawned a worker, 25/30 spend rows at readback
C5 docker stop a dedicated Postgres for 27 s mid-burst (control proxy on 21083): 29 HTTP 200 plus 1 HTTP 429 (provider cooldown), liveliness 200 throughout, /health/readiness reported db: disconnected then recovered, collector 28/30 ids gen=5 leaked, 26/30 spend rows

Dashboard and destination UI

Open http://localhost:21081/ui, log in as admin with the master key, open Logs (/ui/?page=logs): the audited ids are listed
Click the chatcmpl-EO9HBH6On35YrZDOB0BsN8xh0EGXR row: detail page with request, response, tokens and cost
Open Playground, pick proof-model, send a message: the assistant reply renders
Open http://localhost:21106 and find the trace for the same id: Phoenix trace detail present