feat(otel): promote nested request metadata keys to litellm.metadata.* span attributes

[devin-ai-integration]

TLDR

Problem this solves:

• baggage_metadata_keys: [requester_metadata.trace_id] promoted nothing
• caller metadata.trace_id never reached the LLM-call span as its own attribute
• OTEL v1 had no allowlist for caller metadata at all

How it solves it:

• nested request metadata is flattened to dotted paths before the allowlist lookup
• an allowlisted requester_metadata.<path> lands as litellm.metadata.<path>; only the proxy's requester_metadata. wrapper is stripped, every other key keeps its full dotted name
• same allowlist and attribute name on OTEL v2 and the legacy OTEL v1 callback
• requester_metadata as a whole is still never promoted

User Flow

Before: a platform team wants to correlate their own trace id with the gateway's LLM span, but the span never carries it

1. The proxy admin sets callback_settings.otel.baggage_metadata_keys: [requester_metadata.trace_id] and restarts the proxy
2. The developer sends POST https://litellm-domain/v1/chat/completions with "metadata": {"trace_id": "abc"} in the body
3. The request returns 200 with the completion
4. In their tracing backend the request's LLM-call span (chat gpt-4o-mini on OTEL v2, litellm_request on OTEL v1) has no litellm.metadata.trace_id attribute, so they cannot join it to their own trace

After: the same request stamps the caller's trace id on the request's spans

1. The proxy admin sets callback_settings.otel.baggage_metadata_keys: [requester_metadata.trace_id] and restarts the proxy
2. The developer sends POST https://litellm-domain/v1/chat/completions with "metadata": {"trace_id": "abc"} in the body
3. The request returns 200 with the completion
4. In their tracing backend the same LLM-call span (and on OTEL v2 the server and service spans of the same request) carries litellm.metadata.trace_id=abc, while metadata.nested.deep and the rest of the caller's metadata stay out because they were not allowlisted. Allowlisting requester_metadata.nested.deep as well adds litellm.metadata.nested.deep=x

Relevant issues

Customer request via Pylon #8571

Affected release

Linear ticket

Resolves LIT-7760

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup. Four proxies, each started with --num_workers 2 against real OpenAI and a real PostgreSQL, all exporting over OTLP/HTTP to one local Jaeger (jaegertracing/all-in-one:1.62.0). The Before pair runs from the merge-base worktree, the After pair from the PR tip, on both OTEL v2 (LITELLM_OTEL_V2=true) and the legacy OTEL v1 callback. Each arm reports under its own service_name so the traces can be read back per arm from Jaeger's query API, which is what a user opening the Jaeger UI sees. The request bodies are byte-identical across every leg and every arm

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
litellm_settings:
  callbacks: ["otel"]
callback_settings:
  otel:
    exporter: otlp_http
    endpoint: http://127.0.0.1:47791
    service_name: lit7760-<arm>
    baggage_metadata_keys:
      - requester_metadata.trace_id
      - requester_metadata.nested.deep
general_settings:
  master_key: sk-lit7760

cat chat.json
{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Reply with exactly: pong"}], "max_tokens": 5, "metadata": {"trace_id": "abc", "nested": {"deep": "x"}, "empty": ""}}
cat responses.json
{"model": "gpt-4o-mini", "input": "Reply with exactly: pong", "max_output_tokens": 16, "metadata": {"trace_id": "abc", "nested": {"deep": "x"}, "empty": ""}}
cat messages.json
{"model": "gpt-4o-mini", "max_tokens": 5, "messages": [{"role": "user", "content": "Reply with exactly: pong"}], "metadata": {"trace_id": "abc", "nested": {"deep": "x"}, "empty": ""}}

Each case below runs the same three curls against the arm's port, three times each so both uvicorn workers serve requests, then reads the traces back. The v1 arms take the exporter from OTEL_EXPORTER=otlp_http OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:47791/v1/traces OTEL_SERVICE_NAME=lit7760-<arm> because the legacy callback reads exporter settings from the environment

for route in /v1/chat/completions /v1/responses /v1/messages; do
  curl -s http://127.0.0.1:$PORT$route -H "Authorization: Bearer sk-lit7760" -H "Content-Type: application/json" -d @$(basename $route).json
done
curl -s "http://127.0.0.1:47790/api/traces?service=lit7760-$ARM&limit=100&lookback=1h"

Before (4e99640)

/v1/chat/completions

1. PYTHONPATH="$PWD:$PWD/enterprise" python -c "import litellm; print(litellm.__file__)" -> /home/ubuntu/lit7760/base/litellm/__init__.py on both Before arms
2. curl -> 200 object=chat.completion model=gpt-4o-mini text='pong' on v1 (port 47780) and v2 (port 47782)
3. Jaeger, v2 service lit7760-base_v2, litellm.metadata.* tags per span of the three traces:

  3 route=/v1/chat/completions span='POST /v1/chat/completions' litellm.metadata.*={}
  3 route=/v1/chat/completions span='chat gpt-4o-mini' litellm.metadata.*={}

4. Jaeger, v1 service lit7760-base_v1:

  3 route=/v1/chat/completions span='Received Proxy Server Request' litellm.metadata.*={}
  3 route=/v1/chat/completions span='litellm_request' litellm.metadata.*={}

/v1/responses

1. curl -> 200 object=response model=gpt-4o-mini text='pong' on v1 and v2
2. Jaeger v2:

  3 route=/v1/responses span='POST /v1/responses' litellm.metadata.*={}
  3 route=/v1/responses span='chat gpt-4o-mini' litellm.metadata.*={}

3. Jaeger v1:

  3 route=/v1/responses span='Received Proxy Server Request' litellm.metadata.*={}
  3 route=/v1/responses span='litellm_request' litellm.metadata.*={}

/v1/messages

1. curl -> 200 object=message model=gpt-4o-mini text='pong' on v1 and v2
2. Jaeger v2:

  3 route=/v1/messages span='POST /v1/messages' litellm.metadata.*={}
  3 route=/v1/messages span='chat gpt-4o-mini' litellm.metadata.*={}

3. Jaeger v1:

  3 route=/v1/messages span='Received Proxy Server Request' litellm.metadata.*={}
  3 route=/v1/messages span='litellm_request' litellm.metadata.*={}

Every litellm.metadata.* tag Jaeger holds for the Before arms

1. lit7760-base_v1 -> {}
2. lit7760-base_v2 -> {'litellm.metadata.table_name': 7} (the proxy's own PostgreSQL spans, present on both arms and unrelated to caller metadata)

After (8de51df)

/v1/chat/completions

1. PYTHONPATH="$PWD:$PWD/enterprise" python -c "import litellm; print(litellm.__file__)" -> /home/ubuntu/repos/litellm/litellm/__init__.py on both After arms, git rev-parse --short HEAD -> 8de51dfaab
2. curl -> 200 object=chat.completion model=gpt-4o-mini text='pong' on v1 (port 47781) and v2 (port 47783)
3. Jaeger, v2 service lit7760-head_v2:

  3 route=/v1/chat/completions span='POST /v1/chat/completions' litellm.metadata.*={"litellm.metadata.nested.deep": "x", "litellm.metadata.trace_id": "abc"}
  3 route=/v1/chat/completions span='chat gpt-4o-mini' litellm.metadata.*={"litellm.metadata.nested.deep": "x", "litellm.metadata.trace_id": "abc"}

4. Jaeger, v1 service lit7760-head_v1:

  3 route=/v1/chat/completions span='Received Proxy Server Request' litellm.metadata.*={}
  3 route=/v1/chat/completions span='litellm_request' litellm.metadata.*={"litellm.metadata.nested.deep": "x", "litellm.metadata.trace_id": "abc"}

/v1/responses

1. curl -> 200 object=response model=gpt-4o-mini text='pong' on v1 and v2
2. Jaeger v2:

  3 route=/v1/responses span='POST /v1/responses' litellm.metadata.*={"litellm.metadata.nested.deep": "x", "litellm.metadata.trace_id": "abc"}
  3 route=/v1/responses span='chat gpt-4o-mini' litellm.metadata.*={"litellm.metadata.nested.deep": "x", "litellm.metadata.trace_id": "abc"}

3. Jaeger v1:

  3 route=/v1/responses span='Received Proxy Server Request' litellm.metadata.*={}
  3 route=/v1/responses span='litellm_request' litellm.metadata.*={"litellm.metadata.nested.deep": "x", "litellm.metadata.trace_id": "abc"}

/v1/messages

1. curl -> 200 object=message model=gpt-4o-mini text='pong' on v1 and v2
2. Jaeger v2:

  3 route=/v1/messages span='POST /v1/messages' litellm.metadata.*={"litellm.metadata.nested.deep": "x", "litellm.metadata.trace_id": "abc"}
  3 route=/v1/messages span='chat gpt-4o-mini' litellm.metadata.*={"litellm.metadata.nested.deep": "x", "litellm.metadata.trace_id": "abc"}

3. Jaeger v1:

  3 route=/v1/messages span='Received Proxy Server Request' litellm.metadata.*={}
  3 route=/v1/messages span='litellm_request' litellm.metadata.*={"litellm.metadata.nested.deep": "x", "litellm.metadata.trace_id": "abc"}

Every litellm.metadata.* tag Jaeger holds for the After arms

1. lit7760-head_v1 -> {'litellm.metadata.nested.deep': 9, 'litellm.metadata.trace_id': 9} (one litellm_request span per request, 9 requests)
2. lit7760-head_v2 -> {'litellm.metadata.nested.deep': 27, 'litellm.metadata.trace_id': 27, 'litellm.metadata.table_name': 20} (server span, LLM span and the cost-tracking DB span of each request; table_name is the same pre-existing PostgreSQL tag as on Before)
3. No trace on either After arm carries litellm.metadata.requester_metadata, litellm.metadata.deep, litellm.metadata.empty or litellm.metadata.nested.skipped: the wrapper is not promoted as a blob, the leaf keeps its dotted parent, the empty string is dropped and unlisted siblings stay out

Default allowlist unchanged (no baggage_metadata_keys, console exporter, request with "user": "enduser-7760")

1. Captured at 30f02aa6da; the only commit after it, 8de51dfaab, changes a test docstring and no runtime code
2. v2 Before and After both stamp exactly {'litellm.metadata.user_api_key_user_id': 'default_user_id', 'litellm.metadata.user_api_key_end_user_id': 'enduser-7760'} on the server, LLM and DB spans, and nothing from the caller's metadata
3. v1 Before and After both stamp the same metadata.user_api_key_* blob attributes on litellm_request and no litellm.metadata.* key

Type

🆕 New Feature

Caveats (if any)

Medium

• No admin UI control: the OTEL callback form only exposes endpoint, headers and protocol, and has no field for any baggage_*_keys setting today, so this key (like the existing baggage_promoted_keys and baggage_team_metadata_keys) is config.yaml and env var only. Adding a UI field for the whole baggage_* family is a separate PR
• Docs for baggage_metadata_keys dotted paths and the new LITELLM_OTEL_BAGGAGE_METADATA_KEYS env var on v1 belong in litellm-docs and are not part of this PR

Low

• Attribute names are stable for existing configs: a flat dotted key such as a.b still lands as litellm.metadata.a.b, only the requester_metadata. wrapper is dropped. The one collision left is a caller key and a proxy key with the same name (requester_metadata.trace_id and a top-level trace_id); the later allowlist entry wins
• Empty strings are dropped like every other promoted baggage value; 0 and False are stringified and kept
• Only string, bool, int and float leaves are promoted. Lists and empty mappings are not flattened
• The nested walk is an explicit worklist rather than recursion, so the repo's recursive-function check stays clean and a deeply nested payload cannot grow the call stack
• On v2 the auth-time seed still runs without request metadata (the body is not parsed there); the pre-call hook reseeds with it, which is why the server span in the proof carries the value too
• On v1 the promoted keys land on the litellm_request span only, where every other litellm.* attribute of the legacy callback already lives; the v1 server span carries no per-request metadata before or after this PR
• OTEL v1 subclasses that override set_attributes (Arize, Arize Phoenix) never reach the legacy stamping path, so baggage_metadata_keys is inert there; the other v1 subclasses inherit it and are unchanged while the key is unset

Taxonomy audit of the diff. F3: v1 and v2 both covered, sibling endpoints share the same metadata snapshot (see Low above). C5: precedence unchanged, auth-derived user_api_key_* keys still win over request metadata of the same name, and the v1 config-then-env order mirrors baggage_team_metadata_keys. W1: flatten_metadata, metadata_from_request_data and promoted_metadata only read and return new read-only mappings; the pre-call hook test asserts data is unchanged. X1: is not None for the request metadata, empty mapping handled explicitly, 0 covered by a test. Y4: metadata vs litellm_metadata picks the one carrying requester_metadata. H2/H4: no new comments, no Optional, no bare dict, no Any, lines at 120. T1-T5: the new tests fail on the merge base (KeyError: 'litellm.metadata.trace_id', unexpected keyword argument 'baggage_metadata_keys') and pass with the fix; env vars are patched with patch.dict. B, D, E, G, M, N, O, P, R, V, Z: not applicable, no cache, auth, spend, streaming, guardrail, provider or migration code touched

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

ran /live-pr-risk and found no regressions/backward incompatible risks

Link to Devin session: https://app.devin.ai/sessions/c3ea5e13360e4d5391e80461d27f551f
Open in Devin Desktop: https://app.devin.ai/desktop/session/c3ea5e13360e4d5391e80461d27f551f?variant=devin
Requested by: @yassin-berriai

---

Note

Low Risk
Observability-only allowlist changes; default configs unchanged and unlisted metadata is not exported, with minor attribute-name collision risk if allowlists overlap.

Overview
Enables allowlisted caller metadata (e.g. requester_metadata.trace_id from request metadata.trace_id) to appear on traces as litellm.metadata.* span attributes on both OTEL v2 and legacy v1.

Nested metadata is flattened to dotted paths before lookup; allowlisted requester_metadata.<path> keys are promoted with the requester_metadata. wrapper stripped (other dotted keys keep their full path). The full requester_metadata blob is never promoted—only explicitly listed leaves.

OTEL v2 seeds baggage in the pre-call hook using a requester_metadata-only snapshot from the request body (proxy-owned metadata siblings are not read from there). Legacy v1 adds baggage_metadata_keys / LITELLM_OTEL_BAGGAGE_METADATA_KEYS and stamps the same promoted attributes on inference spans when configured (empty by default, unlike v2’s default auth keys).

^{Reviewed by Cursor Bugbot for commit 8de51df. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_otel_promote_nested_request_metadata_keys (8de51df) with main (b04d530)¹}

Footnotes

1. No successful run was found on main (734038a) during the generation of this report, so b04d530 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[greptile-apps]

Greptile Summary

This PR allows explicitly configured nested caller metadata to be promoted into OpenTelemetry span attributes while preserving dotted paths and excluding the enclosing metadata blob

• Flattens scalar metadata leaves into dotted paths
• Applies the same allowlist behavior to OTEL v1 and v2
• Restricts v2 request extraction to the caller-owned requester_metadata snapshot
• Adds regression coverage for nested paths, excluded siblings, empty values, configuration sources, and span propagation

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness or repository-rule findings

The metadata promotion remains explicitly allowlisted, preserves dotted paths, excludes the requester wrapper, and is covered across both OTEL implementations. The earlier collision and unnecessary-docstring threads were manually resolved without explanation. The immutable merge finding was fixed as claimed by devin-ai-integration[bot] and its thread was resolved

Important Files Changed

Filename	Overview
litellm/integrations/otel/model/baggage.py	Promotes allowlisted flattened metadata with stable dotted attribute names and immutable result construction
litellm/integrations/otel/model/metadata.py	Extracts caller-owned request metadata and iteratively flattens supported scalar leaves
litellm/integrations/otel/logger.py	Seeds OTEL v2 request baggage with the extracted caller metadata snapshot
litellm/integrations/opentelemetry.py	Adds OTEL v1 configuration and span stamping for allowlisted metadata
litellm/integrations/otel/model/config.py	Clarifies configuration behavior for nested requester metadata paths
tests/test_litellm/integrations/otel/test_otel_v2_baggage.py	Covers nested promotion, path preservation, scalar handling, and exclusion behavior
tests/test_litellm/integrations/otel/test_otel_v2_logger.py	Covers propagation to server, LLM-call, and service spans without mutating request data
tests/test_litellm/integrations/test_opentelemetry.py	Covers legacy OTEL promotion, defaults, environment configuration, and constructor overrides

_{Reviews (3): Last reviewed commit: "test(otel): describe which request metad..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 97.56098% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/otel/model/metadata.py	95.23%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8de51df. Configure here.}