feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2 by ryan-crabbe-berri · Pull Request #31525 · BerriAI/litellm

ryan-crabbe-berri · 2026-06-27T20:30:56Z

Relevant issues

Linear ticket

Resolve LIT-3810

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Driven against a live proxy on otel_v2 exporting to a local Jaeger, using the in-repo stdio MCP server (tests/mcp_tests/mcp_server.py); no auth, no LLM cost

Proxy:

LITELLM_OTEL_V2=true OTEL_EXPORTER=otlp_http \
  OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 OTEL_SERVICE_NAME=litellm-lit3810 \
  python litellm/proxy/proxy_cli.py --config config.yaml --port 4111 --detailed_debug

A plain MCP session (initialize, then tools/list) now produces a dedicated tools/list CLIENT span. Before this change the discovery call emitted no MCP span at all; it surfaced only as a bare POST /{mcp_server_name}/mcp server span:

$ curl -s 'http://localhost:16686/api/operations?service=litellm-lit3810'
... {"name":"tools/list","spanKind":"client"} ...

The span is its own correctly-timed root trace and records the transport/session span as a link rather than nesting under it:

tools/list trace 2b204d2f...   spans: 1   total duration: 0.42s
  tools/list  [client]   FOLLOWS_FROM -> trace 5056876d... (the initialize/session transport span)
  is own root (no parent): true

Jaeger renders an OTel span link as a FOLLOWS_FROM reference, so the transport/session span shows up as a link, not the parent. That anchoring is also what caused the original report: because every message nested under the initialize anchor, a tools/list run seconds later (44s in that report) rendered that far to the right of its parent with a clock-skew warning, instead of the independent trace above

When a client propagates W3C trace context in params._meta, the span joins that client trace as a child instead of starting its own root. That propagation path and the identity handling (changes 3 and 4: a spoofed params._meta.baggage never lands as a span attribute, and the authenticated team does) are covered by unit tests

Type

🆕 New Feature
🐛 Bug Fix

Changes

Under otel_v2 an MCP tools/call produced its own CLIENT span, but tools/list produced none; a discovery call surfaced only as a bare POST /{mcp_server_name}/mcp server span with no MCP attributes, indistinguishable from initialize. This PR gives tools/list its own span and, along the way, fixes how every MCP span attaches to a trace and where its identity attributes come from

1. Emit a tools/list span. The v2 logger already received the list event, but _emit_mcp_tool_call only matched call_mcp_tool, so list_mcp_tools fell through to the LLM-call path and emitted nothing. A dedicated MCP_LIST_TOOLS span role (with its own typed MCPListToolsSpanData) fixes that. Per the OTel GenAI MCP semconv the span is named tools/list, is CLIENT kind, and carries mcp.method.name; it omits gen_ai.operation.name and gen_ai.tool.name, which the spec reserves for tool executions

2. Attach MCP spans to the caller's trace, not the HTTP session. Streamable-HTTP multiplexes many JSON-RPC messages over one session, so the request-root anchor captured on initialize persisted and every later message nested under it; a tools/list run a few seconds later rendered that far to the right of its parent with a clock-skew warning, because the MCP message and the HTTP transport are independent lifecycles. Per the MCP context-propagation rules, each MCP span now parents to the W3C trace context the client propagates in params._meta (SEP-414), records the transport/session span as a span link rather than the parent, and starts its own root trace when nothing was propagated. This applies to tools/call too, which shared the bug. This is the OTel GenAI MCP semconv topology: MCP and transport contexts are independent, so the transport span is recorded as a span link, not a parent, and every MCP operation is its own trace linked to the HTTP/session span, uniformly across the stateful and stateless paths. Heads-up for reviewers: clients that don't propagate context (the common case today) get their tools/call/tools/list spans in their own trace linked to the transport rather than nested under the HTTP request; that is the intended semconv topology, not a regression

3. Stop trusting identity from the client. params._meta is caller-controlled and also carries W3C baggage, and LiteLLMBaggageSpanProcessor stamps allowlisted baggage keys (litellm.team.id, litellm.metadata.*) onto every span. A client could send params._meta.baggage: litellm.team.id=...,litellm.metadata.user_api_key_user_id=... and have those identity attributes attributed to its own spans, poisoning team/user dashboards and any metric derived from span attributes. The propagator now extracts trace context only (traceparent/tracestate), and the gateway stops collecting the baggage key at the source

4. Stamp the authenticated identity instead. Dropping client baggage (change 3) left the tool-call and tools/list spans with no team/key/metadata, so they couldn't be attributed or filtered by team. A shared _seed_identity_baggage helper now seeds identity from the parsed, authenticated StandardLoggingPayload, the same source the LLM-call span already used, so attribution is restored without reopening the spoofing vector. Regression tests assert the authenticated team lands on both MCP spans and that a spoofed params._meta.baggage value can't override it

greptile-apps · 2026-06-27T20:34:11Z

Greptile Summary

This PR adds otel_v2 spans for MCP tool discovery. The main changes are:

Dedicated tools/list CLIENT spans for MCP discovery calls
MCP span parenting from propagated params._meta trace context, with the HTTP transport recorded as a span link
Client-supplied baggage excluded from MCP trace extraction
Authenticated LiteLLM identity seeded onto MCP spans from the standard logging payload
Tests for span emission, trace topology, baggage handling, identity attribution, and malformed trace context

Confidence Score: 5/5

The MCP OpenTelemetry changes are well-scoped and covered by focused regression tests for span emission, trace topology, baggage filtering, authenticated identity attribution, and malformed context handling.

The changed code paths are isolated to otel_v2 MCP logging and tracing behavior, with tests covering both the new discovery span and the related context/identity edge cases.

T-Rex Logs

What T-Rex did

Observed the pre-change baseline and post-change for the tools-list-span flow to verify that a new Tools List span was emitted as a CLIENT span with the expected attributes after the change.
Validated the client-span wiring for the get_weather and tools/list flows, including behavior with and without meta, and confirmed the presence of transport links in the appropriate contexts.
Validated MCP identity baggage behavior across runs, confirming authenticated identity and no spoofed baggage, with two MCP spans exported and StatusCode.UNSET as expected.

_{Ran code and verified through T-Rex}

_{Reviews (3): Last reviewed commit: "refactor(otel): model MCP spans as roots..." | Re-trigger Greptile}

codecov · 2026-06-27T20:35:11Z

Codecov Report

❌ Patch coverage is 87.35632% with 11 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/_experimental/mcp_server/server.py	70.37%	8 Missing ⚠️
litellm/integrations/otel/logger.py	91.66%	2 Missing ⚠️
litellm/integrations/otel/model/spans.py	83.33%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…l_v2 Under otel_v2 an MCP tools/call already produced a dedicated CLIENT span, but tools/list produced none. The discovery call surfaced only as the bare POST /{mcp_server_name}/mcp server span with no MCP attributes, indistinguishable from initialize and impossible to query by method The list success event already reaches the v2 logger with call_type list_mcp_tools, but _emit_mcp_tool_call only matched call_mcp_tool, so listing fell through to the LLM-call path and emitted nothing. This adds a dedicated MCP_LIST_TOOLS span role with its own MCPListToolsSpanData, emitted from a sibling _emit_mcp_list_tools branch that mirrors the tools/call path Per the OTel GenAI MCP semantic conventions the span is named tools/list (the method name alone, since there is no low-cardinality target), is a CLIENT span parented to the request span, and carries mcp.method.name plus the call id. It deliberately omits gen_ai.operation.name and gen_ai.tool.name, which the convention reserves for tool executions, since listing runs no tool

veria-ai · 2026-06-27T22:49:30Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

…ansport span MCP streamable-HTTP multiplexes many JSON-RPC messages over one session, so the request-root anchor captured on initialize persisted and every later message's span (tools/call, tools/list) nested under it. A tools/list run 44s after the initialize rendered 44s to the right of its parent with a clock-skew warning, because the MCP message and the HTTP transport are independent lifecycles Following the OTel GenAI MCP semantic conventions, an MCP span now parents to the W3C trace context the client propagated in the request's params._meta (a remote parent, per SEP-414), records the transport/session span as a span link rather than the parent, and starts its own root trace when nothing was propagated. The MCP gateway captures traceparent/tracestate/baggage from each message's params._meta into a per-message contextvar that the otel_v2 emitter reads; opentelemetry stays an optional dependency via guarded lazy imports This applies to tools/call as well as the new tools/list span, since both shared the same transport-anchoring bug

…ity spoofing The MCP trace propagation added a W3CBaggagePropagator, so resolve_mcp_span_context extracted the client's W3C Baggage from params._meta into the span's parent context. The LiteLLMBaggageSpanProcessor then stamps allowlisted baggage keys onto the span, and the list-tools/tool-call mappers don't set those identity keys, so nothing overwrites them. A malicious MCP client could send params._meta.baggage: litellm.team.id=...,litellm.metadata.user_api_key_user_id=... and have those identity attributes attributed to its spans. Extract trace context only (traceparent/tracestate) in the propagator, and stop collecting the baggage key at the source in _mcp_meta_trace_carrier. Parenting to the client's trace context, the actual goal, needs only trace context; remote baggage had no legitimate consumer here. Regression tests at both layers assert a spoofed params._meta.baggage never lands as a span identity attribute.

…pers The otel MCP trace-carrier helpers added in this branch pushed the BLE001 and UP006 strict-rule totals past their ceilings. Use PEP 585 `dict[str, str]` instead of `Dict`, and narrow the optional-import guards to `except ImportError` (the only failure these can hit, matching the "when otel_v2 is unavailable" intent) instead of a blind `except Exception`.

…itellm_lit_3810_mcp_otel_toollist

Parenting MCP spans to the client's params._meta trace context over an empty Context() meant the tool-call and tools/list spans carried no team/key/metadata identity at all, so they couldn't be attributed or filtered by team in a traces backend. The LLM-call span already re-seeds identity from the parsed, authenticated StandardLoggingPayload rather than trusting ambient/remote context; extract that into a shared _seed_identity_baggage helper and run both MCP emitters through it. Identity comes only from the authenticated payload, never the client carrier, so this keeps the earlier spoofing fix intact while restoring attribution. Regression tests assert the authenticated team lands on both MCP spans and that a spoofed params._meta.baggage value can't override it.

ryan-crabbe-berri · 2026-06-29T19:17:41Z

@greptileai re review

…PAN_REGISTRY

ryan-crabbe-berri · 2026-06-29T23:13:27Z

@greptileai re review

…ned_destinations Resolve conflicts from staging advancing with the MCP tools/list span work (#31525) and the key permissions admin-gate (#31810): - otel logger.py: keep the PR's multi-span (carrier.spans / emit_fanout) fan-out, adopt staging's _seed_identity_baggage helper in the deferred path - context.py: union ContextVar+Token and TYPE_CHECKING+Mapping imports - key_management_endpoints.py: hoist the regenerate team lookup to the top so both staging's object_permission gate and the PR's logging_exporters gate see it - tests: union the new imports/mocks and keep both sides' added tests

…l_v2 (BerriAI#31525) * feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2 Under otel_v2 an MCP tools/call already produced a dedicated CLIENT span, but tools/list produced none. The discovery call surfaced only as the bare POST /{mcp_server_name}/mcp server span with no MCP attributes, indistinguishable from initialize and impossible to query by method The list success event already reaches the v2 logger with call_type list_mcp_tools, but _emit_mcp_tool_call only matched call_mcp_tool, so listing fell through to the LLM-call path and emitted nothing. This adds a dedicated MCP_LIST_TOOLS span role with its own MCPListToolsSpanData, emitted from a sibling _emit_mcp_list_tools branch that mirrors the tools/call path Per the OTel GenAI MCP semantic conventions the span is named tools/list (the method name alone, since there is no low-cardinality target), is a CLIENT span parented to the request span, and carries mcp.method.name plus the call id. It deliberately omits gen_ai.operation.name and gen_ai.tool.name, which the convention reserves for tool executions, since listing runs no tool * fix(otel): anchor MCP spans to params._meta trace context, not the transport span MCP streamable-HTTP multiplexes many JSON-RPC messages over one session, so the request-root anchor captured on initialize persisted and every later message's span (tools/call, tools/list) nested under it. A tools/list run 44s after the initialize rendered 44s to the right of its parent with a clock-skew warning, because the MCP message and the HTTP transport are independent lifecycles Following the OTel GenAI MCP semantic conventions, an MCP span now parents to the W3C trace context the client propagated in the request's params._meta (a remote parent, per SEP-414), records the transport/session span as a span link rather than the parent, and starts its own root trace when nothing was propagated. The MCP gateway captures traceparent/tracestate/baggage from each message's params._meta into a per-message contextvar that the otel_v2 emitter reads; opentelemetry stays an optional dependency via guarded lazy imports This applies to tools/call as well as the new tools/list span, since both shared the same transport-anchoring bug * fix(otel): drop client baggage from MCP params._meta to prevent identity spoofing The MCP trace propagation added a W3CBaggagePropagator, so resolve_mcp_span_context extracted the client's W3C Baggage from params._meta into the span's parent context. The LiteLLMBaggageSpanProcessor then stamps allowlisted baggage keys onto the span, and the list-tools/tool-call mappers don't set those identity keys, so nothing overwrites them. A malicious MCP client could send params._meta.baggage: litellm.team.id=...,litellm.metadata.user_api_key_user_id=... and have those identity attributes attributed to its spans. Extract trace context only (traceparent/tracestate) in the propagator, and stop collecting the baggage key at the source in _mcp_meta_trace_carrier. Parenting to the client's trace context, the actual goal, needs only trace context; remote baggage had no legitimate consumer here. Regression tests at both layers assert a spoofed params._meta.baggage never lands as a span identity attribute. * style(mcp): clear ruff strict-budget breach in otel trace-carrier helpers The otel MCP trace-carrier helpers added in this branch pushed the BLE001 and UP006 strict-rule totals past their ceilings. Use PEP 585 `dict[str, str]` instead of `Dict`, and narrow the optional-import guards to `except ImportError` (the only failure these can hit, matching the "when otel_v2 is unavailable" intent) instead of a blind `except Exception`. * fix(otel): stamp authenticated identity baggage onto MCP spans Parenting MCP spans to the client's params._meta trace context over an empty Context() meant the tool-call and tools/list spans carried no team/key/metadata identity at all, so they couldn't be attributed or filtered by team in a traces backend. The LLM-call span already re-seeds identity from the parsed, authenticated StandardLoggingPayload rather than trusting ambient/remote context; extract that into a shared _seed_identity_baggage helper and run both MCP emitters through it. Identity comes only from the authenticated payload, never the client carrier, so this keeps the earlier spoofing fix intact while restoring attribution. Regression tests assert the authenticated team lands on both MCP spans and that a spoofed params._meta.baggage value can't override it. * refactor(otel): model MCP spans as roots that link the transport in SPAN_REGISTRY

ryan-crabbe-berri force-pushed the litellm_lit_3810_mcp_otel_toollist branch 2 times, most recently from d3b81a7 to 36784ca Compare June 27, 2026 22:43

veria-ai Bot reviewed Jun 27, 2026

View reviewed changes

Comment thread litellm/integrations/otel/plumbing/context.py Outdated

ryan-crabbe-berri force-pushed the litellm_lit_3810_mcp_otel_toollist branch from 36784ca to c226986 Compare June 27, 2026 22:54

veria-ai Bot reviewed Jun 28, 2026

View reviewed changes

Comment thread litellm/integrations/otel/plumbing/context.py

ryan-crabbe-berri force-pushed the litellm_lit_3810_mcp_otel_toollist branch from 8c6c2dd to 4ba406b Compare June 28, 2026 00:45

ryan-crabbe-berri added 2 commits June 27, 2026 19:25

Merge remote-tracking branch 'origin/litellm_internal_staging' into l…

896d598

…itellm_lit_3810_mcp_otel_toollist

refactor(otel): model MCP spans as roots that link the transport in S…

fae7f5f

…PAN_REGISTRY

yassin-berriai approved these changes Jun 30, 2026

View reviewed changes

ryan-crabbe-berri merged commit 468d11f into litellm_internal_staging Jun 30, 2026
126 checks passed

ryan-crabbe-berri deleted the litellm_lit_3810_mcp_otel_toollist branch June 30, 2026 17:27

Navigation Menu

Uh oh!

Uh oh!

feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2 - #31525

feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2#31525
ryan-crabbe-berri merged 7 commits into
litellm_internal_stagingfrom
litellm_lit_3810_mcp_otel_toollist

ryan-crabbe-berri commented Jun 27, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jun 27, 2026 •

edited

Loading

T-Rex Logs

Uh oh!

codecov Bot commented Jun 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

veria-ai Bot commented Jun 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

ryan-crabbe-berri commented Jun 29, 2026

Uh oh!

ryan-crabbe-berri commented Jun 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

ryan-crabbe-berri commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Type

Changes

Uh oh!

greptile-apps Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

T-Rex Logs

Uh oh!

codecov Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

veria-ai Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Security review

Uh oh!

Uh oh!

ryan-crabbe-berri commented Jun 29, 2026

Uh oh!

ryan-crabbe-berri commented Jun 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ryan-crabbe-berri commented Jun 27, 2026 •

edited

Loading

greptile-apps Bot commented Jun 27, 2026 •

edited

Loading

codecov Bot commented Jun 27, 2026 •

edited

Loading

veria-ai Bot commented Jun 27, 2026 •

edited

Loading