-
Notifications
You must be signed in to change notification settings - Fork 2k
Add OpenTelemetry tracing support #2869
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
046f845
Add OpenTelemetry tracing support
chrisguidry 0d513d7
Fix test_multi_client_with_logging to filter OpenTelemetry logs
chrisguidry 5a94c88
Update telemetry docs for accuracy
chrisguidry 88182d2
Add otel-desktop-viewer to telemetry docs
chrisguidry ad660c0
Address Jeremiah's telemetry feedback
chrisguidry 473b0e3
Address review feedback on telemetry docs
chrisguidry 33f952f
Address CodeRabbit suggestions
chrisguidry 07b056b
Address CodeRabbit feedback
chrisguidry e19cc17
Add tests for session ID capture on telemetry spans
chrisguidry b94b7e9
Update loq limits for telemetry additions
chrisguidry File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,264 @@ | ||
| --- | ||
| title: OpenTelemetry | ||
| sidebarTitle: Telemetry | ||
| description: Native OpenTelemetry instrumentation for distributed tracing. | ||
| icon: chart-line | ||
| --- | ||
|
|
||
| FastMCP includes native OpenTelemetry instrumentation for observability. Traces are automatically generated for tool, prompt, resource, and resource template operations, providing visibility into server behavior, request handling, and provider delegation chains. | ||
|
|
||
| ## How It Works | ||
|
|
||
| FastMCP uses the OpenTelemetry API for instrumentation. This means: | ||
|
|
||
| - **Zero configuration required** - Instrumentation is always active | ||
| - **No overhead when unused** - Without an SDK, all operations are no-ops | ||
| - **Bring your own SDK** - You control collection, export, and sampling | ||
| - **Works with any OTEL backend** - Jaeger, Zipkin, Datadog, New Relic, etc. | ||
|
|
||
| ## Enabling Telemetry | ||
|
|
||
| The easiest way to export traces is using `opentelemetry-instrument`, which configures the SDK automatically: | ||
|
|
||
| ```bash | ||
| pip install opentelemetry-distro opentelemetry-exporter-otlp | ||
| opentelemetry-bootstrap -a install | ||
| ``` | ||
|
|
||
| Then run your server with tracing enabled: | ||
|
|
||
| ```bash | ||
| opentelemetry-instrument \ | ||
| --service_name my-fastmcp-server \ | ||
| --exporter_otlp_endpoint http://localhost:4317 \ | ||
| fastmcp run server.py | ||
| ``` | ||
|
|
||
| Or configure via environment variables: | ||
|
|
||
| ```bash | ||
| export OTEL_SERVICE_NAME=my-fastmcp-server | ||
| export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 | ||
|
|
||
| opentelemetry-instrument fastmcp run server.py | ||
| ``` | ||
|
|
||
| This works with any OTLP-compatible backend (Jaeger, Zipkin, Grafana Tempo, Datadog, etc.) and requires no changes to your FastMCP code. | ||
|
|
||
| <Card title="OpenTelemetry Python Documentation" icon="book" href="https://opentelemetry.io/docs/languages/python/"> | ||
| Learn more about the OpenTelemetry Python SDK, auto-instrumentation, and available exporters. | ||
| </Card> | ||
|
|
||
| ## Tracing | ||
|
|
||
| FastMCP creates spans for all MCP operations, providing end-to-end visibility into request handling. | ||
|
|
||
| ### Server Spans | ||
|
|
||
| The server creates spans for each operation: | ||
|
|
||
| | Span Name | Description | | ||
| |-----------|-------------| | ||
| | `tool {name}` | Tool execution (e.g., `tool get_weather`) | | ||
| | `resource {uri}` | Resource read (e.g., `resource config://database`) | | ||
| | `prompt {name}` | Prompt render (e.g., `prompt greeting`) | | ||
|
|
||
| For mounted servers, an additional `delegate {name}` span shows the delegation to the child server. | ||
|
|
||
| ### Client Spans | ||
|
|
||
| The FastMCP client creates spans for outgoing requests with the same naming pattern (`tool {name}`, `resource {uri}`, `prompt {name}`). | ||
|
|
||
| ### Span Hierarchy | ||
|
|
||
| Spans form a hierarchy showing the request flow. For mounted servers: | ||
|
|
||
| ``` | ||
| tool weather_forecast (CLIENT) | ||
| └── tool weather_forecast (SERVER, provider=FastMCPProvider) | ||
| └── delegate get_weather (INTERNAL) | ||
| └── tool get_weather (SERVER, provider=LocalProvider) | ||
| ``` | ||
|
|
||
| For proxy providers connecting to remote servers: | ||
|
|
||
| ``` | ||
| tool remote_search (CLIENT) | ||
| └── tool remote_search (SERVER, provider=ProxyProvider) | ||
| └── [remote server spans via trace context propagation] | ||
| ``` | ||
|
|
||
| ## Programmatic Configuration | ||
|
|
||
| For more control, configure the SDK in your Python code before importing FastMCP: | ||
|
|
||
| ```python | ||
| from opentelemetry import trace | ||
| from opentelemetry.sdk.trace import TracerProvider | ||
| from opentelemetry.sdk.trace.export import BatchSpanProcessor | ||
| from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter | ||
|
|
||
| # Configure the SDK with OTLP exporter | ||
| provider = TracerProvider() | ||
| processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317")) | ||
| provider.add_span_processor(processor) | ||
| trace.set_tracer_provider(provider) | ||
|
|
||
| # Now import and use FastMCP - traces will be exported automatically | ||
| from fastmcp import FastMCP | ||
|
|
||
| mcp = FastMCP("my-server") | ||
|
|
||
| @mcp.tool() | ||
| def greet(name: str) -> str: | ||
| return f"Hello, {name}!" | ||
| ``` | ||
|
|
||
| <Tip> | ||
| The SDK must be configured **before** importing FastMCP to ensure the tracer provider is set when FastMCP initializes. | ||
| </Tip> | ||
|
|
||
| ### Local Development | ||
|
|
||
| For quick local trace visualization, [otel-desktop-viewer](https://github.com/CtrlSpice/otel-desktop-viewer) is a lightweight single-binary tool: | ||
|
|
||
| ```bash | ||
| # macOS | ||
| brew install nico-barbas/brew/otel-desktop-viewer | ||
|
|
||
| # Or download from GitHub releases | ||
| ``` | ||
|
|
||
| Run it alongside your server: | ||
|
|
||
| ```bash | ||
| # Terminal 1: Start the viewer (UI at http://localhost:8000, OTLP on :4317) | ||
| otel-desktop-viewer | ||
|
|
||
| # Terminal 2: Run your server with tracing | ||
| opentelemetry-instrument fastmcp run server.py | ||
| ``` | ||
|
|
||
| For more features, use [Jaeger](https://www.jaegertracing.io/): | ||
|
|
||
| ```bash | ||
| docker run -d --name jaeger \ | ||
| -p 16686:16686 \ | ||
| -p 4317:4317 \ | ||
| jaegertracing/all-in-one:latest | ||
| ``` | ||
|
|
||
| Then view traces at http://localhost:16686 | ||
|
|
||
| ## Custom Spans | ||
|
|
||
| You can add your own spans using the FastMCP tracer: | ||
|
|
||
| ```python | ||
| from fastmcp import FastMCP | ||
| from fastmcp.telemetry import get_tracer | ||
|
|
||
| mcp = FastMCP("custom-spans") | ||
|
|
||
| @mcp.tool() | ||
| async def complex_operation(input: str) -> str: | ||
| tracer = get_tracer() | ||
|
|
||
| with tracer.start_as_current_span("parse_input") as span: | ||
| span.set_attribute("input.length", len(input)) | ||
| parsed = parse(input) | ||
|
|
||
| with tracer.start_as_current_span("process_data") as span: | ||
| span.set_attribute("data.count", len(parsed)) | ||
| result = process(parsed) | ||
|
|
||
| return result | ||
| ``` | ||
|
|
||
| ## Error Handling | ||
|
|
||
| When errors occur, spans are automatically marked with error status and the exception is recorded: | ||
|
|
||
| ```python | ||
| @mcp.tool() | ||
| def risky_operation() -> str: | ||
| raise ValueError("Something went wrong") | ||
|
|
||
| # The span will have: | ||
| # - status = ERROR | ||
| # - exception event with stack trace | ||
| ``` | ||
|
|
||
| ## Attributes Reference | ||
|
|
||
| ### Standard Semantic Conventions | ||
|
|
||
| FastMCP uses OpenTelemetry semantic conventions where applicable: | ||
|
|
||
| | Attribute | Value | | ||
| |-----------|-------| | ||
| | `rpc.system` | `"mcp"` | | ||
| | `rpc.service` | Server name | | ||
| | `rpc.method` | MCP protocol method | | ||
| | `error.type` | Exception class name (on errors) | | ||
| | `enduser.id` | Client ID from access token (when authenticated) | | ||
| | `enduser.scope` | Space-separated OAuth scopes (when authenticated) | | ||
|
|
||
| ### FastMCP Custom Attributes | ||
|
|
||
| All custom attributes use the `fastmcp.` prefix: | ||
|
|
||
| | Attribute | Description | | ||
| |-----------|-------------| | ||
| | `fastmcp.server.name` | Server name | | ||
| | `fastmcp.component.type` | `tool`, `resource`, `prompt`, or `resource_template` | | ||
| | `fastmcp.component.key` | Full component identifier (e.g., `tool:greet`) | | ||
| | `fastmcp.provider.type` | Provider class (`LocalProvider`, `FastMCPProvider`, `ProxyProvider`) | | ||
| | `fastmcp.session.id` | Client session identifier | | ||
|
|
||
| Provider-specific attributes for delegation context: | ||
|
|
||
| | Attribute | Description | | ||
| |-----------|-------------| | ||
| | `fastmcp.delegate.original_name` | Original tool/prompt name before namespacing | | ||
| | `fastmcp.delegate.original_uri` | Original resource URI before namespacing | | ||
| | `fastmcp.proxy.backend_name` | Remote server tool/prompt name | | ||
| | `fastmcp.proxy.backend_uri` | Remote server resource URI | | ||
|
|
||
| ## Testing with Telemetry | ||
|
|
||
| For testing, use the in-memory exporter: | ||
|
|
||
| ```python | ||
| import pytest | ||
| from collections.abc import Generator | ||
| from opentelemetry import trace | ||
| from opentelemetry.sdk.trace import TracerProvider | ||
| from opentelemetry.sdk.trace.export import SimpleSpanProcessor | ||
| from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter | ||
|
|
||
| from fastmcp import FastMCP | ||
|
|
||
| @pytest.fixture | ||
| def trace_exporter() -> Generator[InMemorySpanExporter, None, None]: | ||
| exporter = InMemorySpanExporter() | ||
| provider = TracerProvider() | ||
| provider.add_span_processor(SimpleSpanProcessor(exporter)) | ||
| original_provider = trace.get_tracer_provider() | ||
| trace.set_tracer_provider(provider) | ||
| yield exporter | ||
| exporter.clear() | ||
| trace.set_tracer_provider(original_provider) | ||
|
|
||
| async def test_tool_creates_span(trace_exporter: InMemorySpanExporter) -> None: | ||
| mcp = FastMCP("test") | ||
|
|
||
| @mcp.tool() | ||
| def hello() -> str: | ||
| return "world" | ||
|
|
||
| await mcp.call_tool("hello", {}) | ||
|
|
||
| spans = trace_exporter.get_finished_spans() | ||
| assert any(s.name == "tool hello" for s in spans) | ||
| ``` | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.