feat(observability): add OpenTelemetry tracing and metrics support#590
Merged
Cali0707 merged 7 commits intocontainers:mainfrom Jan 21, 2026
Merged
feat(observability): add OpenTelemetry tracing and metrics support#590Cali0707 merged 7 commits intocontainers:mainfrom
Cali0707 merged 7 commits intocontainers:mainfrom
Conversation
Cali0707
reviewed
Dec 18, 2025
pkg/metrics/otel_collector.go
Outdated
Comment on lines
60
to
61
| spanName := fmt.Sprintf("%s %s", method, path) | ||
| _, span := o.tracer.Start(ctx, spanName, |
Collaborator
There was a problem hiding this comment.
I'm a little worried we may be missing the traceparent/tracestate on these spans
In the current semconv proposal it seems like these should be stored in the req.Params.Meta of the MCP request. But, this runs before the request is deserialized into the MCP types.
Collaborator
Author
There was a problem hiding this comment.
will add span propagation
Cali0707
reviewed
Dec 18, 2025
Cali0707
reviewed
Jan 8, 2026
Collaborator
Cali0707
left a comment
Cali0707
left a comment
There was a problem hiding this comment.
Sorry it took me so long to review here @nader-ziada - thanks for fixing up all the spans!
pkg/metrics/otel_stats_collector.go
Outdated
Comment on lines
57
to
59
| provider := sdkmetric.NewMeterProvider( | ||
| sdkmetric.WithReader(reader), | ||
| ) |
Collaborator
There was a problem hiding this comment.
I think we need to somewhere hook the provider.Shutdown(ctx) into the server shutdown, so that it can try to flush the metrics
pkg/metrics/otel_collector.go
Outdated
| _, span := o.tracer.Start(ctx, spanName, | ||
| trace.WithSpanKind(trace.SpanKindServer), | ||
| trace.WithAttributes( | ||
| attribute.String("http.method", method), |
Collaborator
There was a problem hiding this comment.
From https://opentelemetry.io/docs/specs/semconv/http/http-spans/#http-server-span I think that they have changed the attribute names 😓
It looks like these should probably be:
- "http.request.method"
- "url.path" (although they also have the "http.route" still - not totally sure for this one)
- "http.response.status_code"
I think?
pkg/telemetry/telemetry.go
Outdated
|
|
||
| // Create OTLP exporter | ||
| // Endpoint is configured via OTEL_EXPORTER_OTLP_ENDPOINT env var | ||
| exporter, err := otlptracegrpc.New(ctx) |
Collaborator
There was a problem hiding this comment.
Can we support allowing HTTP as well as grpc through the OTEL_EXPORTER_OTLP_PROTOCOL env var?
Collaborator
|
/assign Will take a look 👁️ |
Collaborator
Author
Collaborator
Author
Add observability capabilities through OpenTelemetry, providing distributed tracing and metrics for MCP operations and HTTP requests. - Automatic tracing of all MCP tool calls via middleware - HTTP request/response tracing in server mode - New telemetry package with configurable OTLP tracing via environment variables - Metrics collection for tool execution statistics and HTTP request counters - Middleware-based tracing that follows MCP semantic conventions - Session-based span context propagation between tool calls - Enable via OTEL_EXPORTER_OTLP_ENDPOINT environment variable - Custom sampling rates via OTEL_TRACES_SAMPLER environment variables Signed-off-by: Nader Ziada <nziada@redhat.com>
Signed-off-by: Nader Ziada <nziada@redhat.com>
- mcp.tool.duration: Histogram tracking tool call latency in seconds, with tool.name label for per-tool performance analysis - mcp.server.info: Gauge exposing version metadata with version and go_version labels for fleet tracking Signed-off-by: Nader Ziada <nziada@redhat.com>
matzew
reviewed
Jan 20, 2026
matzew
reviewed
Jan 20, 2026
Signed-off-by: Matthias Wessendorf <mwessend@redhat.com>
Signed-off-by: Matthias Wessendorf <mwessend@redhat.com>
matzew
reviewed
Jan 20, 2026
Fixes for Otel - avoid panic
matzew
reviewed
Jan 20, 2026
| ├─ network.transport: tcp | ||
| └─ Status: OK | ||
| ``` | ||
|
|
Collaborator
There was a problem hiding this comment.
how do we feel about including a screenshot ?
matzew
reviewed
Jan 20, 2026
matzew
reviewed
Jan 20, 2026
matzew
reviewed
Jan 20, 2026
matzew
reviewed
Jan 20, 2026
matzew
reviewed
Jan 20, 2026
| image: quay.io/containers/kubernetes_mcp_server:latest | ||
| env: | ||
| # OTLP endpoint (required to enable tracing) | ||
| - name: OTEL_EXPORTER_OTLP_ENDPOINT |
Collaborator
There was a problem hiding this comment.
(in a follow up PR) wanna add something to the helm chart for this subject?
matzew
reviewed
Jan 20, 2026
Signed-off-by: Nader Ziada <nziada@redhat.com>
manusa
added a commit
to marcnuri-forks/kubernetes-mcp-server
that referenced
this pull request
Jan 28, 2026
The toolScopedAuthorizationMiddleware was conceived for tool authorization but the feature was dropped in favor of implementing this in an MCP gateway. This removes: - The middleware function and its registration - TokenScopesContextKey constant and tokenScopesContextKeyType type - Unused imports (fmt, slices) in middleware.go This change was accidentally reverted in PR containers#590 and is now being re-applied (originally from PR containers#633). Signed-off-by: Marc Nuri <marc@marcnuri.com>
Cali0707
pushed a commit
that referenced
this pull request
Jan 28, 2026
) The toolScopedAuthorizationMiddleware was conceived for tool authorization but the feature was dropped in favor of implementing this in an MCP gateway. This removes: - The middleware function and its registration - TokenScopesContextKey constant and tokenScopesContextKeyType type - Unused imports (fmt, slices) in middleware.go This change was accidentally reverted in PR #590 and is now being re-applied (originally from PR #633). Signed-off-by: Marc Nuri <marc@marcnuri.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add observability capabilities through OpenTelemetry, providing distributed tracing and metrics for MCP operations and HTTP requests.