feat: MCP integration for AI tools

[zeitlinger]

Closes #1091

Summary

• Enables MCP servers so AI coding tools (Claude, Cursor, etc.) can query the LGTM stack
• Tempo native MCP for TraceQL queries (built-in, just needs config flag)
• mcp-grafana for PromQL, LogQL, dashboards (external, SA token bootstrapped)
• Generates /etc/lgtm/mcp.json at startup — one config block for any MCP client
• Opt-in collector debug exporter via OTEL_COLLECTOR_DEBUG_EXPORTER=true
• Full docs at docs/mcp-integration.md

Setup

docker exec lgtm cat /etc/lgtm/mcp.json

Paste into your AI tool's MCP configuration.

Test plan

• Start container, verify mcp.json is generated with valid SA token
• Connect mcp-grafana, query metrics and logs
• Connect Tempo MCP, query traces
• Enable OTEL_COLLECTOR_DEBUG_EXPORTER=true, verify debug output in logs
• OATS acceptance tests pass

[Copilot]

Pull request overview

Adds documentation and container-side scaffolding to support Model Context Protocol (MCP) integrations so AI coding tools can query LGTM data (traces/metrics/logs/dashboards), plus an optional OpenTelemetry Collector debug-exporter overlay.

Changes:

• Document MCP setup and add a new MCP integration guide.
• Enable Tempo’s MCP server and emit an MCP config JSON (including a Grafana service-account token) from the container.
• Add an opt-in OTEL Collector “debug exporter” overlay and extend acceptance test log expectations.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
README.md	Adds a short “AI Tool Integration (MCP)” entry point and MCP link.
docs/mcp-integration.md	New guide describing MCP setup, including Grafana + Tempo MCP configuration.
docker/run-all.sh	Generates Grafana service account token and writes /etc/lgtm/mcp.json, plus startup log hints.
docker/tempo-config.yaml	Enables Tempo MCP server in the query frontend.
docker/run-otelcol.sh	Adds OTEL_COLLECTOR_DEBUG_EXPORTER flag to layer an extra collector config.
docker/otelcol-config-debug.yaml	New overlay config that adds debug exporters to traces/metrics/logs pipelines.
examples/python/oats.yaml	Extends acceptance test to assert the new MCP-related startup log lines.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

docker/run-otelcol.sh:43

• When both OTEL_EXPORTER_OTLP_ENDPOINT (adds otelcol-config-export-http.yaml) and OTEL_COLLECTOR_DEBUG_EXPORTER (adds otelcol-config-debug.yaml) are enabled, the later debug config replaces the pipelines' exporters lists, which drops otlphttp/external and stops exporting to the external OTLP endpoint. Consider generating a combined config (or a separate debug+external config) so the exporters list includes both debug/* and otlphttp/external when both flags are set (instead of last-one-wins).

debug_config_file=""
if [[ ${OTEL_COLLECTOR_DEBUG_EXPORTER:-false} == "true" ]]; then
	echo "Enabling debug exporter for OpenTelemetry Collector"
	debug_config_file="--config=file:./otelcol-config-debug.yaml"
fi

run_with_logging "OpenTelemetry Collector ${OPENTELEMETRY_COLLECTOR_VERSION}" "${ENABLE_LOGS_OTELCOL:-false}" \
	./otelcol-contrib/otelcol-contrib --feature-gates service.profilesSupport --config=file:./otelcol-config.yaml ${secondary_config_file} ${debug_config_file}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.

Comments suppressed due to low confidence (2)

build-lgtm.sh:25

• This now prefers Podman over Docker when both are installed, but still unconditionally runs buildx build. On many systems podman buildx isn’t available by default, so this can break builds that previously succeeded with Docker. Consider checking for buildx support (or falling back to podman build / using Docker when available), or allow users to override the runtime selection.

if command -v podman >/dev/null 2>&1; then
	RUNTIME=podman
elif command -v docker >/dev/null 2>&1; then
	RUNTIME=docker
else
	echo "Unable to find a suitable container runtime such as Docker or Podman. Exiting."
	exit 1
fi

if [ "$RUNTIME" = "podman" ]; then
	TAG="localhost/grafana/otel-lgtm:${RELEASE}"
else
	TAG="grafana/otel-lgtm:${RELEASE}"
fi

$RUNTIME buildx build -f docker/Dockerfile docker --tag "$TAG" --build-arg LGTM_VERSION="${RELEASE}"

docker/run-otelcol.sh:43

• When both OTEL_EXPORTER_OTLP_ENDPOINT (external export) and OTEL_COLLECTOR_DEBUG_EXPORTER=true are set, this passes two extra --config files that both override service.pipelines.*.exporters (otelcol-config-export-http.yaml and otelcol-config-debug.yaml). Depending on config merge precedence, the later file can clobber the earlier one, dropping the external exporter (or the debug exporter). Consider generating a combined exporters list (external + debug) or ensuring the overlay config doesn’t overwrite other overlays’ exporter settings.

debug_config_file=""
if [[ ${OTEL_COLLECTOR_DEBUG_EXPORTER:-false} == "true" ]]; then
	echo "Enabling debug exporter for OpenTelemetry Collector"
	debug_config_file="--config=file:./otelcol-config-debug.yaml"
fi

run_with_logging "OpenTelemetry Collector ${OPENTELEMETRY_COLLECTOR_VERSION}" "${ENABLE_LOGS_OTELCOL:-false}" \
	./otelcol-contrib/otelcol-contrib --feature-gates service.profilesSupport --config=file:./otelcol-config.yaml ${secondary_config_file} ${debug_config_file}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zeitlinger]

Keeping in draft - waiting for feedback from human testers

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[alesj]

@zeitlinger any plans to support some non-stdio transport for grafana as well?

[alesj]

Btw, when running the mcp image - docker run -i docker.io/grafana/otel-lgtm (or w/o -i),
I cannot shut it down via ctrl-c ...

[zeitlinger]

@zeitlinger any plans to support some non-stdio transport for grafana as well?

no, mcp-grafana is a process tool, stdio only by design

that being said, I'd love to try out a CLI based approach too, but this is the first thing that's available

[alesj]

no, mcp-grafana is a process tool, stdio only by design

Being a mcp and uvx noob, iiuc mcp-grafana is a separate process that runs,
exposing mcp, and knows how to do queries (for logs, prom, dashboards)?

As such it makes a bit hard for use to wire it I guess,
since we only do Docker images run in devservices.
Dunno how we could also start mcp-grafana ... hence needing a remote transport, afais.

[alesj]

Ah, I've been told this is not true ... we could also start uvx ... nice :-)

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

README.md:265

• The README now documents port-forwarding Tempo on 3200 and suggests mise k8s-port-forward as an alternative, but the k8s-port-forward task in mise.toml still forwards only 3000 4040 4317 4318 9090 (no 3200). This makes the mise shortcut inconsistent with the documented port-forward command; consider updating the mise task to include 3200:3200 as well.

# Configure port forwarding
kubectl port-forward service/lgtm 3000:3000 3200:3200 4040:4040 4317:4317 4318:4318 9090:9090

# Using mise
mise k8s-apply
mise k8s-port-forward

</details>



---

💡 <a href="/grafana/docker-otel-lgtm/new/main?filename=.github/instructions/*.instructions.md" class="Link--inTextBlock" target="_blank" rel="noopener noreferrer">Add Copilot custom instructions</a> for smarter, more guided reviews. <a href="https://docs.github.com/en/copilot/customizing-copilot/adding-repository-custom-instructions-for-github-copilot" class="Link--inTextBlock" target="_blank" rel="noopener noreferrer">Learn how to get started</a>.

[zeitlinger]

Put in draft mode - I want to try GCX first

[zeitlinger]

Put in draft mode - I want to try GCX first

@martincostello as discussed - let's do this PR before trying out a new approach

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[martincostello]

Looks OK to me modulo the GitHub Copilot comments.

[alesj]

@zeitlinger is there a new image version already out with this cool mcp stuff?
(so that I update it in Quarkus as well ...)

[martincostello]

These changes should be in v0.24.0.