Add OpenTelemtry eBPF auto-instrumentation support by zeitlinger · Pull Request #1022 · grafana/docker-otel-lgtm

zeitlinger · 2026-02-10T15:40:41Z

Summary

Add OBI (OpenTelemetry eBPF Instrumentation) support to the grafana/otel-lgtm Docker image, opt-in via ENABLE_OBI=true
Include the OBI binary in the image with config, startup scripts, and health check integration
Add run-lgtm.sh/run-lgtm.ps1 support to auto-add --pid=host --privileged flags when OBI is enabled
Add mise run lgtm-obi task and OBI_TARGET friendly targeting (java, python, node, dotnet, ruby)
Add examples/obi/ — a multi-app demo running 5 uninstrumented apps (Java, Go, Python, Node.js, .NET) with OBI providing zero-code-change observability via eBPF
Align lint setup with upstream opentelemetry-java-contrib

OBI Integration (docker/)

Dockerfile: Pull otel/ebpf-instrument image, copy binary + Java agent into the image
run-obi.sh: Maps OBI_TARGET to OTEL_EBPF_AUTO_TARGET_EXE, defaults to port-based discovery
obi-config.yaml: Sends traces/metrics to the local OTel Collector at 127.0.0.1:4318
run-all.sh: Conditionally starts OBI when ENABLE_OBI=true, excluded from main health check loop

OBI Multi-App Example (examples/obi/)

Runs all 5 language examples without any OpenTelemetry instrumentation — OBI discovers and instruments them automatically via eBPF:

Java/Python: Reuse existing source with custom Dockerfiles that strip the OTel agent/distro
Go/Node.js/.NET: New minimal source with zero OTel dependencies
LGTM container runs with privileged: true and pid: "host" for eBPF access

Test plan

mise run build-lgtm dev1 && mise run lgtm-obi dev1 — verify OBI starts and reports monitoring
cd examples/obi && docker compose up --build — verify all 5 apps start
Open Grafana at http://localhost:3000 → Explore → Tempo: traces from all services
Open Grafana → Explore → Prometheus: http_server_request_duration_seconds_count metrics
Verify no OTel agent/SDK is loaded in any app container
mise run lint:rest — verify lints pass
added an OATs test

Copilot

Pull request overview

This PR adds optional Grafana Beyla eBPF auto-instrumentation to the LGTM container image and local run scripts, allowing traces/metrics to be sent to the existing OpenTelemetry Collector (127.0.0.1:4318) without changing collector config.

Changes:

Add Beyla binary/config/scripts into the Docker image and start Beyla optionally in run-all.sh when ENABLE_BEYLA=true.
Update run-lgtm.sh / run-lgtm.ps1 to auto-add --pid=host --privileged and forward Beyla-related env vars when enabled.
Document Beyla usage and add a mise task for quick startup.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
run-lgtm.sh	Adds opt-in Beyla enablement logic, env forwarding, and conditional TTY behavior.
run-lgtm.ps1	Adds opt-in Beyla enablement logic and env forwarding for Windows/PowerShell runs.
mise.toml	Adds `lgtm-beyla` task to run LGTM with Beyla enabled.
docker/run-beyla.sh	New entrypoint script to run Beyla with target/port-based discovery defaults.
docker/run-all.sh	Starts Beyla optionally and adds non-blocking status messaging.
docker/download-beyla.sh	Downloads/extracts Beyla during the image build.
docker/beyla-config.yaml	Configures Beyla OTLP HTTP export to the existing collector and enables internal metrics.
docker/Dockerfile	Adds Beyla version ARG, downloads Beyla, and includes Beyla scripts/config in the image.
README.md	Documents how to enable/configure Beyla and adds logging env var.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

zeitlinger · 2026-02-10T17:41:11Z

@martincostello please check again

zeitlinger · 2026-02-11T13:35:01Z

Will convert to use OBI:

there's probably no strong reason to choose Beyla over upstream OBI at this point, and it may be worth switching.

Here's the situation:

Beyla is now a thin wrapper around OBI

Grafana https://grafana.com/blog/2025/05/07/opentelemetry-ebpf-instrumentation-beyla-donation/ as the OpenTelemetry eBPF Instrumentation (OBI) project. The relationship is:

OBI (https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation) is the upstream, vendor-neutral project where all core development happens
Beyla (https://github.com/grafana/beyla) is now Grafana's downstream distribution — it vendors OBI as a git submodule and adds Grafana-specific features (Cloud onboarding,
Alloy integration)

All current Beyla maintainers work full-time on the upstream repo, and PRs (unless docs-only) are expected to go to OBI, not Beyla.

Arguments for using OBI instead

Vendor-neutral — this repo (docker-otel-lgtm) is an OpenTelemetry reference stack, not a Grafana-specific product. Using the upstream OTel project is more philosophically
aligned.
Where development happens — OBI has broader community contributions (Splunk, Coralogix, Odigos, etc.) and development has
https://opentelemetry.io/blog/2025/obi-announcing-first-release/.
No Grafana-specific extras needed — the Grafana-specific features Beyla adds (Cloud onboarding, Alloy integration) are irrelevant in this container, which already has a
local OTLP endpoint at 127.0.0.1:4318.
Beyla is shrinking — Grafana is actively migrating code out of Beyla into OBI. Beyla will eventually be just a thin shell.

Arguments for keeping Beyla (weaker)

More mature release cadence — Beyla is at v2.8.5 while OBI just hit its https://opentelemetry.io/blog/2025/obi-announcing-first-release/. If OBI is still alpha, that could
be a concern for stability.
Simpler binary distribution — Beyla ships a single binary via GitHub releases, which the current download-beyla.sh already uses. OBI's distribution story may differ.
Familiarity — more existing docs and blog posts reference Beyla.

My take

For a project named docker-otel-lgtm that's meant to be a vendor-neutral OpenTelemetry dev/demo stack, OBI would be the more natural fit — unless it's still too early in its
release cycle (alpha) to be reliable. It might be worth checking whether OBI has a stable enough binary release to use, and if so, switching from Beyla to OBI. At minimum, the
PR should document why Beyla was chosen over upstream OBI.

Sources:

Copilot

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

grcevski

Looking great! I'd just enable context propagation by default. If it's not supported by the kernel, we will gracefully degrade to black-box context propagation.

Copilot

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Remove OBI references (not yet in repo, coming via PR #1022) - Remove nonexistent `mise run lgtm-obi` command - Fix cosign claim: only OTel Collector uses cosign, others use SHA256 - Fix docker-compose filenames to match actual `docker-compose.oats.yml` - Fix collector config filenames and otelcol-contrib invocation

Copilot

Pull request overview

Copilot reviewed 29 out of 30 changed files in this pull request and generated 6 comments.

Files not reviewed (1)

examples/obi/nodejs/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

already has command for #1022 --------- Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

## Summary - Replace the `include:` directive with multiple `-f` flags when merging docker-compose files - The `include:` approach treats each file as an independent project, preventing service name overrides across files (fails with `services.lgtm conflicts with imported resource`) - Multiple `-f` flags allow later files to merge into services defined by earlier files - Add `--project-directory` based on the first user compose file so that relative paths (build contexts, dockerfiles) resolve correctly ## Motivation This is needed for [docker-otel-lgtm OBI support](grafana/docker-otel-lgtm#1022), where the OATS compose file must override the `lgtm` service to add `privileged: true`, `pid: "host"`, and `ENABLE_OBI: "true"` for eBPF instrumentation. ## Test plan - [x] Verified OBI example (`oats examples/obi`) passes with this change - [x] Verified existing Go example generates identical compose output (no regression)

Copilot

Pull request overview

Copilot reviewed 29 out of 30 changed files in this pull request and generated no new comments.

Files not reviewed (1)

examples/obi/nodejs/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

examples/obi/nodejs/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Include the Beyla binary in the image (opt-in via ENABLE_BEYLA=true). When enabled, run-lgtm.sh/ps1 auto-add --pid=host --privileged flags. Beyla sends traces and metrics to the existing OTel Collector at 127.0.0.1:4318, requiring no collector config changes. Beyla is excluded from the main health check loop so it fails gracefully when eBPF capabilities are unavailable.

…n (OBI) Beyla's core has been donated to the OpenTelemetry project as OBI. Since this repo is a vendor-neutral OpenTelemetry stack, using the upstream project is a better fit. Key changes: - Multi-stage COPY from otel/ebpf-instrument image instead of downloading Beyla release tarballs - Env vars: BEYLA_TARGET → OBI_TARGET, BEYLA_EXECUTABLE_NAME → OTEL_EBPF_AUTO_TARGET_EXE, BEYLA_OPEN_PORT → OTEL_EBPF_OPEN_PORT - Config: internal_metrics.prometheus.port → prometheus_export.port/path - Remove download-beyla.sh (binary comes from container image)

Demonstrates OBI automatically instrumenting 5 apps (Java, Go, Python, Node.js, .NET) without any OpenTelemetry SDK, agent, or code changes. Java and Python reuse existing source with Dockerfiles that strip OTel layers. Go, Node.js, and .NET have new minimal source with zero OTel dependencies. The lgtm container runs with ENABLE_OBI, privileged mode, and host PID namespace to allow eBPF kernel-level instrumentation.

- Fix lgtm-obi --local ignoring the tag argument - Add comment explaining why examples/obi/go/ is excluded from super-linter - Clarify that OTLP exporter vars are forwarded via --env-file .env

- Fix shadow warning in otel.go (rename inner err to errs) - Fix empty-block warning in rolldice.go (nolint:revive) - Remove deprecated otelhttp.WithRouteTag (auto-extracted since Go 1.22+) - Add examples/obi/go to go.work so golangci-lint can typecheck it - Suppress gosec warnings in OBI Go demo (dice roll, simple server) - Switch from VALIDATE_GO_MODULES=false to VALIDATE_GO=false (GO_MODULES no longer times out; non-module GO can't handle two modules) - Remove FIX_GO=true (incompatible with VALIDATE_GO=false)

- Add ebpf context_propagation: all to OBI config (grcevski) - Add Pyroscope port 4040 to OBI docker-compose example - Fix PowerShell runner: conditional TTY allocation like bash runner - Fix OBI nodejs Dockerfile: use npm ci with lockfile and non-root user - Fix OBI go.mod: use go 1.24.0 for consistency with go.work - Fix generate-traffic Dockerfile: clean up apt lists

Required for OBI acceptance tests to pass.

- Add *.cs eol=crlf to .gitattributes for editorconfig compliance - Regenerate renovate-tracked-deps.json with OBI entries - Remove ebpf.context_propagation from obi-config.yaml (needs kernel 5.17+) - Enable OBI logging in oats test compose for debugging

Per reviewer feedback, ebpf.context_propagation: all enables distributed trace propagation and falls back gracefully on unsupported kernels.

Copilot

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

examples/obi/nodejs/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Fix grep portability: use grep -vE instead of GNU-specific \| alternation - Drop -i flag when stdin is not a TTY (bash and PowerShell) - Add comment explaining why oats.yaml has no input section - Fix OBI documentation link to point to opentelemetry.io

zeitlinger self-assigned this Feb 10, 2026

Copilot AI review requested due to automatic review settings February 10, 2026 15:40

Copilot started reviewing on behalf of zeitlinger February 10, 2026 15:41 View session

Copilot AI reviewed Feb 10, 2026

View reviewed changes

Comment thread run-lgtm.ps1 Outdated

Comment thread run-lgtm.sh Outdated

Comment thread docker/download-beyla.sh Outdated

Comment thread docker/run-all.sh Outdated

martincostello reviewed Feb 10, 2026

View reviewed changes

Comment thread README.md Outdated

Comment thread README.md

Comment thread README.md Outdated

Comment thread docker/Dockerfile Outdated

Comment thread docker/download-beyla.sh Outdated

Comment thread docker/download-beyla.sh Outdated

Comment thread run-lgtm.ps1 Outdated

martincostello approved these changes Feb 11, 2026

View reviewed changes

zeitlinger marked this pull request as draft February 11, 2026 13:33

zeitlinger mentioned this pull request Feb 11, 2026

add CLAUDE.md #1023

Merged

zeitlinger changed the title ~~Add Grafana Beyla eBPF auto-instrumentation support~~ Add OpenTelemtry eBPF auto-instrumentation support Feb 11, 2026

Copilot AI review requested due to automatic review settings February 11, 2026 17:16

Copilot started reviewing on behalf of zeitlinger February 11, 2026 17:17 View session

Copilot AI reviewed Feb 11, 2026

View reviewed changes

Comment thread run-lgtm.ps1

Comment thread mise.toml Outdated

Comment thread .github/config/super-linter.env Outdated

zeitlinger force-pushed the add-beyla branch from 3a41b9a to 6c9ccb1 Compare February 12, 2026 08:12

grcevski approved these changes Feb 12, 2026

View reviewed changes

Comment thread docker/run-obi.sh

Comment thread docker/obi-config.yaml

zeitlinger mentioned this pull request Feb 12, 2026

Allow docker-compose files to override the LGTM service grafana/oats#223

Merged

2 tasks

Copilot AI review requested due to automatic review settings February 12, 2026 14:41

Copilot started reviewing on behalf of zeitlinger February 12, 2026 14:41 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

Comment thread run-lgtm.ps1

Comment thread examples/obi/docker-compose.yaml

Comment thread examples/obi/nodejs/Dockerfile Outdated

Copilot AI review requested due to automatic review settings February 12, 2026 16:42

Copilot started reviewing on behalf of zeitlinger February 12, 2026 16:44 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

Comment thread examples/obi/go/go.mod Outdated

Comment thread examples/obi/java.Dockerfile

Comment thread examples/obi/generate-traffic.Dockerfile Outdated

Comment thread run-lgtm.sh Outdated

Comment thread examples/obi/nodejs/Dockerfile

Comment thread examples/obi/nodejs/app.js

Copilot AI review requested due to automatic review settings February 12, 2026 18:04

zeitlinger force-pushed the add-beyla branch from 9eb7366 to 1c14057 Compare February 12, 2026 18:04

Copilot started reviewing on behalf of zeitlinger February 12, 2026 18:04 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

zeitlinger force-pushed the add-beyla branch from 16d5ed9 to 1c14057 Compare February 12, 2026 18:35

Copilot AI review requested due to automatic review settings February 12, 2026 19:03

Copilot started reviewing on behalf of zeitlinger February 12, 2026 19:04 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

Comment thread run-lgtm.sh Outdated

Comment thread run-lgtm.sh

Comment thread run-lgtm.ps1

Comment thread examples/obi/oats.yaml

zeitlinger added 11 commits February 12, 2026 20:21

pr review

3355704

fix

16729e6

Address PR review comments

40793bb

- Fix lgtm-obi --local ignoring the tag argument - Add comment explaining why examples/obi/go/ is excluded from super-linter - Clarify that OTLP exporter vars are forwarded via --env-file .env

fix

6f2b79d

Bump oats to 0.6.1

30c4889

Required for OBI acceptance tests to pass.

zeitlinger force-pushed the add-beyla branch from 5144fc1 to cc1beab Compare February 12, 2026 19:21

Enable OBI context propagation by default

58e93ed

Per reviewer feedback, ebpf.context_propagation: all enables distributed trace propagation and falls back gracefully on unsupported kernels.

Copilot AI review requested due to automatic review settings February 12, 2026 19:44

Copilot started reviewing on behalf of zeitlinger February 12, 2026 19:44 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

Comment thread examples/obi/README.md Outdated

Comment thread run-lgtm.sh

zeitlinger marked this pull request as ready for review February 12, 2026 19:55

zeitlinger enabled auto-merge (squash) February 12, 2026 19:59

zeitlinger merged commit b0c27bd into main Feb 12, 2026
10 checks passed

zeitlinger deleted the add-beyla branch February 12, 2026 20:08

Conversation

zeitlinger commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zeitlinger commented Feb 10, 2026

Uh oh!

zeitlinger commented Feb 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

grcevski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zeitlinger commented Feb 10, 2026 •

edited

Loading