Migrate from individual OTel packages to EDOT by afharo · Pull Request #226609 · elastic/kibana

afharo · 2025-07-04T15:51:52Z

Summary

Migrating as much as we can from the myriad of @opentelemetry/* packages to @elastic/opentelemetry-node/sdk.

We still need some individual packages installed for very specific things.

Checklist

Review the backport guidelines and apply applicable backport:* labels.

afharo

Highlight the fixes to get @elastic/opentelemetry-node/sdk to work in Kibana

afharo · 2025-07-07T23:09:33Z

src/platform/packages/private/kbn-import-resolver/src/import_resolver.ts

+    if (req.startsWith('@elastic/opentelemetry-node/sdk')) {
+      return Path.resolve(REPO_ROOT, `node_modules/@elastic/opentelemetry-node/lib/sdk.js`);
+    }


This fixes the runtime imports (including ESLint).

we can likely add a note here so we don't forget we can remove this once the cjs/esm interop is in

Can we create an issue and reference it here? That way we can track the technical debt and possibly trigger someone to fix it.

afharo · 2025-07-07T23:13:04Z

tsconfig.base.json

      // Allows for importing from `kibana` package for the exported types.
-      "@emotion/core": ["typings/@emotion"]
+      "@emotion/core": ["typings/@emotion"],
+      "@elastic/opentelemetry-node/sdk": ["typings/@elastic/opentelemetry-node/sdk"],


Tell TS where to look for the types of this package.

we can likely add a note here as well so we don't forget to remove this in the future

x-pack/platform/plugins/private/monitoring_collection/server/plugin.ts

kibanamachine · 2025-07-07T23:16:46Z

Dependency Review Bot Analysis 🔍

Found 1 new third-party dependencies:

Package	Version	Vulnerabilities	Health Score
@opentelemetry/sdk-metrics	^2.0.1	🔴 C: 0, 🟠 H: 0, 🟡 M: 0, 🟢 L: 0

Self Checklist

To help with the review, please update the PR description to address the following points for each new third-party dependency listed above:

Purpose: What is this dependency used for? Briefly explain its role in your changes.
Justification: Why is adding this dependency the best approach?
Alternatives explored: Were other options considered (e.g., using existing internal libraries/utilities, implementing the functionality directly)? If so, why was this dependency chosen over them?
Existing dependencies: Does Kibana have a dependency providing similar functionality? If so, why is the new one preferred?

Thank you for providing this information!

mistic

the changes to support @elastic/opentelemetry-node/sdk LGTM

mistic · 2025-07-07T23:27:52Z

src/platform/packages/private/kbn-import-resolver/src/import_resolver.ts

+    if (req.startsWith('@elastic/opentelemetry-node/sdk')) {
+      return Path.resolve(REPO_ROOT, `node_modules/@elastic/opentelemetry-node/lib/sdk.js`);
+    }


we can likely add a note here so we don't forget we can remove this once the cjs/esm interop is in

mistic · 2025-07-07T23:28:14Z

tsconfig.base.json

      // Allows for importing from `kibana` package for the exported types.
-      "@emotion/core": ["typings/@emotion"]
+      "@emotion/core": ["typings/@emotion"],
+      "@elastic/opentelemetry-node/sdk": ["typings/@elastic/opentelemetry-node/sdk"],


we can likely add a note here as well so we don't forget to remove this in the future

pickypg

LGT SM

x-pack/platform/plugins/private/monitoring_collection/server/plugin.ts

dgieselaar · 2025-07-08T04:41:16Z

x-pack/platform/plugins/private/monitoring_collection/server/plugin.ts

-import { diag, DiagLogger, DiagLogLevel } from '@opentelemetry/api';
-import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
+import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';
+// import { ATTR_SERVICE_INSTANCE_ID } from '@opentelemetry/semantic-conventions/incubating';


iirc, the OTel docs actually mention you should copy them over (while they are in development) and not import directly

dgieselaar · 2025-07-08T04:44:41Z

package.json

    "@opentelemetry/api": "^1.9.0",
-    "@opentelemetry/api-metrics": "^0.31.0",
-    "@opentelemetry/context-async-hooks": "^2.0.0",
-    "@opentelemetry/core": "^2.0.0",


These APIs are env-agnostic I think - in the sense that you should be able to use them in both the browser and node.js. How do we plan to handle this if we fully delegate to @elastic/opentelemetry-node which is only for node.js?

That's a good question.

AFAIK, we're focusing on the server for now.

I would expect that there will be an @elastic/opentelemetry-web/sdk or something like that.

If not, we can install the vanilla packages again.

I would expect that there will be an @elastic/opentelemetry-web/sdk or something like that.

I'm not sure. In general I'm not sure if we should hide OpenTelemetry APIs behind an elastic distrib. @trentm what are the advantages here? It just feels off to use re-exported OpenTelemetry types where we also have less control about versioning.

What benefit do we get from maintaining 12 different packages (17 really, but I only managed to remove 12 because we needed the other 5)? Especially with the current state of maturity of OTel.

Our renovate PR didn't manage to identify that @opentelemetry/api-metrics (link), or @opentelemetry/sdk-metrics-base (link) were deprecated 3 years ago!

And don't get me started on potentially incompatibilities across versions. I'd rather push that burden to the owners of EDOT.

I can also see the benefit of "dog-fooding" our products to identify opportunities to improve them: this PR highlights the bad DX of having to move from direct imports of types to namespaced imports: import { MeterProvider } becomes import { metrics } + metrics.MeterProvider 😢

IMO, this is a bad DX, and it's good that we figured it out first.

But I'd love to hear @trentm's thoughts.

We don't currently have a "web SDK" (i.e. an OTel SDK targetting browsers).

The state of a Web SDK in vanilla OTel JS is also much less mature that the packages for Node.js. See https://opentelemetry.io/docs/languages/js/getting-started/browser/ for a start, but one needs to learn more about the details to use it effectively. @david-luna and I can try to help where needed.

These APIs are env-agnostic I think

Dario, I'm not sure if you were talking about all three of those deps:

- "@opentelemetry/api-metrics": "^0.31.0", - "@opentelemetry/context-async-hooks": "^2.0.0", - "@opentelemetry/core": "^2.0.0",

api-metrics is gone now. It was merged into @opentelemetry/api. It being a dep in Kibana's package.json shows that this metrics usage was a quite old usage from before the Metrics parts of @opentelemetry/api were stabilized.

context-async-hooks is only relevant to Node.js, so you shouldn't need to this dep at all. @elastic/opentelemetry-node will setup the appropriate ContextManager for Node.js.

core: The only usage of that in kibana.git that I see is:

x-pack/platform/packages/shared/kbn-inference-tracing/src/create_inference_active_span.ts 8:import { isTracingSuppressed } from '@opentelemetry/core';

Is createActiveInferenceSpan() used in Kibana browser code? If so, having a separate dep on @opentelemetry/core will be necessary -- and fine to have alongside @elastic/opentelemetry-node.
If just in Node.js code, the core module is re-exported by @elastic/opentelemetry-node/sdk:

> var edot = require('@elastic/opentelemetry-node/sdk') > edot.core.isTracingSuppressed [Function: isTracingSuppressed]

Thank you @trentm for such informative comments.

Is there a problem with this:

import {metrics} from '@elastic/opentelemetry-node';
// ...
const myReader = new metrics.MetricReader(...)
or this:

import {metrics} from '@elastic/opentelemetry-node';
const {MetricReader} = metrics;
// ...
const myReader = new MetricReader(...)
other than a single line import {MetricReader} from 'something' looks nicer?

That makes it nicer (I'll adapt the PR to minimize the number of lines changed). However, the DX is still suboptimal: typically your IDE helps with the imports. This package forces you to intentionally type it because the IDE is never going to find the MetricReader that you're after.

Also, I wonder if this affects performance as it may be worse on tree-shaking: I might be completely wrong here but AFAIK the import in the example below is a noop at runtime and the compiler removes it:

import { MetricReader } from '...'; function doSomething(reader: MetricReader) {}

However, I'm not that sure about importing a parent variable metrics (in both flavors that you shared above). I'll need to test to see if the generated compiled code removes it or not.

@trentm, actually, we need to differentiate between runtime code and types 😢:

import { tracing } from '@elastic/opentelemetry-node/sdk'; const { TraceIdRatioBasedSampler } = tracing; type SpanProcessor = tracing.SpanProcessor;

What benefit do we get from maintaining 12 different packages (17 really, but I only managed to remove 12 because we needed the other 5)? Especially with the current state of maturity of OTel.

We don't, but i think we'll end up using some of the packages directly anyway. It just feels off to me, but not a blocker for me in any case.

Dario, I'm not sure if you were talking about all three of those deps:

I'm talking about @opentelemetry/api and @opentelemetry/core. E.g., I have helpers that create spans. I want to be able to use that code in the browser and in Node.js. How can I do that if we are expected to go through @elastic/opentelemetry-node and a hypothetical @elastic/opentelemetry-web lib? We would need @elastic/opentelemetry to be created I guess. And we'd have to do that for every instrumentation package we want to use.

@dgieselaar, I agree with you that this makes it harder to make agnostic code.

TBH, after trying to "simulate" some of the changes in #224933 with this set of packages, I think that creating spans or metrics should be used via imports from @opentelemetry/api (which is a peer dependency of @elastic/opentelemetry-node/sdk, meaning that we'll have to install it separately anyway)

IMO, it's better to do import { metrics } from '@opentelemetry/api'; than import { api } from '@elastic/opentelemetry-node/sdk'; + use api.metrics.

On top of that, given that OTel has this weird requirement about every component in the code having to know the name of the meter/tracer, we'll likely end up with some sort of utils package that exposes the default meter/tracer (the output of metrics.getMeter('kibana')).

So I'd expect that the EDOT imports are reduced to the places where we're configuring the SDK (setting up the providers and exporters), but not the actual code instrumentation.

Does this make sense?

I'm talking about @opentelemetry/api and @opentelemetry/core. E.g., I have helpers that create spans. I want to be able to use that code in the browser and in Node.js. How can I do that if we are expected to go through @elastic/opentelemetry-node and a hypothetical @elastic/opentelemetry-web lib?

My understanding of OTel is that separately importing @opentelemetry/api and using it is intentional and a good thing.

That leaves @opentelemetry/core. I think it would be fine for Kibana to have both @elastic/opentelemetry-node and @opentelemetry/core as deps. However, yes it means that you likely want to watch that the version of both are compatible versions so that the expected functionality works. And possibly want to sure that the same exact version is installed to reduce bundle sizes (if that is a concern).

efegurkan

Search Kibana changes LGTM

kibanamachine · 2025-07-21T19:51:24Z

💔 Some backports could not be created

Status	Branch	Result
❌	8.18	Backport failed because of merge conflicts
❌	8.19	Backport failed because of merge conflicts You might need to backport the following PRs to 8.19: - [EBT] Add Global Session Context Provider (#227881) - Update dependency @elastic/charts to v70.1.1 (main) (#227902) - Update dependency msw to ~2.10.3 (main) (#227672) - [ska] relocate platform banner, cloud, so, custom branding tests (#227269) - [ska][x-pack] relocate spaces_api_integration tests (#225377)
❌	9.0	Backport failed because of merge conflicts You might need to backport the following PRs to 9.0: - Remove playwright-chromium package from dependencies (#225221) - SKA: Relocate "platform" packages that remain on `/packages` (#208704)
✅	9.1

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 226609

Questions ?

Please refer to the Backport tool documentation

(cherry picked from commit cf6458d) # Conflicts: # package.json # src/platform/packages/private/kbn-import-resolver/src/import_resolver.ts # src/platform/packages/shared/kbn-tracing/src/init_tracing.ts # src/platform/packages/shared/kbn-tracing/src/late_binding_span_processor.ts # x-pack/platform/packages/shared/kbn-inference-tracing/src/base_inference_span_processor.ts # x-pack/platform/packages/shared/kbn-inference-tracing/src/create_inference_active_span.ts # x-pack/platform/packages/shared/kbn-inference-tracing/src/langfuse/langfuse_span_processor.ts # x-pack/platform/packages/shared/kbn-inference-tracing/src/phoenix/get_chat_span.ts # x-pack/platform/packages/shared/kbn-inference-tracing/src/phoenix/get_execute_tool_span.ts # x-pack/platform/packages/shared/kbn-inference-tracing/src/phoenix/phoenix_span_processor.ts # x-pack/platform/plugins/private/monitoring_collection/server/plugin.ts # yarn.lock

(cherry picked from commit cf6458d) # Conflicts: # package.json # src/platform/packages/private/kbn-import-resolver/src/import_resolver.ts

afharo · 2025-07-21T20:39:44Z

💚 All backports created successfully

Status	Branch	Result
✅	9.0
✅	8.19
✅	8.18

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

(cherry picked from commit cf6458d) # Conflicts: # package.json # src/platform/packages/private/kbn-import-resolver/src/import_resolver.ts # src/platform/packages/shared/kbn-tracing/src/init_tracing.ts # src/platform/packages/shared/kbn-tracing/src/late_binding_span_processor.ts # x-pack/platform/packages/shared/kbn-inference-tracing/src/base_inference_span_processor.ts # x-pack/platform/packages/shared/kbn-inference-tracing/src/create_inference_active_span.ts # x-pack/platform/packages/shared/kbn-inference-tracing/src/langfuse/langfuse_span_processor.ts # x-pack/platform/packages/shared/kbn-inference-tracing/src/phoenix/get_chat_span.ts # x-pack/platform/packages/shared/kbn-inference-tracing/src/phoenix/get_execute_tool_span.ts # x-pack/platform/packages/shared/kbn-inference-tracing/src/phoenix/phoenix_span_processor.ts # x-pack/platform/plugins/private/monitoring_collection/server/plugin.ts # yarn.lock

# Backport This will backport the following commits from `main` to `9.1`: - [Migrate from individual OTel packages to EDOT (#226609)](#226609)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>

# Backport This will backport the following commits from `main` to `8.19`: - [Migrate from individual OTel packages to EDOT (#226609)](#226609)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Tiago Costa <tiago.costa@elastic.co>

# Backport This will backport the following commits from `main` to `9.0`: - [Migrate from individual OTel packages to EDOT (#226609)](#226609)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: Tiago Costa <tiago.costa@elastic.co>

# Backport This will backport the following commits from `main` to `8.18`: - [Migrate from individual OTel packages to EDOT (#226609)](#226609)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

afharo added 2 commits July 7, 2025 23:10

[PoC] Migrate to EDOT

692bb49

Fix imports and some usages

3d94c59

afharo force-pushed the migrate-to-edot branch from 15dc798 to 3d94c59 Compare July 7, 2025 23:08

clean package.json

8eea9c5

afharo commented Jul 7, 2025

View reviewed changes

afharo marked this pull request as ready for review July 7, 2025 23:16

afharo requested review from a team as code owners July 7, 2025 23:16

afharo self-assigned this Jul 7, 2025

afharo added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting labels Jul 7, 2025

kibanamachine requested a review from a team July 7, 2025 23:16

afharo changed the title ~~[PoC] Migrate to EDOT~~ Migrate from individual OTel packages to EDOT Jul 7, 2025

mistic approved these changes Jul 7, 2025

View reviewed changes

update renovate

d094ef8

afharo requested a review from a team as a code owner July 7, 2025 23:29

Add comments for future work to undo these changes

c26dc9d

afharo mentioned this pull request Jul 7, 2025

[Server-side OTel] Add autoinstrumentation #224839

Closed

afharo linked an issue Jul 7, 2025 that may be closed by this pull request

[Server-side OTel] Add autoinstrumentation #224839

Closed

Remove unnecessary dependency @opentelemetry/sdk-metrics

7499887

pickypg approved these changes Jul 8, 2025

View reviewed changes

x-pack/platform/plugins/private/monitoring_collection/server/plugin.ts Show resolved Hide resolved

dgieselaar reviewed Jul 8, 2025

View reviewed changes

efegurkan approved these changes Jul 8, 2025

View reviewed changes

afharo added 2 commits July 9, 2025 11:54

Merge branch 'main' of github.com:elastic/kibana into migrate-to-edot

df45e7a

Add missing entry to renovate

ce7353b

kibanamachine mentioned this pull request Jul 21, 2025

[9.1] Migrate from individual OTel packages to EDOT (#226609) #228861

Merged

afharo mentioned this pull request Jul 21, 2025

[9.0] Migrate from individual OTel packages to EDOT (#226609) #228866

Merged

This was referenced Jul 21, 2025

[8.19] Migrate from individual OTel packages to EDOT (#226609) #228867

Merged

[8.18] Migrate from individual OTel packages to EDOT (#226609) #228872

Merged

kibanamachine added the v9.1.0 label Jul 21, 2025

kibanamachine added the v9.0.4 label Jul 22, 2025

Bluefinger pushed a commit to Bluefinger/kibana that referenced this pull request Jul 22, 2025

Migrate from individual OTel packages to EDOT (elastic#226609)

7c29858

mistic added v8.18.5 and removed v9.0.4 v8.18.4 labels Jul 22, 2025

kertal pushed a commit to kertal/kibana that referenced this pull request Jul 25, 2025

Migrate from individual OTel packages to EDOT (elastic#226609)

e067f98

Conversation

afharo commented Jul 4, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

afharo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kibanamachine commented Jul 7, 2025

Self Checklist

Uh oh!

mistic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pickypg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trentm Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

afharo Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efegurkan left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Jul 21, 2025

💔 Some backports could not be created

Manual backport

Questions ?

Uh oh!

afharo commented Jul 21, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

afharo commented Jul 4, 2025 •

edited by kibanamachine

Loading

trentm Jul 8, 2025 •

edited

Loading

afharo Jul 9, 2025 •

edited

Loading