[Inference] Move tracing config to @kbn/tracing by dgieselaar · Pull Request #225417 · elastic/kibana

dgieselaar · 2025-06-26T09:10:59Z

This moves the initialization of phoenix/langfuse processors+exporters to @kbn/tracing. Previously the Inference plugin initialized and registered them.

Why

We need to use the tracing spans outside of the Kibana application as well - for instance when running evaluations or other CLI-based workflows. This doesn't work if Kibana server manages the exporter, as it has no visibility in spans created outside of the process.

The configuration path moves from xpack.inference.tracing.exporter.{phoenix,langfuse} to telemetry.tracing.exporter.{phoenix,langfuse}, without backwards compatibility.

Notes

These are registered as exporters but implemented as processors. The reason is that they need to use baggage to make sure all spans created in the context of an LLM workflow get marked and exported. So exporters are somewhat of a misnomer but it seems overkill to me to register both exporters and processors separately.
I'm backporting to 8.19 as well, given A) the risk of merge conflicts, B) the fact that this is disabled by default.
I'm assuming this is not a breaking change because this is not on by default and not document. If it is a breaking change I would need some guidance on how to use the
I used o3 to write documentation for @kbn/tracing and @kbn/inference-tracing.

dgieselaar · 2025-06-28T19:51:01Z

@elasticmachine merge upstream

rudolf

If any clusters have values set for xpack.inference.tracing.exporter.{phoenix,langfuse} upgrading will cause their cluster to fail to start. It would be safer to deprecate these settings.

afharo

LGTM! I think that the breaking change is assumable since the original configs were introduced in 8.19 and 9.1, and they haven't been released yet. Let's make sure to backport these changes before the releases.

nit: I think that we should start grouping OTel packages in a directory (platform/packags/shared/opentelemetry/*). WDYT?

afharo · 2025-06-30T11:15:48Z

src/core/packages/http/router-server-internal/src/router.ts

+         * and is temporary until we fully migrate to [OpenTelemetry
+         * tracing](https://github.com/elastic/kibana/issues/220914).
+         */
+        const ctx = propagation.extract(context.active(), request.headers, defaultTextMapGetter);


I wonder if it's worth moving this to a previous handler. This way, the context can be reused for any previous lifecycle steps like authentication/authorization, validation, etc.

I don't think that it should block the PR, though. We can move it in a follow-up PR.

WDYT?

yes makes sense: #225817

sorry, replied to the wrong comment - wdym with:

I wonder if it's worth moving this to a previous handler.

It should definitely happen as early as possible. where would I need to put it then?

I was thinking about something similar to

kibana/src/core/packages/http/server-internal/src/http_server.ts

Line 525 in c3a26c3

private setupRequestStateAssignment(

.

Maybe we can store the context in the request.app in a similar fashion (only if it makes sense).

ah that does not actually work I think, you need to be able to wrap a callback

afharo · 2025-06-30T11:25:21Z

src/platform/packages/shared/kbn-opentelemetry-attributes/src/resource_attributes/index.ts

+export const RESOURCE_ATTR_SERVICE_NODE_NAME = 'service.node.name';
+export const RESOURCE_ATTR_SERVICE_ENVIRONMENT = 'service.environment';


Do these conform to SemConv?

AFAIK, service.environment should be deployment.environment.name: https://opentelemetry.io/docs/specs/semconv/resource/deployment-environment/

Based on the definition of ServiceNodeName, should service.node.name be `container.name?

cheers, will rename!

I've changed to service.instance.id and service.namespace (which are both in development, so I'm defining them in @kbn/opentelemetry-attributes.

src/platform/packages/shared/kbn-opentelemetry-utils/src/flatten_to_attributes.ts

afharo · 2025-06-30T11:28:05Z

src/platform/packages/shared/kbn-opentelemetry-utils/src/from_traceparent.ts

+// Helpers to check if ID is not all zeros (spec prohibits all-zero ids)
+const isAllZeros = (str: string) => /^0+$/.test(str);
+
+export function fromTraceparent(traceparent: string): SpanContext | undefined {


😮 I'm surprised that this helper is not provided in the @opentelemetry libraries

me too, I'll have another look to see if I haven't missed anything.

src/platform/packages/shared/kbn-tracing-config/README.md

afharo · 2025-06-30T11:30:33Z

src/platform/packages/shared/kbn-tracing-config/kibana.jsonc

+  "id": "@kbn/tracing-config",
+  "owner": "@elastic/kibana-core",
+  "group": "platform",
+  "visibility": "shared"


nit: I wonder if this should be "private"

tbh I really don't get the point of private packages. Why should everything in x-pack/platform be able to import it, but nothing in x-pack/solutions/observability? but a topic for another day.. 😄

src/cli/apm.js

…processors

…ana into move-inference-processors

dgieselaar · 2025-06-30T14:47:25Z

@elasticmachine merge upstream

elasticmachine · 2025-06-30T16:50:31Z

💚 Build Succeeded

Buildkite Build
Commit: e747130

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/inference-tracing`	17	19	+2
`@kbn/inference-tracing-config`	-	1	+1
`@kbn/opentelemetry-attributes`	-	9	+9
`@kbn/opentelemetry-utils`	-	7	+7
`@kbn/telemetry-config`	0	1	+1
total			+20

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/inference-common`	12	10	-2
`@kbn/inference-tracing`	1	2	+1
`@kbn/inference-tracing-config`	-	1	+1
total			-0

Unknown metric groups

API count

id	before	after	diff
`@kbn/inference-tracing`	20	22	+2
`@kbn/inference-tracing-config`	-	12	+12
`@kbn/opentelemetry-attributes`	-	9	+9
`@kbn/opentelemetry-utils`	-	9	+9
`@kbn/telemetry-config`	7	4	-3
`@kbn/tracing-config`	-	6	+6
total			+35

ESLint disabled line counts

id	before	after	diff
`@kbn/opentelemetry-utils`	-	1	+1

Total ESLint disabled count

id	before	after	diff
`@kbn/opentelemetry-utils`	-	1	+1

History

💔 Build #314033 failed 6544e5f
💚 Build #313772 succeeded 79bffdf
💚 Build #313434 succeeded 8d56c51
💚 Build #313053 succeeded 83742ed

peteharverson

AI infra owned changes LGTM

kibanamachine · 2025-07-01T08:29:06Z

Starting backport for target branches: 8.19, 9.1

https://github.com/elastic/kibana/actions/runs/15994035227

This moves the initialization of phoenix/langfuse processors+exporters to @kbn/tracing. Previously the Inference plugin initialized and registered them. ## Why We need to use the tracing spans outside of the Kibana application as well - for instance when running evaluations or other CLI-based workflows. This doesn't work if Kibana server manages the exporter, as it has no visibility in spans created outside of the process. **The configuration path moves from `xpack.inference.tracing.exporter.{phoenix,langfuse}` to `telemetry.tracing.exporter.{phoenix,langfuse}`, without backwards compatibility.** ## Notes - These are registered as _exporters_ but implemented as _processors_. The reason is that they need to use baggage to make sure all spans created in the context of an LLM workflow get marked and exported. So `exporters` are somewhat of a misnomer but it seems overkill to me to register both `exporters` and `processors` separately. - I'm backporting to 8.19 as well, given A) the risk of merge conflicts, B) the fact that this is disabled by default. - I'm assuming this is not a breaking change because this is not on by default and not document. If it _is_ a breaking change I would need some guidance on how to use the - I used o3 to write documentation for @kbn/tracing and @kbn/inference-tracing. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit cbd3a97)

kibanamachine · 2025-07-01T08:35:41Z

💔 Some backports could not be created

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts You might need to backport the following PRs to 8.19: - [ska][x-pack] Move painless_lab, upgrade_assistant and reporting_management functional tests (#225829) - Update codeowners for observability ESQL_RECOMMENDED_QUERIES (#225429) - [ska][x-pack] Move canvas, watcher and dev_tools functional tests - [ska] relocate 'shared/lib/security' helpers to 'osquery/cypress' dir (#225438) - [ska] relocate api_integration_basic & defend_workflows_cypress tests (#225374) - Load huggingface content datasets (#224543) - [Lens] Add internal CRUD api routes (#223296) - Move apps/security and apps/spaces tests under platform
✅	9.1

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 225417

Questions ?

Please refer to the Backport tool documentation

) # Backport This will backport the following commits from `main` to `9.1`: - [[Inference] Move tracing config to @kbn/tracing (#225417)](#225417)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

This moves the initialization of phoenix/langfuse processors+exporters to @kbn/tracing. Previously the Inference plugin initialized and registered them. ## Why We need to use the tracing spans outside of the Kibana application as well - for instance when running evaluations or other CLI-based workflows. This doesn't work if Kibana server manages the exporter, as it has no visibility in spans created outside of the process. **The configuration path moves from `xpack.inference.tracing.exporter.{phoenix,langfuse}` to `telemetry.tracing.exporter.{phoenix,langfuse}`, without backwards compatibility.** ## Notes - These are registered as _exporters_ but implemented as _processors_. The reason is that they need to use baggage to make sure all spans created in the context of an LLM workflow get marked and exported. So `exporters` are somewhat of a misnomer but it seems overkill to me to register both `exporters` and `processors` separately. - I'm backporting to 8.19 as well, given A) the risk of merge conflicts, B) the fact that this is disabled by default. - I'm assuming this is not a breaking change because this is not on by default and not document. If it _is_ a breaking change I would need some guidance on how to use the - I used o3 to write documentation for @kbn/tracing and @kbn/inference-tracing. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit cbd3a97) # Conflicts: # .github/CODEOWNERS # src/core/packages/http/router-server-internal/src/router.ts

dgieselaar · 2025-07-02T14:07:23Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…6224) # Backport This will backport the following commits from `main` to `8.19`: - [[Inference] Move tracing config to @kbn/tracing (#225417)](#225417)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

dgieselaar added release_note:skip Skip the PR/issue when compiling release notes backport:version Backport to applied version labels v9.1.0 v8.19.0 v9.2.0 labels Jun 26, 2025

[Inference] Move tracing config to @kbn/tracing

bb55e90

dgieselaar force-pushed the move-inference-processors branch from 844e9ee to bb55e90 Compare June 26, 2025 09:26

Add documentation

0841cad

dgieselaar marked this pull request as ready for review June 26, 2025 12:54

dgieselaar requested review from a team as code owners June 26, 2025 12:54

kibanamachine and others added 2 commits June 26, 2025 13:06

[CI] Auto-commit changed files from 'node scripts/generate codeowners'

83742ed

Propagate traceparent headers

8d56c51

Merge branch 'main' into move-inference-processors

79bffdf

rudolf requested a review from afharo June 30, 2025 10:10

rudolf reviewed Jun 30, 2025

View reviewed changes

afharo approved these changes Jun 30, 2025

View reviewed changes

dgieselaar mentioned this pull request Jun 30, 2025

OpenTelemetry: move OTel-related packages into shared directory #225817

Open

dgieselaar and others added 4 commits June 30, 2025 14:06

Merge branch 'main' of github.com:elastic/kibana into move-inference-…

4c5a46d

…processors

Review feedback

3289d96

Merge branch 'move-inference-processors' of github.com:dgieselaar/kib…

71064aa

…ana into move-inference-processors

[CI] Auto-commit changed files from 'node scripts/notice'

6544e5f

Merge branch 'main' into move-inference-processors

e747130

peteharverson approved these changes Jul 1, 2025

View reviewed changes

dgieselaar merged commit cbd3a97 into elastic:main Jul 1, 2025
13 checks passed

dgieselaar deleted the move-inference-processors branch July 1, 2025 08:28

kibanamachine mentioned this pull request Jul 1, 2025

[9.1] [Inference] Move tracing config to @kbn/tracing (#225417) #225953

Merged

kibanamachine mentioned this pull request Jul 2, 2025

[Streams] Add Streamlang package #224915

Merged

dgieselaar mentioned this pull request Jul 2, 2025

[8.19] [Inference] Move tracing config to @kbn/tracing (#225417) #226224

Merged

		export const RESOURCE_ATTR_SERVICE_NODE_NAME = 'service.node.name';
		export const RESOURCE_ATTR_SERVICE_ENVIRONMENT = 'service.environment';

Conversation

dgieselaar commented Jun 26, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Notes

Uh oh!

dgieselaar commented Jun 28, 2025

Uh oh!

rudolf left a comment

Choose a reason for hiding this comment

Uh oh!

afharo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

afharo Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dgieselaar commented Jun 30, 2025

Uh oh!

elasticmachine commented Jun 30, 2025

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Public APIs missing exports

API count

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

peteharverson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kibanamachine commented Jul 1, 2025

Uh oh!

kibanamachine commented Jul 1, 2025

💔 Some backports could not be created

Manual backport

Questions ?

Uh oh!

dgieselaar commented Jul 2, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Comments

dgieselaar commented Jun 26, 2025 •

edited by kibanamachine

Loading

afharo Jun 30, 2025 •

edited

Loading