Agent builder exporter

[machadoum]

Depends on elastic/elasticsearch#147811

Resolves https://github.com/elastic/search-team/issues/14191
Resolves https://github.com/elastic/search-team/issues/14190

Summary

This PR adds a dedicated OpenTelemetry trace export path for Agent Builder inference spans, so they can land in Elasticsearch under a distinct dataset (agent_builder) while reusing Kibana’s Elasticsearch connection (auth, TLS, transport). Generic tracing can remain sampled down without silently dropping inference work: inference spans identified via kibana.inference.tracing baggage are preserved through sampling so downstream processors can export them on a copy.

Why: Agent Builder observability needs reliable inference-span export and routing into its own traces data stream, aligned with Elasticsearch’s native OTLP traces ingestion (/_otlp/v1/traces from elastic/elasticsearch#147811).

Architecture

• @kbn/tracing — InferencePreservingSampler wraps the existing ParentBasedSampler in init_tracing.ts. Non-inference spans pass through unchanged. Inference spans upgrade NOT_RECORD to RECORD (without forcing SAMPLED) so domain processors can clone and set SAMPLED for their pipeline.
• @kbn/inference-tracing — ElasticsearchOtlpExporter serializes spans with @opentelemetry/otlp-transformer and POSTs OTLP-protobuf to ES /_otlp/v1/traces via the ES client transport (same connection settings as Kibana).
• should_track_span.ts / isInferenceSpan() extracts “should track” logic from BaseInferenceSpanProcessor.onStart; shared by BaseInferenceSpanProcessor and Agent Builder.
• agent_builder — AgentBuilderSpanProcessor copies eligible spans, forces SAMPLED on the copy for export, adds data_stream.dataset: agent_builder, and feeds a BatchSpanProcessor. Enabled state comes from an LRU-backed saved-objects check against AGENT_BUILDER_EXPERIMENTAL_FEATURES_SETTING_ID.
• register_tracing.ts chooses OTLPTraceExporter when a trace URL is configured, otherwise ElasticsearchOtlpExporter; registers via LateBindingSpanProcessor.register().
• Lifecycle: exporter registration in plugin.ts start(), async teardown in stop().

flowchart LR
  subgraph tracing["Global tracing"]
    S["InferencePreservingSampler"]
    P["Span processors"]
  end
  subgraph ab["Agent Builder"]
    AB["AgentBuilderSpanProcessor"]
    BSP["BatchSpanProcessor"]
    E["OTLP URL exporter OR ElasticsearchOtlpExporter"]
  end
  S --> P
  P --> AB
  AB --> BSP --> E --> ES["Elasticsearch traces"]

Loading

Package exports

• @kbn/tracing: InferencePreservingSampler (wired in init_tracing.ts).
• @kbn/inference-tracing: ElasticsearchOtlpExporter, isInferenceSpan / should_track_span helpers (via new module), existing processors updated to use shared inference detection.

How to test

1. Make sure your ES instance has the otlp endpoint enabled Enable OTLP logs and traces by default elasticsearch#147811
2. Enable agent builder experimental setting agentBuilder:experimentalFeatures
3. Enable Kibana tracing telemetry.tracing.enabled: false
4. Run a query through Agent Builder.
5. Confirm Agent Builder spans exist under .ds-traces-agent_builder*.

If the evals plugin is enabled xpack.evals.enabled: true you will see a view traces button in agent builder the reasoning panel.

Checklist

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support (no product UI strings in this PR — server tracing/config only)
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios (not in this PR yet — follow-up)
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list (agent_builder.tracing.* added — cloud/docker follow-up required before merge)
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations. (no breaking public HTTP API changes)
• Flaky Test Runner was used on any tests changed (no tests changed in this PR yet)
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
• Review the backport guidelines and apply applicable backport:* labels. backport:skip — new feature; no backport planned via this PR.

Third-party Dependency

Purpose: Serializes ReadableSpan[] into OTLP-protobuf binary format (ProtobufTraceSerializer.serializeRequest()) so the ElasticsearchOtlpExporter can POST spans to ES's /_otlp/v1/traces via the ES client transport — no separate OTLP collector needed.

Justification: The existing OTLP exporters (exporter-trace-otlp-proto) bundle their own HTTP transport and can't route through the ES client. We need the serialization layer standalone to reuse Kibana's ES connection (auth, TLS).

Alternatives explored:

• Use exporter-trace-otlp-proto directly: Can't — it owns its HTTP connection and can't use the ES client transport. We do use it for the external OTLP URL path; the ES path needs standalone serialization.
• Implement serialization manually: OTLP-protobuf encoding is non-trivial (protobuf schema, resource/scope/span mapping, attribute encoding). Fragile and would drift from the spec.

Existing dependencies: Already a direct dep in root package.json (0.214.0) and resolved in yarn.lock (3 versions). Transitively pulled by exporter-trace-otlp-proto, exporter-trace-otlp-http, exporter-logs-otlp-*, otlp-exporter-base, and sdk-node. No new package enters node_modules — this PR just adds a direct import from kbn-inference-tracing.

Identify risks

• See some risk examples

Risk	Severity	Mitigation
Global sampling interaction — InferencePreservingSampler changes when inference spans are recorded vs dropped relative to parent-based sampling.	Medium	Scoped to baggage-marked inference spans; non-inference spans unchanged. Review trace volume and cardinality in staging; validate alongside inference and platform tracing owners.
Elasticsearch OTLP dependency — native /_otlp/v1/traces must be available and compatible for the fallback exporter path; misconfiguration could mean lost or failed exports.	Medium	Depends on elastic/elasticsearch#147811; test both OTLP URL and ES-transport paths; monitor exporter errors and ES responses.

[trentm]

@machadoum From only reading the code (I haven't run this) I think this looks good.

[trentm]

This span attribute is dropped before export for this span processor.
Note that the other span processors (and their exported spans) will have this span attribute.
That's probably fine, but will you want a more self-explanatory attribute name?
Also perhaps name using dot-separators as is more common in OTel usage.
Is there a naming pattern from other span attributes added by other in-Kibana self-instrumentation?

If this attribute really isn't wanted, then you would need something like a custom SpanProcessor that wrapped all the other processors added via LateBindingSpanProcessor.get().register(...) to have them drop the attribute.

[machadoum]

That makes sense. I shamelessly copied this code from the base_inference_span_processor.ts.

kibana/x-pack/platform/packages/shared/kbn-inference-tracing/src/base_inference_span_processor.ts

Lines 30 to 33 in 3cdecbf

	if (shouldTrack) {
	span.setAttribute('_should_track', true);
	this.delegate.onStart(span, parentContext);
	}

Could we address it on another PR, since it already affects other span processors.

[trentm]

I'm fine with it. But I'm not a privileged reviewer in this repo. :)

[trentm]

The OTel Node.js usage looks sane to me.

[SrdjanLL]

There's a note that this dependency is intended for internal use only, https://www.npmjs.com/package/@opentelemetry/otlp-transformer.

Might be worth probing out whether this is an actual risk or we can expect this to stabilize?

[machadoum]

@trentm Do you think that using otlp-transformer for serializing the spam is a risk?

    const serialized = ProtobufTraceSerializer.serializeRequest(spans);

https://github.com/elastic/kibana/pull/265290/changes#diff-aebb43311f0beef2700524e5d3da07aeef6950094a687c9a48b1241aa2d1d0beR37-R38

There is no public alternative — OTel JS provides no other way to serialize ReadableSpan[] into OTLP protobuf/JSON. The only options are: use this package, or hand-roll protobuf serialization. Which sounds like a much worse alternative. And very OTLP exporter in the OTel JS ecosystem depends on it — exporter-trace-otlp-proto, exporter-trace-otlp-http, exporter-logs-otlp-grpc, otlp-exporter-base, sdk-node.

[trentm]

OTel JS is just not fully at 1.x (aka "stable") yet. One component that isn't yet "stable" is its OTLP exporters (of which the otlp-transformer package is a part).

No, I don't think having otlp-transformer as a transitive dep is a "risk".
These packages are what every user of OTel JS is using to export OTLP data.

However, why was this dep explicitly added? I don't see it used explicitly anywhere.
I see export { ElasticsearchOtlpExporter } from './src/elasticsearch_otlp_exporter'; in the PR. Is there a new "elasticsearch_otlp_exporter.ts" file that hasn't been added to this PR?

Note that the require buildkite check is failing.

[machadoum]

However, why was this dep explicitly added? I don't see it used explicitly anywhere.

Sorry hooky mistake. I failed to push the file after it was moved during a code review improvement. Sean also caught the same problem here: #265290 (comment)

If you refresh the page it will be there.

[trentm]

Okay, I see.

As @SrdjanLL points out, otlp-transformer is primarily written as an internal package that is shared between OTel JS's OTLP exporters for the different signals (traces, metrics, logs). Its interface won't change without a noted breaking change. Because it is still 0.x, per semver, that "breaking" change version will be a new minor. So, for example, 0.214.0 -> 0.215.0 will potentially be breaking and you'd need to watch for that. Type changes, if any, would most likely automatically point out a breaking issue.

[dmlemeshko]

x-pack/platform/test/tsconfig.json changes LGTM

[qn895]

AI Infra LGTM

[elena-shostak]

@opentelemetry/otlp-transformer dependency LGTM

[seanstory]

LGTM!

[afharo]

I knew about /_otlp/v1/metrics. Is traces now supported as well?

[machadoum]

Yes, it was recently merged together with logs elastic/elasticsearch#147811

[kibanamachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 834e2f3

Failed CI Steps

• Jest Tests #2
• affected Scout: [ platform / spaces ] plugin

Test Failures

• [job] [logs] Jest Tests #2 / useWorkflowExecutions should pass executionTypes filter to query params

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/inference-tracing	25	30	+5
@kbn/tracing	21	28	+7
total			+12

Unknown metric groups

API count

id	before	after	diff
@kbn/inference-tracing	29	34	+5
@kbn/tracing	23	31	+8
total			+13

History

• 💔 Build #438655 failed a354514
• 💔 Build #438530 failed 3981cbe
• 💚 Build #436839 succeeded 7817a27
• 💔 Build #436828 failed 0133bb8
• 💔 Build #436477 failed f35789b
• 💔 Build #436468 failed 734bd85

cc @machadoum