Enable OTLP logs and traces by default#147811
Merged
Merged
Conversation
Register OTLP logs and traces endpoints without feature flag gates so they are available in release builds.
elasticsearchmachine
added
Team:StorageEngine
external-contributor
Collaborator
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
Collaborator
|
Hi @felixbarny, I've created a changelog YAML for you. |
Contributor
🔍 Preview links for changed docs⏳ Building and deploying preview... View progress This comment will be updated with preview links when the build is complete. |
Contributor
ℹ️ Important: Docs version tagging👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version. We use applies_to tags to mark version-specific features and changes. Expand for a quick overviewWhen to use applies_to tags:✅ At the page level to indicate which products/deployments the content applies to (mandatory) What NOT to do:❌ Don't remove or replace information that applies to an older version 🤔 Need help?
|
9 tasks
Member
Author
|
This is ready now. @kkrik-es could you have a look, please? |
kkrik-es
approved these changes
May 5, 2026
machadoum
added a commit
to elastic/kibana
that referenced
this pull request
May 6, 2026
Depends on elastic/elasticsearch#147811 Resolves elastic/search-team#14191 Resolves elastic/search-team#14190 ## Summary This PR adds a dedicated OpenTelemetry trace export path for **Agent Builder inference spans**, so they can land in Elasticsearch under a distinct dataset (`agent_builder`) while reusing Kibana’s Elasticsearch connection (auth, TLS, transport). Generic tracing can remain sampled down without silently dropping inference work: inference spans identified via `kibana.inference.tracing` baggage are preserved through sampling so downstream processors can export them on a copy. **Why:** Agent Builder observability needs reliable inference-span export and routing into its own traces data stream, aligned with Elasticsearch’s native OTLP traces ingestion (`/_otlp/v1/traces` from elastic/elasticsearch#147811). ## Architecture - **`@kbn/tracing` — `InferencePreservingSampler`** wraps the existing `ParentBasedSampler` in `init_tracing.ts`. Non-inference spans pass through unchanged. Inference spans upgrade `NOT_RECORD` to `RECORD` (without forcing `SAMPLED`) so domain processors can clone and set `SAMPLED` for their pipeline. - **`@kbn/inference-tracing` — `ElasticsearchOtlpExporter`** serializes spans with `@opentelemetry/otlp-transformer` and POSTs OTLP-protobuf to ES `/_otlp/v1/traces` via the ES client transport (same connection settings as Kibana). - **`should_track_span.ts` / `isInferenceSpan()`** extracts “should track” logic from `BaseInferenceSpanProcessor.onStart`; shared by `BaseInferenceSpanProcessor` and Agent Builder. - **`agent_builder` — `AgentBuilderSpanProcessor`** copies eligible spans, forces `SAMPLED` on the copy for export, adds `data_stream.dataset: agent_builder`, and feeds a `BatchSpanProcessor`. Enabled state comes from an LRU-backed saved-objects check against `AGENT_BUILDER_EXPERIMENTAL_FEATURES_SETTING_ID`. - **`register_tracing.ts`** chooses **`OTLPTraceExporter`** when a trace URL is configured, otherwise **`ElasticsearchOtlpExporter`**; registers via `LateBindingSpanProcessor.register()`. - **Lifecycle:** exporter registration in `plugin.ts` `start()`, async teardown in `stop()`. ```mermaid flowchart LR subgraph tracing["Global tracing"] S["InferencePreservingSampler"] P["Span processors"] end subgraph ab["Agent Builder"] AB["AgentBuilderSpanProcessor"] BSP["BatchSpanProcessor"] E["OTLP URL exporter OR ElasticsearchOtlpExporter"] end S --> P P --> AB AB --> BSP --> E --> ES["Elasticsearch traces"] ``` ### Package exports - **`@kbn/tracing`:** `InferencePreservingSampler` (wired in `init_tracing.ts`). - **`@kbn/inference-tracing`:** `ElasticsearchOtlpExporter`, `isInferenceSpan` / `should_track_span` helpers (via new module), existing processors updated to use shared inference detection. ## How to test 1. Make sure your ES instance has the otlp endpoint enabled elastic/elasticsearch#147811 2. Enable agent builder experimental setting `agentBuilder:experimentalFeatures` 3. Enable Kibana tracing `telemetry.tracing.enabled: false` 4. Run a query through Agent Builder. 5. Confirm Agent Builder spans exist under `.ds-traces-agent_builder*`. If the evals plugin is enabled `xpack.evals.enabled: true` you will see a view traces button in agent builder the reasoning panel. <img width="787" height="277" alt="Screenshot 2026-04-30 at 11 02 04" src="https://github.com/user-attachments/assets/9a891a63-436c-4302-af48-3027406c8d1f" /> <img width="624" height="805" alt="Screenshot 2026-04-30 at 11 02 06" src="https://github.com/user-attachments/assets/d1493730-8ef4-4c66-a50d-8237a24d0180" /> ### Checklist Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) _(no product UI strings in this PR — server tracing/config only)_ - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios **(not in this PR yet — follow-up)** - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) **(`agent_builder.tracing.*` added — cloud/docker follow-up required before merge)** - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. _(no breaking public HTTP API changes)_ - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed _(no tests changed in this PR yet)_ - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. **`backport:skip`** — new feature; no backport planned via this PR. ### Third-party Dependency **Purpose:** Serializes ReadableSpan[] into OTLP-protobuf binary format (ProtobufTraceSerializer.serializeRequest()) so the ElasticsearchOtlpExporter can POST spans to ES's /_otlp/v1/traces via the ES client transport — no separate OTLP collector needed. **Justification:** The existing OTLP exporters (exporter-trace-otlp-proto) bundle their own HTTP transport and can't route through the ES client. We need the serialization layer standalone to reuse Kibana's ES connection (auth, TLS). **Alternatives explored:** * Use exporter-trace-otlp-proto directly: Can't — it owns its HTTP connection and can't use the ES client transport. We do use it for the external OTLP URL path; the ES path needs standalone serialization. * Implement serialization manually: OTLP-protobuf encoding is non-trivial (protobuf schema, resource/scope/span mapping, attribute encoding). Fragile and would drift from the spec. **Existing dependencies:** Already a direct dep in root package.json (0.214.0) and resolved in yarn.lock (3 versions). Transitively pulled by exporter-trace-otlp-proto, exporter-trace-otlp-http, exporter-logs-otlp-*, otlp-exporter-base, and sdk-node. No new package enters node_modules — this PR just adds a direct import from kbn-inference-tracing. ### Identify risks - [x] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) | Risk | Severity | Mitigation | | --- | --- | --- | | **Global sampling interaction** — `InferencePreservingSampler` changes when inference spans are recorded vs dropped relative to parent-based sampling. | Medium | Scoped to baggage-marked inference spans; non-inference spans unchanged. Review trace volume and cardinality in staging; validate alongside inference and platform tracing owners. | | **Elasticsearch OTLP dependency** — native `/_otlp/v1/traces` must be available and compatible for the fallback exporter path; misconfiguration could mean lost or failed exports. | Medium | Depends on elastic/elasticsearch#147811; test both OTLP URL and ES-transport paths; monitor exporter errors and ES responses. | --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
ersin-erdal
pushed a commit
to ersin-erdal/kibana
that referenced
this pull request
May 6, 2026
Depends on elastic/elasticsearch#147811 Resolves elastic/search-team#14191 Resolves elastic/search-team#14190 ## Summary This PR adds a dedicated OpenTelemetry trace export path for **Agent Builder inference spans**, so they can land in Elasticsearch under a distinct dataset (`agent_builder`) while reusing Kibana’s Elasticsearch connection (auth, TLS, transport). Generic tracing can remain sampled down without silently dropping inference work: inference spans identified via `kibana.inference.tracing` baggage are preserved through sampling so downstream processors can export them on a copy. **Why:** Agent Builder observability needs reliable inference-span export and routing into its own traces data stream, aligned with Elasticsearch’s native OTLP traces ingestion (`/_otlp/v1/traces` from elastic/elasticsearch#147811). ## Architecture - **`@kbn/tracing` — `InferencePreservingSampler`** wraps the existing `ParentBasedSampler` in `init_tracing.ts`. Non-inference spans pass through unchanged. Inference spans upgrade `NOT_RECORD` to `RECORD` (without forcing `SAMPLED`) so domain processors can clone and set `SAMPLED` for their pipeline. - **`@kbn/inference-tracing` — `ElasticsearchOtlpExporter`** serializes spans with `@opentelemetry/otlp-transformer` and POSTs OTLP-protobuf to ES `/_otlp/v1/traces` via the ES client transport (same connection settings as Kibana). - **`should_track_span.ts` / `isInferenceSpan()`** extracts “should track” logic from `BaseInferenceSpanProcessor.onStart`; shared by `BaseInferenceSpanProcessor` and Agent Builder. - **`agent_builder` — `AgentBuilderSpanProcessor`** copies eligible spans, forces `SAMPLED` on the copy for export, adds `data_stream.dataset: agent_builder`, and feeds a `BatchSpanProcessor`. Enabled state comes from an LRU-backed saved-objects check against `AGENT_BUILDER_EXPERIMENTAL_FEATURES_SETTING_ID`. - **`register_tracing.ts`** chooses **`OTLPTraceExporter`** when a trace URL is configured, otherwise **`ElasticsearchOtlpExporter`**; registers via `LateBindingSpanProcessor.register()`. - **Lifecycle:** exporter registration in `plugin.ts` `start()`, async teardown in `stop()`. ```mermaid flowchart LR subgraph tracing["Global tracing"] S["InferencePreservingSampler"] P["Span processors"] end subgraph ab["Agent Builder"] AB["AgentBuilderSpanProcessor"] BSP["BatchSpanProcessor"] E["OTLP URL exporter OR ElasticsearchOtlpExporter"] end S --> P P --> AB AB --> BSP --> E --> ES["Elasticsearch traces"] ``` ### Package exports - **`@kbn/tracing`:** `InferencePreservingSampler` (wired in `init_tracing.ts`). - **`@kbn/inference-tracing`:** `ElasticsearchOtlpExporter`, `isInferenceSpan` / `should_track_span` helpers (via new module), existing processors updated to use shared inference detection. ## How to test 1. Make sure your ES instance has the otlp endpoint enabled elastic/elasticsearch#147811 2. Enable agent builder experimental setting `agentBuilder:experimentalFeatures` 3. Enable Kibana tracing `telemetry.tracing.enabled: false` 4. Run a query through Agent Builder. 5. Confirm Agent Builder spans exist under `.ds-traces-agent_builder*`. If the evals plugin is enabled `xpack.evals.enabled: true` you will see a view traces button in agent builder the reasoning panel. <img width="787" height="277" alt="Screenshot 2026-04-30 at 11 02 04" src="https://github.com/user-attachments/assets/9a891a63-436c-4302-af48-3027406c8d1f" /> <img width="624" height="805" alt="Screenshot 2026-04-30 at 11 02 06" src="https://github.com/user-attachments/assets/d1493730-8ef4-4c66-a50d-8237a24d0180" /> ### Checklist Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) _(no product UI strings in this PR — server tracing/config only)_ - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios **(not in this PR yet — follow-up)** - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) **(`agent_builder.tracing.*` added — cloud/docker follow-up required before merge)** - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. _(no breaking public HTTP API changes)_ - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed _(no tests changed in this PR yet)_ - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. **`backport:skip`** — new feature; no backport planned via this PR. ### Third-party Dependency **Purpose:** Serializes ReadableSpan[] into OTLP-protobuf binary format (ProtobufTraceSerializer.serializeRequest()) so the ElasticsearchOtlpExporter can POST spans to ES's /_otlp/v1/traces via the ES client transport — no separate OTLP collector needed. **Justification:** The existing OTLP exporters (exporter-trace-otlp-proto) bundle their own HTTP transport and can't route through the ES client. We need the serialization layer standalone to reuse Kibana's ES connection (auth, TLS). **Alternatives explored:** * Use exporter-trace-otlp-proto directly: Can't — it owns its HTTP connection and can't use the ES client transport. We do use it for the external OTLP URL path; the ES path needs standalone serialization. * Implement serialization manually: OTLP-protobuf encoding is non-trivial (protobuf schema, resource/scope/span mapping, attribute encoding). Fragile and would drift from the spec. **Existing dependencies:** Already a direct dep in root package.json (0.214.0) and resolved in yarn.lock (3 versions). Transitively pulled by exporter-trace-otlp-proto, exporter-trace-otlp-http, exporter-logs-otlp-*, otlp-exporter-base, and sdk-node. No new package enters node_modules — this PR just adds a direct import from kbn-inference-tracing. ### Identify risks - [x] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) | Risk | Severity | Mitigation | | --- | --- | --- | | **Global sampling interaction** — `InferencePreservingSampler` changes when inference spans are recorded vs dropped relative to parent-based sampling. | Medium | Scoped to baggage-marked inference spans; non-inference spans unchanged. Review trace volume and cardinality in staging; validate alongside inference and platform tracing owners. | | **Elasticsearch OTLP dependency** — native `/_otlp/v1/traces` must be available and compatible for the fallback exporter path; misconfiguration could mean lost or failed exports. | Medium | Depends on elastic/elasticsearch#147811; test both OTLP URL and ES-transport paths; monitor exporter errors and ES responses. | --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
felixbarny
added a commit
to elastic/docs-content
that referenced
this pull request
May 7, 2026
#6219) Elasticsearch 9.5 adds OTLP/HTTP endpoints for logs (`/_otlp/v1/logs`) and traces (`/_otlp/v1/traces`) as tech preview, alongside the existing metrics endpoint. This PR documents the new endpoints and restructures the page accordingly. The OTLP endpoint page moves from the TSDS subtree to `manage-data/ingest/otlp-endpoint.md`, since it's no longer metrics-only. Cross-references in `quickstart-tsds.md`, `advanced-topics-tsds.md`, `time-series-data-stream-tsds.md`, and `manage-data/toc.yml` are repointed. The page is reframed to position the OTLP endpoint as a non-default ingestion path. Users on ECH or serverless should still use mOTLP; users on ECE, ECK, or self-managed should still use a gateway Collector with the Elasticsearch exporter. The OTLP endpoint is recommended for applications that export OTLP natively without running a Collector, and for lightweight dev setups (SDK to Elasticsearch). The Collector example switches from basic auth to API key auth using the `Authorization: ApiKey <api_key>` header. A new section explains how to create an API key with the minimum required privileges (`create_doc` and `auto_configure`), with per-signal index patterns that match the OTLP REST integration tests. Traces ingestion requires both `traces-*` and `logs-*` because span events are written to logs data streams. The page also documents the body map mapping mode for logs, opt-in via the `X-Elastic-Mapping-Mode: bodymap` header or the `elastic.mapping.mode` scope attribute. The Limitations section calls out that logs and traces can be duplicated on retry because acknowledgement is per-batch; metrics aren't affected because TSDS deduplicates documents based on their dimensions and timestamp. Depends on elastic/elasticsearch#147811 (feature-flag removal). Only merge after that lands. --------- Co-authored-by: Fabrizio Ferri-Benedetti <fabri.ferribenedetti@elastic.co> Co-authored-by: Kostas Krikellas <131142368+kkrik-es@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
alighahramani-alig
pushed a commit
to alighahramani-alig/elasticsearch
that referenced
this pull request
May 8, 2026
Register OTLP logs and traces endpoints without feature flag gates so they are available in release builds.
henningandersen
pushed a commit
to henningandersen/elasticsearch
that referenced
this pull request
May 11, 2026
Register OTLP logs and traces endpoints without feature flag gates so they are available in release builds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
otlp_tracesandotlp_logsfeature-flag gates so OTLP logs and traces endpoints are registered by default.Should be merged after these PRs are in: