diff --git a/manage-data/data-store/data-streams/tsds-ingest-otlp.md b/manage-data/data-store/data-streams/tsds-ingest-otlp.md index 00e09e0b1e..d19fe7c404 100644 --- a/manage-data/data-store/data-streams/tsds-ingest-otlp.md +++ b/manage-data/data-store/data-streams/tsds-ingest-otlp.md @@ -1,135 +1,20 @@ --- navigation_title: "OTLP/HTTP endpoint" +description: "Ingest OpenTelemetry metrics into time series data streams through the Elasticsearch OTLP/HTTP endpoint." applies_to: deployment: - self: - ece: - eck: + self: ga 9.2 + ece: ga + eck: ga products: - id: elasticsearch --- -# OTLP/HTTP endpoint +# Ingest metrics into a TSDS using the OTLP/HTTP endpoint -In addition to the ingestion of metrics data through the bulk API, -{{es}} offers an alternative way to ingest data through the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp). +{{es}} supports metrics ingestion through the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp). -The endpoint is available under `/_otlp/v1/metrics`. +For OpenTelemetry metrics, prefer the {{es}} OTLP/HTTP endpoint over the Bulk API because it's optimized for OTLP ingest performance. +It also simplifies setup by automatically creating [TSDS](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md) through built-in index templates, and deriving dimensions and metric mappings from OTLP metadata. -## Prerequisites - -An {{es}} cluster running {{stack}} 9.2 or later. The OTLP endpoint is available on ECE, ECK, and self-managed deployments that run this stack version. - -## Overview and deployment options - -:::{important} -The recommended approach for sending OpenTelemetry Protocol (OTLP) data depends on your deployment: - -- **ECH and {{serverless-short}}:** Use the [{{motlp}}](opentelemetry://reference/motlp.md) directly. -- **ECE, ECK, and self-managed:** Use the {{es}} OTLP endpoint described on this page, ideally through an OpenTelemetry Collector in [Gateway mode](elastic-agent://reference/edot-collector/config/default-config-standalone.md#gateway-mode). - -For details on the recommended way to set up OpenTelemetry-based data ingestion, refer to the [EDOT reference architecture](opentelemetry://reference/architecture/index.md). -::: - -Ingesting metrics data using the OTLP endpoint has the following advantages: - -* Improved ingestion performance, especially if the data contains many resource attributes. -* Simplified index mapping: - there's no need to manually create data streams, index templates, or define dimensions and metrics. - Metrics are dynamically mapped using the metadata included in the OTLP requests. - -:::{note} -{{es}} only supports [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp), -not [OTLP/gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc). -::: - -Don't send metrics from applications directly to the {{es}} OTLP endpoint, especially if there are many individual applications that periodically send a small amount of metrics. Instead, send data to an OpenTelemetry Collector first. This helps with handling many connections, and with creating bigger batches to improve ingestion performance. - -## How to send data to the OTLP endpoint - -To send data from an OpenTelemetry Collector to the {{es}} OTLP endpoint, -use the [`OTLP/HTTP` exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter). -This is an example configuration: - -```yaml -extensions: - basicauth/elasticsearch: - client_auth: - username: - password: -exporters: - otlphttp/elasticsearch-metrics: - endpoint: /_otlp - sending_queue: - enabled: true - sizer: bytes - queue_size: 50_000_000 # 50MB uncompressed - block_on_overflow: true - batch: - flush_timeout: 1s - min_size: 1_000_000 # 1MB uncompressed - max_size: 4_000_000 # 4MB uncompressed - auth: - authenticator: basicauth/elasticsearch -service: - extensions: [basicauth/elasticsearch] - pipelines: - metrics: - exporters: [otlphttp/elasticsearch-metrics] - receivers: ... -``` - -The supported options for `compression` are `gzip` (default value of the `OTLP/HTTP` exporter) and `none`. - -% TODO we might actually also support snappy and zstd, test and update accordingly) - -To track metrics in your custom application, -use the [OpenTelemetry language SDK](https://opentelemetry.io/docs/getting-started/dev/) of your choice. - -:::{note} -Only `encoding: proto` is supported, which the `OTLP/HTTP` exporter uses by default. -::: - -## Send data to different data streams - -By default, metrics are ingested into the `metrics-generic.otel-default` data stream. You can influence the target data stream by setting specific attributes on your data: - -- `data_stream.dataset` or `data_stream.namespace` in attributes, with the following order of precedence: data point attribute -> scope attribute -> resource attribute -- Otherwise, if the scope name contains `/receiver/`, `data_stream.dataset` is set to the receiver name. -- Otherwise, `data_stream.dataset` falls back to `generic` and `data_stream.namespace` falls back to `default`. - -The target data stream name is constructed as `metrics-${data_stream.dataset}.otel-${data_stream.namespace}`. - -## Configure histogram handling -```{applies_to} -stack: preview =9.3, ga 9.4+ -``` - -You can configure how OTLP histograms are mapped using the `xpack.otel_data.histogram_field_type` cluster setting. Valid values are: - - - `histogram` (default on {applies_to}`stack: preview =9.3`): Map histograms as T-Digests using the `histogram` field type - - `exponential_histogram` (default on {applies_to}`stack: ga 9.4+` {applies_to}`serverless: ga`): Map histograms as exponential histograms using the `exponential_histogram` field type - -The setting is dynamic and can be updated at runtime: - -```console -PUT /_cluster/settings -{ - "persistent" : { - "xpack.otel_data.histogram_field_type" : "exponential_histogram" - } -} -``` - -Because both `histogram` and `exponential_histogram` support [coerce](elasticsearch://reference/elasticsearch/mapping-reference/coerce.md), changing this setting dynamically does not risk mapping conflicts or ingestion failures. - -This setting only applies to metrics ingested using the [{{es}} OTLP endpoint](./tsds-ingest-otlp.md). -Documents ingested with the _bulk API (for example using the {{es}} exporter for the OpenTelemetry Collector) are not affected. - -## Limitations - -* Only the OTLP metrics endpoint (`/_otlp/v1/metrics`) is supported. - To ingest logs, traces, and profiles, use a distribution of the OpenTelemetry Collector that includes the [{{es}} exporter](opentelemetry://reference/edot-collector/components/elasticsearchexporter.md), - such as the [{{edot}} (EDOT) Collector](opentelemetry://reference/edot-collector/index.md). -* Histograms are only supported in delta temporality. Set the temporality preference to delta in your SDKs, or use the [`cumulativetodelta` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/cumulativetodeltaprocessor) to avoid cumulative histograms to be dropped. -* Exemplars are not supported. +For more details, refer to the [{{es}} OTLP/HTTP endpoint](/manage-data/ingest/otlp-endpoint.md) reference. diff --git a/manage-data/ingest/otlp-endpoint.md b/manage-data/ingest/otlp-endpoint.md new file mode 100644 index 0000000000..1272429448 --- /dev/null +++ b/manage-data/ingest/otlp-endpoint.md @@ -0,0 +1,235 @@ +--- +navigation_title: "OTLP/HTTP endpoint" +description: "Send metrics, logs, and traces directly to Elasticsearch through its native OTLP/HTTP endpoints, without running an OpenTelemetry Collector." +applies_to: + deployment: + self: ga 9.2 + ece: ga + eck: ga +products: + - id: elasticsearch +--- + +# {{es}} OTLP/HTTP endpoint + +In addition to ingesting data through the Bulk API, {{es}} accepts data through the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp). +The {{es}} OTLP/HTTP endpoint exposes three signal-specific paths: + +| Signal | Path | Availability | +| --- | --- | --- | +| Metrics | `/_otlp/v1/metrics` | {applies_to}`stack: ga 9.2+` | +| Logs | `/_otlp/v1/logs` | {applies_to}`stack: preview 9.5` | +| Traces | `/_otlp/v1/traces` | {applies_to}`stack: preview 9.5` | + +:::{important} +{{es}} only supports [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp), not [OTLP/gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc). +::: + +## When to use the {{es}} OTLP endpoint + +For most users, one of the following higher-level ingestion paths is recommended: + +| Deployment | Recommended ingestion path | +| --- | --- | +| {{ech}} and {{serverless-short}} | [{{motlp}}](opentelemetry://reference/motlp.md) | +| {{ece}}, {{eck}}, and self-managed | OpenTelemetry Collector in [Gateway mode](elastic-agent://reference/edot-collector/config/default-config-standalone.md#gateway-mode), using the [{{es}} exporter](opentelemetry://reference/edot-collector/components/elasticsearchexporter.md) | + +Use {{motlp}} if it's available in your deployment, even when an application can target the {{es}} OTLP endpoint directly. + +For an overview of the recommended OpenTelemetry-based ingestion architecture, refer to the [EDOT reference architecture](opentelemetry://reference/architecture/index.md). + +Use the {{es}} OTLP endpoint directly when one of the following applies: + +* You have an application that exports OTLP natively and you want it to send data to {{es}} without running an OpenTelemetry Collector. + For example, a lightweight development setup (SDK to {{es}}). +* You operate a self-managed gateway Collector and prefer the `OTLP/HTTP` exporter over the [{{es}} exporter](opentelemetry://reference/edot-collector/components/elasticsearchexporter.md). + +:::{warning} +Don't send telemetry from many individual applications directly to the {{es}} OTLP endpoint at the same time. +Send to an OpenTelemetry Collector first so it can absorb connection churn and batch records to improve ingestion performance. +::: + +## Advantages of OTLP ingest over Bulk API + +Compared to the Bulk API, ingesting through OTLP offers: + +* Improved ingestion performance, especially for payloads with many resource attributes. +* Simplified mapping: data streams, index templates, dimensions, and metrics are derived dynamically from OTLP metadata. + There's no need to set them up manually. + +## How to send data to the {{es}} OTLP endpoint + +### Create an API key + +Authenticate to the {{es}} OTLP endpoint with an API key. +Refer to the API key documentation for your deployment type for instructions on how to create one: + +* [{{es}} API keys](/deploy-manage/api-keys/elasticsearch-api-keys.md) (self-managed, {{ece}}, {{eck}}) +* [{{ech}} API keys](/deploy-manage/api-keys/elastic-cloud-api-keys.md) +* [{{ece}} API keys](/deploy-manage/api-keys/elastic-cloud-enterprise-api-keys.md) +* [{{serverless-short}} project API keys](/deploy-manage/api-keys/serverless-project-api-keys.md) + +The API key needs `create_doc` and `auto_configure` privileges on the data stream patterns it writes to. +`create_doc` allows writing documents without overwriting existing ones. +`auto_configure` allows the endpoint to create the target data streams on first write. + +The minimum index patterns depend on which signals you ingest: + +| Signals ingested | Required `names` patterns | +| --- | --- | +| Metrics | `metrics-*` | +| Logs | `logs-*` | +| Traces | `traces-*`, `logs-*` | +| All three | `metrics-*`, `logs-*`, `traces-*` | + +Traces ingestion also writes span events to `logs-*` data streams, so it requires both patterns. + +For example, an API key role descriptor that allows ingesting all three signals: + +```json +{ + "indices": [ + { + "names": ["logs-*", "metrics-*", "traces-*"], + "privileges": ["create_doc", "auto_configure"] + } + ] +} +``` + +### Configure an OpenTelemetry Collector + +To send data from an OpenTelemetry Collector to an {{es}} OTLP endpoint, configure the [`OTLP/HTTP` exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter): + +```yaml +exporters: + otlphttp/elasticsearch: + endpoint: /_otlp + headers: + Authorization: "ApiKey " + sending_queue: + enabled: true + sizer: bytes <1> + queue_size: 50_000_000 <2> + block_on_overflow: true + batch: <3> + flush_timeout: 1s + min_size: 1_000_000 + max_size: 4_000_000 +service: + pipelines: + logs: + exporters: [otlphttp/elasticsearch] + receivers: ... + traces: + exporters: [otlphttp/elasticsearch] + receivers: ... + metrics: + exporters: [otlphttp/elasticsearch] + receivers: ... +``` + +1. Sizes the queue and batches by uncompressed bytes. +2. Limits the queue to 50 MB of uncompressed data. + Increasing this value can absorb longer {{es}} outages or traffic bursts, but also increases Collector memory usage. +3. Controls the uncompressed batch size sent to {{es}}. + In this example, batches are sent at 1 MB and capped at 4 MB. + Larger batches reduce request overhead, but increase peak memory usage and the amount of data retried after a failed request. + +The exporter appends the signal-specific path (`/v1/logs`, `/v1/traces`, `/v1/metrics`) to the configured `endpoint`. + +These values are starting points for a gateway Collector. +Tune them for your workload and Collector resources. +They are local to each Collector instance and don't increase {{es}} ingest capacity. +If many applications need to send telemetry, scale out the gateway Collector instead of sending directly from each application. + +Supported `compression` values are `gzip` (the `OTLP/HTTP` exporter default) and `none`. + +To send data from a custom application, use the [OpenTelemetry language SDK](https://opentelemetry.io/docs/getting-started/dev/) of your choice and point its OTLP/HTTP exporter at the corresponding {{es}} OTLP endpoint path. + +:::{note} +Only `encoding: proto` is supported, which the `OTLP/HTTP` exporter uses by default. +::: + +## Routing to data streams + +By default, records are written to the following data streams: + +| Signal | Default data stream | +| --- | --- | +| Logs | `logs-generic.otel-default` | +| Traces | `traces-generic.otel-default` | +| Metrics | `metrics-generic.otel-default` | + +For more about how OTLP metrics are stored as time series data streams, refer to [Ingest metrics into a TSDS using the OTLP/HTTP endpoint](/manage-data/data-store/data-streams/tsds-ingest-otlp.md). + +The target data stream name follows the pattern `-.otel-`. +You can influence `dataset` and `namespace` by setting attributes on your data: + +* Set `data_stream.dataset` and/or `data_stream.namespace` as attributes. + Precedence: data point or log record attribute, then scope attribute, then resource attribute. +* Otherwise, if the scope name contains `/receiver/`, `data_stream.dataset` is set to the receiver name. +* Otherwise, `data_stream.dataset` falls back to `generic` and `data_stream.namespace` falls back to `default`. + +Examples: + +| Signal | Attributes or scope name | Target data stream | +| --- | --- | --- | +| Logs | `data_stream.dataset: nginx.access`, `data_stream.namespace: prod` | `logs-nginx.access.otel-prod` | +| Traces | `data_stream.dataset: checkout`, `data_stream.namespace: staging` | `traces-checkout.otel-staging` | +| Metrics | Scope name contains `/receiver/hostmetrics`, no `data_stream.*` attributes | `metrics-hostmetrics.otel-default` | +| Metrics | No matching attributes or receiver scope name | `metrics-generic.otel-default` | + +## Body map mode for logs +```{applies_to} +stack: preview 9.5 +``` + +By default, OTLP log records are mapped into {{es}}'s standard OTel document structure, preserving resource, scope, and record metadata. + +If an upstream component has already shaped the log body to match the desired document structure, you can opt into the body map mode. +In this mode, the log record's body map is used as the complete document, without copying the surrounding OTLP metadata. + +Enable body map mode in either of two ways: + +* Per request, by setting the `X-Elastic-Mapping-Mode` HTTP header to `bodymap`. +* Per instrumentation scope, by setting the `elastic.mapping.mode` scope attribute to `bodymap`. + The scope attribute takes precedence over the header. + +## Configure histogram handling for metrics +```{applies_to} +stack: preview =9.3, ga 9.4+ +``` + +You can configure how OTLP histogram metrics are mapped using the `xpack.otel_data.histogram_field_type` cluster setting. +Valid values are: + + - `histogram` (default on {applies_to}`stack: preview =9.3`): Map histograms as T-Digests using the `histogram` field type + - `exponential_histogram` (default on {applies_to}`stack: ga 9.4+`): Map histograms as exponential histograms using the `exponential_histogram` field type + +The setting is dynamic and can be updated at runtime: + +```console +PUT /_cluster/settings +{ + "persistent" : { + "xpack.otel_data.histogram_field_type" : "exponential_histogram" + } +} +``` + +Because both `histogram` and `exponential_histogram` support [coerce](elasticsearch://reference/elasticsearch/mapping-reference/coerce.md), changing this setting dynamically does not risk mapping conflicts or ingestion failures. + +This setting only applies to metrics ingested through the {{es}} OTLP endpoint. +Documents ingested using the Bulk API (for example through the {{es}} exporter for the OpenTelemetry Collector) are not affected. + +## Limitations + +* **Delivery guarantees:** {{es}} can only acknowledge an OTLP request as a whole, not on a per-record basis. + If part of a request fails, the client retries the entire batch, which can produce duplicate logs or trace spans. + Metrics are not affected because metric points written to time series data streams are [deduplicated based on their dimensions and timestamp](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-dimension). +* **Profiles:** Profiles are not supported. + To ingest profiles, use a distribution of the OpenTelemetry Collector that includes the [{{es}} exporter](opentelemetry://reference/edot-collector/components/elasticsearchexporter.md), such as the [{{edot}} (EDOT) Collector](opentelemetry://reference/edot-collector/index.md). +* **Histogram temporality:** Histograms are only supported in delta temporality. + Set the temporality preference to delta in your SDKs, or use the [`cumulativetodelta` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/cumulativetodeltaprocessor) so cumulative histograms aren't dropped. +* **Exemplars:** Exemplars are not supported yet. diff --git a/manage-data/toc.yml b/manage-data/toc.yml index b394eaac8f..45e0a86aed 100644 --- a/manage-data/toc.yml +++ b/manage-data/toc.yml @@ -112,6 +112,7 @@ toc: children: - file: ingest/agentless/cloud-connector-deployment.md - file: ingest/agentless/agentless-integrations-faq.md + - file: ingest/otlp-endpoint.md - file: ingest/sample-data.md - file: ingest/upload-data-files.md - file: ingest/transform-enrich.md