Skip to content

[exporter/elasticsearch] Set require_data_stream=true for ECS mapping mode #46632

Merged
ChrsMark merged 8 commits into
open-telemetry:mainfrom
isaacaflores2:es-exporter-ecs-require-data_stream
Mar 5, 2026
Merged

[exporter/elasticsearch] Set require_data_stream=true for ECS mapping mode #46632
ChrsMark merged 8 commits into
open-telemetry:mainfrom
isaacaflores2:es-exporter-ecs-require-data_stream

Conversation

@isaacaflores2

Copy link
Copy Markdown
Contributor

Description

#46499 will update the elasticsearch exporter to use dynamic templates for the ECS mapping mode.

This PR updates the exporter to use set require_data_stream=true for ECS mapping mode.

Link to tracking issue

Fixes

Testing

  1. Updated and added unit test cases
  2. Build collector and indexed metrics into a local Elasticsearch instance
Local collector testing details

Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4319
        include_metadata: true
      http:
        endpoint: localhost:4320
        include_metadata: true

exporters:
  elasticsearch:
    endpoint: ${env:ES_URL}
    auth:
      authenticator: basicauth
  debug:
    verbosity: detailed

extensions:
  basicauth:
    client_auth:
      username: <username>
      password: <password>

service:
  extensions: [ basicauth ]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [elasticsearch]
    metrics:
      receivers: [otlp]
      exporters: [elasticsearch,debug]
    logs:
      receivers: [otlp]
      exporters: [elasticsearch]
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: '127.0.0.1'
                port: 8889

Test data

Sent data using telemetrygen

telemetrygen metrics --otlp-insecure 1 --otlp-header x-elastic-mapping-mode=\"ecs\" --service "ecs_test_1" --otlp-endpoint localhost:4320 --otlp-http
telemetrygen logs --otlp-insecure 1 --otlp-header x-elastic-mapping-mode=\"ecs\" --service "ecs_test_1" --otlp-endpoint localhost:4320 --otlp-http
telemetrygen traces --otlp-insecure 1 --otlp-header x-elastic-mapping-mode=\"ecs\" --service "ecs_test_1" --otlp-endpoint localhost:4320 --otlp-http

Kibana

Example for indexed log
image

Documentation

Updated README.md to specify the new minimum version for ECS mapping mode

@carsonip carsonip left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on error hint, otherwise code change looks good

Comment thread exporter/elasticsearchexporter/bulkindexer.go Outdated
ChrsMark pushed a commit that referenced this pull request Mar 4, 2026
…ng mode metrics encoder (#46499)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
The current ECS mapping mode metrics encoder does not correctly map
histograms. Legacy APM metrics rely on an ingest pipeline that requires
metrics to be wrapped in `metricset.sample`.



<details><summary>Mapping example </summary>
<p>

## Before
Current behavior when indexing a histogram with name
`http.request.duration`
```json
"http": {
  "properties": {
    "request": {
      "properties": {
        "duration": {
          "properties": {
            "counts": {
              "type": "long",
              "index": false
            },
            "values": {
              "type": "float",
              "index": false
            }
          }
        }
      }
    }
  }
}
```

## After
When the exporter sends the `dynamic_templates` the histogram is
correctly mapped
```json
"http": {
    "properties": {
      "request": {
        "properties": {
          "duration": {
            "type": "histogram"
          }
        }
      }
    }
```

</p>
</details> 

This PR updates the ECS mapping mode metics encode to send
`dynamic_templates` when indexing metrics using the same approach used
today for the OTel mapping mode

<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Related to
#46632

<!--Describe what testing was performed and which tests were added.-->
#### Testing
1. Updated existing ecs mode test cases
2. Add additional test cases
3. Build collector and indexed metrics into a local Elasticsearch
instance:
<details><summary>Local collector testing details</summary>
<p>

## Collector configuration
```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4319
        include_metadata: true
      http:
        endpoint: localhost:4320
        include_metadata: true

exporters:
  elasticsearch:
    endpoint: ${env:ES_URL}
    auth:
      authenticator: basicauth
  debug:
    verbosity: detailed

extensions:
  basicauth:
    client_auth:
      username: <username>
      password: <password>

service:
  extensions: [ basicauth ]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [elasticsearch]
    metrics:
      receivers: [otlp]
      exporters: [elasticsearch,debug]
    logs:
      receivers: [otlp]
      exporters: [elasticsearch]
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: '127.0.0.1'
                port: 8889

```

## Test data
```bash
#!/usr/bin/env bash
# Send OTLP metrics (gauge, counter, histogram, summary) via HTTP/JSON.
# Default: http://localhost:4320/v1/metrics (override with OTLP_METRICS_ENDPOINT).
#
# To test: start the minimal collector, then run this script:
#   ./bin/otelcontribcol-minimal_$(go env GOOS)_$(go env GOARCH) --config exporter_test_config_local.yaml
#   ./.send_otlp_metrics
#
set -euo pipefail

OTLP_METRICS_ENDPOINT="${OTLP_METRICS_ENDPOINT:-http://localhost:4320/v1/metrics}"

exit_with_hint() {
  local code=$1
  if [[ $code -ne 0 ]]; then
    echo "Send failed (exit $code). Is the collector running with OTLP HTTP on the endpoint?" >&2
    echo "  Example: ./bin/otelcontribcol-minimal_\$(go env GOOS)_\$(go env GOARCH) --config exporter_test_config_local.yaml" >&2
  fi
  exit "$code"
}

trap 'exit_with_hint $?' EXIT

# Nanoseconds since epoch (second precision; portable on macOS and Linux)
NOW_NS="$(date +%s)000000000"

curl --fail-with-body -sS \
  -X POST "${OTLP_METRICS_ENDPOINT}" \
  -H "Content-Type: application/json" \
  -H "X-Elastic-Mapping-Mode: ecs" \
  --data-binary @- <<EOF
{
  "resourceMetrics": [
    {
      "resource": {
        "attributes": [
          { "key": "service.name", "value": { "stringValue": "metricset-samples-curl" } },
          { "key": "service.namespace", "value": { "stringValue": "dev" } }
        ]
      },
      "scopeMetrics": [
        {
          "scope": {
            "name": "curl.manual",
            "version": "1.0.0"
          },
          "metrics": [
            {
              "name": "demo.gauge",
              "description": "Gauge in same request",
              "unit": "1",
              "gauge": {
                "dataPoints": [
                  {
                    "startTimeUnixNano": "${NOW_NS}",
                    "timeUnixNano": "${NOW_NS}",
                    "asDouble": 42.5,
                    "attributes": [
                      { "key": "metric.type", "value": { "stringValue": "gauge" } }
                    ]
                  }
                ]
              }
            },
            {
              "name": "demo.counter",
              "description": "Monotonic sum (counter) in same request",
              "unit": "1",
              "sum": {
                "aggregationTemporality": 2,
                "isMonotonic": true,
                "dataPoints": [
                  {
                    "startTimeUnixNano": "${NOW_NS}",
                    "timeUnixNano": "${NOW_NS}",
                    "asInt": 7,
                    "attributes": [
                      { "key": "metric.type", "value": { "stringValue": "counter" } }
                    ]
                  }
                ]
              }
            },
            {
              "name": "demo.histogram",
              "description": "Delta histogram in same request",
              "unit": "ms",
              "histogram": {
                "aggregationTemporality": 1,
                "dataPoints": [
                  {
                    "startTimeUnixNano": "${NOW_NS}",
                    "timeUnixNano": "${NOW_NS}",
                    "count": 10,
                    "sum": 123.4,
                    "bucketCounts": [1, 2, 3, 4],
                    "explicitBounds": [10, 50, 100],
                    "attributes": [
                      { "key": "metric.type", "value": { "stringValue": "histogram" } }
                    ]
                  }
                ]
              }
            },
            {
              "name": "demo.summary",
              "description": "Summary in same request",
              "unit": "ms",
              "summary": {
                "dataPoints": [
                  {
                    "startTimeUnixNano": "${NOW_NS}",
                    "timeUnixNano": "${NOW_NS}",
                    "count": 5,
                    "sum": 250.5,
                    "quantileValues": [
                      { "quantile": 0.5, "value": 45.0 },
                      { "quantile": 0.9, "value": 90.0 }
                    ],
                    "attributes": [
                      { "key": "metric.type", "value": { "stringValue": "summary" } }
                    ]
                  }
                ]
              }
            }
          ]
        }
      ]
    }
  ]
}
EOF
```

## Kibana Validation
<img width="1723" height="811" alt="image"
src="https://github.com/user-attachments/assets/2323fa81-0e2e-489a-afbe-12ed1c67c6ba"
/>

`GET metrics-generic-default/_mapping`
<img width="1723" height="811" alt="image"
src="https://github.com/user-attachments/assets/8c3699f2-aefb-4031-b789-9cd7d5222c12"
/>


### mappings
```json/
"demo": {
          "properties": {
            "counter": {
              "type": "double",
              "index": false
            },
            "gauge": {
              "type": "double",
              "index": false
            },
            "histogram": {
              "type": "histogram"
            },
            "summary": {
              "type": "aggregate_metric_double",
              "metrics": [
                "sum",
                "value_count"
              ],
              "default_metric": "value_count"
            }
          }
        }
```




</p>
</details> 

<!--Describe the documentation added.-->
#### Documentation

Updated the README.md to document the existing OTel mapping mode dynamic
templating and added details for the ECS mapping mode changes

@carsonip carsonip left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking great, thanks!

@ChrsMark ChrsMark merged commit 831bfcb into open-telemetry:main Mar 5, 2026
191 checks passed
@otelbot

otelbot Bot commented Mar 5, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution @isaacaflores2! 🎉 We would like to hear from you about your experience contributing to OpenTelemetry by taking a few minutes to fill out this survey. If you are getting started contributing, you can also join the CNCF Slack channel #opentelemetry-new-contributors to ask for guidance and get help.

antonio-mazzini pushed a commit to antonio-mazzini/opentelemetry-collector-contrib that referenced this pull request Mar 5, 2026
…ng mode metrics encoder (open-telemetry#46499)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
The current ECS mapping mode metrics encoder does not correctly map
histograms. Legacy APM metrics rely on an ingest pipeline that requires
metrics to be wrapped in `metricset.sample`.



<details><summary>Mapping example </summary>
<p>

## Before
Current behavior when indexing a histogram with name
`http.request.duration`
```json
"http": {
  "properties": {
    "request": {
      "properties": {
        "duration": {
          "properties": {
            "counts": {
              "type": "long",
              "index": false
            },
            "values": {
              "type": "float",
              "index": false
            }
          }
        }
      }
    }
  }
}
```

## After
When the exporter sends the `dynamic_templates` the histogram is
correctly mapped
```json
"http": {
    "properties": {
      "request": {
        "properties": {
          "duration": {
            "type": "histogram"
          }
        }
      }
    }
```

</p>
</details> 

This PR updates the ECS mapping mode metics encode to send
`dynamic_templates` when indexing metrics using the same approach used
today for the OTel mapping mode

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Related to
open-telemetry#46632

<!--Describe what testing was performed and which tests were added.-->
#### Testing
1. Updated existing ecs mode test cases
2. Add additional test cases
3. Build collector and indexed metrics into a local Elasticsearch
instance:
<details><summary>Local collector testing details</summary>
<p>

## Collector configuration
```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4319
        include_metadata: true
      http:
        endpoint: localhost:4320
        include_metadata: true

exporters:
  elasticsearch:
    endpoint: ${env:ES_URL}
    auth:
      authenticator: basicauth
  debug:
    verbosity: detailed

extensions:
  basicauth:
    client_auth:
      username: <username>
      password: <password>

service:
  extensions: [ basicauth ]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [elasticsearch]
    metrics:
      receivers: [otlp]
      exporters: [elasticsearch,debug]
    logs:
      receivers: [otlp]
      exporters: [elasticsearch]
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: '127.0.0.1'
                port: 8889

```

## Test data
```bash
#!/usr/bin/env bash
# Send OTLP metrics (gauge, counter, histogram, summary) via HTTP/JSON.
# Default: http://localhost:4320/v1/metrics (override with OTLP_METRICS_ENDPOINT).
#
# To test: start the minimal collector, then run this script:
#   ./bin/otelcontribcol-minimal_$(go env GOOS)_$(go env GOARCH) --config exporter_test_config_local.yaml
#   ./.send_otlp_metrics
#
set -euo pipefail

OTLP_METRICS_ENDPOINT="${OTLP_METRICS_ENDPOINT:-http://localhost:4320/v1/metrics}"

exit_with_hint() {
  local code=$1
  if [[ $code -ne 0 ]]; then
    echo "Send failed (exit $code). Is the collector running with OTLP HTTP on the endpoint?" >&2
    echo "  Example: ./bin/otelcontribcol-minimal_\$(go env GOOS)_\$(go env GOARCH) --config exporter_test_config_local.yaml" >&2
  fi
  exit "$code"
}

trap 'exit_with_hint $?' EXIT

# Nanoseconds since epoch (second precision; portable on macOS and Linux)
NOW_NS="$(date +%s)000000000"

curl --fail-with-body -sS \
  -X POST "${OTLP_METRICS_ENDPOINT}" \
  -H "Content-Type: application/json" \
  -H "X-Elastic-Mapping-Mode: ecs" \
  --data-binary @- <<EOF
{
  "resourceMetrics": [
    {
      "resource": {
        "attributes": [
          { "key": "service.name", "value": { "stringValue": "metricset-samples-curl" } },
          { "key": "service.namespace", "value": { "stringValue": "dev" } }
        ]
      },
      "scopeMetrics": [
        {
          "scope": {
            "name": "curl.manual",
            "version": "1.0.0"
          },
          "metrics": [
            {
              "name": "demo.gauge",
              "description": "Gauge in same request",
              "unit": "1",
              "gauge": {
                "dataPoints": [
                  {
                    "startTimeUnixNano": "${NOW_NS}",
                    "timeUnixNano": "${NOW_NS}",
                    "asDouble": 42.5,
                    "attributes": [
                      { "key": "metric.type", "value": { "stringValue": "gauge" } }
                    ]
                  }
                ]
              }
            },
            {
              "name": "demo.counter",
              "description": "Monotonic sum (counter) in same request",
              "unit": "1",
              "sum": {
                "aggregationTemporality": 2,
                "isMonotonic": true,
                "dataPoints": [
                  {
                    "startTimeUnixNano": "${NOW_NS}",
                    "timeUnixNano": "${NOW_NS}",
                    "asInt": 7,
                    "attributes": [
                      { "key": "metric.type", "value": { "stringValue": "counter" } }
                    ]
                  }
                ]
              }
            },
            {
              "name": "demo.histogram",
              "description": "Delta histogram in same request",
              "unit": "ms",
              "histogram": {
                "aggregationTemporality": 1,
                "dataPoints": [
                  {
                    "startTimeUnixNano": "${NOW_NS}",
                    "timeUnixNano": "${NOW_NS}",
                    "count": 10,
                    "sum": 123.4,
                    "bucketCounts": [1, 2, 3, 4],
                    "explicitBounds": [10, 50, 100],
                    "attributes": [
                      { "key": "metric.type", "value": { "stringValue": "histogram" } }
                    ]
                  }
                ]
              }
            },
            {
              "name": "demo.summary",
              "description": "Summary in same request",
              "unit": "ms",
              "summary": {
                "dataPoints": [
                  {
                    "startTimeUnixNano": "${NOW_NS}",
                    "timeUnixNano": "${NOW_NS}",
                    "count": 5,
                    "sum": 250.5,
                    "quantileValues": [
                      { "quantile": 0.5, "value": 45.0 },
                      { "quantile": 0.9, "value": 90.0 }
                    ],
                    "attributes": [
                      { "key": "metric.type", "value": { "stringValue": "summary" } }
                    ]
                  }
                ]
              }
            }
          ]
        }
      ]
    }
  ]
}
EOF
```

## Kibana Validation
<img width="1723" height="811" alt="image"
src="https://github.com/user-attachments/assets/2323fa81-0e2e-489a-afbe-12ed1c67c6ba"
/>

`GET metrics-generic-default/_mapping`
<img width="1723" height="811" alt="image"
src="https://github.com/user-attachments/assets/8c3699f2-aefb-4031-b789-9cd7d5222c12"
/>


### mappings
```json/
"demo": {
          "properties": {
            "counter": {
              "type": "double",
              "index": false
            },
            "gauge": {
              "type": "double",
              "index": false
            },
            "histogram": {
              "type": "histogram"
            },
            "summary": {
              "type": "aggregate_metric_double",
              "metrics": [
                "sum",
                "value_count"
              ],
              "default_metric": "value_count"
            }
          }
        }
```




</p>
</details> 

<!--Describe the documentation added.-->
#### Documentation

Updated the README.md to document the existing OTel mapping mode dynamic
templating and added details for the ECS mapping mode changes
antonio-mazzini pushed a commit to antonio-mazzini/opentelemetry-collector-contrib that referenced this pull request Mar 5, 2026
…ng mode (open-telemetry#46632)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

open-telemetry#46499
will update the elasticsearch exporter to use dynamic templates for the
ECS mapping mode.

This PR updates the exporter to use set `require_data_stream=true` for
ECS mapping mode.

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes

<!--Describe what testing was performed and which tests were added.-->
#### Testing
1. Updated and added unit test cases
2. Build collector and indexed metrics into a local Elasticsearch
instance
<details><summary>Local collector testing details</summary>
<p>

## Collector configuration
```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4319
        include_metadata: true
      http:
        endpoint: localhost:4320
        include_metadata: true

exporters:
  elasticsearch:
    endpoint: ${env:ES_URL}
    auth:
      authenticator: basicauth
  debug:
    verbosity: detailed

extensions:
  basicauth:
    client_auth:
      username: <username>
      password: <password>

service:
  extensions: [ basicauth ]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [elasticsearch]
    metrics:
      receivers: [otlp]
      exporters: [elasticsearch,debug]
    logs:
      receivers: [otlp]
      exporters: [elasticsearch]
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: '127.0.0.1'
                port: 8889

```

### Test data
Sent data using `telemetrygen`
```bash
telemetrygen metrics --otlp-insecure 1 --otlp-header x-elastic-mapping-mode=\"ecs\" --service "ecs_test_1" --otlp-endpoint localhost:4320 --otlp-http
telemetrygen logs --otlp-insecure 1 --otlp-header x-elastic-mapping-mode=\"ecs\" --service "ecs_test_1" --otlp-endpoint localhost:4320 --otlp-http
telemetrygen traces --otlp-insecure 1 --otlp-header x-elastic-mapping-mode=\"ecs\" --service "ecs_test_1" --otlp-endpoint localhost:4320 --otlp-http
```

### Kibana
Example for indexed log
<img width="2553" height="662" alt="image"
src="https://github.com/user-attachments/assets/7881b217-61ba-4372-95b7-132fba103de6"
/>


</p>
</details> 

<!--Describe the documentation added.-->
#### Documentation
Updated `README.md` to specify the new minimum version for ECS mapping
mode
avleentwilio pushed a commit to avleentwilio/opentelemetry-collector-contrib that referenced this pull request Apr 1, 2026
…ng mode metrics encoder (open-telemetry#46499)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
The current ECS mapping mode metrics encoder does not correctly map
histograms. Legacy APM metrics rely on an ingest pipeline that requires
metrics to be wrapped in `metricset.sample`.



<details><summary>Mapping example </summary>
<p>

## Before
Current behavior when indexing a histogram with name
`http.request.duration`
```json
"http": {
  "properties": {
    "request": {
      "properties": {
        "duration": {
          "properties": {
            "counts": {
              "type": "long",
              "index": false
            },
            "values": {
              "type": "float",
              "index": false
            }
          }
        }
      }
    }
  }
}
```

## After
When the exporter sends the `dynamic_templates` the histogram is
correctly mapped
```json
"http": {
    "properties": {
      "request": {
        "properties": {
          "duration": {
            "type": "histogram"
          }
        }
      }
    }
```

</p>
</details> 

This PR updates the ECS mapping mode metics encode to send
`dynamic_templates` when indexing metrics using the same approach used
today for the OTel mapping mode

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Related to
open-telemetry#46632

<!--Describe what testing was performed and which tests were added.-->
#### Testing
1. Updated existing ecs mode test cases
2. Add additional test cases
3. Build collector and indexed metrics into a local Elasticsearch
instance:
<details><summary>Local collector testing details</summary>
<p>

## Collector configuration
```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4319
        include_metadata: true
      http:
        endpoint: localhost:4320
        include_metadata: true

exporters:
  elasticsearch:
    endpoint: ${env:ES_URL}
    auth:
      authenticator: basicauth
  debug:
    verbosity: detailed

extensions:
  basicauth:
    client_auth:
      username: <username>
      password: <password>

service:
  extensions: [ basicauth ]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [elasticsearch]
    metrics:
      receivers: [otlp]
      exporters: [elasticsearch,debug]
    logs:
      receivers: [otlp]
      exporters: [elasticsearch]
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: '127.0.0.1'
                port: 8889

```

## Test data
```bash
#!/usr/bin/env bash
# Send OTLP metrics (gauge, counter, histogram, summary) via HTTP/JSON.
# Default: http://localhost:4320/v1/metrics (override with OTLP_METRICS_ENDPOINT).
#
# To test: start the minimal collector, then run this script:
#   ./bin/otelcontribcol-minimal_$(go env GOOS)_$(go env GOARCH) --config exporter_test_config_local.yaml
#   ./.send_otlp_metrics
#
set -euo pipefail

OTLP_METRICS_ENDPOINT="${OTLP_METRICS_ENDPOINT:-http://localhost:4320/v1/metrics}"

exit_with_hint() {
  local code=$1
  if [[ $code -ne 0 ]]; then
    echo "Send failed (exit $code). Is the collector running with OTLP HTTP on the endpoint?" >&2
    echo "  Example: ./bin/otelcontribcol-minimal_\$(go env GOOS)_\$(go env GOARCH) --config exporter_test_config_local.yaml" >&2
  fi
  exit "$code"
}

trap 'exit_with_hint $?' EXIT

# Nanoseconds since epoch (second precision; portable on macOS and Linux)
NOW_NS="$(date +%s)000000000"

curl --fail-with-body -sS \
  -X POST "${OTLP_METRICS_ENDPOINT}" \
  -H "Content-Type: application/json" \
  -H "X-Elastic-Mapping-Mode: ecs" \
  --data-binary @- <<EOF
{
  "resourceMetrics": [
    {
      "resource": {
        "attributes": [
          { "key": "service.name", "value": { "stringValue": "metricset-samples-curl" } },
          { "key": "service.namespace", "value": { "stringValue": "dev" } }
        ]
      },
      "scopeMetrics": [
        {
          "scope": {
            "name": "curl.manual",
            "version": "1.0.0"
          },
          "metrics": [
            {
              "name": "demo.gauge",
              "description": "Gauge in same request",
              "unit": "1",
              "gauge": {
                "dataPoints": [
                  {
                    "startTimeUnixNano": "${NOW_NS}",
                    "timeUnixNano": "${NOW_NS}",
                    "asDouble": 42.5,
                    "attributes": [
                      { "key": "metric.type", "value": { "stringValue": "gauge" } }
                    ]
                  }
                ]
              }
            },
            {
              "name": "demo.counter",
              "description": "Monotonic sum (counter) in same request",
              "unit": "1",
              "sum": {
                "aggregationTemporality": 2,
                "isMonotonic": true,
                "dataPoints": [
                  {
                    "startTimeUnixNano": "${NOW_NS}",
                    "timeUnixNano": "${NOW_NS}",
                    "asInt": 7,
                    "attributes": [
                      { "key": "metric.type", "value": { "stringValue": "counter" } }
                    ]
                  }
                ]
              }
            },
            {
              "name": "demo.histogram",
              "description": "Delta histogram in same request",
              "unit": "ms",
              "histogram": {
                "aggregationTemporality": 1,
                "dataPoints": [
                  {
                    "startTimeUnixNano": "${NOW_NS}",
                    "timeUnixNano": "${NOW_NS}",
                    "count": 10,
                    "sum": 123.4,
                    "bucketCounts": [1, 2, 3, 4],
                    "explicitBounds": [10, 50, 100],
                    "attributes": [
                      { "key": "metric.type", "value": { "stringValue": "histogram" } }
                    ]
                  }
                ]
              }
            },
            {
              "name": "demo.summary",
              "description": "Summary in same request",
              "unit": "ms",
              "summary": {
                "dataPoints": [
                  {
                    "startTimeUnixNano": "${NOW_NS}",
                    "timeUnixNano": "${NOW_NS}",
                    "count": 5,
                    "sum": 250.5,
                    "quantileValues": [
                      { "quantile": 0.5, "value": 45.0 },
                      { "quantile": 0.9, "value": 90.0 }
                    ],
                    "attributes": [
                      { "key": "metric.type", "value": { "stringValue": "summary" } }
                    ]
                  }
                ]
              }
            }
          ]
        }
      ]
    }
  ]
}
EOF
```

## Kibana Validation
<img width="1723" height="811" alt="image"
src="https://github.com/user-attachments/assets/2323fa81-0e2e-489a-afbe-12ed1c67c6ba"
/>

`GET metrics-generic-default/_mapping`
<img width="1723" height="811" alt="image"
src="https://github.com/user-attachments/assets/8c3699f2-aefb-4031-b789-9cd7d5222c12"
/>


### mappings
```json/
"demo": {
          "properties": {
            "counter": {
              "type": "double",
              "index": false
            },
            "gauge": {
              "type": "double",
              "index": false
            },
            "histogram": {
              "type": "histogram"
            },
            "summary": {
              "type": "aggregate_metric_double",
              "metrics": [
                "sum",
                "value_count"
              ],
              "default_metric": "value_count"
            }
          }
        }
```




</p>
</details> 

<!--Describe the documentation added.-->
#### Documentation

Updated the README.md to document the existing OTel mapping mode dynamic
templating and added details for the ECS mapping mode changes
avleentwilio pushed a commit to avleentwilio/opentelemetry-collector-contrib that referenced this pull request Apr 1, 2026
…ng mode (open-telemetry#46632)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

open-telemetry#46499
will update the elasticsearch exporter to use dynamic templates for the
ECS mapping mode.

This PR updates the exporter to use set `require_data_stream=true` for
ECS mapping mode.

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes

<!--Describe what testing was performed and which tests were added.-->
#### Testing
1. Updated and added unit test cases
2. Build collector and indexed metrics into a local Elasticsearch
instance
<details><summary>Local collector testing details</summary>
<p>

## Collector configuration
```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4319
        include_metadata: true
      http:
        endpoint: localhost:4320
        include_metadata: true

exporters:
  elasticsearch:
    endpoint: ${env:ES_URL}
    auth:
      authenticator: basicauth
  debug:
    verbosity: detailed

extensions:
  basicauth:
    client_auth:
      username: <username>
      password: <password>

service:
  extensions: [ basicauth ]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [elasticsearch]
    metrics:
      receivers: [otlp]
      exporters: [elasticsearch,debug]
    logs:
      receivers: [otlp]
      exporters: [elasticsearch]
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: '127.0.0.1'
                port: 8889

```

### Test data
Sent data using `telemetrygen`
```bash
telemetrygen metrics --otlp-insecure 1 --otlp-header x-elastic-mapping-mode=\"ecs\" --service "ecs_test_1" --otlp-endpoint localhost:4320 --otlp-http
telemetrygen logs --otlp-insecure 1 --otlp-header x-elastic-mapping-mode=\"ecs\" --service "ecs_test_1" --otlp-endpoint localhost:4320 --otlp-http
telemetrygen traces --otlp-insecure 1 --otlp-header x-elastic-mapping-mode=\"ecs\" --service "ecs_test_1" --otlp-endpoint localhost:4320 --otlp-http
```

### Kibana
Example for indexed log
<img width="2553" height="662" alt="image"
src="https://github.com/user-attachments/assets/7881b217-61ba-4372-95b7-132fba103de6"
/>


</p>
</details> 

<!--Describe the documentation added.-->
#### Documentation
Updated `README.md` to specify the new minimum version for ECS mapping
mode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants