[exporter/elasticsearch] OTel mode serialization #33290

aleksmaus · 2024-05-29T15:26:34Z

Description:

Implements OTel (OpenTelemetry-native) mode serialization for elasticsearch exporter.
This is an initial cut in order to get the discussion going.
This is approach was tested as internal POC.

It leverages Elasticsearch "passthrough" fields mapping initially introduced in Elasticsearch 8.13 allowing users to query the document/scope/resources attributes as top level fields, making the ECS queries compatible with OTel sematic convention schema. Another benefit is the simplicity of conversion of stored document from Elasticsearch back to Otel data model format.

The document/scope/resources attributes are dynamically mapped and stored as flattened keys.

Here is an example of index template mappings with "passthrough" fields:

PUT _index_template/logs_otel
{
  "priority": 250,
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "logs"
        },
        "codec": "best_compression",
        "mapping": {
          "ignore_malformed": "true"
        }
      }
    },
    "mappings": {
      "_source": {
        "enabled": true
      },
      "date_detection": false,
      "dynamic": "strict",
      "dynamic_templates": [
        {
          "all_strings_to_keywords": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        },
        {
          "complex_attributes": {
            "path_match": [
              "resource.attributes.*",
              "scope.attributes.*",
              "attributes.*"
            ],
            "match_mapping_type": "object",
            "mapping": {
              "type": "flattened"
            }
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date_nanos",
          "ignore_malformed": false
        },
        "data_stream": {
          "type": "object",
          "properties": {
            "type": {
              "type": "constant_keyword"
            },
            "dataset": {
              "type": "constant_keyword"
            },
            "namespace": {
              "type": "constant_keyword"
            }
          }
        },
        "observed_timestamp": {
          "type": "date_nanos",
          "ignore_malformed": true
        },
        "severity_number": {
          "type": "long"
        },
        "severity_text": {
          "type": "keyword"
        },
        "body_text": {
          "type": "match_only_text"
        },
        "body_structured": {
          "type": "flattened"
        },
        "attributes": {
          "type": "passthrough",
          "dynamic": true,
          "priority": 2
        },
        "dropped_attributes_count": {
          "type": "long"
        },
        "trace_flags": {
          "type": "byte"
        },
        "trace_id": {
          "type": "keyword"
        },
        "span_id": {
          "type": "keyword"
        },
        "scope": {
          "properties": {
            "name": {
              "type": "keyword"
            },
            "version": {
              "type": "keyword"
            },
            "attributes": {
              "type": "passthrough",
              "dynamic": true,
              "priority": 1
            },
            "dropped_attributes_count": {
              "type": "long"
            },
            "schema_url": {
              "type": "keyword"
            }
          }
        },
        "resource": {
          "properties": {
            "dropped_attributes_count": {
              "type": "long"
            },
            "schema_url": {
              "type": "keyword"
            },
            "attributes": {
              "type": "passthrough",
              "dynamic": true,
              "priority": 0
            }
          }
        }
      }
    }
  },
  "index_patterns": [
    "logs-*.otel-*"
  ],
  "data_stream": {}
}

Here is an example of the auditd document in Elasticsearch abbreviated:

{
    "@timestamp": "2024-05-29T13:30:25.085926000Z",
    "attributes": {
        "foo": "bar",
        "some.bool": true
    },
    "body_structured": {
        "MESSAGE": "AVC apparmor=\"STATUS\" operation=\"profile_replace\" info=\"same as current profile, skipping\" profile=\"unconfined\" name=\"/usr/bin/evince-previewer\" pid=2702 comm=\"apparmor_parser\"",
        "SYSLOG_FACILITY": "4",
        "SYSLOG_IDENTIFIER": "audit",
        "_SOURCE_REALTIME_TIMESTAMP": "1716989425080000",
        "_TRANSPORT": "audit",
    },
    "dropped_attributes_count": 0,
    "observed_timestamp": "2024-05-29T14:49:26.534908898Z",
    "resource": {
        "attributes": {
            "data_stream.dataset": "auditd.otel",
            "data_stream.namespace": "default",
            "data_stream.type": "logs",
            "host.arch": "arm64",
            "host.cpu.cache.l2.size": 0,
            "host.cpu.family": "",
            "host.cpu.model.id": "0x000",
            "host.cpu.model.name": "",
            "host.cpu.stepping": "0",
            "host.cpu.vendor.id": "Apple",
            "host.id": "cae0e0147d454a80971b0b747c8b62b9",
            "host.ip": [
                "172.16.3.131",
                "fe80::20c:29ff:fe66:3012",
            "host.name": "lebuntu",
            "host.os.description": "Ubuntu 22.04.4 LTS (Jammy Jellyfish) (Linux lebuntu 5.15.0-107-generic #117-Ubuntu SMP Mon Apr 29 14:37:09 UTC 2024 aarch64)",
            "host.os.type": "linux",
            "os.description": "Ubuntu 22.04.4 LTS (Jammy Jellyfish) (Linux lebuntu 5.15.0-107-generic #117-Ubuntu SMP Mon Apr 29 14:37:09 UTC 2024 aarch64)",
            "os.type": "linux"
        },
        "dropped_attributes_count": 0,
        "schema_url": "https://opentelemetry.io/schemas/1.6.1"
    },
    "severity_number": 0,
    "trace_flags": 0
}

Here is an example of ECS compatible query that works on this Otel native schema:

GET logs-auditd.otel-default/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "host.name": "lebuntu"
          }
        }
      ]
    }
  }
}

Link to tracking Issue:
No tracking issue yet.

Testing:
Added unit test for OTel transformation.
Tested with journald OTel receiver.

Documentation:
No documentation is added yet.

felixbarny · 2024-06-06T13:56:21Z

Functionally, the PR looks good to me. The only missing piece is the data stream routing. Maybe a test case with complex (map, list) attributes could be useful. I'll defer the actual code review to folks more familiar with go and the code base.

github-actions · 2024-06-21T05:20:29Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

aleksmaus · 2024-06-24T19:16:56Z

Updated PR based on the some code review feedback. Added the data_stream routing support, that was initially implemented in Josh's POC per reviewer request.

aleksmaus · 2024-06-24T19:18:41Z

Don't have permissions to remove the Stale label. Please advise.

felixbarny · 2024-06-28T09:11:22Z

#33794 is doing some changes wrt. routing. After that is merged, please merge in the changes and make sure the same routing mechanism is used. Let us know if you think that would work.

aleksmaus · 2024-06-28T13:57:38Z

#33794 is doing some changes wrt. routing. After that is merged, please merge in the changes and make sure the same routing mechanism is used. Let us know if you think that would work.

Yeah, that why I mentioned in DM before that I didn't want to add the routing since the work was happening in that area. It's easy to rollback.

exporter/elasticsearchexporter/exporter.go

carsonip · 2024-07-10T15:38:23Z

exporter/elasticsearchexporter/internal/objmodel/objmodel.go

@@ -245,19 +245,19 @@ func (doc *Document) Dedup() {
 // Serialize writes the document to the given writer. The serializer will create nested objects if dedot is true.
 //
 // NOTE: The documented MUST be sorted if dedot is true.
-func (doc *Document) Serialize(w io.Writer, dedot bool) error {
+func (doc *Document) Serialize(w io.Writer, dedot bool, otel bool) error {


Do we have to change objmodel for otel mode? Is there a reason why it isn't done like https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/model.go#L124 where we just add the attributes under a key?

As far as I remember, at the time of implementing this feature, I stumbled on the issue where the document could only be serialized as completely flat from the root or "dedotted" (by default https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/factory.go#L80), all the comma separated properties were unwrapped into the nested object, which was not stored correctly "flattened" with the passthrough attributes in ES.

With OTel-native serialization the serialized document structure is a mix, needed to keep only the ".attributes*" flattened, while the rest of the document from the root is not flattened.

Let me know if anything changed in that area recently, this PR was open for some time.

Thanks for the context. I'm just learning about how passthrough field type works, and what you describe here makes sense. I wonder if it could be structured better in the code, but I don't have any good ideas at the moment and it is not a blocker.

Maybe it makes sense to revisit this once we remove the dedot option: #33772.

exporter/elasticsearchexporter/config.go

Co-authored-by: Felix Barnsteiner <[email protected]>

aleksmaus · 2024-07-15T15:13:00Z

Accepted the README suggestion, merged the latest main

carsonip

thanks! changelog nit, otherwise lgtm

.chloggen/feature_elasticsearch_otel_model.yaml

Co-authored-by: Carson Ip <[email protected]>

aleksmaus · 2024-07-16T19:37:57Z

Accepted the suggestions, resolved the latest set of conflicts with main.

djaglowski · 2024-07-23T01:25:43Z

Please resolve CI failures and merge conflict

aleksmaus · 2024-07-23T12:39:00Z

Resolved conflicts

carsonip

=== Failed
=== FAIL: internal/objmodel TestDocument_Serialize_Otel/otel (0.00s)
    objmodel_test.go:387: 
                Error Trace:    /home/carson/projects/opentelemetry-collector-contrib/exporter/elasticsearchexporter/internal/objmodel/objmodel_test.go:387
                Error:          Not equal: 
                                expected: "{\"@timestamp\":\"2024-03-18T21:09:53.645578000Z\",\"attributes\":{\"auditd.log.op\":\"PAM:session_open\",\"auditd.log.record_type\":\"USER_START\",\"auditd.log.sequence\":6082,\"auditd.log.subj\":\"unconfined\",\"auditd.log.uid\":\"1000\"},\"resource\":{\"attributes\":{\"blah.num\":234,\"blah.str\":\"something\"}},\"scope\":{\"attributes\":{\"bar.one\":\"boo\",\"foo.two\":\"bar\"}}}"
                                actual  : "{\"@timestamp\":\"2024-03-18T21:09:53.645578000Z\",\"attributes\":{\"auditd.log.op\":\"PAM:session_open\",\"auditd.log.record_type\":\"USER_START\",\"auditd.log.sequence\":6082,\"auditd.log.subj\":\"unconfined\",\"auditd.log.uid\":\"1000\"},\"resource\":{\"attributes\":{\"blah\":{\"num\":234,\"str\":\"something\"}}},\"scope\":{\"attributes\":{\"bar\":{\"one\":\"boo\"},\"foo\":{\"two\":\"bar\"}}}}"
                                
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -{"@timestamp":"2024-03-18T21:09:53.645578000Z","attributes":{"auditd.log.op":"PAM:session_open","auditd.log.record_type":"USER_START","auditd.log.sequence":6082,"auditd.log.subj":"unconfined","auditd.log.uid":"1000"},"resource":{"attributes":{"blah.num":234,"blah.str":"something"}},"scope":{"attributes":{"bar.one":"boo","foo.two":"bar"}}}
                                +{"@timestamp":"2024-03-18T21:09:53.645578000Z","attributes":{"auditd.log.op":"PAM:session_open","auditd.log.record_type":"USER_START","auditd.log.sequence":6082,"auditd.log.subj":"unconfined","auditd.log.uid":"1000"},"resource":{"attributes":{"blah":{"num":234,"str":"something"}}},"scope":{"attributes":{"bar":{"one":"boo"},"foo":{"two":"bar"}}}}
                Test:           TestDocument_Serialize_Otel/otel

=== FAIL: internal/objmodel TestDocument_Serialize_Otel (0.00s)

=== FAIL: internal/objmodel TestDocument_Serialize_Otel/otel (re-run 1) (0.00s)
    objmodel_test.go:387: 
                Error Trace:    /home/carson/projects/opentelemetry-collector-contrib/exporter/elasticsearchexporter/internal/objmodel/objmodel_test.go:387
                Error:          Not equal: 
                                expected: "{\"@timestamp\":\"2024-03-18T21:09:53.645578000Z\",\"attributes\":{\"auditd.log.op\":\"PAM:session_open\",\"auditd.log.record_type\":\"USER_START\",\"auditd.log.sequence\":6082,\"auditd.log.subj\":\"unconfined\",\"auditd.log.uid\":\"1000\"},\"resource\":{\"attributes\":{\"blah.num\":234,\"blah.str\":\"something\"}},\"scope\":{\"attributes\":{\"bar.one\":\"boo\",\"foo.two\":\"bar\"}}}"
                                actual  : "{\"@timestamp\":\"2024-03-18T21:09:53.645578000Z\",\"attributes\":{\"auditd.log.op\":\"PAM:session_open\",\"auditd.log.record_type\":\"USER_START\",\"auditd.log.sequence\":6082,\"auditd.log.subj\":\"unconfined\",\"auditd.log.uid\":\"1000\"},\"resource\":{\"attributes\":{\"blah\":{\"num\":234,\"str\":\"something\"}}},\"scope\":{\"attributes\":{\"bar\":{\"one\":\"boo\"},\"foo\":{\"two\":\"bar\"}}}}"
                                
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -{"@timestamp":"2024-03-18T21:09:53.645578000Z","attributes":{"auditd.log.op":"PAM:session_open","auditd.log.record_type":"USER_START","auditd.log.sequence":6082,"auditd.log.subj":"unconfined","auditd.log.uid":"1000"},"resource":{"attributes":{"blah.num":234,"blah.str":"something"}},"scope":{"attributes":{"bar.one":"boo","foo.two":"bar"}}}
                                +{"@timestamp":"2024-03-18T21:09:53.645578000Z","attributes":{"auditd.log.op":"PAM:session_open","auditd.log.record_type":"USER_START","auditd.log.sequence":6082,"auditd.log.subj":"unconfined","auditd.log.uid":"1000"},"resource":{"attributes":{"blah":{"num":234,"str":"something"}}},"scope":{"attributes":{"bar":{"one":"boo"},"foo":{"two":"bar"}}}}
                Test:           TestDocument_Serialize_Otel/otel

aleksmaus added 2 commits May 29, 2024 09:56

Implement OTel mode serialization

3f2f30f

Enforce flattened attributes

d00a835

aleksmaus requested review from a team and djaglowski May 29, 2024 15:26

github-actions bot assigned djaglowski May 29, 2024

github-actions bot added the exporter/elasticsearch label May 29, 2024

github-actions bot requested review from carsonip, JaredTan95 and ycombinator May 29, 2024 15:26

aleksmaus and others added 4 commits May 30, 2024 10:43

Merge branch 'main' into feature/elasticsearch_otel_model

51e0c62

Merge branch 'main' into feature/elasticsearch_otel_model

f308e88

Merge branch 'main' into feature/elasticsearch_otel_model

5d80a84

Merge branch 'main' into feature/elasticsearch_otel_model

bd189d2

github-actions bot added the Stale label Jun 21, 2024

aleksmaus added 2 commits June 24, 2024 14:53

Add support for data_stream.* routing

87d9929

Merge branch 'main' into feature/elasticsearch_otel_model

eedd947

github-actions bot removed the Stale label Jun 25, 2024

Merge branch 'main' into feature/elasticsearch_otel_model

3fba3f7

aleksmaus and others added 2 commits June 28, 2024 09:57

Merge branch 'main' into feature/elasticsearch_otel_model

3efdbec

Merge branch 'main' into feature/elasticsearch_otel_model

ca8a2ec

felixbarny reviewed Jul 1, 2024

View reviewed changes

exporter/elasticsearchexporter/exporter.go Outdated Show resolved Hide resolved

carsonip reviewed Jul 10, 2024

View reviewed changes

exporter/elasticsearchexporter/config.go Outdated Show resolved Hide resolved

Merge branch 'main' into feature/elasticsearch_otel_model

460138c

aleksmaus and others added 2 commits July 15, 2024 11:11

Update exporter/elasticsearchexporter/README.md

5b0b4ba

Co-authored-by: Felix Barnsteiner <[email protected]>

Merge branch 'main' into feature/elasticsearch_otel_model

dd78d3f

carsonip reviewed Jul 16, 2024

View reviewed changes

.chloggen/feature_elasticsearch_otel_model.yaml Outdated Show resolved Hide resolved

.chloggen/feature_elasticsearch_otel_model.yaml Outdated Show resolved Hide resolved

felixbarny approved these changes Jul 16, 2024

View reviewed changes

aleksmaus and others added 3 commits July 16, 2024 15:27

Update .chloggen/feature_elasticsearch_otel_model.yaml

2537852

Co-authored-by: Carson Ip <[email protected]>

Update .chloggen/feature_elasticsearch_otel_model.yaml

182eb37

Co-authored-by: Carson Ip <[email protected]>

Merge branch 'main' into feature/elasticsearch_otel_model

64d0bd5

carsonip approved these changes Jul 16, 2024

View reviewed changes

aleksmaus added 3 commits July 17, 2024 09:00

Merge branch 'main' into feature/elasticsearch_otel_model

54a52f0

Merge branch 'main' into feature/elasticsearch_otel_model

5bcdf7e

Merge branch 'main' into feature/elasticsearch_otel_model

e24cd4c

Merge branch 'main' into feature/elasticsearch_otel_model

74c1d77

carsonip approved these changes Jul 23, 2024

View reviewed changes

carsonip suggested changes Jul 23, 2024

View reviewed changes

aleksmaus and others added 2 commits July 23, 2024 09:20

Return the set of prefixes that need to stay flattened

f411f0d

Merge branch 'main' into feature/elasticsearch_otel_model

10370e9

carsonip approved these changes Jul 23, 2024

View reviewed changes

aleksmaus and others added 3 commits July 23, 2024 09:28

Merge branch 'main' into feature/elasticsearch_otel_model

8f027f5

Merge branch 'main' into feature/elasticsearch_otel_model

b7a5def

Fix linter failures due to naming convention

abd9159

djaglowski approved these changes Jul 23, 2024

View reviewed changes

djaglowski merged commit 13366cc into open-telemetry:main Jul 23, 2024
154 checks passed

github-actions bot added this to the next release milestone Jul 23, 2024

axw mentioned this pull request Jul 29, 2024

Map all OTel attributes elastic/apm-data#183

Open

carsonip mentioned this pull request Aug 15, 2024

[exporter/elasticsearch] Add OTel mapping mode for traces #34472

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[exporter/elasticsearch] OTel mode serialization #33290

[exporter/elasticsearch] OTel mode serialization #33290

aleksmaus commented May 29, 2024 •

edited

Loading

felixbarny commented Jun 6, 2024

github-actions bot commented Jun 21, 2024

aleksmaus commented Jun 24, 2024

aleksmaus commented Jun 24, 2024

felixbarny commented Jun 28, 2024

aleksmaus commented Jun 28, 2024

carsonip Jul 10, 2024

aleksmaus Jul 10, 2024 •

edited

Loading

carsonip Jul 10, 2024

felixbarny Jul 11, 2024

aleksmaus commented Jul 15, 2024

carsonip left a comment

aleksmaus commented Jul 16, 2024

djaglowski commented Jul 23, 2024

aleksmaus commented Jul 23, 2024

carsonip left a comment

[exporter/elasticsearch] OTel mode serialization #33290

[exporter/elasticsearch] OTel mode serialization #33290

Conversation

aleksmaus commented May 29, 2024 • edited Loading

felixbarny commented Jun 6, 2024

github-actions bot commented Jun 21, 2024

aleksmaus commented Jun 24, 2024

aleksmaus commented Jun 24, 2024

felixbarny commented Jun 28, 2024

aleksmaus commented Jun 28, 2024

carsonip Jul 10, 2024

Choose a reason for hiding this comment

aleksmaus Jul 10, 2024 • edited Loading

Choose a reason for hiding this comment

carsonip Jul 10, 2024

Choose a reason for hiding this comment

felixbarny Jul 11, 2024

Choose a reason for hiding this comment

aleksmaus commented Jul 15, 2024

carsonip left a comment

Choose a reason for hiding this comment

aleksmaus commented Jul 16, 2024

djaglowski commented Jul 23, 2024

aleksmaus commented Jul 23, 2024

carsonip left a comment

Choose a reason for hiding this comment

aleksmaus commented May 29, 2024 •

edited

Loading

aleksmaus Jul 10, 2024 •

edited

Loading