Export LabelsFromName func from CollectD receiver #117
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Exporting this function since it can be useful in other cases that parse text metrics that embedded collectd tags.
pjanotti
requested review from
bogdandrutu,
flands,
owais,
rghetia,
songy23 and
tigrannajaryan
as code owners
owais
approved these changes
Feb 6, 2020
tigrannajaryan
approved these changes
Feb 6, 2020
mxiamxia
referenced
this pull request
in mxiamxia/opentelemetry-collector-contrib
Jul 22, 2020
ljmsc
referenced
this pull request
in ljmsc/opentelemetry-collector-contrib
Feb 21, 2022
* SDK: SpanProcessor Interface. * add simple span processor. * rename span processor. * fix logic to export or process span data.
codeboten
pushed a commit
that referenced
this pull request
Nov 23, 2022
Move instrumentation sqlalchemy
djaglowski
pushed a commit
that referenced
this pull request
Jul 23, 2024
**Description:**
Implements OTel (OpenTelemetry-native) mode serialization for
elasticsearch exporter.
This is an initial cut in order to get the discussion going.
This is approach was tested as internal POC.
It leverages Elasticsearch ```"passthrough"``` fields mapping initially
introduced in Elasticsearch 8.13 allowing users to query the
document/scope/resources attributes as top level fields, making the ECS
queries compatible with OTel sematic convention schema. Another benefit
is the simplicity of conversion of stored document from Elasticsearch
back to Otel data model format.
The document/scope/resources attributes are dynamically mapped and
stored as flattened keys.
Here is an example of index template mappings with ```"passthrough"```
fields:
```
PUT _index_template/logs_otel
{
"priority": 250,
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "logs"
},
"codec": "best_compression",
"mapping": {
"ignore_malformed": "true"
}
}
},
"mappings": {
"_source": {
"enabled": true
},
"date_detection": false,
"dynamic": "strict",
"dynamic_templates": [
{
"all_strings_to_keywords": {
"mapping": {
"ignore_above": 1024,
"type": "keyword"
},
"match_mapping_type": "string"
}
},
{
"complex_attributes": {
"path_match": [
"resource.attributes.*",
"scope.attributes.*",
"attributes.*"
],
"match_mapping_type": "object",
"mapping": {
"type": "flattened"
}
}
}
],
"properties": {
"@timestamp": {
"type": "date_nanos",
"ignore_malformed": false
},
"data_stream": {
"type": "object",
"properties": {
"type": {
"type": "constant_keyword"
},
"dataset": {
"type": "constant_keyword"
},
"namespace": {
"type": "constant_keyword"
}
}
},
"observed_timestamp": {
"type": "date_nanos",
"ignore_malformed": true
},
"severity_number": {
"type": "long"
},
"severity_text": {
"type": "keyword"
},
"body_text": {
"type": "match_only_text"
},
"body_structured": {
"type": "flattened"
},
"attributes": {
"type": "passthrough",
"dynamic": true,
"priority": 2
},
"dropped_attributes_count": {
"type": "long"
},
"trace_flags": {
"type": "byte"
},
"trace_id": {
"type": "keyword"
},
"span_id": {
"type": "keyword"
},
"scope": {
"properties": {
"name": {
"type": "keyword"
},
"version": {
"type": "keyword"
},
"attributes": {
"type": "passthrough",
"dynamic": true,
"priority": 1
},
"dropped_attributes_count": {
"type": "long"
},
"schema_url": {
"type": "keyword"
}
}
},
"resource": {
"properties": {
"dropped_attributes_count": {
"type": "long"
},
"schema_url": {
"type": "keyword"
},
"attributes": {
"type": "passthrough",
"dynamic": true,
"priority": 0
}
}
}
}
}
},
"index_patterns": [
"logs-*.otel-*"
],
"data_stream": {}
}
```
Here is an example of the auditd document in Elasticsearch abbreviated:
```
{
"@timestamp": "2024-05-29T13:30:25.085926000Z",
"attributes": {
"foo": "bar",
"some.bool": true
},
"body_structured": {
"MESSAGE": "AVC apparmor=\"STATUS\" operation=\"profile_replace\" info=\"same as current profile, skipping\" profile=\"unconfined\" name=\"/usr/bin/evince-previewer\" pid=2702 comm=\"apparmor_parser\"",
"SYSLOG_FACILITY": "4",
"SYSLOG_IDENTIFIER": "audit",
"_SOURCE_REALTIME_TIMESTAMP": "1716989425080000",
"_TRANSPORT": "audit",
},
"dropped_attributes_count": 0,
"observed_timestamp": "2024-05-29T14:49:26.534908898Z",
"resource": {
"attributes": {
"data_stream.dataset": "auditd.otel",
"data_stream.namespace": "default",
"data_stream.type": "logs",
"host.arch": "arm64",
"host.cpu.cache.l2.size": 0,
"host.cpu.family": "",
"host.cpu.model.id": "0x000",
"host.cpu.model.name": "",
"host.cpu.stepping": "0",
"host.cpu.vendor.id": "Apple",
"host.id": "cae0e0147d454a80971b0b747c8b62b9",
"host.ip": [
"172.16.3.131",
"fe80::20c:29ff:fe66:3012",
"host.name": "lebuntu",
"host.os.description": "Ubuntu 22.04.4 LTS (Jammy Jellyfish) (Linux lebuntu 5.15.0-107-generic #117-Ubuntu SMP Mon Apr 29 14:37:09 UTC 2024 aarch64)",
"host.os.type": "linux",
"os.description": "Ubuntu 22.04.4 LTS (Jammy Jellyfish) (Linux lebuntu 5.15.0-107-generic #117-Ubuntu SMP Mon Apr 29 14:37:09 UTC 2024 aarch64)",
"os.type": "linux"
},
"dropped_attributes_count": 0,
"schema_url": "https://opentelemetry.io/schemas/1.6.1"
},
"severity_number": 0,
"trace_flags": 0
}
```
Here is an example of ECS compatible query that works on this Otel
native schema:
```
GET logs-auditd.otel-default/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"host.name": "lebuntu"
}
}
]
}
}
}
```
**Link to tracking Issue:**
No tracking issue yet.
**Testing:**
Added unit test for OTel transformation.
Tested with journald OTel receiver.
**Documentation:**
No documentation is added yet.
---------
Co-authored-by: Felix Barnsteiner <[email protected]>
Co-authored-by: Carson Ip <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Exporting this function since it can be useful in other cases that parse
text metrics that embedded collectd tags.
After this is merged the duplication of this function mentioned on PR #116 can be avoided.