[receiver/datadog] Span Obfuscation#1
Merged
Conversation
bezmax
reviewed
Oct 31, 2024
bezmax
approved these changes
Oct 31, 2024
lopes-felipe
pushed a commit
that referenced
this pull request
Dec 23, 2024
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Generates simple histograms using telemetrygen <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes <!--Describe what testing was performed and which tests were added.--> #### Testing Test with a local otel collector with debug output ``` bin/telemetrygen metrics --metrics 5 --otlp-http --otlp-endpoint "localhost:4318" --metric-type Histogram --otlp-insecure ``` Output from debug Exporter: ``` Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.633365 +0000 UTC Timestamp: 2024-11-13 16:22:51.633367 +0000 UTC Count: 0 Sum: 0.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 0 Buckets #3, Count: 0 Buckets #4, Count: 0 ResourceMetrics #1 Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.639942 +0000 UTC Timestamp: 2024-11-13 16:22:51.639942 +0000 UTC Count: 1 Sum: 1.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 0 Buckets #1, Count: 1 Buckets #2, Count: 0 Buckets #3, Count: 0 Buckets #4, Count: 0 ResourceMetrics #2 Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.6404 +0000 UTC Timestamp: 2024-11-13 16:22:51.640401 +0000 UTC Count: 2 Sum: 4.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 0 Buckets #1, Count: 1 Buckets #2, Count: 0 Buckets #3, Count: 1 Buckets #4, Count: 0 ResourceMetrics #3 Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.640729 +0000 UTC Timestamp: 2024-11-13 16:22:51.640729 +0000 UTC Count: 3 Sum: 3.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 1 Buckets #1, Count: 1 Buckets #2, Count: 1 Buckets #3, Count: 0 Buckets #4, Count: 0 ResourceMetrics #4 Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.641073 +0000 UTC Timestamp: 2024-11-13 16:22:51.641073 +0000 UTC Count: 4 Sum: 12.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 1 Buckets #3, Count: 2 Buckets #4, Count: 1 {"kind": "exporter", "data_type": "metrics", "name": "debug"} ``` <!--Describe the documentation added.--> #### Documentation <!--Please delete paragraphs that you did not use before submitting.--> --------- Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>
lopes-felipe
pushed a commit
that referenced
this pull request
Feb 11, 2025
#### Description Vulnerability #1: GO-2025-3420 Sensitive headers incorrectly sent after cross-domain redirect in net/http More info: https://pkg.go.dev/vuln/GO-2025-3420 Standard library Found in: net/http@go1.22.8 Fixed in: net/http@go1.22.11 Example traces found: Error: #1: codeowners.go:212:55: githubgen.codeownersGenerator.getGithubMembers calls github.OrganizationsService.ListMembers, which eventually calls http.Client.Do Vulnerability #2: GO-[20](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/13003223509/job/36265594395?pr=37492#step:6:21)25-3373 Usage of IPv6 zone IDs can bypass URI name constraints in crypto/x509 More info: https://pkg.go.dev/vuln/GO-2025-3373 Standard library Found in: crypto/x509@go1.[22](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/13003223509/job/36265594395?pr=37492#step:6:23).8 Fixed in: crypto/x509@go1.22.11 Example traces found: Related: open-telemetry/opentelemetry-collector#12197
lopes-felipe
pushed a commit
that referenced
this pull request
Apr 18, 2025
…ry#38146) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description This PR adds support for configuring metric temporality (delta or cumulative) in the telemetrygen tool. This allows users to generate metrics with different temporality types, which is particularly useful for testing different metric collection scenarios. ##### Changes - Added `temporalityType` flag for metrics pipeline that accepts values: `delta` or `cumulative` <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#38073 <!--Describe what testing was performed and which tests were added.--> #### Testing Testing was performed by setting up as simple collector with `otlp `receiver and `debug` exporter ##### Sum Datapoint ``` Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 Resource attributes: -> service.name: Str(telemetrygen) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Delta ``` ##### Histogram Datapoint ``` Count: 0 Sum: 3940.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 5.000000 ExplicitBounds #2: 10.000000 ExplicitBounds #3: 25.000000 ExplicitBounds #4: 50.000000 ExplicitBounds #5: 75.000000 ExplicitBounds #6: 100.000000 ExplicitBounds open-telemetry#7: 250.000000 ExplicitBounds open-telemetry#8: 500.000000 ExplicitBounds open-telemetry#9: 750.000000 ExplicitBounds open-telemetry#10: 1000.000000 ExplicitBounds open-telemetry#11: 2500.000000 ExplicitBounds open-telemetry#12: 5000.000000 ExplicitBounds open-telemetry#13: 7500.000000 ExplicitBounds open-telemetry#14: 10000.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 1 Buckets #3, Count: 0 Buckets #4, Count: 0 Buckets #5, Count: 0 Buckets #6, Count: 3 Buckets open-telemetry#7, Count: 4 Buckets open-telemetry#8, Count: 1 Buckets open-telemetry#9, Count: 1 Buckets open-telemetry#10, Count: 0 Buckets open-telemetry#11, Count: 0 Buckets open-telemetry#12, Count: 0 Buckets open-telemetry#13, Count: 0 Buckets open-telemetry#14, Count: 0 ``` - [x] Need to add tests for the config --------- Signed-off-by: Nikos Angelopoulos <nikolaos.angelopoulos@grafana.com>
lopes-felipe
pushed a commit
that referenced
this pull request
Apr 18, 2025
…-telemetry#38500) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Breaking change. Overhaul in document routing. New document routing logic: ``` Documents are statically or dynamically routed to the target index / data stream in the following order. The first routing mode that applies will be used. 1. "Static mode": Route to `logs_index` for log records, `metrics_index` for data points and `traces_index` for spans, if these configs are not empty respectively. [^3] 2. "Dynamic - Index attribute mode": Route to index name specified in `elasticsearch.index` attribute (precedence: log record / data point / span attribute > scope attribute > resource attribute) if the attribute exists. [^3] 3. "Dynamic - Data stream routing mode": Route to data stream constructed from `${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}`, where `data_stream.type` is `logs` for log records, `metrics` for data points, and `traces` for spans, and is static. [^3] In a special case with `mapping::mode: bodymap`, `data_stream.type` field (valid values: `logs`, `metrics`) can be dynamically set from attributes. The resulting documents will contain the corresponding `data_stream.*` fields, see restrictions applied to [Data Stream Fields](https://www.elastic.co/guide/en/ecs/current/ecs-data_stream.html). 1. `data_stream.dataset` or `data_stream.namespace` in attributes (precedence: log record / data point / span attribute > scope attribute > resource attribute) 2. Otherwise, if scope name matches regex `/receiver/(\w*receiver)`, `data_stream.dataset` will be capture group #1 3. Otherwise, `data_stream.dataset` falls back to `generic` and `data_stream.namespace` falls back to `default`. ``` ``` In OTel mapping mode (`mapping::mode: otel`), there is special handling in addition to the above document routing rules in [Elasticsearch document routing](#elasticsearch-document-routing). The order to determine the routing mode is the same as [Elasticsearch document routing](#elasticsearch-document-routing). 1. "Static mode": Span events are separate documents routed to `logs_index` if non-empty. 2. "Dynamic - Index attribute mode": Span events are separate documents routed using attribute `elasticsearch.index` (precedence: span event attribute > scope attribute > resource attribute) if the attribute exists. 3. "Dynamic - Data stream routing mode": - For all documents, `data_stream.dataset` will always be appended with `.otel`. - A special case to (3)(1) in [Elasticsearch document routing](#elasticsearch-document-routing), span events are separate documents that have `data_stream.type: logs` and are routed using data stream attributes (precedence: span event attribute > scope attribute > resource attribute) ``` Effective changes: - Deprecate and make `{logs,metrics,traces}_dynamic_index` config no-op - Config validation error on `{logs,metrics,traces}_dynamic_index::enabled` and `{logs,metrics,traces}_index` set at the same time, as users who rely on dynamic index should not set `{logs,metrics,traces}_index`. - Remove `elasticsearch.index.{prefix,suffix}` handling. Replace it with `elasticsearch.index` handling that uses attribute value as index directly. Users rely on the previously supported `elasticsearch.index.prefix` and `elasticsearch.index.suffix` should migrate to a transform processor that sets `elasticsearch.index`. - Fix a bug where receiver-based routing overwrites data_stream.dataset. Should be released together with open-telemetry#38458 <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#38361 <!--Describe what testing was performed and which tests were added.--> #### Testing <!--Describe the documentation added.--> #### Documentation <!--Please delete paragraphs that you did not use before submitting.--> --------- Co-authored-by: Andrzej Stencel <andrzej.stencel@elastic.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Datadog Agent's span processor has an obfuscation pipeline that—among other things—obfuscates sensitive data, cleans up span resource names, and extracts particular components into specific span metadata.
This PR implements the same behavior into the
datadogreceiver, but disabled by default for retro compatibility with existing users.PS: This is already running in our environments. Just replicating here to remove my personal fork.