[otap-dataflow] Debug Exporter by c1ly · Pull Request #644 · open-telemetry/otel-arrow

c1ly · 2025-06-23T23:06:00Z

Add a debug exporter to the otlp crate

The exporter will output to stdout various levels of verbosity (basic -> normal -> detailed) on the otlp objects going through the pipeline

…, takes in otlp data and outputs information to console

…! with push_str

codecov · 2025-06-23T23:07:32Z

Codecov Report

Attention: Patch coverage is 95.27058% with 104 lines in your changes missing coverage. Please review.

Project coverage is 73.43%. Comparing base (a1aa4a5) to head (1607cb5).
Report is 18 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #644      +/-   ##
==========================================
+ Coverage   72.46%   73.43%   +0.97%     
==========================================
  Files         304      310       +6     
  Lines       49258    51457    +2199     
==========================================
+ Hits        35695    37789    +2094     
- Misses      13029    13134     +105     
  Partials      534      534

Components	Coverage Δ
otap-dataflow	`85.35% <95.27%> (+1.82%)`	⬆️
beaubourg	`66.86% <ø> (ø)`
otel-arrow-rust	`81.93% <ø> (ø)`
query_abstraction	`80.61% <ø> (ø)`
query_engine	`74.54% <ø> (ø)`
syslog_cef_receivers	`99.02% <ø> (ø)`
otel-arrow-go	`52.95% <ø> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull Request Overview

This PR adds a new debug exporter to the otlp crate that outputs OTLP objects to stdout with configurable verbosity levels. Key changes include:

Integration of the new debug_exporter module into lib.rs.
Addition of a Verbosity enum and a marshaler implementation for logging OTLP data in various formats.
Implementation of a DebugExporter that handles different OTLP data types and configuration-related control messages.

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
rust/otap-dataflow/crates/otlp/src/lib.rs	Added a module reference for the new debug exporter.
rust/otap-dataflow/crates/otlp/src/debug_exporter/verbosity.rs	Introduces the Verbosity enum to control output detail.
rust/otap-dataflow/crates/otlp/src/debug_exporter/mod.rs	Exposes exporter, marshaler, and verbosity modules.
rust/otap-dataflow/crates/otlp/src/debug_exporter/marshaler.rs	Implements a normal marshaler to convert OTLP messages into string reports.
rust/otap-dataflow/crates/otlp/src/debug_exporter/exporter.rs	Implements the DebugExporter and its associated push functions for different OTLP data types.

Comments suppressed due to low confidence (1)

rust/otap-dataflow/crates/otlp/src/debug_exporter/exporter.rs:151

The variable 'resouce_metrics' appears to be misspelled. Consider renaming it to 'resource_metrics' for better clarity.

    let resouce_metrics = metric_request.resource_metrics.len();

c1ly · 2025-06-30T22:28:25Z

Example Output => Basic Verbosity

Timer tick received
OTLP Metric objects received: 0
OTLP Trace objects received: 0
OTLP Profile objects received: 0
OTLP Log objects received: 0
Received 1 resource metrics
Received 5 metrics
Received 5 data points
Received 1 resource spans
Received 1 spans
Received 1 events
Received 1 links
Received 1 resource logs
Received 1 log records
Received 1 events
Received 1 resource profiles
Received 0 samples
Shutdown message received

Example Output => Normal Verbosity

Timer tick received
OTLP Metric objects received: 0
OTLP Trace objects received: 0
OTLP Profile objects received: 0
OTLP Log objects received: 0
Received 1 resource metrics
Received 5 metrics
Received 5 data points
ResourceMetric #0, Schema:[http://schema.opentelemetry.io], Attributes: ip=192.168.0.2
ScopeMetric #0, Name: library, Version: @v1, Schema: [http://schema.opentelemetry.io], Attributes: instrumentation_scope_k1=k1 value
system.cpu.time 0
system.cpu.time freq=3GHz count=0 sum=56 min=12 max=100.1
system.cpu.time freq=3GHz count=0 sum=56 min=12 max=100.1 le94.17542094619048=0
system.cpu.time cpu_logical_processors=8 0
system.cpu.time cpu_cores=4 count=0 sum=56 q0=0
Received 1 resource spans
Received 1 spans
Received 1 events
Received 1 links
ResourceSpan #0, Schema:[http://schema.opentelemetry.io], Attributes: ip=192.168.0.1
ScopeSpan #0, Name: library, Version: @v1, Schema: [http://schema.opentelemetry.io], Attributes: hostname=host5.retailer.com
Name: user-account, Trace ID: 4327e52011a22f9662eac217d77d1ec0, Span ID: 7271ee06d7e5925f, Attributes: hostname=host4.gov
Received 1 resource logs
Received 1 log records
Received 1 events
ResourceLog #0, Schema:[http://schema.opentelemetry.io], Attributes: version=2.0
ScopeLog #0, Name: library, Version: @v1, Schema: [http://schema.opentelemetry.io], Attributes: hostname=host5.retailer.com
Body: Sint impedit non ut eligendi nisi neque harum maxime adipisci., Attributes: hostname=host3.thedomain.edu
Received 1 resource profiles
Received 0 samples
ResourceProfile #0, Schema:[http://schema.opentelemetry.io], Attributes: hostname=host7.com
ScopeProfile #0, Name: library, Version: @v1, Schema: [http://schema.opentelemetry.io], Attributes: hostname=host5.retailer.com
Shutdown message received

Example Output => Detailed Verbosity

Timer tick received
OTLP Metric objects received: 0
OTLP Trace objects received: 0
OTLP Profile objects received: 0
OTLP Log objects received: 0
Received 1 resource metrics
Received 5 metrics
Received 5 data points
ResourceMetric #0
Resource SchemaURL: http://schema.opentelemetry.io
Resource attributes:
-> ip: 192.168.0.2
ScopeMetrics #0
ScopeMetrics SchemaURL: http://schema.opentelemetry.io
Instrumentation Scope library @v1
Instrumentation Scope Attributes:
-> instrumentation_scope_k1: k1 value
Metric #0
Descriptor:
-> Name: system.cpu.time
-> Description: time cpu has ran
-> Unit: s
-> DataType: Gauge
NumberDataPoints #0
Attributes:
StartTimestamp: 1650499200000000100
Timestamp: 1663718400000001400
Value: 0
Metric #1
Descriptor:
-> Name: system.cpu.time
-> Description: time cpu has ran
-> Unit: s
-> DataType: Exponential Histogram
-> AggregationTemporality: 4
ExponentialHistogramDataPoints #0
Attributes:
-> freq: 3GHz
StartTimestamp: 1650499200000000000
Timestamp: 1663718400000001400
Count: 0
Sum: 56
Min: 12
Max: 100.1
Bucket [-4.113250378782927, -1), Count: 0
Bucket (1, 4.113250378782927], Count: 0
Exemplars:
Exemplar #0
-> Trace ID: 4327e52011a22f9662eac217d77d1ec0
-> Span ID: 7271ee06d7e5925f
-> Timestamp: 1663718400000001400
-> Value: 22.2
-> FilteredAttributes:
-> cpu: 0
Metric #2
Descriptor:
-> Name: system.cpu.time
-> Description: time cpu has ran
-> Unit: s
-> DataType: Histogram
-> AggregationTemporality: 4
HistogramDataPoints 0
Attributes:
-> freq: 3GHz
StartTimestamp: 1650499200000000000
Timestamp: 1663718400000001400
Count: 0
Sum: 56
Min: 12
Max: 100.1
ExplicitBound 0: 94.17542094619048
ExplicitBound 1: 65.66722851519177
Buckets 0, Count: 0
Exemplars:
Exemplar #0
-> Trace ID: 4327e52011a22f9662eac217d77d1ec0
-> Span ID: 7271ee06d7e5925f
-> Timestamp: 1663718400000001400
-> Value: 22.2
-> FilteredAttributes:
-> cpu: 0
Metric #3
Descriptor:
-> Name: system.cpu.time
-> Description: time cpu has ran
-> Unit: s
-> DataType: Sum
-> IsMonotonic: true
-> AggregationTemporality: 4
NumberDataPoints #0
Attributes:
-> cpu_logical_processors: 8
StartTimestamp: 1650499200000000000
Timestamp: 1663718400000001400
Value: 0
Exemplars:
Exemplar #0
-> Trace ID: 4327e52011a22f9662eac217d77d1ec0
-> Span ID: 7271ee06d7e5925f
-> Timestamp: 1663718400000001400
-> Value: 22.2
-> FilteredAttributes:
-> ************: true
Metric #4
Descriptor:
-> Name: system.cpu.time
-> Description: time cpu has ran
-> Unit: s
-> DataType: Summary
SummaryDataPoints 0
Attributes:
-> cpu_cores: 4
StartTimestamp: 1650499200000000100
Timestamp: 1663718400000001400
Count: 0
Sum: 56
QuantileValue 0: Quantile 0, Value 0
Received 1 resource spans
Received 1 spans
Received 1 events
Received 1 links
ResourceSpan #0
Resource SchemaURL: http://schema.opentelemetry.io
Resource attributes
-> ip: 192.168.0.1
ScopeSpans #0
ScopeSpans SchemaURL: http://schema.opentelemetry.io
Instrumentation Scope library @v1
Instrumentation Scope Attributes:
-> hostname: host5.retailer.com
Span 0
Trace ID: 4327e52011a22f9662eac217d77d1ec0
Parent ID: 7271ee06d7e5925f
ID: 7271ee06d7e5925f
Name: user-account
Kind: 4
TraceState: ended
Start time: 1647648000000000106
End time: 1647648000000000104
Status code: 2
Status message: Error
Attributes:
-> hostname: host4.gov
Events:
SpanEvent 0
-> Name: message-receive
-> Timestamp: 1647648000000000108
-> DroppedAttributesCount: 0
-> Attributes:
-> hostname: host5.retailer.com
Links:
SpanLink: 0
-> Trace ID: 4327e52011a22f9662eac217d77d1ec0
-> Span ID: 7271ee06d7e5925f
-> TraceState: ended
-> DroppedAttributesCount: 0
-> Attributes:
-> hostname: host2.org
Received 1 resource logs
Received 1 log records
Received 1 events
ResourceLog #0
Resource SchemaURL: http://schema.opentelemetry.io
Resource attributes:
-> version: 2.0
ScopeLogs #0
ScopeLogs SchemaURL: http://schema.opentelemetry.io
Instrumentation Scope library @v1
Instrumentation Scope Attributes:
-> hostname: host5.retailer.com
LogRecord #0
ObservedTimestamp: 1663718400000001300
Timestamp: 2000000000
SeverityText: INFO
SeverityNumber: 2
EventName: event1
Body: Sint impedit non ut eligendi nisi neque harum maxime adipisci.
Attributes:
-> hostname: host3.thedomain.edu
Trace ID: 4327e52011a22f9662eac217d77d1ec0
Span ID: 7271ee06d7e5925f
Flags: 8
Received 1 resource profiles
Received 0 samples
ResourceProfile #0
Resource SchemaURL: http://schema.opentelemetry.io
Resource attributes
-> hostname: host7.com
ScopeProfiles #0
ScopeProfiles SchemaURL: http://schema.opentelemetry.io
Instrumentation Scope library @v1
Instrumentation Scope Attributes:
-> hostname: host5.retailer.com
Profile 0
Profile ID:
Start time: 0
Duration: 0
Dropped attributes count: 0
Location indices: []
Shutdown message received

lquerel

Suggested few modifications. Once updated, we can merge this PR.

…utput it at the end when the exporter shuts down

Add a debug exporter to the otlp crate The exporter will output to stdout various levels of verbosity (basic -> normal -> detailed) on the otlp objects going through the pipeline

c1ly added 4 commits June 16, 2025 17:19

create debug exporter struct that implements the local exporter trait…

a3b487c

…, takes in otlp data and outputs information to console

check in progress

3f9ea73

use stringbuffer to format and build the output log instead of format…

a12bc11

…! with push_str

Merge branch 'main' into debug-exporter

0841f5b

c1ly requested a review from a team as a code owner June 23, 2025 23:06

c1ly marked this pull request as draft June 23, 2025 23:06

lquerel reviewed Jun 24, 2025

View reviewed changes

Comment thread rust/otap-dataflow/crates/otlp/src/debug_exporter/exporter.rs Outdated

lquerel reviewed Jun 24, 2025

View reviewed changes

Comment thread rust/otap-dataflow/crates/otlp/src/debug_exporter/verbosity.rs Outdated

lquerel reviewed Jun 24, 2025

View reviewed changes

Comment thread rust/otap-dataflow/crates/otlp/src/debug_exporter/exporter.rs

lquerel reviewed Jun 24, 2025

View reviewed changes

Comment thread rust/otap-dataflow/crates/otlp/src/debug_exporter/exporter.rs

jmacd requested a review from Copilot June 24, 2025 23:56

Copilot AI reviewed Jun 24, 2025

View reviewed changes

Comment thread rust/otap-dataflow/crates/otlp/src/debug_exporter/verbosity.rs Outdated

Comment thread rust/otap-dataflow/crates/otlp/src/debug_exporter/marshaler.rs Outdated

Comment thread rust/otap-dataflow/crates/otlp/src/debug_exporter/exporter.rs Outdated

c1ly added 3 commits June 26, 2025 17:26

added tests and marshaler for detailed verbosity

eb96059

add the detailed marshaler

d5dc977

added additional tests for the exporter and the marshalers

54d7570

c1ly changed the title ~~[WIP] Debug Exporter~~ [otap-dataflow] Debug Exporter Jun 30, 2025

fix some typos

6abb010

c1ly marked this pull request as ready for review June 30, 2025 22:27

c1ly added 3 commits June 30, 2025 15:53

fix tests

026d145

Merge branch 'main' into debug-exporter

50d16c1

merge main

8c6e4b5

lquerel approved these changes Jul 1, 2025

View reviewed changes

c1ly and others added 6 commits July 1, 2025 17:08

added a counter struct that will store all the accumulated data and o…

0af9c50

…utput it at the end when the exporter shuts down

clean up readme

842a0d2

fix clippy warning

8d57ae6

Merge branch 'main' into debug-exporter

e0df496

clean up comments

0af6001

ran cargo fmt

1607cb5

jmacd added this pull request to the merge queue Jul 9, 2025

Merged via the queue into open-telemetry:main with commit 4f55ba7 Jul 9, 2025
47 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[otap-dataflow] Debug Exporter#644

[otap-dataflow] Debug Exporter#644
jmacd merged 17 commits into
open-telemetry:mainfrom
c1ly:debug-exporter

c1ly commented Jun 23, 2025

Uh oh!

codecov Bot commented Jun 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

c1ly commented Jun 30, 2025

Uh oh!

lquerel left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

c1ly commented Jun 23, 2025

Uh oh!

codecov Bot commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

c1ly commented Jun 30, 2025

Example Output => Basic Verbosity

Example Output => Normal Verbosity

Example Output => Detailed Verbosity

Uh oh!

lquerel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Jun 23, 2025 •

edited

Loading