Add Trace Span Pruning Processor

[portertech]

Summary

This PR introduces the spanpruningprocessor, a new trace processor that reduces trace storage costs while preserving observability value. It intelligently identifies and aggregates repetitive leaf spans within traces, replacing groups of similar operations with single summary spans that capture the full statistical picture.

This is a reduced-scope MVP of #45617 (now closed), focusing on the core aggregation algorithm. Advanced features like outlier detection, outlier preservation, histogram buckets, attribute loss analysis, and byte-size metrics will follow in subsequent PRs once the foundation is merged.

Component donation issue: #45654

The Problem

Modern distributed systems generate enormous volumes of trace data. A significant portion consists of repetitive, similar spans -- think N+1 database queries, batch HTTP calls, or fan-out operations. Storing every individual span is expensive and often provides diminishing analytical value beyond the first few instances.

Current solutions are inadequate:

• Head sampling loses entire traces, breaking root cause analysis
• Tail sampling helps but still keeps every span in sampled traces
• Manual instrumentation changes require code modifications across services

The Solution

The Span Pruning Processor identifies duplicate or similar leaf spans within a single trace, groups them, and replaces each group with a single aggregated summary span. When leaf spans are aggregated, the processor also recursively aggregates their parent spans if all children of those parents are being aggregated.

Leaf spans are spans that are not referenced as a parent by any other span in the trace. They typically represent the last actions in an execution call stack (e.g., individual database queries, HTTP calls to external services).

Spans are grouped by:

1. Span name - spans must have the same name
2. Span kind - spans must have the same kind (Internal, Server, Client, Producer, Consumer)
3. Status code - spans must have the same status (OK, Error, or Unset)
4. TraceState - spans must have identical TraceState values (for Consistent Probability Sampling compatibility)
5. Configured attributes - spans must have matching values for attributes specified in group_by_attributes
6. Parent span name - leaf spans must share the same parent span name to be grouped together

Parent spans are eligible for aggregation when all of their children are aggregated, they share the same name, kind, and status code, and they are not root spans.

Use Cases

• Database query optimization: When an application makes many similar database queries (e.g., N+1 queries), aggregate them into a single summary span
• Batch operations: Consolidate many similar leaf operations into a single representative span
• Cost reduction: Reduce trace storage costs by eliminating redundant span data

Configuration

processors:
  spanpruning:
    # Attributes to use for grouping similar leaf spans (supports glob patterns)
    # Spans with the same name AND same values for matching attributes will be grouped
    # Examples:
    #   - "db.*" matches db.operation, db.name, db.statement, etc.
    #   - "http.request.*" matches http.request.method, http.request.header, etc.
    #   - "db.operation" matches only the exact key "db.operation"
    group_by_attributes:
      - "db.*"
      - "http.method"

    # Minimum number of similar leaf spans required before aggregation
    # Default: 5
    min_spans_to_aggregate: 3

    # Maximum depth of parent span aggregation above leaf spans
    # 0 = only aggregate leaf spans (no parent aggregation)
    # -1 = unlimited depth
    # Default: 1
    max_parent_depth: 1

    # Prefix for aggregation statistics attributes
    # Default: "aggregation."
    aggregation_attribute_prefix: "batch."

Configuration Options

Field	Type	Default	Description
group_by_attributes	[]string	[]	Attribute patterns for grouping (supports glob patterns like db.*)
min_spans_to_aggregate	int	5	Minimum group size before aggregation occurs
max_parent_depth	int	1	Max depth of parent aggregation (0=none, -1=unlimited)
aggregation_attribute_prefix	string	"aggregation."	Prefix for aggregation statistics attributes

Glob Pattern Support

The group_by_attributes field supports glob patterns for matching attribute keys:

Pattern	Matches
db.*	db.operation, db.name, db.statement, etc.
http.request.*	http.request.method, http.request.header.content-type, etc.
rpc.*	rpc.method, rpc.service, rpc.system, etc.
db.operation	Only the exact key db.operation

When multiple attributes match a pattern, they are all included in the grouping key (sorted alphabetically for consistency).

Summary Span

When spans are aggregated, the summary span includes:

Properties

• Name: Original span name (e.g., SELECT)
• TraceID: Same as original spans
• SpanID: Newly generated unique ID
• ParentSpanID: Same as original spans (common parent)
• Kind: Same as template span (inherited from slowest span)
• StartTimestamp: Earliest start time of all spans in the group
• EndTimestamp: Latest end time of all spans in the group
• Status: Same as original spans (spans are grouped by status code)
• TraceState: Inherited from the template span (preserved for Consistent Probability Sampling compatibility)
• Attributes: Inherited from the slowest span in the group
• Events: Inherited from the template (slowest) span
• Links: Inherited from the template span

Note: The summary span's duration (EndTimestamp - StartTimestamp) represents the total time window covered by all aggregated spans, which may exceed duration_max_ns. For example, if spans overlap or are staggered, the time range can be larger than any individual span's duration. Use duration_max_ns to find the slowest individual operation.

What Gets Aggregated Away

When spans are aggregated into a summary span, the following data from non-template spans is lost:

Data	Behavior
Span Events	Only the template (slowest) span's events are preserved
Span Links	Only the template span's links are preserved
Attributes	Non-matching attribute values are lost
Individual Timestamps	Original start/end times replaced by the group's time range
SpanIDs	Original SpanIDs are replaced by a single summary SpanID

Aggregation Attributes

The following attributes are added to the summary span (shown with default aggregation_attribute_prefix: "aggregation."):

Attribute	Type	Description
<prefix>is_summary	bool	Always true to identify summary spans
<prefix>span_count	int64	Number of spans that were aggregated
<prefix>duration_min_ns	int64	Minimum duration in nanoseconds
<prefix>duration_max_ns	int64	Maximum duration in nanoseconds
<prefix>duration_avg_ns	int64	Average duration in nanoseconds
<prefix>duration_total_ns	int64	Total duration in nanoseconds

Pipeline Placement

This processor is designed to work best when placed after processors that ensure complete traces are available:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [groupbytrace, spanpruning, batch]
      exporters: [otlp]

Or with tail sampling:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, spanpruning, batch]
      exporters: [otlp]

Examples

Basic Example

A trace with repeated database queries (some failing):

Before Processing:

root-span (parent)
├── SELECT (leaf) - duration: 10ms, db.operation: select, status: OK
├── SELECT (leaf) - duration: 15ms, db.operation: select, status: OK
├── SELECT (leaf) - duration: 12ms, db.operation: select, status: OK
├── SELECT (leaf) - duration: 50ms, db.operation: select, status: Error
├── SELECT (leaf) - duration: 45ms, db.operation: select, status: Error
└── INSERT (leaf) - duration: 20ms, db.operation: insert, status: OK

After Processing (with min_spans_to_aggregate: 2):

root-span (parent)
├── SELECT (summary, status: OK)
│   - aggregation.is_summary: true
│   - aggregation.span_count: 3
│   - aggregation.duration_min_ns: 10000000
│   - aggregation.duration_max_ns: 15000000
│   - aggregation.duration_avg_ns: 12333333
├── SELECT (summary, status: Error)
│   - aggregation.is_summary: true
│   - aggregation.span_count: 2
│   - aggregation.duration_min_ns: 45000000
│   - aggregation.duration_max_ns: 50000000
│   - aggregation.duration_avg_ns: 47500000
└── INSERT (unchanged - only 1 span, below threshold)

Note: Spans with different status codes are grouped separately, preserving error information.

Recursive Parent Aggregation Example

When spans are aggregated, the processor also checks if their parent spans can be aggregated. Parent spans are eligible for aggregation when:

1. All of their children are being aggregated
2. They share the same name, kind, and status code with other eligible parents
3. They are not root spans (must have a parent)
4. At least 2 parents meet the criteria

Before Processing (with min_spans_to_aggregate: 2, group_by_attributes: ["db.op"]):

root
├── handler (status: OK)
│   └── SELECT (db.op=select, status: OK) ───┐
├── handler (status: OK)                      │ leaf group A: 3 OK SELECTs
│   └── SELECT (db.op=select, status: OK) ───┤
├── handler (status: OK)                      │
│   └── SELECT (db.op=select, status: OK) ───┘
├── handler (status: Error)
│   └── SELECT (db.op=select, status: Error) ┐ leaf group B: 2 Error SELECTs
├── handler (status: Error)                   │
│   └── SELECT (db.op=select, status: Error) ┘
├── handler (status: OK)
│   └── INSERT (db.op=insert, status: OK) ──── only 1, below threshold
└── worker (status: OK)
    └── SELECT (db.op=select, status: OK) ──── different parent name

After Processing:

root
├── handler (summary, status: OK, span_count: 3)
│   └── SELECT (summary, status: OK, span_count: 3)
├── handler (summary, status: Error, span_count: 2)
│   └── SELECT (summary, status: Error, span_count: 2)
├── handler (status: OK)
│   └── INSERT (status: OK) ─────────────────────────── unchanged
└── worker (status: OK)
    └── SELECT (status: OK) ─────────────────────────── unchanged

Why each span was handled this way:

Span	Result	Reason
3x handler (OK) with SELECT children	Aggregated	All children aggregated, same name+kind+status
3x SELECT (OK) under handler	Aggregated	Same name + kind + status + attributes + parent name
2x handler (Error) with SELECT children	Aggregated	All children aggregated, same name+kind+status
2x SELECT (Error) under handler	Aggregated	Same name + kind + status + attributes + parent name
handler (OK) with INSERT child	Unchanged	Child not aggregated (only 1 INSERT)
INSERT (OK)	Unchanged	Below threshold (only 1 span)
worker (OK)	Unchanged	Child not aggregated
SELECT (OK) under worker	Unchanged	Different parent name than other SELECTs

Consistent Probability Sampling (CPS) Compatibility

The processor is designed to be compatible with Consistent Probability Sampling (CPS). CPS uses TraceState to carry sampling metadata (ot=th:...;rv:...) where:

• th (threshold) indicates the sampling probability threshold
• rv (randomness value) provides consistent randomness for sampling decisions

Why TraceState matters for aggregation:

Spans with different TraceState values represent different sampling populations with different "adjusted counts" (weights). Aggregating them together would produce statistically incorrect summaries and break downstream sampling decisions.

The processor uses exact TraceState matching (not just the th value) because:

• The rv value affects sampling decisions
• Vendor-specific keys may have semantic meaning
• Key ordering may be significant

Limitations

• Requires complete traces for accurate leaf detection
• Summary span inherits attributes from the slowest span in the group
• Parent spans are only aggregated when ALL their children are aggregated

Telemetry

The processor emits the following metrics to help monitor its operation:

Counters

Metric	Description
otelcol_processor_spanpruning_spans_received	Total number of spans received by the processor
otelcol_processor_spanpruning_spans_pruned	Total number of spans removed by aggregation
otelcol_processor_spanpruning_aggregations_created	Total number of aggregation summary spans created
otelcol_processor_spanpruning_traces_processed	Total number of traces processed

Histograms

Metric	Description
otelcol_processor_spanpruning_aggregation_group_size	Distribution of the number of spans per aggregation group
otelcol_processor_spanpruning_processing_duration	Time taken to process each batch of traces (in seconds)

These metrics can be used to:

• Monitor the effectiveness of span pruning (compare spans_received vs spans_pruned)
• Track the compression ratio achieved by aggregation
• Identify processing bottlenecks via processing_duration
• Understand aggregation patterns via aggregation_group_size

Scope / Future Work

This MVP focuses on the core aggregation engine. The following features from the original PR (#45617) are planned for follow-up PRs:

• Outlier detection: IQR and MAD-based statistical outlier detection
• Outlier preservation: Keep slow spans as individual spans while aggregating normal ones
• Attribute correlation: Identify attributes that correlate with slow operations
• Histogram buckets: Latency distribution in summary spans
• Attribute loss analysis: Track and report attribute diversity lost during aggregation
• Byte-size metrics: Measure serialized trace sizes before/after pruning

Architecture

The processor operates in three phases per trace:

1. Tree Construction (tree.go): Builds parent-child relationships, identifies leaves and orphans
2. Analysis (processor.go, grouping.go): Groups similar leaf spans by key, then walks up the tree to find eligible parent spans for recursive aggregation
3. Execution (aggregation.go): Sorts groups top-down, creates summary spans with preassigned SpanIDs, and batch-removes originals

Key design decisions:

• Tree-based analysis avoids O(n^2) parent lookups by pre-computing relationships
• Type-safe attribute encoding (grouping.go) ensures correct grouping for all pdata value types (maps, slices, bytes)
• Pooled string builders minimize allocations in the hot grouping-key path
• Single-pass statistics (stats.go) computes min/max/avg/total and time ranges without extra traversals

Link to tracking issue

Fixes #45654

Testing

• Comprehensive unit tests (processor_test.go) covering: leaf span aggregation, recursive parent aggregation at multiple depths, grouping by attributes with glob patterns, status code separation, TraceState/CPS compatibility, span kind grouping, edge cases (empty traces, single spans, orphans, multiple roots), configuration validation, and template span selection (events, links, attributes inherited from slowest span)
• Configuration validation tests (config_test.go) covering all fields and error cases
• Aggregation logic tests (aggregation_test.go) for duration calculation and template selection
• Benchmark tests (processor_benchmark_test.go) measuring throughput across varying trace sizes (100-10000 spans) and group counts
• Generated component lifecycle tests and telemetry tests via mdatagen

Documentation

• Comprehensive README.md with configuration reference, glob pattern examples, summary span schema, pipeline placement guidance, before/after examples (including recursive parent aggregation), CPS compatibility notes, limitations, and telemetry reference
• documentation.md generated from metadata.yaml describing all 6 custom telemetry metrics

[iblancasa]

I think we should add first all the "scaffolding" and, as part of another PR, add functionality step by step.
It is not easy to review with everything in a single PR.

[portertech]

@iblancasa this implementation is a stripped down version, only core functionality. I'll explore options for further decomposition, although I am concerned it will be challenging to navigate the steps.

[portertech]

A decomposition plan like https://gist.github.com/portertech/bc131edd61c7edc94aff4663b4891935 could work, but I'm not convinced this path is truly easier to digest. Fewer steps/phases wouldn't result in much decomposition.

@evan-bradley curious to get your take as well 🙏

[jmacd]

I do not think we as a community benefit from asking @portertech to refactor this component into smaller units for review. However, it would be nice to split it into two PRs: (1) the skeleton with all the changes under ./github and ./chloggen, ./internal/tidylist, as well as the skeleton of the component with minimal config.go, factory.go, Makefile, metadata.yaml, generated code, and processor.go. (2) everything else

[portertech]

@jmacd I created a pr for the skeleton #47452 👍

[iblancasa]

I do not think we as a community benefit from asking @portertech to refactor this component into smaller units for review. However, it would be nice to split it into two PRs: (1) the skeleton with all the changes under ./github and ./chloggen, ./internal/tidylist, as well as the skeleton of the component with minimal config.go, factory.go, Makefile, metadata.yaml, generated code, and processor.go. (2) everything else

I'm fine with that.

[atoulme]

Please rebase.

[portertech]

@atoulme rebased and ready to rock. govulncheck is out of my hands.

[jaydeluca]

@portertech do you plan on adding this back in another PR? Just asking because I can do it if not, but I think we should have this field

[portertech]

Please do 👍

[jaydeluca]

#47572