Skip to content

Add Trace Span Pruning Processor #47277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
atoulme merged 5 commits into from
Apr 10, 2026
Merged

Add Trace Span Pruning Processor #47277

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d5281b7
span pruning processor MVP implementation
portertech Apr 9, 2026
dbd58fa
Add span pruning processor implementation changelog entry
portertech Apr 9, 2026
823b7ca
Align spanpruningprocessor collector module pseudo-versions
portertech Apr 9, 2026
3f96ff4
Regenerate contrib distributions for spanpruning processor
portertech Apr 9, 2026
cebf97a
Merge branch 'main' into trace-span-pruning-mvp
portertech Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .chloggen/spanpruningprocessor-impl.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the component, or a single word describing the area of concern, (e.g. receiver/filelog)
component: processor/spanpruning

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Add full implementation of the span pruning processor.

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [45654]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: [user]
290 changes: 286 additions & 4 deletions processor/spanpruningprocessor/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,294 @@
<!-- status autogenerated section -->
# Span Pruning Processor

<!-- status autogenerated section -->
| Status | |
| ------------- |-----------|
| Stability | [development]: traces |
| Distributions | [] |
| Stability | [alpha]: traces |
| Distributions | [contrib] |
| Issues | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Aprocessor%2Fspanpruning%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aprocessor%2Fspanpruning) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Aprocessor%2Fspanpruning%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aprocessor%2Fspanpruning) |
| Code coverage | [![codecov](https://codecov.io/github/open-telemetry/opentelemetry-collector-contrib/graph/main/badge.svg?component=processor_spanpruning)](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=processor_spanpruning&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@portertech](https://www.github.com/portertech) |

[development]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#development
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
<!-- end autogenerated section -->

## Overview

The Span Pruning Processor identifies duplicate or similar leaf spans within a single trace, groups them, and replaces each group with a single aggregated summary span. When leaf spans are aggregated, the processor also recursively aggregates their parent spans if all children of those parents are being aggregated.

**Leaf spans** are spans that are not referenced as a parent by any other span in the trace. They typically represent the last actions in an execution call stack (e.g., individual database queries, HTTP calls to external services).

Spans are grouped by:
1. **Span name** - spans must have the same name
2. **Span kind** - spans must have the same kind (Internal, Server, Client, Producer, Consumer)
3. **Status code** - spans must have the same status (OK, Error, or Unset)
4. **TraceState** - spans must have identical TraceState values (for Consistent Probability Sampling compatibility)
5. **Configured attributes** - spans must have matching values for attributes specified in `group_by_attributes`
6. **Parent span name** - leaf spans must share the same parent span name to be grouped together

Parent spans are eligible for aggregation when all of their children are aggregated, they share the same name, kind, and status code, and they are not root spans.

This processor is useful for reducing trace data volume while preserving meaningful information about repeated operations.

## Use Cases

- **Database query optimization**: When an application makes many similar database queries (e.g., N+1 queries), aggregate them into a single summary span
- **Batch operations**: Consolidate many similar leaf operations into a single representative span
- **Cost reduction**: Reduce trace storage costs by eliminating redundant span data

## Configuration

```yaml
processors:
spanpruning:
# Attributes to use for grouping similar leaf spans (supports glob patterns)
# Spans with the same name AND same values for matching attributes will be grouped
# Examples:
# - "db.*" matches db.operation, db.name, db.statement, etc.
# - "http.request.*" matches http.request.method, http.request.header, etc.
# - "db.operation" matches only the exact key "db.operation"
group_by_attributes:
- "db.*"
- "http.method"

# Minimum number of similar leaf spans required before aggregation
# Default: 5
min_spans_to_aggregate: 3

# Maximum depth of parent span aggregation above leaf spans
# 0 = only aggregate leaf spans (no parent aggregation)
# -1 = unlimited depth
# Default: 1
max_parent_depth: 1

# Prefix for aggregation statistics attributes
# Default: "aggregation."
aggregation_attribute_prefix: "batch."
```

## Configuration Options

| Field | Type | Default | Description |
|-----|--|---------|-------|
| `group_by_attributes` | []string | [] | Attribute patterns for grouping (supports glob patterns like `db.*`) |
| `min_spans_to_aggregate` | int | 5 | Minimum group size before aggregation occurs |
| `max_parent_depth` | int | 1 | Max depth of parent aggregation (0=none, -1=unlimited) |
| `aggregation_attribute_prefix` | string | "aggregation." | Prefix for aggregation statistics attributes |

### Glob Pattern Support

The `group_by_attributes` field supports glob patterns for matching attribute keys:

| Pattern | Matches |
|-----|--|
| `db.*` | `db.operation`, `db.name`, `db.statement`, etc. |
| `http.request.*` | `http.request.method`, `http.request.header.content-type`, etc. |
| `rpc.*` | `rpc.method`, `rpc.service`, `rpc.system`, etc. |
| `db.operation` | Only the exact key `db.operation` |

When multiple attributes match a pattern, they are all included in the grouping key (sorted alphabetically for consistency).

## Summary Span

When spans are aggregated, the summary span includes:

### Properties
- **Name**: Original span name (e.g., `SELECT`)
- **TraceID**: Same as original spans
- **SpanID**: Newly generated unique ID
- **ParentSpanID**: Same as original spans (common parent)
- **Kind**: Same as template span (inherited from slowest span)
- **StartTimestamp**: Earliest start time of all spans in the group
- **EndTimestamp**: Latest end time of all spans in the group
- **Status**: Same as original spans (spans are grouped by status code)
- **TraceState**: Inherited from the template span (preserved for Consistent Probability Sampling compatibility)
- **Attributes**: Inherited from the slowest span in the group

> **Note**: The summary span's duration (`EndTimestamp - StartTimestamp`) represents the total time window covered by all aggregated spans, which may exceed `duration_max_ns`. For example, if spans overlap or are staggered, the time range can be larger than any individual span's duration. Use `duration_max_ns` to find the slowest individual operation.

### What Gets Aggregated Away

When spans are aggregated into a summary span, the following data from non-template spans is **lost**:

| Data | Behavior |
|------|----------|
| **Span Events** | Events from the template (slowest) span are preserved |
| **Span Links** | Links from the template span are preserved |
| **Attributes** | Non-matching attribute values are lost |
| **Individual Timestamps** | Original start/end times replaced by the group's time range |
| **SpanIDs** | Original SpanIDs are replaced by a single summary SpanID |

### Aggregation Attributes
The following attributes are added to the summary span (shown with default `aggregation_attribute_prefix: "aggregation."`):

| Attribute | Type | Description |
|-----------|------|-------------|
| `<prefix>is_summary` | bool | Always `true` to identify summary spans |
| `<prefix>span_count` | int64 | Number of spans that were aggregated |
| `<prefix>duration_min_ns` | int64 | Minimum duration in nanoseconds |
| `<prefix>duration_max_ns` | int64 | Maximum duration in nanoseconds |
| `<prefix>duration_avg_ns` | int64 | Average duration in nanoseconds |
| `<prefix>duration_total_ns` | int64 | Total duration in nanoseconds |

## Pipeline Placement

This processor is designed to work best when placed after processors that ensure complete traces are available:

```yaml
service:
pipelines:
traces:
receivers: [otlp]
processors: [groupbytrace, spanpruning, batch]
exporters: [otlp]
```

Or with tail sampling:

```yaml
service:
pipelines:
traces:
receivers: [otlp]
processors: [tail_sampling, spanpruning, batch]
exporters: [otlp]
```

## Example

### Basic Example

A trace with repeated database queries (some failing):

**Before Processing:**
```
root-span (parent)
├── SELECT (leaf) - duration: 10ms, db.operation: select, status: OK
├── SELECT (leaf) - duration: 15ms, db.operation: select, status: OK
├── SELECT (leaf) - duration: 12ms, db.operation: select, status: OK
├── SELECT (leaf) - duration: 50ms, db.operation: select, status: Error
├── SELECT (leaf) - duration: 45ms, db.operation: select, status: Error
└── INSERT (leaf) - duration: 20ms, db.operation: insert, status: OK
```

**After Processing (with `min_spans_to_aggregate: 2`):**
```
root-span (parent)
├── SELECT (summary, status: OK)
│ - aggregation.is_summary: true
│ - aggregation.span_count: 3
│ - aggregation.duration_min_ns: 10000000
│ - aggregation.duration_max_ns: 15000000
│ - aggregation.duration_avg_ns: 12333333
├── SELECT (summary, status: Error)
│ - aggregation.is_summary: true
│ - aggregation.span_count: 2
│ - aggregation.duration_min_ns: 45000000
│ - aggregation.duration_max_ns: 50000000
│ - aggregation.duration_avg_ns: 47500000
└── INSERT (unchanged - only 1 span, below threshold)
```

Note: Spans with different status codes are grouped separately, preserving error information.

### Recursive Parent Aggregation Example

When spans are aggregated, the processor also checks if their parent spans can be aggregated. Parent spans are eligible for aggregation when:
1. All of their children are being aggregated
2. They share the same name, kind, and status code with other eligible parents
3. They are not root spans (must have a parent)
4. At least 2 parents meet the criteria

**Before Processing (with `min_spans_to_aggregate: 2`, `group_by_attributes: ["db.op"]`):**
```
root
├── handler (status: OK)
│ └── SELECT (db.op=select, status: OK) ───┐
├── handler (status: OK) │ leaf group A: 3 OK SELECTs
│ └── SELECT (db.op=select, status: OK) ───┤
├── handler (status: OK) │
│ └── SELECT (db.op=select, status: OK) ───┘
├── handler (status: Error)
│ └── SELECT (db.op=select, status: Error) ┐ leaf group B: 2 Error SELECTs
├── handler (status: Error) │
│ └── SELECT (db.op=select, status: Error) ┘
├── handler (status: OK)
│ └── INSERT (db.op=insert, status: OK) ──── only 1, below threshold
└── worker (status: OK)
└── SELECT (db.op=select, status: OK) ──── different parent name
```

**After Processing:**
```
root
├── handler (summary, status: OK, span_count: 3)
│ └── SELECT (summary, status: OK, span_count: 3)
├── handler (summary, status: Error, span_count: 2)
│ └── SELECT (summary, status: Error, span_count: 2)
├── handler (status: OK)
│ └── INSERT (status: OK) ─────────────────────────── unchanged
└── worker (status: OK)
└── SELECT (status: OK) ─────────────────────────── unchanged
```

**Why each span was handled this way:**

| Span | Result | Reason |
|------|--------|--------|
| 3x handler (OK) with SELECT children | Aggregated | All children aggregated, same name+kind+status |
| 3x SELECT (OK) under handler | Aggregated | Same name + kind + status + attributes + parent name |
| 2x handler (Error) with SELECT children | Aggregated | All children aggregated, same name+kind+status |
| 2x SELECT (Error) under handler | Aggregated | Same name + kind + status + attributes + parent name |
| handler (OK) with INSERT child | Unchanged | Child not aggregated (only 1 INSERT) |
| INSERT (OK) | Unchanged | Below threshold (only 1 span) |
| worker (OK) | Unchanged | Child not aggregated |
| SELECT (OK) under worker | Unchanged | Different parent name than other SELECTs |

## Limitations

- Requires complete traces for accurate leaf detection
- Summary span inherits attributes from the slowest span in the group
- Parent spans are only aggregated when ALL their children are aggregated

## Consistent Probability Sampling (CPS) Compatibility

The processor is designed to be compatible with [Consistent Probability Sampling](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/) (CPS). CPS uses TraceState to carry sampling metadata (`ot=th:...;rv:...`) where:

- `th` (threshold) indicates the sampling probability threshold
- `rv` (randomness value) provides consistent randomness for sampling decisions

**Why TraceState matters for aggregation:**

Spans with different TraceState values represent different sampling populations with different "adjusted counts" (weights). Aggregating them together would produce statistically incorrect summaries and break downstream sampling decisions.

The processor uses **exact TraceState matching** (not just the `th` value) because:
- The `rv` value affects sampling decisions
- Vendor-specific keys may have semantic meaning
- Key ordering may be significant

## Telemetry

The processor emits the following metrics to help monitor its operation:

### Counters

| Metric | Description |
|--------|-------------|
| `otelcol_processor_spanpruning_spans_received` | Total number of spans received by the processor |
| `otelcol_processor_spanpruning_spans_pruned` | Total number of spans removed by aggregation |
| `otelcol_processor_spanpruning_aggregations_created` | Total number of aggregation summary spans created |
| `otelcol_processor_spanpruning_traces_processed` | Total number of traces processed |

### Histograms

| Metric | Description |
|--------|-------------|
| `otelcol_processor_spanpruning_aggregation_group_size` | Distribution of the number of spans per aggregation group |
| `otelcol_processor_spanpruning_processing_duration` | Time taken to process each batch of traces (in seconds) |

These metrics can be used to:
- Monitor the effectiveness of span pruning (compare `spans_received` vs `spans_pruned`)
- Track the compression ratio achieved by aggregation
- Identify processing bottlenecks via `processing_duration`
- Understand aggregation patterns via `aggregation_group_size`
Loading
Loading