Skip to content

[DOCS-13090] Add OP utilization metrics #33838

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maycmlee merged 8 commits into from
Jan 22, 2026
Merged

[DOCS-13090] Add OP utilization metrics #33838

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
28ab498
add metrics
maycmlee Jan 13, 2026
663ff36
use buffer shortcode
maycmlee Jan 13, 2026
09bf9f2
Update content/en/observability_pipelines/monitoring_and_troubleshoot…
maycmlee Jan 13, 2026
f169e95
Update layouts/shortcodes/observability_pipelines/metrics/buffer.en.md
maycmlee Jan 16, 2026
12773e0
Apply suggestions from code review
maycmlee Jan 16, 2026
792b987
remove buffer max event size metrics
maycmlee Jan 22, 2026
447528b
Merge branch 'master' into may/add-op-comp-buffer-metrics
maycmlee Jan 22, 2026
03db895
Update content/en/observability_pipelines/monitoring_and_troubleshoot…
maycmlee Jan 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions ...bservability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,16 @@ Data dropped intentionally or unintentionally
: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped.
: **Available for**: Sources, processors, and destinations.

Timed out events
Comment thread
maycmlee marked this conversation as resolved.
: **Metric**: `pipelines.component_timed_out_events_total`
: **Description**: The number of events that timed out.
Comment thread
maycmlee marked this conversation as resolved.
Outdated
: **Available for**: Sources, processors, and destinations.
Comment thread
maycmlee marked this conversation as resolved.
Outdated

Time out requests
Comment thread
maycmlee marked this conversation as resolved.
Outdated
: **Metric**: `pipelines.component_timed_out_requests_total`
: **Description**: The number of requests that timed out.
Comment thread
maycmlee marked this conversation as resolved.
Outdated
: **Available for**: Sources, processors, and destinations.
Comment thread
maycmlee marked this conversation as resolved.
Outdated

@iadjivon iadjivon Jan 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be addressed in a different PR, but visually, I wonder if it might be best to emulate the Buffer metrics (when buffering is enabled) format without mentioning
Metric, Description, and Available for

@maycmlee maycmlee Jan 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @iadjivon! I don't quite understand. Are you saying to put the metric name as the header instead of Timed out request?

Utilization
: **Metric**: `pipelines.utilization`
: **Description**: The component's activity. A value of `0` indicates an idle component that is waiting for input. A value of `1` indicates a component that is never idle, which means that the component is likely a bottleneck in the processing topology that is creating backpressure, which might cause events to be dropped.
Expand Down
20 changes: 19 additions & 1 deletion layouts/shortcodes/observability_pipelines/metrics/buffer.en.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,22 @@ Track buffer behavior with these additional metrics:
: Total bytes successfully flushed from the buffer.

`pipelines.buffer_discarded_events_total`
: Events discarded from the buffer (for example, due to overflow).
: Events discarded from the buffer (for example, due to overflow).
Comment thread
maycmlee marked this conversation as resolved.
Outdated

`pipelines.source_buffer_max_event_size`
: Maximum event size for a source's buffer.

@bruceg bruceg Jan 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an unfortunate choice of metric name, as it actually means the maximum buffer size in terms of number of events. It probably should have been max_size_events which would follow other standard metric naming conventions. I will ask the Vector team their thoughts about renaming this. It'll be a breaking change but we haven't really advertised this yet.

@bruceg bruceg Jan 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this followed the Vector buffer metric naming spec which contradicts the Vector metric naming spec

@maycmlee maycmlee Jan 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh okay, that actually makes more sense. If you and Vector want to make the name change, let me know if you I should hold off publishing this.

Suggested change
: Maximum event size for a source's buffer.
: A source's maximum buffer size, defined as the number of events it can hold.

@bruceg bruceg Jan 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WIP: enhancement(observability): Standardize buffer size metric names

@maycmlee maycmlee Jan 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed these metrics since the names are getting changed and released with 2.13 release:

  • pipelines.source_buffer_max_event_size
  • pipelines.transform_buffer_max_event_size


`pipelines.source_buffer_utilization`
: The percentage of a source's buffer that is used.

`pipelines.source_buffer_utilization_level`
: Utilization level for a source's buffer.
Comment thread
maycmlee marked this conversation as resolved.
Outdated

`pipelines.transform_buffer_max_event_size`
: Max event size for a transform's buffer.

`pipelines.transform_buffer_utilization`
: The percentage of a transform's buffer that is used.

`pipelines.transform_buffer_utilization_level`
: Utilization level for a transform's buffer.
Comment thread
maycmlee marked this conversation as resolved.
Outdated
Loading