From 28ab4985ba1591a970840305ce2037f2e8fe00b9 Mon Sep 17 00:00:00 2001 From: May Lee Date: Tue, 13 Jan 2026 10:24:33 -0500 Subject: [PATCH 1/7] add metrics --- .../pipeline_usage_metrics.md | 10 ++++++++++ .../metrics/buffer.en.md | 20 ++++++++++++++++++- 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md index 3f6ddfb448d..90a9b1d5232 100644 --- a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md +++ b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md @@ -77,6 +77,16 @@ Data dropped intentionally or unintentionally : **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. : **Available for**: Sources, processors, and destinations. +Timed out events +: **Metric**: `pipelines.component_timed_out_events_total` +: **Description**: The number of events that timed out. +: **Available for**: Sources, processors, and destinations. + +Time out requests +: **Metric**: `pipelines.component_timed_out_requests_total` +: **Description**: The number of requests that timed out. +: **Available for**: Sources, processors, and destinations. + Utilization : **Metric**: `pipelines.utilization` : **Description**: The component's activity. A value of `0` indicates an idle component that is waiting for input. A value of `1` indicates a component that is never idle, which means that the component is likely a bottleneck in the processing topology that is creating backpressure, which might cause events to be dropped. diff --git a/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md b/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md index d1c636b2fcb..b66bf47d47b 100644 --- a/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md +++ b/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md @@ -19,4 +19,22 @@ Track buffer behavior with these additional metrics: : Total bytes successfully flushed from the buffer. `pipelines.buffer_discarded_events_total` -: Events discarded from the buffer (for example, due to overflow). \ No newline at end of file +: Events discarded from the buffer (for example, due to overflow). + +`pipelines.source_buffer_max_event_size` +: Maximum event size for a source's buffer. + +`pipelines.source_buffer_utilization` +: The percentage of a source's buffer that is used. + +`pipelines.source_buffer_utilization_level` +: Utilization level for a source's buffer. + +`pipelines.transform_buffer_max_event_size` +: Max event size for a transform's buffer. + +`pipelines.transform_buffer_utilization` +: The percentage of a transform's buffer that is used. + +`pipelines.transform_buffer_utilization_level` +: Utilization level for a transform's buffer. \ No newline at end of file From 663ff36c51201bd61e2c46a5357193ca5f1fb229 Mon Sep 17 00:00:00 2001 From: May Lee Date: Tue, 13 Jan 2026 10:52:10 -0500 Subject: [PATCH 2/7] use buffer shortcode --- .../pipeline_usage_metrics.md | 21 +------------------ 1 file changed, 1 insertion(+), 20 deletions(-) diff --git a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md index 90a9b1d5232..c513ff77230 100644 --- a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md +++ b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md @@ -96,26 +96,7 @@ Utilization Track buffer behavior with these metrics: -`pipelines.buffer_events` -: **Description**: Number of events currently in the buffer. - -`pipelines.buffer_byte_size` -: **Description**: Current buffer size in bytes. - -`pipelines.buffer_received_events_total` -: **Description**: Total events added to the buffer. - -`pipelines.buffer_received_event_bytes_total` -: **Description**: Total bytes added to the buffer. - -`pipelines.buffer_sent_events_total` -: **Description**: Total events successfully flushed from the buffer. - -`pipelines.buffer_sent_event_bytes_total` -: **Description**: Total bytes successfully flushed from the buffer. - -`pipelines.buffer_discarded_events_total` -: **Description**: Events discarded from the buffer (for example, due to overflow). +{{% observability_pipelines/metrics/buffer %}} ## Further reading From 09bf9f2845210b79540700226515383e0a399508 Mon Sep 17 00:00:00 2001 From: May Lee Date: Tue, 13 Jan 2026 12:14:57 -0500 Subject: [PATCH 3/7] Update content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md Co-authored-by: Bruce Guenter --- .../monitoring_and_troubleshooting/pipeline_usage_metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md index c513ff77230..d57907949f6 100644 --- a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md +++ b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md @@ -82,7 +82,7 @@ Timed out events : **Description**: The number of events that timed out. : **Available for**: Sources, processors, and destinations. -Time out requests +Timed out requests : **Metric**: `pipelines.component_timed_out_requests_total` : **Description**: The number of requests that timed out. : **Available for**: Sources, processors, and destinations. From f169e95c4b2654c584a457d6aa51b87d1161cf4d Mon Sep 17 00:00:00 2001 From: May Lee Date: Fri, 16 Jan 2026 10:11:23 -0500 Subject: [PATCH 4/7] Update layouts/shortcodes/observability_pipelines/metrics/buffer.en.md --- layouts/shortcodes/observability_pipelines/metrics/buffer.en.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md b/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md index b66bf47d47b..5022959c2e5 100644 --- a/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md +++ b/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md @@ -19,7 +19,7 @@ Track buffer behavior with these additional metrics: : Total bytes successfully flushed from the buffer. `pipelines.buffer_discarded_events_total` -: Events discarded from the buffer (for example, due to overflow). +: Total number of events discarded from the buffer (for example, due to overflow). `pipelines.source_buffer_max_event_size` : Maximum event size for a source's buffer. From 12773e0379255bd0f3554c3a15f2e384ab729f9d Mon Sep 17 00:00:00 2001 From: May Lee Date: Fri, 16 Jan 2026 11:54:59 -0500 Subject: [PATCH 5/7] Apply suggestions from code review --- .../pipeline_usage_metrics.md | 8 ++++---- .../observability_pipelines/metrics/buffer.en.md | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md index d57907949f6..22e0bde0864 100644 --- a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md +++ b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md @@ -79,13 +79,13 @@ Data dropped intentionally or unintentionally Timed out events : **Metric**: `pipelines.component_timed_out_events_total` -: **Description**: The number of events that timed out. -: **Available for**: Sources, processors, and destinations. +: **Description**: The number of events that waited more than 5 seconds to be sent to the first processor and resulted in a HTTP 503 error. This could happen when delivery of events are blocked. +: **Available for**: HTTP-based sources that have a configured timeout, such as the Datadog Agent. Timed out requests : **Metric**: `pipelines.component_timed_out_requests_total` -: **Description**: The number of requests that timed out. -: **Available for**: Sources, processors, and destinations. +: **Description**: The number of requests that timed out for sources that send events to the Worker in batches using HTTP requests. +: **Available for**: HTTP-based sources that have a configured timeout, such as the Datadog Agent. Utilization : **Metric**: `pipelines.utilization` diff --git a/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md b/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md index 5022959c2e5..e02650b6352 100644 --- a/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md +++ b/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md @@ -28,7 +28,7 @@ Track buffer behavior with these additional metrics: : The percentage of a source's buffer that is used. `pipelines.source_buffer_utilization_level` -: Utilization level for a source's buffer. +: The number of events in a source's buffer. `pipelines.transform_buffer_max_event_size` : Max event size for a transform's buffer. @@ -37,4 +37,4 @@ Track buffer behavior with these additional metrics: : The percentage of a transform's buffer that is used. `pipelines.transform_buffer_utilization_level` -: Utilization level for a transform's buffer. \ No newline at end of file +: The number of events in a transform's buffer. \ No newline at end of file From 792b987b16cf93c63d425ee1c37a284300902bfc Mon Sep 17 00:00:00 2001 From: May Lee Date: Thu, 22 Jan 2026 15:50:19 -0500 Subject: [PATCH 6/7] remove buffer max event size metrics --- .../shortcodes/observability_pipelines/metrics/buffer.en.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md b/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md index e02650b6352..4796c5427e9 100644 --- a/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md +++ b/layouts/shortcodes/observability_pipelines/metrics/buffer.en.md @@ -21,18 +21,12 @@ Track buffer behavior with these additional metrics: `pipelines.buffer_discarded_events_total` : Total number of events discarded from the buffer (for example, due to overflow). -`pipelines.source_buffer_max_event_size` -: Maximum event size for a source's buffer. - `pipelines.source_buffer_utilization` : The percentage of a source's buffer that is used. `pipelines.source_buffer_utilization_level` : The number of events in a source's buffer. -`pipelines.transform_buffer_max_event_size` -: Max event size for a transform's buffer. - `pipelines.transform_buffer_utilization` : The percentage of a transform's buffer that is used. From 03db8953b9017a86bf216af8981945341dc697ad Mon Sep 17 00:00:00 2001 From: May Lee Date: Thu, 22 Jan 2026 16:43:36 -0500 Subject: [PATCH 7/7] Update content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md --- .../monitoring_and_troubleshooting/pipeline_usage_metrics.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md index 22e0bde0864..452fdb1e74e 100644 --- a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md +++ b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md @@ -94,7 +94,6 @@ Utilization ## Buffer metrics (when buffering is enabled) -Track buffer behavior with these metrics: {{% observability_pipelines/metrics/buffer %}}