From 815620cc0b0c7ec0b5e25ec54dcb101fb9a6517d Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 4 Aug 2026 09:51:23 +0200 Subject: [PATCH] docs: administration: document pause behavior for HTTP-based inputs The http, splunk, elasticsearch, opentelemetry, and prometheus_remote_write inputs now implement pause and resume callbacks that pause the shared HTTP listener. Previously these plugins had no pause callback. - backpressure.md: add a section describing what clients observe while one of these inputs is paused. The listener stays open but each incoming connection is accepted and immediately closed, keep-alive is disabled, and in-flight requests are dropped, so senders must retry. - pipeline-section.md: cross-reference the new section from the shared HTTP listener settings. - backpressure.md: add alt text to the tracking pixel to clear a markdownlint MD045 warning on the page. Signed-off-by: Eric D. Schabell --- administration/backpressure.md | 13 ++++++++++++- .../configuring-fluent-bit/yaml/pipeline-section.md | 2 ++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/administration/backpressure.md b/administration/backpressure.md index ea43460ce..e93d27e46 100644 --- a/administration/backpressure.md +++ b/administration/backpressure.md @@ -1,6 +1,6 @@ # Backpressure - +tracking It's possible for Fluent Bit to ingest or create data faster than it can flush that data to the intended destinations. This creates a condition known as _backpressure_. @@ -54,6 +54,17 @@ In a few seconds, if the scheduler was able to flush the initial 700 KB of - If the plugin is paused, it invokes a `resume` callback. - The input plugin can continue appending more data. +## Pause behavior for HTTP-based inputs + +The HTTP-based input plugins [`http`](../pipeline/inputs/http.md), [`splunk`](../pipeline/inputs/splunk.md), [`elasticsearch`](../pipeline/inputs/elasticsearch.md), [`opentelemetry`](../pipeline/inputs/opentelemetry.md), and [`prometheus_remote_write`](../pipeline/inputs/prometheus-remote-write.md) act on the `pause` and `resume` callbacks by pausing their [shared HTTP listener](configuring-fluent-bit/yaml/pipeline-section.md#shared-http-listener-settings-for-inputs). This applies whichever backpressure mechanism paused the input. + +While one of these inputs is paused: + +- The listening socket stays open, but Fluent Bit accepts and immediately closes each incoming connection. Clients see a connection that closes before a response arrives rather than a refused connection. +- Connections that are already open are closed instead of being kept alive, and any request still in flight is dropped. + +Because the client never receives a successful response for these requests, it's responsible for retrying. Configure your senders to retry on connection failure so data isn't lost while an input is paused. When the `resume` callback runs, the listener starts accepting connections again. + ## Manage backpressure for filesystem buffering If one or more active input plugins use [filesystem buffering](../pipeline/buffering.md#filesystem-buffering-hybrid), use the following settings to manage backpressure. diff --git a/administration/configuring-fluent-bit/yaml/pipeline-section.md b/administration/configuring-fluent-bit/yaml/pipeline-section.md index 3fea1f1b1..1f613f45d 100644 --- a/administration/configuring-fluent-bit/yaml/pipeline-section.md +++ b/administration/configuring-fluent-bit/yaml/pipeline-section.md @@ -108,6 +108,8 @@ effect. For backward compatibility, some plugins also accept the legacy aliases `http2`, `buffer_max_size`, `buffer_chunk_size`, `max_connections`, and `workers`. +These plugins also pause and resume the shared listener in response to [backpressure](../../backpressure.md#pause-behavior-for-http-based-inputs). While an input is paused, Fluent Bit accepts and immediately closes incoming connections instead of buffering the requests, so clients must retry. + ### Incoming `OAuth 2.0` `JWT` validation settings The HTTP-based input plugins that support bearer token validation share the following `oauth2.*` settings: