diff --git a/.gitbook.yaml b/.gitbook.yaml index 9ee9d0179..db96bf164 100644 --- a/.gitbook.yaml +++ b/.gitbook.yaml @@ -101,3 +101,4 @@ redirects: about/sandbox-and-lab-resources: ./about/resources.md installation/downloads/amazon-ec2: ./installation/downloads/linux/amazon-linux.md administration/configuring-fluent-bit/yaml/configuration-file: ./administration/configuring-fluent-bit/yaml.md + administration/configuring-fluent-bit/unit-sizes: ./administration/configuring-fluent-bit.md#unit-sizes diff --git a/SUMMARY.md b/SUMMARY.md index 36c4e9a95..d61397a29 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -38,7 +38,7 @@ ## Administration * [Configure Fluent Bit](administration/configuring-fluent-bit.md) - * [YAML configuration](administration/configuring-fluent-bit/yaml.md) + * [YAML configuration files](administration/configuring-fluent-bit/yaml.md) * [Environment variables](administration/configuring-fluent-bit/yaml/environment-variables-section.md) * [Includes](administration/configuring-fluent-bit/yaml/includes-section.md) * [Service](administration/configuring-fluent-bit/yaml/service-section.md) @@ -47,7 +47,7 @@ * [Pipeline](administration/configuring-fluent-bit/yaml/pipeline-section.md) * [Plugins](administration/configuring-fluent-bit/yaml/plugins-section.md) * [Upstream servers](administration/configuring-fluent-bit/yaml/upstream-servers-section.md) - * [Classic mode](administration/configuring-fluent-bit/classic-mode.md) + * [Classic configuration files](administration/configuring-fluent-bit/classic-mode.md) * [Configuration file](administration/configuring-fluent-bit/classic-mode/configuration-file.md) * [Commands](administration/configuring-fluent-bit/classic-mode/commands.md) * [Format and schema](administration/configuring-fluent-bit/classic-mode/format-schema.md) @@ -55,7 +55,6 @@ * [Upstream servers](administration/configuring-fluent-bit/classic-mode/upstream-servers.md) * [Variables](administration/configuring-fluent-bit/classic-mode/variables.md) * [Multiline parsing](administration/configuring-fluent-bit/multiline-parsing.md) - * [Unit Sizes](administration/configuring-fluent-bit/unit-sizes.md) * [AWS credentials](administration/aws-credentials.md) * [Backpressure](administration/backpressure.md) * [Buffering and storage](administration/buffering-and-storage.md) diff --git a/administration/configuring-fluent-bit.md b/administration/configuring-fluent-bit.md index d4ccab94f..01a17db98 100644 --- a/administration/configuring-fluent-bit.md +++ b/administration/configuring-fluent-bit.md @@ -1,13 +1,26 @@ # Configure Fluent Bit -Fluent Bit supports two configuration formats: +Fluent Bit uses configuration files to store information about your specified [inputs](../data-pipeline/inputs.md), [outputs](../data-pipeline/outputs.md), [filters](../data-pipeline/filters.md), and more. You can write these configuration files in one of these formats: -- [YAML](./configuring-fluent-bit/yaml.md): Standard configuration format as of v3.2. -- [Classic mode](./configuring-fluent-bit/classic-mode.md): To be deprecated at the end of 2026. +- [YAML configuration files](./configuring-fluent-bit/yaml.md) are the standard configuration format as of Fluent Bit v3.2. They use the `.yaml` file extension. +- [Classic configuration files](./configuring-fluent-bit/classic-mode.md) will be deprecated at the end of 2026. They use the `.conf` file extension. + +## Unit sizes + +Some configuration settings in Fluent Bit use standardized unit sizes to define data and storage limits. For example, the `buffer_chunk_size` and `buffer_max_size` parameters for the [Tail](../data-pipeline/inputs/tail.md) input plugin use unit sizes. + +The following table describes the unit sizes you can use and what they mean. + +| Suffix | Description | Example | +| :--- | :--- | :--- | +| _none_ | **Bytes**: If you specify an integer without a unit size, Fluent Bit interprets that value as a bytes representation. | `32000` means 32,000 bytes. | +| `k`, `kb`, `K`, `KB` | **Kilobytes**: A unit of memory equal to 1,000 bytes. | `32k` means 32,000 bytes. | +| `m`, `mb`, `M`, `MB` | **Megabytes**: A unit of memory equal to 1,000,000 bytes. | `32m` means 32,000,000 bytes. | +| `g`, `gb`, `G`, `GB` | **Gigabytes**: A unit of memory equal to 1,000,000,000 bytes. | `32g` means 32,000,000,000 bytes. | ## Command line interface -Fluent Bit exposes most of it features through the command line interface. Running the `-h` option you can get a list of the options available: +Fluent Bit exposes most of its configuration features through the command line interface. Use the `-h` or `--help` flag to see a list of available options. ```shell # Podman container tooling. @@ -16,167 +29,3 @@ podman run -rm -ti fluent/fluent-bit --help # Docker container tooling. docker run --rm -it fluent/fluent-bit --help ``` - -Which returns the following help text: - -```text -Usage: /fluent-bit/bin/fluent-bit [OPTION] - -Available Options - -b --storage_path=PATH specify a storage buffering path - -c --config=FILE specify an optional configuration file - -d, --daemon run Fluent Bit in background mode - --supervisor run under a supervising parent process, uses "fork()" to execute child processes - -D, --dry-run dry run - -f, --flush=SECONDS flush timeout in seconds (default: 1) - -C, --custom=CUSTOM enable a custom plugin - -i, --input=INPUT set an input - -F --filter=FILTER set a filter - -m, --match=MATCH set plugin match, same as '-p match=abc' - -o, --output=OUTPUT set an output - -p, --prop="A=B" set plugin configuration property - -R, --parser=FILE specify a parser configuration file - -e, --plugin=FILE load an external plugin (shared lib) - -l, --log_file=FILE write log info to a file - -t, --tag=TAG set plugin tag, same as '-p tag=abc' - -T, --sp-task=SQL define a stream processor task - -v, --verbose increase logging verbosity (default: info) - -Z, --enable-chunk-traceenable chunk tracing, it can be activated either through the http api or the command line - --trace-input input to start tracing on startup. - --trace-output output to use for tracing on startup. - --trace-output-property set a property for output tracing on startup. - --trace setup a trace pipeline on startup. Uses a single line, ie: "input=dummy.0 output=stdout output.format='json'" - -w, --workdir set the working directory - -H, --http enable monitoring HTTP server - -P, --port set HTTP server TCP port (default: 2020) - -s, --coro_stack_size set coroutines stack size in bytes (default: 24576) - -q, --quiet quiet mode - -S, --sosreport support report for Enterprise customers - -Y, --enable-hot-reload enable for hot reloading - -W, --disable-thread-safety-on-hot-reloadingdisable thread safety on hot reloading - -V, --version show version number - -h, --help print this help - -Inputs - blob Blob (binary) files - cpu CPU Usage - mem Memory Usage - thermal Thermal - kmsg Kernel Log Buffer - proc Check Process health - disk Diskstats - systemd Systemd (Journal) reader - netif Network Interface Usage - docker Docker containers metrics - docker_events Docker events - podman_metrics Podman metrics - process_exporter_metricsProcess Exporter Metrics (Prometheus Compatible) - gpu_metrics GPU Metrics - node_exporter_metrics Node Exporter Metrics (Prometheus Compatible) - kubernetes_events Kubernetes Events - kafka Kafka consumer input plugin - fluentbit_metrics Fluent Bit internal metrics - prometheus_scrape Scrape metrics from Prometheus Endpoint - prometheus_textfile Read Prometheus metrics from text files - tail Tail files - dummy Generate dummy data - head Head Input - health Check TCP server health - http HTTP - collectd collectd input plugin - statsd StatsD input plugin - opentelemetry OpenTelemetry - elasticsearch HTTP Endpoints for Elasticsearch (Bulk API) - splunk Input plugin for Splunk HEC payloads - prometheus_remote_write Prometheus Remote Write input - event_type Event tests for input plugins - nginx_metrics Nginx status metrics - serial Serial input - stdin Standard Input - syslog Syslog - udp UDP - exec_wasi Exec WASI Input - tcp TCP - mqtt MQTT, listen for Publish messages - forward Fluentd in-forward - random Random - -Processors - content_modifier Modify the content of Logs, Metrics and Traces - labels Modifies metrics labels - metrics_selector select metrics by specified name - opentelemetry_envelope Package log records inside an OpenTelemetry Logs schema - sampling Sampling - sql SQL processor - -Filters - alter_size Alter incoming chunk size - aws Add AWS Metadata - checklist Check records and flag them - ecs Add AWS ECS Metadata - record_modifier modify record - sysinfo Filter for system info - throttle Throttle messages using sliding window algorithm - type_converter Data type converter - kubernetes Filter to append Kubernetes metadata - modify modify records by applying rules - multiline Concatenate multiline messages - nest nest events by specified field values - parser Parse events - expect Validate expected keys and values - grep grep events by specified field values - rewrite_tag Rewrite records tags - log_to_metrics generate log derived metrics - lua Lua Scripting Filter - stdout Filter events to STDOUT - geoip2 add geoip information to records - nightfall scans records for sensitive content - wasm WASM program filter - -Outputs - azure Send events to Azure HTTP Event Collector - azure_blob Azure Blob Storage - azure_logs_ingestion Send logs to Log Analytics with Log Ingestion API - azure_kusto Send events to Kusto (Azure Data Explorer) - bigquery Send events to BigQuery via streaming insert - counter Records counter - datadog Send events to DataDog HTTP Event Collector - es Elasticsearch - exit Exit after a number of flushes (test purposes) - file Generate log file - forward Forward (Fluentd protocol) - http HTTP Output - influxdb InfluxDB Time Series - logdna LogDNA - loki Loki - kafka Kafka - kafka-rest Kafka REST Proxy - nats NATS Server - nrlogs New Relic - null Throws away events - opensearch OpenSearch - oracle_log_analytics Oracle log analytics - plot Generate data file for GNU Plot - pgsql PostgreSQL - skywalking Send logs into log collector on SkyWalking OAP - slack Send events to a Slack channel - splunk Send events to Splunk HTTP Event Collector - stackdriver Send events to Google Stackdriver Logging - stdout Prints events to STDOUT - syslog Syslog - tcp TCP Output - udp UDP Output - td Treasure Data - flowcounter FlowCounter - gelf GELF Output - websocket Websocket - cloudwatch_logs Send logs to Amazon CloudWatch - kinesis_firehose Send logs to Amazon Kinesis Firehose - kinesis_streams Send logs to Amazon Kinesis Streams - opentelemetry OpenTelemetry - prometheus_exporter Prometheus Exporter - prometheus_remote_write Prometheus remote write - s3 Send to S3 - vivo_exporter Vivo Exporter - chronicle Send logs to Google Chronicle as unstructured log -``` diff --git a/administration/configuring-fluent-bit/classic-mode.md b/administration/configuring-fluent-bit/classic-mode.md index 83b124a0f..4aeda9695 100644 --- a/administration/configuring-fluent-bit/classic-mode.md +++ b/administration/configuring-fluent-bit/classic-mode.md @@ -1,4 +1,4 @@ -# Configure Fluent Bit using classic mode +# Classic configuration files {% hint style="info" %} Fluent Bit classic mode configuration will be deprecated at the end of 2026. diff --git a/administration/configuring-fluent-bit/security.md b/administration/configuring-fluent-bit/security.md deleted file mode 100644 index 3ebfc6fb6..000000000 --- a/administration/configuring-fluent-bit/security.md +++ /dev/null @@ -1,2 +0,0 @@ -# Security - diff --git a/administration/configuring-fluent-bit/unit-sizes.md b/administration/configuring-fluent-bit/unit-sizes.md deleted file mode 100644 index 8808b589e..000000000 --- a/administration/configuring-fluent-bit/unit-sizes.md +++ /dev/null @@ -1,12 +0,0 @@ -# Unit sizes - -Some configuration directives in [Fluent Bit](http://fluentbit.io) refer to unit sizes such as when defining the size of a buffer or specific limits. Plugins like [Tail Input](../../pipeline/inputs/tail.md), [Forward Input](../../pipeline/inputs/forward.md), or generic properties like [`Mem_Buf_Limit`](../backpressure.md) use unit sizes. - -Fluent Bit v0.11.10 standardized unit sizes across the core and plugins. The following table describes the options that can be used and what they mean: - -| Suffix | Description | Example | -| :--- | :--- | :--- | -| | When a suffix isn't specified, assume that the value given is a bytes representation. | Specifying a value of `32000` means 32000 bytes. | -| `k`, `K`, `KB`, `kb` | Kilobyte: a unit of memory equal to 1,000 bytes. | `32k` means 32000 bytes. | -| `m`, `M`, `MB`, `mb` | Megabyte: a unit of memory equal to 1,000,000 bytes. | `1M` means 1000000 bytes. | -| `g`, `G`, `GB`, `gb` | Gigabyte: a unit of memory equal to 1,000,000,000 bytes. | `1G` means 1000000000 bytes. | diff --git a/administration/configuring-fluent-bit/yaml.md b/administration/configuring-fluent-bit/yaml.md index 16297bb3a..19998f906 100644 --- a/administration/configuring-fluent-bit/yaml.md +++ b/administration/configuring-fluent-bit/yaml.md @@ -1,4 +1,4 @@ -# YAML configuration +# YAML configuration files diff --git a/pipeline/filters/kubernetes.md b/pipeline/filters/kubernetes.md index 114907f4a..8b1088e0f 100644 --- a/pipeline/filters/kubernetes.md +++ b/pipeline/filters/kubernetes.md @@ -25,7 +25,7 @@ The plugin supports the following configuration parameters: | Key | Description | Default | | :--- | :--- | :--- | -| `Buffer_Size` | Set the buffer size for HTTP client when reading responses from Kubernetes API server. The value must conform to the [unit size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. A value of `0` results in no limit, and the buffer will expand as-needed. If pod specifications exceed the buffer limit, the API response is discarded when retrieving metadata, and some Kubernetes metadata will fail to be injected to the logs. | `32k` | +| `Buffer_Size` | Set the buffer size for HTTP client when reading responses from Kubernetes API server. The value must conform to the [unit size](../../administration/configuring-fluent-bit.md#unit-sizes) specification. A value of `0` results in no limit, and the buffer will expand as-needed. If pod specifications exceed the buffer limit, the API response is discarded when retrieving metadata, and some Kubernetes metadata will fail to be injected to the logs. | `32k` | | `Kube_URL` | API Server endpoint | `https://kubernetes.default.svc:443` | | `Kube_CA_File` | CA certificate file | `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt` | | `Kube_CA_Path` | Absolute path to scan for certificate files | _none_ | diff --git a/pipeline/filters/wasm.md b/pipeline/filters/wasm.md index 12740ef0b..81e3b463b 100644 --- a/pipeline/filters/wasm.md +++ b/pipeline/filters/wasm.md @@ -22,8 +22,8 @@ The plugin supports the following configuration parameters: | `Event_Format` | Define event format to interact with Wasm programs: `msgpack` or `json`. Default: `json`. | | `Function_Name` | Wasm function name that will be triggered to do filtering. It's assumed that the function is built inside the Wasm program specified previously. | | `Accessible_Paths` | Specify the allowlist of paths to be able to access paths from Wasm programs. | -| `Wasm_Heap_Size` | Size of the heap size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. | -| `Wasm_Stack_Size` | Size of the stack size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. | +| `Wasm_Heap_Size` | Size of the heap size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit.md#unit-sizes) for allowed values. | +| `Wasm_Stack_Size` | Size of the stack size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit.md#unit-sizes) for allowed values. | ## Configuration example @@ -46,7 +46,7 @@ pipeline: function_name: filter_function_name # Note: run Fluent Bit from the 'wasm_path' location. accessible_paths: /path/to/accessible - + outputs: - name: stdout match: '*' @@ -74,4 +74,4 @@ pipeline: ``` {% endtab %} -{% endtabs %} \ No newline at end of file +{% endtabs %} diff --git a/pipeline/inputs/blob.md b/pipeline/inputs/blob.md index a370c2cf4..eb9aa822a 100644 --- a/pipeline/inputs/blob.md +++ b/pipeline/inputs/blob.md @@ -13,10 +13,10 @@ The plugin supports the following configuration parameters: | `exclude_pattern` | Set one or multiple shell patterns separated by commas to exclude files matching certain criteria. For example, `exclude_pattern *.tmp,*.bak` will exclude temporary and backup files from processing. | _none_ | | `log_level` | Specifies the log level for this input plugin. If not set here, the plugin uses the global log level specified in the `service` section. Valid values: `off`, `error`, `warn`, `info`, `debug`, `trace`. | `info` | | `log_suppress_interval` | Suppresses log messages from this input plugin that appear similar within a specified time interval. Set to `0` to disable suppression. The value must be specified in seconds. This helps reduce log noise when the same error or warning occurs repeatedly. | `0` | -| `mem_buf_limit` | Set a memory buffer limit for the input plugin instance in bytes. If the limit is reached, the plugin will pause until the buffer is drained. If set to `0`, the buffer limit is disabled. If the plugin has enabled filesystem buffering, this limit won't apply. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `0` | +| `mem_buf_limit` | Set a memory buffer limit for the input plugin instance in bytes. If the limit is reached, the plugin will pause until the buffer is drained. If set to `0`, the buffer limit is disabled. If the plugin has enabled filesystem buffering, this limit won't apply. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit.md#unit-sizes) specification. | `0` | | `path` | Path to scan for blob (binary) files. Supports wildcards and glob patterns. For example, `/var/log/binaries/*.bin` or `/data/artifacts/**/*.dat`. This is a required parameter. | _none_ | | `routable` | If `true`, the data generated by the plugin can be forwarded to other plugins or outputs. If `false`, the data will be discarded. This is used for testing or when you want to process data but not forward it. | `true` | -| `scan_refresh_interval` | Set the interval time to scan for new files. The plugin periodically scans the specified path for new or modified files. The value must be specified according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification (`2s`, `30m`, `1h`). | `2s` | +| `scan_refresh_interval` | Set the interval time to scan for new files. The plugin periodically scans the specified path for new or modified files. The value must be specified according to the [Unit Size](../../administration/configuring-fluent-bit.md#unit-sizes) specification (`2s`, `30m`, `1h`). | `2s` | | `storage.pause_on_chunks_overlimit` | Enable pausing on an input when it reaches its chunks limit. When enabled, the plugin will pause processing if the number of chunks exceeds the limit, preventing memory issues during backpressure scenarios. | `false` | | `storage.type` | Sets the storage type for this input. Options: `filesystem` (persists data to disk), `memory` (stores data in memory only), or `memrb` (memory ring buffer). For production environments with high data volumes, consider using `filesystem` to prevent data loss during restarts. | `memory` | | `tag` | Set a tag for the events generated by this input plugin. Tags are used for routing records to specific outputs. Supports tag expansion with wildcards. | _none_ | diff --git a/pipeline/inputs/exec-wasi.md b/pipeline/inputs/exec-wasi.md index d8e7992ee..6c2212195 100644 --- a/pipeline/inputs/exec-wasi.md +++ b/pipeline/inputs/exec-wasi.md @@ -13,9 +13,9 @@ The plugin supports the following configuration parameters: | `Accessible_Paths` | Specify the allowed list of paths to be able to access paths from Wasm programs. | | `Interval_Sec` | Polling interval (seconds). | | `Interval_NSec` | Polling interval (nanosecond). | -| `Wasm_Heap_Size` | Size of the heap size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. | -| `Wasm_Stack_Size` | Size of the stack size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. | -| `Buf_Size` | Size of the buffer See [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. | +| `Wasm_Heap_Size` | Size of the heap size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit.md#unit-sizes) for allowed values. | +| `Wasm_Stack_Size` | Size of the stack size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit.md#unit-sizes) for allowed values. | +| `Buf_Size` | Size of the buffer See [unit sizes](../../administration/configuring-fluent-bit.md#unit-sizes) for allowed values. | | `Oneshot` | Only run once at startup. This allows collection of data precedent to the Fluent Bit startup (Boolean, default: `false`). | | `Threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). Default: `false`. | diff --git a/pipeline/inputs/exec.md b/pipeline/inputs/exec.md index 10896d6e4..151028f55 100644 --- a/pipeline/inputs/exec.md +++ b/pipeline/inputs/exec.md @@ -24,7 +24,7 @@ The plugin supports the following configuration parameters: | `Parser` | Specify the name of a parser to interpret the entry as a structured message. | | `Interval_Sec` | Polling interval (seconds). | | `Interval_NSec` | Polling interval (nanosecond). | -| `Buf_Size` | Size of the buffer. See [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. | +| `Buf_Size` | Size of the buffer. See [unit sizes](../../administration/configuring-fluent-bit.md#unit-sizes) for allowed values. | | `Oneshot` | Only run once at startup. This allows collection of data precedent to Fluent Bit startup (Boolean, default: `false`). | | `Exit_After_Oneshot` | Exit as soon as the one-shot command exits. This allows the `exec` plugin to be used as a wrapper for another command, sending the target command's output to any Fluent Bit sink, then exits. (Boolean, default: `false`). | | `Propagate_Exit_Code` | When exiting due to `Exit_After_Oneshot`, cause Fluent Bit to exit with the exit code of the command exited by this plugin. Follows [shell conventions for exit code propagation](https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html). (Boolean, default: `false`). | diff --git a/pipeline/inputs/forward.md b/pipeline/inputs/forward.md index 581ccdc46..646fd6e38 100644 --- a/pipeline/inputs/forward.md +++ b/pipeline/inputs/forward.md @@ -12,8 +12,8 @@ The plugin supports the following configuration parameters: | `Port` | TCP port to listen for incoming connections. | `24224` | | `Unix_Path` | Specify the path to Unix socket to receive a Forward message. If set, `Listen` and `Port` are ignored. | _none_ | | `Unix_Perm` | Set the permission of the Unix socket file. If `Unix_Path` isn't set, this parameter is ignored. | _none_ | -| `Buffer_Max_Size` | Specify the maximum buffer memory size used to receive a Forward message. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `6144000` | -| `Buffer_Chunk_Size` | By default the buffer to store the incoming Forward messages, don't allocate the maximum memory allowed, instead it allocate memory when it's required. The rounds of allocations are set by `Buffer_Chunk_Size`. The value must be according to the [Unit Size ](../../administration/configuring-fluent-bit/unit-sizes.md)specification. | `1024000` | +| `Buffer_Max_Size` | Specify the maximum buffer memory size used to receive a Forward message. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit.md#unit-sizes) specification. | `6144000` | +| `Buffer_Chunk_Size` | By default the buffer to store the incoming Forward messages, don't allocate the maximum memory allowed, instead it allocate memory when it's required. The rounds of allocations are set by `Buffer_Chunk_Size`. The value must be according to the [Unit Size ](../../administration/configuring-fluent-bit.md#unit-sizes)specification. | `1024000` | | `Tag_Prefix` | Prefix incoming tag with the defined value. | _none_ | | `Tag` | Override the tag of the forwarded events with the defined value. | _none_ | | `Shared_Key` | Shared key for secure forward authentication. | _none_ | diff --git a/pipeline/inputs/standard-input.md b/pipeline/inputs/standard-input.md index 0f2062cdf..0f7f3a2d9 100644 --- a/pipeline/inputs/standard-input.md +++ b/pipeline/inputs/standard-input.md @@ -15,7 +15,7 @@ The plugin supports the following configuration parameters: | Key | Description | Default | |:--------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------| -| `Buffer_Size` | Set the buffer size to read data. This value is used to increase buffer size and must be set according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `16k` | +| `Buffer_Size` | Set the buffer size to read data. This value is used to increase buffer size and must be set according to the [Unit Size](../../administration/configuring-fluent-bit.md#unit-sizes) specification. | `16k` | | `Parser` | The name of the parser to invoke instead of the default JSON input parser. | _none_ | | `Threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | @@ -83,7 +83,7 @@ To demonstrate how the plugin works, you can use a `bash` script that generates ```bash #!/bin/sh - + for ((i=0; i<=5; i++)); do echo -n " [ @@ -198,13 +198,13 @@ You can then use the parsers file in a `stdin` plugin in the main Fluent Bit con ```yaml service: parsers_file: parsers.yaml - + pipeline: inputs: - name: stdin tag: stdin parser: stringify_message - + outputs: - name: stdout match: '*' @@ -216,7 +216,7 @@ pipeline: ```text [SERVICE] parsers_file parsers.conf - + [INPUT] Name stdin Tag stdin @@ -252,4 +252,4 @@ Which returns output similar to: ... ``` -In production deployments it's best to use a parser that splits messages into real fields and adds appropriate tags. \ No newline at end of file +In production deployments it's best to use a parser that splits messages into real fields and adds appropriate tags. diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index c39d8f730..9396c81df 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -11,7 +11,7 @@ The plugin supports the following configuration parameters: | Key | Description | Default | |:----|:------------|:--------| | `buffer_chunk_size` | Set the initial buffer size to read file data. This value is used to increase buffer size. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `32k` | -| `buffer_max_size` | Set the limit of the buffer size per monitored file. When a buffer needs to be increased, this value is used to restrict the memory buffer growth. If reading a file exceeds this limit, the file is removed from the monitored file list. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `32k` | +| `buffer_max_size` | Set the limit of the buffer size per monitored file. When a buffer needs to be increased, this value is used to restrict the memory buffer growth. If reading a file exceeds this limit, the file is removed from the monitored file list. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit.md#unit-sizes) specification. | `32k` | | `db` | Specify the database file to keep track of monitored files and offsets. Recommended to be unique per plugin. | _none_ | | `db.compare_filename` | This option determines whether to review both `inode` and `filename` when retrieving stored file information from the database. `true` verifies both `inode` and `filename`, while `false` checks only the `inode`. To review the `inode` and `filename` in the database, refer [see `keep_state`](#tailing-files-keeping-state). | `false` | | `db.journal_mode` | Sets the journal mode for databases (`wal`). Enabling `wal` provides higher performance. `wal` isn't compatible with shared network file systems. | `wal` | diff --git a/pipeline/outputs/elasticsearch.md b/pipeline/outputs/elasticsearch.md index 18b36b6c9..96ee4a9bd 100644 --- a/pipeline/outputs/elasticsearch.md +++ b/pipeline/outputs/elasticsearch.md @@ -16,7 +16,7 @@ This plugin has the following configuration parameters: | `Port` | TCP port of the target Elasticsearch instance | `9200` | | `Path` | Elasticsearch accepts new data on HTTP query path `/_bulk`. You can also serve Elasticsearch behind a reverse proxy on a sub-path. Define the path by adding a path prefix in the indexing HTTP POST URI. | Empty string | | `compress` | Set payload compression mechanism. Option available is `gzip`. | _none_ | -| `Buffer_Size` | Specify the buffer size used to read the response from the Elasticsearch HTTP service. Use for debugging purposes where required to read full responses. Response size grows depending of the number of records inserted. To use an unlimited amount of memory, set this value to `False`. Otherwise set the value according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md). | `512KB` | +| `Buffer_Size` | Specify the buffer size used to read the response from the Elasticsearch HTTP service. Use for debugging purposes where required to read full responses. Response size grows depending of the number of records inserted. To use an unlimited amount of memory, set this value to `False`. Otherwise set the value according to the [Unit Size](../../administration/configuring-fluent-bit.md#unit-sizes). | `512KB` | | `Pipeline` | Define which pipeline the database should use. For performance reasons, it's strongly suggested to do parsing and filtering on Fluent Bit side, and avoid pipelines. | _none_ | | `AWS_Auth` | Enable AWS Sigv4 Authentication for Amazon OpenSearch Service. | `Off` | | `AWS_Region` | Specify the AWS region for Amazon OpenSearch Service. | _none_ | diff --git a/pipeline/outputs/opensearch.md b/pipeline/outputs/opensearch.md index bf5a71b47..a19d58d32 100644 --- a/pipeline/outputs/opensearch.md +++ b/pipeline/outputs/opensearch.md @@ -15,7 +15,7 @@ This plugin supports the following parameters: | `Host` | IP address or hostname of the target OpenSearch instance. | `127.0.0.1` | | `Port` | TCP port of the target OpenSearch instance. | `9200` | | `Path` | OpenSearch accepts new data on HTTP query path `/_bulk`. It's possible to serve OpenSearch behind a reverse proxy on a sub-path. This option defines such path on the Fluent Bit side. It adds a path prefix in the indexing HTTP POST URI. | Empty string | -| `Buffer_Size` | Specify the buffer size used to read the response from the OpenSearch HTTP service. Use for debugging purposes where it's required to read full responses. The response size grows depending of the number of records inserted. Set this value to `False` to set an unlimited amount of memory. Otherwise set the value according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `512KB` | +| `Buffer_Size` | Specify the buffer size used to read the response from the OpenSearch HTTP service. Use for debugging purposes where it's required to read full responses. The response size grows depending of the number of records inserted. Set this value to `False` to set an unlimited amount of memory. Otherwise set the value according to the [Unit Size](../../administration/configuring-fluent-bit.md#unit-sizes) specification. | `512KB` | | `Pipeline` | OpenSearch lets you set up filters called pipelines. This option defines which pipeline the database should use. For performance reasons, it's strongly suggested to do parsing and filtering on Fluent Bit side, avoid pipelines. | _none_ | | `AWS_Auth` | Enable AWS Sigv4 Authentication for Amazon OpenSearch Service. | `Off` | | `AWS_Region` | Specify the AWS region for Amazon OpenSearch Service. | _none_ |