diff --git a/administration/configuring-fluent-bit/yaml/pipeline-section.md b/administration/configuring-fluent-bit/yaml/pipeline-section.md index 1f613f45d..74dc93afb 100644 --- a/administration/configuring-fluent-bit/yaml/pipeline-section.md +++ b/administration/configuring-fluent-bit/yaml/pipeline-section.md @@ -79,6 +79,39 @@ The `name` parameter is required and defines for Fluent Bit which input plugin s There is no hard-coded limit on the number of input plugins. The practical maximum depends on available system resources such as memory and file descriptors. +### Count log records by tag for inputs + +Each input can override the service-wide [`telemetry.metrics.logs.tag_records.enabled`](service-section.md#count-log-records-by-tag) setting with a nested `telemetry` block of its own. An input that doesn't set this block inherits the `service` value. + +Only the `enabled` key is valid at the input level. Setting `max_series` or `max_tag_length` on an input is a startup error, because those limits are service-wide. + +The following example enables per-tag counting for the whole service and disables it for one noisy input: + +```yaml +service: + telemetry: + metrics: + logs: + tag_records: + enabled: true + +pipeline: + inputs: + - name: tail + tag: app.logs + path: /var/log/app/*.log + + - name: tail + alias: high_cardinality + tag: dynamic.* + path: /var/log/dynamic/*.log + telemetry: + metrics: + logs: + tag_records: + enabled: false +``` + ### Shared HTTP listener settings for inputs Some HTTP-based input plugins share the same listener implementation and support the following common settings in addition to their plugin-specific parameters: diff --git a/administration/configuring-fluent-bit/yaml/service-section.md b/administration/configuring-fluent-bit/yaml/service-section.md index 497aed19e..7d4535c75 100644 --- a/administration/configuring-fluent-bit/yaml/service-section.md +++ b/administration/configuring-fluent-bit/yaml/service-section.md @@ -96,6 +96,47 @@ Enabling FIPS mode changes the behavior of plugins that rely on MD5: - The [Amazon S3](../../../pipeline/outputs/s3.md) output rejects `send_content_md5` at startup, because that header requires MD5. - The [Azure Blob](../../../pipeline/outputs/azure_blob.md) output derives block IDs using SHA-256 instead of MD5. +## Telemetry configuration + +The `telemetry` key holds a nested block that controls optional internal metrics. Unlike the other `service` keys, these settings have no flat dotted-key form and no [classic mode](../classic-mode/configuration-file.md) equivalent. Define them as a nested map in YAML. + +### Count log records by tag + +Fluent Bit can count the log records each input ingests for every tag and expose the result as the `fluentbit_input_logs_tag_records_total` metric. For details about the metrics this feature produces, see [Monitoring](../../monitoring.md#v2-metrics). + +This tracking is disabled by default because it creates one metric series for each combination of input and tag, which can increase cardinality in your metrics backend. + +The following keys can be set as children to the `telemetry.metrics.logs.tag_records` key: + +| Key | Description | Default Value | +| --- | ----------- | ------------- | +| `enabled` | Enables per-tag counting of ingested log records. Individual inputs can override this value in the [`pipeline.inputs`](../yaml/pipeline-section.md#count-log-records-by-tag-for-inputs) section. Possible values: `false` or `true`. | `false` | +| `max_series` | Sets the maximum number of distinct input and tag combinations to track. This budget is shared across all inputs. After the limit is reached, records carrying a tag that isn't already tracked are counted in `fluentbit_input_logs_tag_records_untracked_total` with the reason `max_series`. Tags that are already tracked continue to be counted. Set to `0` or less to remove the limit. | `500` | +| `max_tag_length` | Sets the maximum length in bytes of a tag to track. Records with a longer tag are counted in `fluentbit_input_logs_tag_records_untracked_total` with the reason `tag_length_limit`. Longer tags are skipped rather than truncated. Set to `0` or less to remove the limit. | `128` | + +Both `max_series` and `max_tag_length` can only be set in the `service` section. + +Each of these two limits accepts an integer, or a string that contains only an integer. Strings are expanded for [environment variables](environment-variables-section.md) before they're parsed, so `${MAX_SERIES}` is valid. The resulting value must fit in a signed 32-bit integer. + +Fluent Bit fails to start when a limit isn't a complete integer, such as `notanumber` or `10abc`, or falls outside the signed 32-bit range. Neither limit enforces a minimum. Both `0` and negative values are accepted and remove the limit instead of causing an error. + +Fluent Bit also rejects unknown keys in this block at startup, so a misspelled key prevents the service from starting. + +The following example enables per-tag counting and lowers both limits: + +```yaml +service: + flush: 1 + http_server: on + telemetry: + metrics: + logs: + tag_records: + enabled: true + max_series: 200 + max_tag_length: 64 +``` + ## Storage configuration The following storage-related keys can be set as children to the `storage` key: diff --git a/administration/monitoring.md b/administration/monitoring.md index e51ba8c20..027220557 100644 --- a/administration/monitoring.md +++ b/administration/monitoring.md @@ -211,6 +211,8 @@ Some metrics are available only for specific plugins or runtime modes. For examp | `fluentbit_input_http_server_ingress_queue_pending_events` | name: the name or alias for the input instance | The current number of deferred ingress queue entries. Only available for HTTP-based inputs using `http_server.workers` greater than `1`. | gauge | events | | `fluentbit_input_long_line_skipped_total` | name: the name or alias for the input instance | The total number of skipped occurrences for long lines. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin when `skip_long_lines` is enabled. | counter | occurrences | | `fluentbit_input_long_line_truncated_total` | name: the name or alias for the input instance | The total number of truncated occurrences for long lines. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin when `truncate_long_lines` is enabled. | counter | occurrences | +| `fluentbit_input_logs_tag_records_total` | name: the name or alias for the input instance, tag: the tag carried by the records | The number of log records this input ingested for each tag. Only available when [`telemetry.metrics.logs.tag_records`](configuring-fluent-bit/yaml/service-section.md#count-log-records-by-tag) is enabled. | counter | records | +| `fluentbit_input_logs_tag_records_untracked_total` | name: the name or alias for the input instance, reason: why the records weren't tracked | The number of log records that couldn't be counted by tag. The `reason` label is `max_series` when the tag cardinality limit is reached, `tag_length_limit` when the tag is longer than `max_tag_length`, or `error` when Fluent Bit fails internally while tracking the tag. Only available when [`telemetry.metrics.logs.tag_records`](configuring-fluent-bit/yaml/service-section.md#count-log-records-by-tag) is enabled. | counter | records | | `fluentbit_input_memrb_dropped_bytes` | name: the name or alias for the input instance | The number of bytes dropped by the memory ring buffer (`memrb`) storage type when the buffer is full. Only available for input plugins with `storage.type` set to `memrb`. | counter | bytes | | `fluentbit_input_memrb_dropped_chunks` | name: the name or alias for the input instance | The number of chunks dropped by the memory ring buffer (`memrb`) storage type when the buffer is full. Only available for input plugins with `storage.type` set to `memrb`. | counter | chunks | | `fluentbit_input_multiline_truncated_total` | name: the name or alias for the input instance | The total number of truncated occurrences for multiline messages. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin when `multiline.parser` is configured. | counter | occurrences |