From a7ed4c310d12c536874d1f06895cb95a78576a90 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 10:25:10 -0800 Subject: [PATCH 01/20] Vale: spelling update Signed-off-by: Lynette Miles --- vale-styles/FluentBit/Spelling-exceptions.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt index c83abd557..e4e17bd47 100644 --- a/vale-styles/FluentBit/Spelling-exceptions.txt +++ b/vale-styles/FluentBit/Spelling-exceptions.txt @@ -67,6 +67,7 @@ dri Dynatrace Elasticsearch endcode +endembed endhint endtab endtabs From 8a173658f53302ae62f4a9b3ca2138daf792a7d3 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 10:30:33 -0800 Subject: [PATCH 02/20] Vale: correcting suggestions plus making a table more readable in text Signed-off-by: Lynette Miles --- pipeline/filters/log_to_metrics.md | 440 ++++++++++++++--------------- 1 file changed, 220 insertions(+), 220 deletions(-) diff --git a/pipeline/filters/log_to_metrics.md b/pipeline/filters/log_to_metrics.md index 1ef152072..2bdfdcfde 100644 --- a/pipeline/filters/log_to_metrics.md +++ b/pipeline/filters/log_to_metrics.md @@ -8,7 +8,7 @@ description: Generate metrics from logs The _log to metrics_ filter lets you generate log-derived metrics. It supports modes to count records, provide a gauge for field values, or create a histogram. You can also match or exclude specific records based on regular expression patterns for values or nested values. -This filter doesn't actually act as a record filter and therefore does not change or drop records. All records will pass through this filter untouched, and any generated metrics will be emitted into a separate metric pipeline. +This filter doesn't actually act as a record filter and therefore doesn't change or drop records. All records will pass through this filter untouched, and any generated metrics will be emitted into a separate metric pipeline. {% hint style="warning" %} @@ -20,31 +20,31 @@ This filter is an experimental feature and isn't recommended for production use. The plugin supports the following configuration parameters: -| Key | Description | Default | Format | -|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------| -| `add_label` | Adds a custom label `NAME` and set the value to the value of `KEY`. | _none_ | `NAME KEY` | -| `alias` | Sets an alias, useful for multiple instances of the same filter plugin. | _none_ | | -| `bucket` | Optional for `metric_mode` `histogram`. If not set, default Prometheus-style buckets are used. | _none_ | For example, `0.75`. | -| `discard_logs` | Flag that defines if logs should be discarded after processing. This applies for all logs, whether they have emitted metrics or not. | `false` | | -| `emitter_name` | Name of the emitter (advanced users). | _none_ | | -| `emitter_mem_buf_limit` | Set a buffer limit to restrict memory usage of metrics emitter. | `10M` | | -| `exclude` | Excludes records in which the content of `KEY` matches the regular expression `REGEX`. | _none_ | `KEY REGEX` | -| `flush_interval_sec` | The interval for metrics emission, in seconds. If `flush_interval_sec` and `flush_interval_nsec` are either both unset or both set to `0`, the filter emits metrics immediately after each filter match. Otherwise, if either parameter is set to a non-zero value, the filter emits metrics at the specified interval. Longer intervals help lower resource consumption in high-load situations. | `0` | | -| `flush_interval_nsec` | The interval for metrics emission, in nanoseconds. This parameter works in conjunction with `flush_interval_sec`. | `0` | | -| `kubernetes_mode` | If enabled, adds `pod_id`, `pod_name`, `namespace_name`, `docker_id` and `container_name` to the metric as labels. This option is intended to be used in combination with the [Kubernetes](./kubernetes.md) filter plugin, which fills those fields. | `false` | | -| `label_field` | Includes a record field as label dimension in the metric. | _none_ | Name of record key. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | -| `log_level` | Specifies the log level for filter plugin. If not set here, plugin uses global log level in `service` section. | `info` | | -| `log_supress_interval` | Suppresses log messages from filter plugin that appear similar within a specified time interval. Setting to `0` indicates no suppression. | `0` | | -| `match` | Set a tag pattern to match records that filter should process. Exact matches or wildcards. | _none_ | | -| `match_regex` | Set a regular expression to match tags for filter routing. This allows more flexible matching compared to wildcards. | _none_ | | -| `metric_description` | Sets a description for the metric. | _none_ | | -| `metric_mode` | Defines the mode for the metric. Valid values are `counter`, `gauge` or `histogram`. | `counter` | | -| `metric_name` | Sets the name of the metric. | `a` | | -| `metric_namespace` | Sets the namespace of the metric. | `log_metric` | | -| `metric_subsystem` | Subsystem of the metric. | _none_ | | -| `regex` | Includes records in which the content of `KEY` matches the regular expression `REGEX`. | _none_ | `KEY REGEX` | -| `tag` | Defines the tag for the generated metrics record. | _none_ | | -| `value_field` | Required for modes `gauge` and `histogram`. Specifies the record field that holds a numerical value. | _none_ | Name of record key. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | +| Key | Description | Default | Format | +|-----|-------------|---------|--------| +| `add_label` | Adds a custom label `NAME` and set the value to the value of `KEY`. | _none_ | `NAME KEY` | +| `alias` | Sets an alias, use to set multiple instances of the same filter plugin. | _none_ | | +| `bucket` | Optional for `metric_mode` `histogram`. If not set, default Prometheus-style buckets are used. | _none_ | For example, `0.75`. | +| `discard_logs` | Flag that defines if logs should be discarded after processing. This applies for all logs, whether they have emitted metrics or not. | `false` | | +| `emitter_name` | Name of the emitter (advanced users). | _none_ | | +| `emitter_mem_buf_limit` | Set a buffer limit to restrict memory usage of metrics emitter. | `10M` | | +| `exclude` | Excludes records in which the content of `KEY` matches the regular expression `REGEX`. | _none_ | `KEY REGEX` | +| `flush_interval_sec`| The interval for metrics emission, in seconds. If `flush_interval_sec` and `flush_interval_nsec` are either both unset or both set to `0`, the filter emits metrics immediately after each filter match. Otherwise, if either parameter is set to a non-zero value, the filter emits metrics at the specified interval. Longer intervals help lower resource consumption in high-load situations. | `0` | | +| `flush_interval_nsec` | The interval for metrics emission, in nanoseconds. This parameter works in conjunction with `flush_interval_sec`. | `0` | | +| `kubernetes_mode` | If enabled, adds `pod_id`, `pod_name`, `namespace_name`, `docker_id` and `container_name` to the metric as labels. This option is intended to be used in combination with the [Kubernetes](./kubernetes.md) filter plugin, which fills those fields. | `false` | | +| `label_field` | Includes a record field as label dimension in the metric. | _none_ | Name of record key. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | +| `log_level` | Specifies the log level for filter plugin. If not set here, plugin uses global log level in `service` section.| `info` | | +| `log_supress_interval` | Suppresses log messages from filter plugin that appear similar within a specified time interval. Setting to `0` indicates no suppression. | `0` | | +| `match` | Set a tag pattern to match records that filter should process. Exact matches or wildcards.| _none_ | | +| `match_regex` | Set a regular expression to match tags for filter routing. This allows more flexible matching compared to wildcards. | _none_ | | +| `metric_description`| Sets a description for the metric.| _none_ | | +| `metric_mode` | Defines the mode for the metric. Valid values are `counter`, `gauge` or `histogram`. | `counter`| | +| `metric_name` | Sets the name of the metric. | `a` | | +| `metric_namespace` | Sets the namespace of the metric. | `log_metric` | | +| `metric_subsystem` | Subsystem of the metric. | _none_ | | +| `regex` | Includes records in which the content of `KEY` matches the regular expression `REGEX`.| _none_ | `KEY REGEX` | +| `tag` | Defines the tag for the generated metrics record. | _none_ | | +| `value_field` | Required for modes `gauge` and `histogram`. Specifies the record field that holds a numerical value. | _none_ | Name of record key. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | ## Examples @@ -69,27 +69,27 @@ service: pipeline: inputs: - - name: dummy - dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' - tag: dummy.log +- name: dummy + dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' + tag: dummy.log - - name: dummy - dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' - tag: dummy.log2 +- name: dummy + dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' + tag: dummy.log2 filters: - - name: log_to_metrics - match: 'dummy.log*' - tag: test_metric - metric_mode: counter - metric_name: count_all_dummy_messages - metric_description: 'This metric counts dummy messages' +- name: log_to_metrics + match: 'dummy.log*' + tag: test_metric + metric_mode: counter + metric_name: count_all_dummy_messages + metric_description: 'This metric counts dummy messages' outputs: - - name: prometheus_exporter - match: '*' - host: 0.0.0.0 - port: 9999 +- name: prometheus_exporter + match: '*' + host: 0.0.0.0 + port: 9999 ``` {% endtab %} @@ -97,32 +97,32 @@ pipeline: ```text [SERVICE] - flush 1 - log_level info + flush 1 + log_level info [INPUT] - Name dummy - Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} - Tag dummy.log + Name dummy + Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} + Tagdummy.log [INPUT] - Name dummy - Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} - Tag dummy.log2 + Name dummy + Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} + Tagdummy.log2 [FILTER] - name log_to_metrics - match dummy.log* - tag test_metric - metric_mode counter - metric_name count_all_dummy_messages + name log_to_metrics + match dummy.log* + tagtest_metric + metric_modecounter + metric_namecount_all_dummy_messages metric_description This metric counts dummy messages [OUTPUT] - name prometheus_exporter - match * - host 0.0.0.0 - port 9999 + name prometheus_exporter + match * + host 0.0.0.0 + port 9999 ``` {% endtab %} @@ -138,8 +138,8 @@ global: scrape_configs: # Scraping Fluent Bit example. - job_name: "fluentbit" - static_configs: - - targets: ["localhost:9999"] +static_configs: + - targets: ["localhost:9999"] ``` {% endtab %} @@ -170,34 +170,34 @@ service: pipeline: inputs: - - name: dummy - dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' - tag: dummy.log +- name: dummy + dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' + tag: dummy.log - - name: dummy - dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' - tag: dummy.log2 +- name: dummy + dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' + tag: dummy.log2 filters: - - name: log_to_metrics - match: 'dummy.log*' - tag: test_metric - metric_mode: gauge - metric_name: current_duration - metric_description: 'This metric shows the current duration' - value_field: duration - kubernetes_mode: on - regex: 'message .*el.*' - add_label: app $kubernetes['labels']['app'] - label_field: - - color - - shape +- name: log_to_metrics + match: 'dummy.log*' + tag: test_metric + metric_mode: gauge + metric_name: current_duration + metric_description: 'This metric shows the current duration' + value_field: duration + kubernetes_mode: on + regex: 'message .*el.*' + add_label: app $kubernetes['labels']['app'] + label_field: +- color +- shape outputs: - - name: prometheus_exporter - match: '*' - host: 0.0.0.0 - port: 9999 +- name: prometheus_exporter + match: '*' + host: 0.0.0.0 + port: 9999 ``` {% endtab %} @@ -205,38 +205,38 @@ pipeline: ```text [SERVICE] - flush 1 - log_level info + flush 1 + log_level info [INPUT] - Name dummy - Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} - Tag dummy.log + Name dummy + Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} + Tagdummy.log [INPUT] - Name dummy - Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} - Tag dummy.log2 + Name dummy + Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} + Tagdummy.log2 [FILTER] - name log_to_metrics - match dummy.log* - tag test_metric - metric_mode gauge - metric_name current_duration + name log_to_metrics + match dummy.log* + tagtest_metric + metric_modegauge + metric_namecurrent_duration metric_description This metric shows the current duration - value_field duration - kubernetes_mode on - regex message .*el.* - add_label app $kubernetes['labels']['app'] - label_field color - label_field shape + value_fieldduration + kubernetes_modeon + regex message .*el.* + add_label app $kubernetes['labels']['app'] + label_fieldcolor + label_fieldshape [OUTPUT] - name prometheus_exporter - match * - host 0.0.0.0 - port 9999 + name prometheus_exporter + match * + host 0.0.0.0 + port 9999 ``` {% endtab %} @@ -252,8 +252,8 @@ global: scrape_configs: # Scraping Fluent Bit example. - job_name: "fluentbit" - static_configs: - - targets: ["localhost:9999"] +static_configs: + - targets: ["localhost:9999"] ``` {% endtab %} @@ -294,34 +294,34 @@ service: pipeline: inputs: - - name: dummy - dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' - tag: dummy.log +- name: dummy + dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' + tag: dummy.log - - name: dummy - dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' - tag: dummy.log2 +- name: dummy + dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' + tag: dummy.log2 filters: - - name: log_to_metrics - match: 'dummy.log*' - tag: test_metric - metric_mode: histogram - metric_name: current_duration - metric_description: 'This metric shows the request duration' - value_field: duration - kubernetes_mode: on - regex: 'message .*el.*' - add_label: app $kubernetes['labels']['app'] - label_field: - - color - - shape +- name: log_to_metrics + match: 'dummy.log*' + tag: test_metric + metric_mode: histogram + metric_name: current_duration + metric_description: 'This metric shows the request duration' + value_field: duration + kubernetes_mode: on + regex: 'message .*el.*' + add_label: app $kubernetes['labels']['app'] + label_field: +- color +- shape outputs: - - name: prometheus_exporter - match: '*' - host: 0.0.0.0 - port: 9999 +- name: prometheus_exporter + match: '*' + host: 0.0.0.0 + port: 9999 ``` {% endtab %} @@ -329,38 +329,38 @@ pipeline: ```text [SERVICE] - flush 1 - log_level info + flush 1 + log_level info [INPUT] - Name dummy - Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} - Tag dummy.log + Name dummy + Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} + Tagdummy.log [INPUT] - Name dummy - Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} - Tag dummy.log2 + Name dummy + Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} + Tagdummy.log2 [FILTER] - name log_to_metrics - match dummy.log* - tag test_metric - metric_mode histogram - metric_name current_duration + name log_to_metrics + match dummy.log* + tagtest_metric + metric_modehistogram + metric_namecurrent_duration metric_description This metric shows the request duration - value_field duration - kubernetes_mode on - regex message .*el.* - add_label app $kubernetes['labels']['app'] - label_field color - label_field shape + value_fieldduration + kubernetes_modeon + regex message .*el.* + add_label app $kubernetes['labels']['app'] + label_fieldcolor + label_fieldshape [OUTPUT] - name prometheus_exporter - match * - host 0.0.0.0 - port 9999 + name prometheus_exporter + match * + host 0.0.0.0 + port 9999 ``` {% endtab %} @@ -376,8 +376,8 @@ global: scrape_configs: # Scraping Fluent Bit example. - job_name: "fluentbit" - static_configs: - - targets: ["localhost:9999"] +static_configs: + - targets: ["localhost:9999"] ``` {% endtab %} @@ -433,42 +433,42 @@ service: pipeline: inputs: - - name: dummy - dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' - tag: dummy.log +- name: dummy + dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' + tag: dummy.log - - name: dummy - dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' - tag: dummy.log2 +- name: dummy + dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' + tag: dummy.log2 filters: - - name: log_to_metrics - match: 'dummy.log*' - tag: test_metric - metric_mode: histogram - metric_name: current_duration - metric_description: 'This metric shows the HTTP request duration as histogram in milliseconds' - value_field: duration - kubernetes_mode: on - bucket: - - 1 - - 5 - - 10 - - 50 - - 1000 - - 250 - - 500 - - 1000 - regex: 'message .*el.*' - label_field: - - color - - shape +- name: log_to_metrics + match: 'dummy.log*' + tag: test_metric + metric_mode: histogram + metric_name: current_duration + metric_description: 'This metric shows the HTTP request duration as histogram in milliseconds' + value_field: duration + kubernetes_mode: on + bucket: +- 1 +- 5 +- 10 +- 50 +- 1000 +- 250 +- 500 +- 1000 + regex: 'message .*el.*' + label_field: +- color +- shape outputs: - - name: prometheus_exporter - match: '*' - host: 0.0.0.0 - port: 9999 +- name: prometheus_exporter + match: '*' + host: 0.0.0.0 + port: 9999 ``` {% endtab %} @@ -476,45 +476,45 @@ pipeline: ```text [SERVICE] - flush 1 - log_level info + flush 1 + log_level info [INPUT] - Name dummy - Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} - Tag dummy.log + Name dummy + Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} + Tagdummy.log [INPUT] - Name dummy - Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} - Tag dummy.log2 + Name dummy + Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} + Tagdummy.log2 [FILTER] - name log_to_metrics - match dummy.log* - tag test_metric - metric_mode histogram - metric_name current_duration + name log_to_metrics + match dummy.log* + tagtest_metric + metric_modehistogram + metric_namecurrent_duration metric_description This metric shows the HTTP request duration as histogram in milliseconds - value_field duration - kubernetes_mode on - bucket 1 - bucket 5 - bucket 10 - bucket 50 - bucket 100 - bucket 250 - bucket 500 - bucket 1000 - regex message .*el.* - label_field color - label_field shape + value_fieldduration + kubernetes_modeon + bucket 1 + bucket 5 + bucket 10 + bucket 50 + bucket 100 + bucket 250 + bucket 500 + bucket 1000 + regex message .*el.* + label_fieldcolor + label_fieldshape [OUTPUT] - name prometheus_exporter - match * - host 0.0.0.0 - port 9999 + name prometheus_exporter + match * + host 0.0.0.0 + port 9999 ``` {% endtab %} @@ -530,8 +530,8 @@ global: scrape_configs: # Scraping Fluent Bit example. - job_name: "fluentbit" - static_configs: - - targets: ["localhost:9999"] +static_configs: + - targets: ["localhost:9999"] ``` {% endtab %} From f8f19567431688af1d374b743a51a8974bf98361 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 10:31:19 -0800 Subject: [PATCH 03/20] Vale: correcting suggestions Signed-off-by: Lynette Miles --- pipeline/processors/conditional-processing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pipeline/processors/conditional-processing.md b/pipeline/processors/conditional-processing.md index 51f5272e6..94db09af1 100644 --- a/pipeline/processors/conditional-processing.md +++ b/pipeline/processors/conditional-processing.md @@ -91,9 +91,9 @@ The `conditions.rules.op` parameter has the following possible values: - `gte`: greater than or equal to - `lte`: less than or equal to - `regex`: matches a regular expression -- `not_regex`: does not match a regular expression +- `not_regex`: doesn't match a regular expression - `in`: is included in the specified array -- `not_in`: is not included in the specified array +- `not_in`: isn't included in the specified array ## Examples From 5e45468106e0e13a0a447616251f098a69206881 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 10:36:18 -0800 Subject: [PATCH 04/20] Style: Uncorrecting and recorrecting teh table spacing for readability Signed-off-by: Lynette Miles --- pipeline/filters/log_to_metrics.md | 423 ++++++++++++++--------------- 1 file changed, 211 insertions(+), 212 deletions(-) diff --git a/pipeline/filters/log_to_metrics.md b/pipeline/filters/log_to_metrics.md index 2bdfdcfde..c1d0a61e3 100644 --- a/pipeline/filters/log_to_metrics.md +++ b/pipeline/filters/log_to_metrics.md @@ -28,24 +28,23 @@ The plugin supports the following configuration parameters: | `discard_logs` | Flag that defines if logs should be discarded after processing. This applies for all logs, whether they have emitted metrics or not. | `false` | | | `emitter_name` | Name of the emitter (advanced users). | _none_ | | | `emitter_mem_buf_limit` | Set a buffer limit to restrict memory usage of metrics emitter. | `10M` | | -| `exclude` | Excludes records in which the content of `KEY` matches the regular expression `REGEX`. | _none_ | `KEY REGEX` | -| `flush_interval_sec`| The interval for metrics emission, in seconds. If `flush_interval_sec` and `flush_interval_nsec` are either both unset or both set to `0`, the filter emits metrics immediately after each filter match. Otherwise, if either parameter is set to a non-zero value, the filter emits metrics at the specified interval. Longer intervals help lower resource consumption in high-load situations. | `0` | | -| `flush_interval_nsec` | The interval for metrics emission, in nanoseconds. This parameter works in conjunction with `flush_interval_sec`. | `0` | | -| `kubernetes_mode` | If enabled, adds `pod_id`, `pod_name`, `namespace_name`, `docker_id` and `container_name` to the metric as labels. This option is intended to be used in combination with the [Kubernetes](./kubernetes.md) filter plugin, which fills those fields. | `false` | | -| `label_field` | Includes a record field as label dimension in the metric. | _none_ | Name of record key. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | -| `log_level` | Specifies the log level for filter plugin. If not set here, plugin uses global log level in `service` section.| `info` | | -| `log_supress_interval` | Suppresses log messages from filter plugin that appear similar within a specified time interval. Setting to `0` indicates no suppression. | `0` | | -| `match` | Set a tag pattern to match records that filter should process. Exact matches or wildcards.| _none_ | | -| `match_regex` | Set a regular expression to match tags for filter routing. This allows more flexible matching compared to wildcards. | _none_ | | -| `metric_description`| Sets a description for the metric.| _none_ | | -| `metric_mode` | Defines the mode for the metric. Valid values are `counter`, `gauge` or `histogram`. | `counter`| | -| `metric_name` | Sets the name of the metric. | `a` | | -| `metric_namespace` | Sets the namespace of the metric. | `log_metric` | | -| `metric_subsystem` | Subsystem of the metric. | _none_ | | -| `regex` | Includes records in which the content of `KEY` matches the regular expression `REGEX`.| _none_ | `KEY REGEX` | -| `tag` | Defines the tag for the generated metrics record. | _none_ | | -| `value_field` | Required for modes `gauge` and `histogram`. Specifies the record field that holds a numerical value. | _none_ | Name of record key. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | - +| `exclude` | Excludes records in which the content of `KEY` matches the regular expression `REGEX`. | _none_ | `KEY REGEX` | +| `flush_interval_sec` | The interval for metrics emission, in seconds. If `flush_interval_sec` and `flush_interval_nsec` are either both unset or both set to `0`, the filter emits metrics immediately after each filter match. Otherwise, if either parameter is set to a non-zero value, the filter emits metrics at the specified interval. Longer intervals help lower resource consumption in high-load situations. | `0` | | +| `flush_interval_nsec` | The interval for metrics emission, in nanoseconds. This parameter works in conjunction with `flush_interval_sec`.| `0` | | +| `kubernetes_mode` | If enabled, adds `pod_id`, `pod_name`, `namespace_name`, `docker_id` and `container_name` to the metric as labels. This option is intended to be used in combination with the [Kubernetes](./kubernetes.md) filter plugin, which fills those fields. | `false` | | +| `label_field` | Includes a record field as label dimension in the metric. | _none_ | Name of record key. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | +| `log_level` | Specifies the log level for filter plugin. If not set here, plugin uses global log level in `service` section.| `info` | | +| `log_supress_interval` | Suppresses log messages from filter plugin that appear similar within a specified time interval. Setting to `0` indicates no suppression. | `0` | | +| `match` | Set a tag pattern to match records that filter should process. Exact matches or wildcards. | _none_ | | +| `match_regex` | Set a regular expression to match tags for filter routing. This allows more flexible matching compared to wildcards. | _none_ | | +| `metric_description` | Sets a description for the metric. | _none_ | | +| `metric_mode` | Defines the mode for the metric. Valid values are `counter`, `gauge` or `histogram`.| `counter` | | +| `metric_name` | Sets the name of the metric. | `a` | | +| `metric_namespace` | Sets the namespace of the metric. | `log_metric` | | +| `metric_subsystem` | Subsystem of the metric. | _none_ | | +| `regex` | Includes records in which the content of `KEY` matches the regular expression `REGEX`. | _none_ | `KEY REGEX` | +| `tag` | Defines the tag for the generated metrics record. | _none_ | | +| `value_field` | Required for modes `gauge` and `histogram`. Specifies the record field that holds a numerical value. | _none_ | Name of record key. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | ## Examples @@ -69,27 +68,27 @@ service: pipeline: inputs: -- name: dummy - dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' - tag: dummy.log + - name: dummy + dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' + tag: dummy.log -- name: dummy - dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' - tag: dummy.log2 + - name: dummy + dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' + tag: dummy.log2 filters: -- name: log_to_metrics - match: 'dummy.log*' - tag: test_metric - metric_mode: counter - metric_name: count_all_dummy_messages - metric_description: 'This metric counts dummy messages' + - name: log_to_metrics + match: 'dummy.log*' + tag: test_metric + metric_mode: counter + metric_name: count_all_dummy_messages + metric_description: 'This metric counts dummy messages' outputs: -- name: prometheus_exporter - match: '*' - host: 0.0.0.0 - port: 9999 + - name: prometheus_exporter + match: '*' + host: 0.0.0.0 + port: 9999 ``` {% endtab %} @@ -97,32 +96,32 @@ pipeline: ```text [SERVICE] - flush 1 - log_level info + flush 1 + log_level info [INPUT] - Name dummy - Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} - Tagdummy.log + Name dummy + Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} + Tag dummy.log [INPUT] - Name dummy - Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} - Tagdummy.log2 + Name dummy + Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} + Tag dummy.log2 [FILTER] - name log_to_metrics - match dummy.log* - tagtest_metric - metric_modecounter - metric_namecount_all_dummy_messages + name log_to_metrics + match dummy.log* + tag test_metric + metric_mode counter + metric_name count_all_dummy_messages metric_description This metric counts dummy messages [OUTPUT] - name prometheus_exporter - match * - host 0.0.0.0 - port 9999 + name prometheus_exporter + match * + host 0.0.0.0 + port 9999 ``` {% endtab %} @@ -138,8 +137,8 @@ global: scrape_configs: # Scraping Fluent Bit example. - job_name: "fluentbit" -static_configs: - - targets: ["localhost:9999"] + static_configs: + - targets: ["localhost:9999"] ``` {% endtab %} @@ -170,34 +169,34 @@ service: pipeline: inputs: -- name: dummy - dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' - tag: dummy.log + - name: dummy + dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' + tag: dummy.log -- name: dummy - dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' - tag: dummy.log2 + - name: dummy + dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' + tag: dummy.log2 filters: -- name: log_to_metrics - match: 'dummy.log*' - tag: test_metric - metric_mode: gauge - metric_name: current_duration - metric_description: 'This metric shows the current duration' - value_field: duration - kubernetes_mode: on - regex: 'message .*el.*' - add_label: app $kubernetes['labels']['app'] - label_field: -- color -- shape + - name: log_to_metrics + match: 'dummy.log*' + tag: test_metric + metric_mode: gauge + metric_name: current_duration + metric_description: 'This metric shows the current duration' + value_field: duration + kubernetes_mode: on + regex: 'message .*el.*' + add_label: app $kubernetes['labels']['app'] + label_field: + - color + - shape outputs: -- name: prometheus_exporter - match: '*' - host: 0.0.0.0 - port: 9999 + - name: prometheus_exporter + match: '*' + host: 0.0.0.0 + port: 9999 ``` {% endtab %} @@ -205,38 +204,38 @@ pipeline: ```text [SERVICE] - flush 1 - log_level info + flush 1 + log_level info [INPUT] - Name dummy - Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} - Tagdummy.log + Name dummy + Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} + Tag dummy.log [INPUT] - Name dummy - Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} - Tagdummy.log2 + Name dummy + Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} + Tag dummy.log2 [FILTER] - name log_to_metrics - match dummy.log* - tagtest_metric - metric_modegauge - metric_namecurrent_duration + name log_to_metrics + match dummy.log* + tag test_metric + metric_mode gauge + metric_name current_duration metric_description This metric shows the current duration - value_fieldduration - kubernetes_modeon - regex message .*el.* - add_label app $kubernetes['labels']['app'] - label_fieldcolor - label_fieldshape + value_field duration + kubernetes_mode on + regex message .*el.* + add_label app $kubernetes['labels']['app'] + label_field color + label_field shape [OUTPUT] - name prometheus_exporter - match * - host 0.0.0.0 - port 9999 + name prometheus_exporter + match * + host 0.0.0.0 + port 9999 ``` {% endtab %} @@ -252,8 +251,8 @@ global: scrape_configs: # Scraping Fluent Bit example. - job_name: "fluentbit" -static_configs: - - targets: ["localhost:9999"] + static_configs: + - targets: ["localhost:9999"] ``` {% endtab %} @@ -294,34 +293,34 @@ service: pipeline: inputs: -- name: dummy - dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' - tag: dummy.log + - name: dummy + dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' + tag: dummy.log -- name: dummy - dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' - tag: dummy.log2 + - name: dummy + dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' + tag: dummy.log2 filters: -- name: log_to_metrics - match: 'dummy.log*' - tag: test_metric - metric_mode: histogram - metric_name: current_duration - metric_description: 'This metric shows the request duration' - value_field: duration - kubernetes_mode: on - regex: 'message .*el.*' - add_label: app $kubernetes['labels']['app'] - label_field: -- color -- shape + - name: log_to_metrics + match: 'dummy.log*' + tag: test_metric + metric_mode: histogram + metric_name: current_duration + metric_description: 'This metric shows the request duration' + value_field: duration + kubernetes_mode: on + regex: 'message .*el.*' + add_label: app $kubernetes['labels']['app'] + label_field: + - color + - shape outputs: -- name: prometheus_exporter - match: '*' - host: 0.0.0.0 - port: 9999 + - name: prometheus_exporter + match: '*' + host: 0.0.0.0 + port: 9999 ``` {% endtab %} @@ -329,38 +328,38 @@ pipeline: ```text [SERVICE] - flush 1 - log_level info + flush 1 + log_level info [INPUT] - Name dummy - Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} - Tagdummy.log + Name dummy + Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} + Tag dummy.log [INPUT] - Name dummy - Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} - Tagdummy.log2 + Name dummy + Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} + Tag dummy.log2 [FILTER] - name log_to_metrics - match dummy.log* - tagtest_metric - metric_modehistogram - metric_namecurrent_duration + name log_to_metrics + match dummy.log* + tag test_metric + metric_mode histogram + metric_name current_duration metric_description This metric shows the request duration - value_fieldduration - kubernetes_modeon - regex message .*el.* - add_label app $kubernetes['labels']['app'] - label_fieldcolor - label_fieldshape + value_field duration + kubernetes_mode on + regex message .*el.* + add_label app $kubernetes['labels']['app'] + label_field color + label_field shape [OUTPUT] - name prometheus_exporter - match * - host 0.0.0.0 - port 9999 + name prometheus_exporter + match * + host 0.0.0.0 + port 9999 ``` {% endtab %} @@ -376,8 +375,8 @@ global: scrape_configs: # Scraping Fluent Bit example. - job_name: "fluentbit" -static_configs: - - targets: ["localhost:9999"] + static_configs: + - targets: ["localhost:9999"] ``` {% endtab %} @@ -433,42 +432,42 @@ service: pipeline: inputs: -- name: dummy - dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' - tag: dummy.log + - name: dummy + dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' + tag: dummy.log -- name: dummy - dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' - tag: dummy.log2 + - name: dummy + dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' + tag: dummy.log2 filters: -- name: log_to_metrics - match: 'dummy.log*' - tag: test_metric - metric_mode: histogram - metric_name: current_duration - metric_description: 'This metric shows the HTTP request duration as histogram in milliseconds' - value_field: duration - kubernetes_mode: on - bucket: -- 1 -- 5 -- 10 -- 50 -- 1000 -- 250 -- 500 -- 1000 - regex: 'message .*el.*' - label_field: -- color -- shape + - name: log_to_metrics + match: 'dummy.log*' + tag: test_metric + metric_mode: histogram + metric_name: current_duration + metric_description: 'This metric shows the HTTP request duration as histogram in milliseconds' + value_field: duration + kubernetes_mode: on + bucket: + - 1 + - 5 + - 10 + - 50 + - 1000 + - 250 + - 500 + - 1000 + regex: 'message .*el.*' + label_field: + - color + - shape outputs: -- name: prometheus_exporter - match: '*' - host: 0.0.0.0 - port: 9999 + - name: prometheus_exporter + match: '*' + host: 0.0.0.0 + port: 9999 ``` {% endtab %} @@ -476,45 +475,45 @@ pipeline: ```text [SERVICE] - flush 1 - log_level info + flush 1 + log_level info [INPUT] - Name dummy - Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} - Tagdummy.log + Name dummy + Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} + Tag dummy.log [INPUT] - Name dummy - Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} - Tagdummy.log2 + Name dummy + Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} + Tag dummy.log2 [FILTER] - name log_to_metrics - match dummy.log* - tagtest_metric - metric_modehistogram - metric_namecurrent_duration + name log_to_metrics + match dummy.log* + tag test_metric + metric_mode histogram + metric_name current_duration metric_description This metric shows the HTTP request duration as histogram in milliseconds - value_fieldduration - kubernetes_modeon - bucket 1 - bucket 5 - bucket 10 - bucket 50 - bucket 100 - bucket 250 - bucket 500 - bucket 1000 - regex message .*el.* - label_fieldcolor - label_fieldshape + value_field duration + kubernetes_mode on + bucket 1 + bucket 5 + bucket 10 + bucket 50 + bucket 100 + bucket 250 + bucket 500 + bucket 1000 + regex message .*el.* + label_field color + label_field shape [OUTPUT] - name prometheus_exporter - match * - host 0.0.0.0 - port 9999 + name prometheus_exporter + match * + host 0.0.0.0 + port 9999 ``` {% endtab %} @@ -530,8 +529,8 @@ global: scrape_configs: # Scraping Fluent Bit example. - job_name: "fluentbit" -static_configs: - - targets: ["localhost:9999"] + static_configs: + - targets: ["localhost:9999"] ``` {% endtab %} From bfaed6495db57f79939dfb9d7fc25df17f72a1d9 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 10:50:45 -0800 Subject: [PATCH 05/20] Vale: Update tests to reduce suggestions and making table more readable in text Signed-off-by: Lynette Miles --- pipeline/inputs/tail.md | 86 +++++++++---------- vale-styles/FluentBit/Acronyms.yml | 4 + vale-styles/FluentBit/Headings.yml | 2 + vale-styles/FluentBit/Spelling-exceptions.txt | 1 + 4 files changed, 50 insertions(+), 43 deletions(-) diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 786d94962..7d1ac7fa1 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -8,37 +8,37 @@ The plugin reads every matched file in the `Path` pattern. For every new line fo The plugin supports the following configuration parameters: -| Key | Description | Default | -|:----------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------| -| `buffer_chunk_size` | Set the initial buffer size to read file data. This value is used to increase buffer size. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `32k` | -| `buffer_max_size` | Set the limit of the buffer size per monitored file. When a buffer needs to be increased, this value is used to restrict the memory buffer growth. If reading a file exceeds this limit, the file is removed from the monitored file list. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `32k` | -| `path` | Pattern specifying a specific log file or multiple ones through the use of common wildcards. Allows multiple patterns separated by commas. | _none_ | -| `path_key` | If enabled, it appends the name of the monitored file as part of the record. The value assigned becomes the key in the map. | _none_ | -| `exclude_path` | Set one or multiple shell patterns separated by commas to exclude files matching certain criteria, For example, `exclude_path *.gz,*.zip`. | _none_ | -| `offset_key` | If enabled, Fluent Bit appends the offset of the current monitored file as part of the record. The value assigned becomes the key in the map. | _none_ | -| `read_from_head` | For new discovered files on start (without a database offset/position), read the content from the head of the file, not tail. | `false` | -| `refresh_interval` | The interval of refreshing the list of watched files in seconds. | `60` | -| `rotate_wait` | Specify the number of extra time in seconds to monitor a file once is rotated in case some pending data is flushed. | `5` | -| `ignore_older` | Ignores files older than `ignore_older`. Supports `m`, `h`, `d` (minutes, hours, days) syntax. | Read all. | -| `ignore_active_older_files`| Ignore files that are older than the value set in `ignore_older` even if the file is being ingested. | `false` | -| `skip_long_lines` | When a monitored file reaches its buffer capacity due to a very long line (`buffer_max_size`), the default behavior is to stop monitoring that file. `skip_long_lines` alter that behavior and instruct Fluent Bit to skip long lines and continue processing other lines that fit into the buffer size. | `off` | -| `skip_empty_lines` | Skips empty lines in the log file from any further processing or output. | `off` | -| `db` | Specify the database file to keep track of monitored files and offsets. Recommended to be unique per plugin. | _none_ | +| Key | Description | Default | +|:----|:---------------|:----------| +| `buffer_chunk_size` | Set the initial buffer size to read file data. This value is used to increase buffer size. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `32k` | +| `buffer_max_size` | Set the limit of the buffer size per monitored file. When a buffer needs to be increased, this value is used to restrict the memory buffer growth. If reading a file exceeds this limit, the file is removed from the monitored file list. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `32k` | +| `path` | Pattern specifying a specific log file or multiple ones through the use of common wildcards. Allows multiple patterns separated by commas. | _none_ | +| `path_key` | If enabled, it appends the name of the monitored file as part of the record. The value assigned becomes the key in the map. | _none_ | +| `exclude_path` | Set one or multiple shell patterns separated by commas to exclude files matching certain criteria, For example, `exclude_path *.gz,*.zip`. | _none_ | +| `offset_key` | If enabled, Fluent Bit appends the offset of the current monitored file as part of the record. The value assigned becomes the key in the map. | _none_ | +| `read_from_head` | For new discovered files on start (without a database offset/position), read the content from the head of the file, not tail. | `false` | +| `refresh_interval` | The interval of refreshing the list of watched files in seconds. | `60` | +| `rotate_wait` | Specify the number of extra time in seconds to monitor a file once is rotated in case some pending data is flushed. | `5` | +| `ignore_older` | Ignores files older than `ignore_older`. Supports `m`, `h`, `d` (minutes, hours, days) syntax. | Read all. | +| `ignore_active_older_files`| Ignore files that are older than the value set in `ignore_older` even if the file is being ingested. | `false` | +| `skip_long_lines` | When a monitored file reaches its buffer capacity due to a very long line (`buffer_max_size`), the default behavior is to stop monitoring that file. `skip_long_lines` alter that behavior and instruct Fluent Bit to skip long lines and continue processing other lines that fit into the buffer size. | `off` | +| `skip_empty_lines` | Skips empty lines in the log file from any further processing or output. | `off` | +| `db` | Specify the database file to keep track of monitored files and offsets. Recommended to be unique per plugin. | _none_ | | `db.sync` | Set a default synchronization (I/O) method. This flag affects how the internal SQLite engine do synchronization to disk, for more details about each option see [the SQLite documentation](https://www.sqlite.org/pragma.html#pragma_synchronous). Most scenarios will be fine with `normal` mode. If you need full synchronization after every write operation set `full` mode. `full` has a high I/O performance cost. Values: `extra`, `full`, `normal`, `off`. | `normal` | -| `db.locking` | Specify that the database will be accessed only by Fluent Bit. Enabling this feature helps increase performance when accessing the database but restricts externals tool from querying the content. | `false` | -| `db.journal_mode` | Sets the journal mode for databases (`wal`). Enabling `wal` provides higher performance. `wal` isn't compatible with shared network file systems. | `wal` | -| `db.compare_filename` | This option determines whether to review both `inode` and `filename` when retrieving stored file information from the database. `true` verifies both `inode` and `filename`, while `false` checks only the `inode`. To review the `inode` and `filename` in the database, refer [see `keep_state`](#tailing-files-keeping-state). | `false` | -| `mem_buf_limit` | Set a memory limit that Tail plugin can use when appending data to the engine. If the limit is reached, it will be paused. When the data is flushed it resumes. | _none_ | -| `exit_on_eof` | When reading a file will exit as soon as it reach the end of the file. Used for bulk load and tests. | `false` | -| `parser` | Specify the name of a parser to interpret the entry as a structured message. | _none_ | -| `key` | When a message is unstructured (no parser applied), it's appended as a string under the key name `log`. This option lets you define an alternative name for that key. | `log` | -| `inotify_watcher` | Set to `false` to use file stat watcher instead of `inotify`. | `true` | +| `db.locking` | Specify that the database will be accessed only by Fluent Bit. Enabling this feature helps increase performance when accessing the database but restricts externals tool from querying the content. | `false` | +| `db.journal_mode` | Sets the journal mode for databases (`wal`). Enabling `wal` provides higher performance. `wal` isn't compatible with shared network file systems. | `wal` | +| `db.compare_filename` | This option determines whether to review both `inode` and `filename` when retrieving stored file information from the database. `true` verifies both `inode` and `filename`, while `false` checks only the `inode`. To review the `inode` and `filename` in the database, refer [see `keep_state`](#tailing-files-keeping-state). | `false` | +| `mem_buf_limit` | Set a memory limit that Tail plugin can use when appending data to the engine. If the limit is reached, it will be paused. When the data is flushed it resumes. | _none_ | +| `exit_on_eof` | When reading a file will exit as soon as it reach the end of the file. Used for bulk load and tests. | `false` | +| `parser` | Specify the name of a parser to interpret the entry as a structured message. | _none_ | +| `key` | When a message is unstructured (no parser applied), it's appended as a string under the key name `log`. This option lets you define an alternative name for that key. | `log` | +| `inotify_watcher` | Set to `false` to use file stat watcher instead of `inotify`. | `true` | | `tag` | Set a tag with `regexextract` fields that will be placed on lines read. For example, `kube....`. Tag expansion is supported: if the tag includes an asterisk (`*`), that asterisk will be replaced with the absolute path of the monitored file, with slashes replaced by dots. See [Workflow of Tail + Kubernetes Filter](../filters/kubernetes.md#workflow-of-tail--kubernetes-filter). | _none_ | -| `tag_regex` | Set a regular expression to extract fields from the filename. For example: `(?[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$`. | _none_ | -| `static_batch_size` | Set the maximum number of bytes to process per iteration for the monitored static files (files that already exist upon Fluent Bit start). | `50M` | -| `file_cache_advise` | Set the `posix_fadvise` in `POSIX_FADV_DONTNEED` mode. This reduces the usage of the kernel file cache. This option is ignored if not running on Linux. | `on` | -| `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | -| `Unicode.Encoding` | Set the Unicode character encoding of the file data. This parameter requests two-byte aligned chunk and buffer sizes. If data is not aligned for two bytes, Fluent Bit will use two-byte alignment automatically to avoid character breakages on consuming boundaries. Supported values: `UTF-16LE`, `UTF-16BE`, and `auto`. | `none` | +| `tag_regex` | Set a regular expression to extract fields from the filename. For example: `(?[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$`. | _none_ | +| `static_batch_size` | Set the maximum number of bytes to process per iteration for the monitored static files (files that already exist upon Fluent Bit start). | `50M` | +| `file_cache_advise` | Set the `posix_fadvise` in `POSIX_FADV_DONTNEED` mode. This reduces the usage of the kernel file cache. This option is ignored if not running on Linux. | `on` | +| `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | +| `Unicode.Encoding` | Set the Unicode character encoding of the file data. This parameter requests two-byte aligned chunk and buffer sizes. If data isn't aligned for two bytes, Fluent Bit will use two-byte alignment automatically to avoid character breakages on consuming boundaries. Supported values: `UTF-16LE`, `UTF-16BE`, and `auto`. | `none` | | `Generic.Encoding` | Set the non-Unicode encoding of the file data. Supported values: `ShiftJIS`, `UHC`, `GBK`, `GB18030`, `Big5`, `Win866`, `Win874`, `Win1250`, `Win1251`, `Win1252`, `Win2513`, `Win1254`, `Win1255`, and `Win1256`. | `none` | ## Buffers and memory management @@ -81,7 +81,7 @@ If no database file is present, positioning behavior depends on the value of `re The database file essentially stores `inode=offset` so it should be unique per instance of the plugin, for example if you have two tail inputs then use two separate `db` files for each. That way each tail input can independently track its own state. {% hint style="info" %} -The `Unicode.Encoding` parameter is dependent on the simdutf library, which is itself dependent on C++ version 11 or later. In environments that use earlier versions of C++, the `Unicode.Encoding` parameter will fail. +The `Unicode.Encoding` parameter is dependent on the `simdutf` library, which is itself dependent on C++ version 11 or later. In environments that use earlier versions of C++, the `Unicode.Encoding` parameter will fail. Additionally, the `auto` setting for `Unicode.Encoding` isn't supported in all cases, and can make mistakes when it tries to guess the correct encoding. For best results, use either the `UTF-16LE` or `UTF-16BE` setting if you know the encoding type of the target file. {% endhint %} @@ -154,8 +154,8 @@ Fluent Bit 1.8 and later supports multiline core capabilities for the Tail input Multiline core is exposed by the following configuration: -| Key | Description | -|:-------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------| +| Key | Description | +|:-------------------|:---------------| | `multiline.parser` | Specify one or multiple [Multiline Parser definitions](../../administration/configuring-fluent-bit/multiline-parsing.md) to apply to the content. | [Multiline Parser](../../administration/configuring-fluent-bit/multiline-parsing.md) provides built-in configuration modes. When using a new `multiline.parser` definition, you must disable the old configuration from your tail section like: @@ -205,10 +205,10 @@ For example, it will first try `docker`, and if `docker` doesn't match, it will For the old multiline configuration, the following options exist to configure the handling of multiline logs: -| Key | Description | Default | -|:-------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------| +| Key | Description | Default | +|:-------------------|:------------------|:--------| | `multiline` | If enabled, the plugin will try to discover multiline messages and use the proper parsers to compose the outgoing messages. When this option is enabled the Parser option isn't used. | `off` | -| `multiline_flush` | Wait period time in seconds to process queued multiline messages. | `4` | +| `multiline_flush` | Wait period time in seconds to process queued multiline messages. | `4` | | `parser_firstline` | Name of the parser that matches the beginning of a multiline message. The regular expression defined in the parser must include a group name (named `capture`), and the value of the last match group must be a string. | _none_ | | `parser_N` | Optional. Extra parser to interpret and structure multiline entries. This option can be used to define multiple parsers. For example, `parser_1 ab1`, `parser_2 ab2`, `parser_N abN`. | _none_ | @@ -216,10 +216,10 @@ For the old multiline configuration, the following options exist to configure th Docker mode exists to recombine JSON log lines split by the Docker daemon due to its line length limit. To use this feature, configure the tail plugin with the corresponding parser and then enable Docker mode: -| Key | Description | Default | -|:---------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------|:--------| +| Key | Description | Default | +|:------|:------------|:--------| | `docker_mode` | If enabled, the plugin will recombine split Docker log lines before passing them to any parser. This mode can't be used at the same time as Multiline. | `Off` | -| `docker_mode_flush` | Wait period time in seconds to flush queued unfinished split lines. | `4` | +| `docker_mode_flush` | Wait period time in seconds to flush queued unfinished split lines. | `4` | | `docker_mode_parser` | Specify an optional parser for the first line of the Docker multiline mode. The parser name to be specified must be registered in the `parsers.conf` file. | _none_ | ## Get started @@ -489,13 +489,13 @@ This is common in environments that use: - Legacy Windows systems with applications that use traditional code pages (for example, ShiftJIS, GBK, Win1252). -### Configuration parameters +### Character encoding configuration parameters To enable encoding conversion, you will use one of the following two parameters within an input plugin configuration. 1. `Unicode.Encoding` - Use this parameter for high-performance conversion of UTF-16 encoded logs to UTF-8. This method utilizes modern processor features (SIMD instructions) to accelerate the conversion process, making it highly efficient. + Use this parameter for high-performance conversion of UTF-16 encoded logs to UTF-8. This method utilizes modern processor features (SIMD instructions) to accelerate the conversion process, making it highly efficient. - Use Case: Ideal for logs coming from modern Windows environments that default to UTF-16. - Supported Values: @@ -507,9 +507,9 @@ To enable encoding conversion, you will use one of the following two parameters Use this parameter to convert from a wide variety of other character encodings, particularly legacy Windows code pages. - Use Case: Essential for logs from older systems or applications configured for specific regions, common in East Asia and Eastern Europe. - - Supported Values: You can use any of the names or aliases listed below. + - Supported values: You can use any of the names or aliases listed below. -### East Asian Encodings +### East Asian encodings - `ShiftJIS` (Aliases: `SJIS`, `CP932`, `Windows-31J`) - `GB18030` @@ -558,4 +558,4 @@ pipeline: ``` {% endtab %} -{% endtabs %} \ No newline at end of file +{% endtabs %} diff --git a/vale-styles/FluentBit/Acronyms.yml b/vale-styles/FluentBit/Acronyms.yml index 2ce57fee4..af1fa09fd 100644 --- a/vale-styles/FluentBit/Acronyms.yml +++ b/vale-styles/FluentBit/Acronyms.yml @@ -31,6 +31,7 @@ exceptions: - DEBUG - DESC - DOM + - DOS - DNS - DPI - DPPS @@ -40,6 +41,7 @@ exceptions: - EKS - FAQ - FIPS + - GBK - GCC - GCP - GCS @@ -85,6 +87,7 @@ exceptions: - NSIS - NVDA - OAP + - OEM - OOM - OSS - PATH @@ -106,6 +109,7 @@ exceptions: - SDS - SHA - SIEM + - SIMD - SLA - SLES - SNI diff --git a/vale-styles/FluentBit/Headings.yml b/vale-styles/FluentBit/Headings.yml index 807bd020f..2f48456d5 100644 --- a/vale-styles/FluentBit/Headings.yml +++ b/vale-styles/FluentBit/Headings.yml @@ -52,6 +52,7 @@ exceptions: - Docker - Docker Hub - DogStatsD + - DOS - eBPF - EC2 - ECS @@ -108,6 +109,7 @@ exceptions: - NATS - New Relic - Observability Platform + - OEM - OIDC - Okta - OpenMetrics diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt index e4e17bd47..63e49815a 100644 --- a/vale-styles/FluentBit/Spelling-exceptions.txt +++ b/vale-styles/FluentBit/Spelling-exceptions.txt @@ -81,6 +81,7 @@ FluentBit Fluentd formatter formatters +GBK github glibc Golang From 71903907ccadc580609c1d70f152af0129fc569c Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 10:56:54 -0800 Subject: [PATCH 06/20] Vale: reducing suggestions Signed-off-by: Lynette Miles --- pipeline/outputs/azure_kusto.md | 138 ++++++++++++++--------------- vale-styles/FluentBit/Acronyms.yml | 1 + 2 files changed, 70 insertions(+), 69 deletions(-) diff --git a/pipeline/outputs/azure_kusto.md b/pipeline/outputs/azure_kusto.md index 2e078cbb6..c29275b92 100644 --- a/pipeline/outputs/azure_kusto.md +++ b/pipeline/outputs/azure_kusto.md @@ -48,79 +48,79 @@ By default, Kusto will insert incoming ingestion data into a table by inferring ## Configuration parameters -| Key | Description | Default | -|----------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------| -| `alias` | Sets an alias, useful for multiple instances of the same output plugin. | _none_ | +| Key | Description | Default | +|------|-------------|--------| +| `alias` | Sets an alias, use to define multiple instances of the same output plugin. | _none_ | | `auth_type` | Set the authentication type: `service_principal`, `managed_identity`, or `workload_identity`. For `managed_identity`, use `system` as `client_id` for `system-assigned identity`, or specify the managed identity's client ID. | `service_principal` | | `azure_kusto_buffer_key` | When buffering is `true`, set the Azure Kusto buffer key which must be specified when using multiple instances of Azure Kusto output plugin and buffering is enabled. | `key` | -| `blob_uri_length` | Set the length of generated blob URI before ingesting to Kusto. | `64` | -| `buffer_dir` | When buffering is `true`, specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-kusto/` | -| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Kusto. | `false` | -| `buffer_file_delete_early` | When buffering is `true`, whether to delete the buffered file early after successful blob creation. | `false` | -| `unify_tag` | Optional. This creates a single buffer file when the buffering mode is `On`. | `On` | -| `client_id` | Required if `managed_identity_client_id` isn't set. The client ID of the AAD registered application. | _none_ | -| `client_secret` | Set the client secret (Application Password) of the AAD application used for authentication. | _none_ | -| `compression_enabled` | If enabled, sends compressed HTTP payload (gzip) to Kusto. | `true` | -| `database_name` | The database name. | _none_ | -| `delete_on_max_upload_error` | When buffering is `true`, whether to delete the buffer file on maximum upload errors. | `false` | -| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | -| `io_timeout` | Configure the HTTP IO timeout for uploads. | `60s` | -| `include_tag_key` | If enabled, a tag is appended to output. The key name is used `tag_key` property. | `true` | -| `include_time_key` | If enabled, a timestamp is appended to output. The key name is used `time_key` property. | `true` | -| `ingestion_endpoint` | The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net`. | _none_ | -| `ingestion_endpoint_connect_timeout` | The connection timeout of various Kusto endpoints in seconds. | `60` | +| `blob_uri_length` | Set the length of generated blob URI before ingesting to Kusto. | `64` | +| `buffer_dir` | When buffering is `true`, specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-kusto/` | +| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Kusto. | `false` | +| `buffer_file_delete_early` | When buffering is `true`, whether to delete the buffered file early after successful blob creation. | `false` | +| `unify_tag` | Optional. This creates a single buffer file when the buffering mode is `On`. | `On` | +| `client_id` | Required if `managed_identity_client_id` isn't set. The client ID of the AAD registered application. | _none_ | +| `client_secret` | Set the client secret (Application Password) of the AAD application used for authentication. | _none_ | +| `compression_enabled` | If enabled, sends compressed HTTP payload (gzip) to Kusto. | `true` | +| `database_name` | The database name. | _none_ | +| `delete_on_max_upload_error` | When buffering is `true`, whether to delete the buffer file on maximum upload errors. | `false` | +| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | +| `io_timeout` | Configure the HTTP IO timeout for uploads. | `60s` | +| `include_tag_key` | If enabled, a tag is appended to output. The key name is used `tag_key` property. | `true` | +| `include_time_key` | If enabled, a timestamp is appended to output. The key name is used `time_key` property. | `true` | +| `ingestion_endpoint` | The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net`. | _none_ | +| `ingestion_endpoint_connect_timeout` | The connection timeout of various Kusto endpoints in seconds. | `60` | | `ingestion_mapping_reference` | The name of a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping) that will be used to map the ingested payload into the table columns. | _none_ | -| `ingestion_resources_refresh_interval` | Set the Azure Kusto ingestion resources refresh interval. | `3600` | -| `log_key` | Key name of the log content. | `log` | -| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | -| `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | -| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | -| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to simple wildcards. | _none_ | -| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` | -| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | -| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | -| `net.dns.prefer_ipv4` | Select the primary DNS resolver type (LEGACY or ASYNC). | `false` | -| `net.dns.prefer_ipv6` | Prioritize IPv6 DNS results when trying to establish a connection. | `false` | -| `net.dns.resolver` | Select the primary DNS resolver type (LEGACY or ASYNC). | _none_ | -| `net.io_timeout` | Set maximum time a connection can stay idle while assigned. | `0s` | -| `net.keepalive` | Enable or disable `Keepalive` support. | `false` | -| `net.keepalive_idle_timeout` | Set maximum time allowed for an idle `Keepalive` connection.. | `false` | -| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | -| `net.proxy_env_ignore` | Ignore the environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` when set. | `false` | -| `net.source_address` | Specify network address to bind for data traffic. | _none_ | -| `net.tcp_keepalive` | Enable or disable Keepalive support. | `off` | -| `net.tcp_keepalive_interval` | Interval between TCP keepalive probes when no response is received on a keepidle probe. | `-1` | -| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it is retried. | `2000` | -| `net.tcp_keepalive_probes` | Number of unacknowledged probes to consider a connection dead. | `-1` | -| `net.tcp_keepalive_time` | Interval between the last data packet sent and the first TCP keepalive probe. | `-1` | -| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | -| `port` | TCP port of the target HTTP server. | `0` | -| `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` | -| `scheduler_max_retries` | Optional. When buffering is `On`, set the maximum number of retries for ingestion using the scheduler. | `3` | -| `store_dir_limit_size` | When buffering is `true`, set the max size of the buffer directory. | `8G` | -| `tag_key` | The key name of tag. If `include_tag_key` is false, This property is ignored. | `tag` | -| `table_name` | The table name. | _none_ | -| `tenant_id` | Required if `managed_identity_client_id` isn't set. The tenant/domain ID of the AAD registered application. | _none_ | -| `time_key` | The key name of time. If `include_time_key` is false, this property is ignored. | `timestamp` | -| `tls` | Enable or disable TLS/SSL support. | `off` | -| `tls.ca_file` | Absolute path to CA certificate file. | _none_ | -| `tls.ca_path` | Absolute path to scan for certificate files. | _none_ | -| `tls.cert_file` | Absolute path to Certificate file. | _none_ | -| `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | _none_ | -| `tls.debug` | Set TLS debug level. Accepts `0` (No debug), `1`(Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | -| `tls.key_file` | Absolute path to private Key file. | _none_ | -| `tls.key_passwd` | Optional password for tls.key_file file. | _none_ | -| `tls.max_version` | Specify the maximum version of TLS. | _none_ | -| `tls.min_version` | Specify the minimum version of TLS. | _none_ | -| `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | -| `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ | -| `tls.verify` | Force certificate validation. | `on` | -| `tls.windows.certstore_name` | Sets the certstore name on an output (Windows). | _none_ | -| `tls.windows.use_enterprise_store` | Sets whether using enterprise certificate store or not on an output (Windows). | _none_ | -| `unify_tag` | This creates a single buffer file when the buffering mode is `true`. | `true` | -| `upload_file_size` | Specifies the size of files to be uploaded in megabytes. | `200MB` | +| `ingestion_resources_refresh_interval` | Set the Azure Kusto ingestion resources refresh interval. | `3600` | +| `log_key` | Key name of the log content. | `log` | +| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | +| `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | +| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | +| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to wildcards. | _none_ | +| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` | +| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | +| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | +| `net.dns.prefer_ipv4` | Select the primary DNS resolver type (LEGACY or ASYNC). | `false` | +| `net.dns.prefer_ipv6` | Prioritize IPv6 DNS results when trying to establish a connection. | `false` | +| `net.dns.resolver` | Select the primary DNS resolver type (LEGACY or ASYNC). | _none_ | +| `net.io_timeout` | Set maximum time a connection can stay idle while assigned. | `0s` | +| `net.keepalive` | Enable or disable `Keepalive` support. | `false` | +| `net.keepalive_idle_timeout` | Set maximum time allowed for an idle `Keepalive` connection.. | `false` | +| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | +| `net.proxy_env_ignore` | Ignore the environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` when set. | `false` | +| `net.source_address` | Specify network address to bind for data traffic. | _none_ | +| `net.tcp_keepalive` | Enable or disable Keepalive support. | `off` | +| `net.tcp_keepalive_interval` | Interval between TCP keepalive probes when no response is received on a `keepidle` probe. | `-1` | +| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it's retried. | `2000` | +| `net.tcp_keepalive_probes` | Number of unacknowledged probes to consider a connection dead. | `-1` | +| `net.tcp_keepalive_time` | Interval between the last data packet sent and the first TCP keepalive probe.| `-1` | +| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | +| `port` | TCP port of the target HTTP server. | `0` | +| `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` | +| `scheduler_max_retries` | Optional. When buffering is `On`, set the maximum number of retries for ingestion using the scheduler. | `3` | +| `store_dir_limit_size` | When buffering is `true`, set the max size of the buffer directory. | `8G` | +| `tag_key` | The key name of tag. If `include_tag_key` is false, This property is ignored.| `tag` | +| `table_name` | The table name. | _none_ | +| `tenant_id` | Required if `managed_identity_client_id` isn't set. The tenant/domain ID of the AAD registered application. | _none_ | +| `time_key` | The key name of time. If `include_time_key` is false, this property is ignored. | `timestamp` | +| `tls` | Enable or disable TLS/SSL support. | `off` | +| `tls.ca_file` | Absolute path to CA certificate file. | _none_ | +| `tls.ca_path` | Absolute path to scan for certificate files. | _none_ | +| `tls.cert_file` | Absolute path to Certificate file. | _none_ | +| `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | _none_ | +| `tls.debug` | Set TLS debug level. Accepts `0` (No debug), `1`(Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | +| `tls.key_file` | Absolute path to private Key file. | _none_ | +| `tls.key_passwd` | Optional password for tls.key_file file. | _none_ | +| `tls.max_version` | Specify the maximum version of TLS. | _none_ | +| `tls.min_version` | Specify the minimum version of TLS. | _none_ | +| `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | +| `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ | +| `tls.verify` | Force certificate validation. | `on` | +| `tls.windows.certstore_name` | Sets the `certstore` name on an output (Windows). | _none_ | +| `tls.windows.use_enterprise_store` | Sets whether using enterprise certificate store or not on an output (Windows). | _none_ | +| `unify_tag` | This creates a single buffer file when the buffering mode is `true`. | `true` | +| `upload_file_size` | Specifies the size of files to be uploaded in megabytes. | `200MB` | | `upload_timeout` | Optionally specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than x minutes and haven't reached `upload_file_size` limit yet. | `30m` | -| `workload_identity_token_file` | Set the token path for workload identity authentication. | _none_ | +| `workload_identity_token_file` | Set the token path for workload identity authentication. | _none_ | ### Configuration file diff --git a/vale-styles/FluentBit/Acronyms.yml b/vale-styles/FluentBit/Acronyms.yml index af1fa09fd..6577a763a 100644 --- a/vale-styles/FluentBit/Acronyms.yml +++ b/vale-styles/FluentBit/Acronyms.yml @@ -16,6 +16,7 @@ exceptions: - ARN - ASC - ASP + - ASYNC - AWS - BSD - CIDR From 84267b94231eaafbf694f0034060044d12edbbcd Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 11:05:10 -0800 Subject: [PATCH 07/20] Vale: reducing suggestions Signed-off-by: Lynette Miles --- pipeline/outputs/opentelemetry.md | 172 ++++++++++++++--------------- vale-styles/FluentBit/Acronyms.yml | 1 + 2 files changed, 85 insertions(+), 88 deletions(-) diff --git a/pipeline/outputs/opentelemetry.md b/pipeline/outputs/opentelemetry.md index cff4d14ae..918fa5457 100644 --- a/pipeline/outputs/opentelemetry.md +++ b/pipeline/outputs/opentelemetry.md @@ -1,97 +1,93 @@ -— - -== description: An output plugin to submit Logs, Metrics, or Traces to an OpenTelemetry endpoint - -= OpenTelemetry +# OpenTelemetry The OpenTelemetry plugin lets you take logs, metrics, and traces from Fluent Bit and submit them to an OpenTelemetry HTTP endpoint. Only HTTP endpoints are supported. -| Key | Description | Default | -|-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------| -| `add_label` | Adds a custom label to the metrics use format: `add_label name value`. | _none_ | -| `alias` | Sets an alias, useful for multiple instances of the same output plugin. | _none_ | -| `aws_auth` | Enable AWS SigV4 authentication. | `false` | -| `aws_external_id` | Specify an external ID for the STS API, can be used with the `aws_role_arn` parameter. | _none_ | -| `aws_profile` | AWS Profile name. AWS Profiles can be configured with AWS CLI. | _none_ | -| `aws_region` | AWS region of your service. | _none_ | -| `aws_role_arn` | ARN of an IAM role to assume (ex. for cross account access). | _none_ | -| `aws_service` | AWS destination service code, used by SigV4 authentication. | `logs` | -| `aws_sts_endpoint` | Custom endpoint for the AWS STS API, used with the `aws_role_arn` option. | _none_ | -| `grpc` | Enable, disable or force gRPC usage. Accepted values : `on`, `off`, `auto`. | `off` | -| `batch_size` | Set the maximum number of log records to be flushed at a time. | `1000` | -| `compress` | Set payload compression mechanism. Options available are `gzip` and `zstd`. | _none_ | -| `grpc_logs_uri` | Specify an optional gRPCß URI for the target OTel endpoint. | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` | -| `grpc_metrics_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export` | -| `grpc_profiles_uri` | Specify an optional gRPC URI for profiles OTel endpoint. | `/opentelemetry.proto.collector.profiles.v1experimental.ProfilesService/Export` | -| `grpc_traces_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.trace.v1.TraceService/Export` | -| `header` | Add a HTTP header key/value pair. Multiple headers can be set. | _none_ | -| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | -| `http2` | Enable, disable or force HTTP/2 usage. Accepted values : `on`, `off`, or `force`. | `off` | -| `http_passwd` | Set HTTP auth password. | _none_ | -| `http_user` | Set HTTP auth user. | _none_ | -| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | -| `logs_attributes_metadata_key` | Specify an `Attributes` key. | `$Attributes` | -| `logs_body_key` | Specify an optional HTTP URI for the target OTel endpoint. | _none_ | -| `logs_body_key_attributes` | If set and it matched a pattern, it includes the remaining fields in the record as attributes. | `false` | -| `logs_instrumentation_scope_metadata_key` | Specify an `InstrumentationScope` key. | `InstrumentationScope` | -| `logs_observed_timestamp_metadata_key` | Specify an `ObservedTimestamp` key. | `$ObservedTimestamp` | -| `logs_max_resources` | Set the maximum number of OTLP log resources per export request (`0` disables the limit). | `0` | -| `logs_max_scopes` | Set the maximum number of OTLP log scopes per resource (`0` disables the limit). | `0` | -| `logs_metadata_key` | Set the key to lookup in the metadata. | `otlp` | -| `logs_resource_metadata_key` | Specify a `Resource` key. | `Resource` | -| `log_response_payload` | Specify if the response payload should be logged or not. | `true` | -| `logs_severity_number_message_key` | Specify a `SeverityNumber` key. | `$severityNumber` | -| `logs_severity_number_metadata_key` | Specify a `SeverityNumber` key. | `$SeverityNumber` | -| `logs_severity_text_message_key` | Specify a `SeverityText` key. | `$SeverityText` | -| `log_severity_text_metadata_key` | Specify a `SeverityText` key. | `$SeverityText` | -| `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | -| `logs_span_id_message_key` | Specify a `SpanId` key. | `$SpanId` | -| `logs_span_id_metadata_key` | Specify a `SpanId` key. | `$SpanId` | -| `logs_timestamp_metadata_key` | Specify a `Timestamp` key. | `$Timestamp` | -| `logs_trace_flags_metadata_key` | Specify a `TraceFlags` key. | `$TraceFlags` | -| `logs_trace_id_message_key` | Specify a `TraceId` key. | `$TraceId` | -| `logs_trace_id_metadata_key` | Specify a `TraceId` key. | `$TraceId` | -| `logs_uri` | Specify an optional HTTP URI for the target OTel endpoint. | `/v1/logs` | -| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | -| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to simple wildcards. | _none_ | -| `metrics_uri` | Specify an optional HTTP URI for the target OTel endpoint. | `/v1/metrics` | -| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` | -| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | -| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it is retried. | `2000` | -| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | -| `net.dns.prefer_ipv4` | Select the primary DNS resolver type (LEGACY or ASYNC). | _none_ | -| `net.dns.prefer_ipv6` | Prioritize IPv6 DNS results when trying to establish a connection. | _none_ | -| `net.io_timeout` | Set maximum time a connection can stay idle while assigned. | `0s` | -| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | -| `net.proxy_env_ignore` | Ignore the environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` when set. | `false` | -| `net.source_address` | Specify network address to bind for data traffic. | _none_ | -| `net.tcp_keepalive` | Enable or disable Keepalive support. | `off` | -| `net.tcp_keepalive_time` | Interval between the last data packet sent and the first TCP keepalive probe. | `-1` | -| `net.tcp_keepalive_interval` | Interval between TCP keepalive probes when no response is received on a keepidle probe. | `-1` | -| `net.tcp_keepalive_probes` | Number of unacknowledged probes to consider a connection dead. | `-1` | -| `port` | TCP port of the target HTTP server. | `80` | -| `profiles_uri` | Specify an optional HTTP URI for the profiles OTel endpoint. | `/v1development/profiles` | -| `proxy` | Specify an HTTP Proxy. The expected format of this value is `http://host:port`. | _none_ | -| `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` | -| `tls` | Enable or disable TLS/SSL support. | `off` | -| `tls.ca_file` | Absolute path to CA certificate file. | _none_ | -| `tls.ca_path` | Absolute path to scan for certificate files. | _none_ | -| `tls.cert_file` | Absolute path to Certificate file. | _none_ | -| `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | _none_ | -| `tls.debug` | Set TLS debug level. Accepts `0` (No debug), `1`(Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | -| `tls.key_file` | Absolute path to private Key file. | _none_ | -| `tls.key_passwd` | Optional password for tls.key_file file. | _none_ | -| `tls.max_version` | Specify the maximum version of TLS. | _none_ | -| `tls.min_version` | Specify the minimum version of TLS. | _none_ | -| `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | -| `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ | -| `tls.verify` | Force certificate validation. | `on` | -| `tls.windows.certstore_name` | Sets the certstore name on an output (Windows). | _none_ | -| `tls.windows.use_enterprise_store` | Sets whether using enterprise certstore or not on an output (Windows). | _none_ | -| `traces_uri` | Specify an optional HTTP URI for the target OTel endpoint. | `/v1/traces` | +| Key | Description | Default | +|---------|--------------|---------| +| `add_label` | Adds a custom label to the metrics use format: `add_label name value`. | _none_ | +| `alias` | Sets an alias, use for multiple instances of the same output plugin. | _none_ | +| `aws_auth` | Enable AWS SigV4 authentication. | `false`| +| `aws_external_id` | Specify an external ID for the STS API, can be used with the `aws_role_arn` parameter. | _none_ | +| `aws_profile` | AWS Profile name. AWS Profiles can be configured with AWS CLI. | _none_ | +| `aws_region` | AWS region of your service. | _none_ | +| `aws_role_arn` | ARN of an IAM role to assume (ex. for cross account access). | _none_ | +| `aws_service` | AWS destination service code, used by SigV4 authentication. | `logs` | +| `aws_sts_endpoint` | Custom endpoint for the AWS STS API, used with the `aws_role_arn` option. | _none_ | +| `grpc` | Enable, disable or force gRPC usage. Accepted values : `on`, `off`, `auto`. | `off` | +| `batch_size` | Set the maximum number of log records to be flushed at a time. | `1000` | +| `compress` | Set payload compression mechanism. Options available are `gzip` and `zstd`. | _none_ | +| `grpc_logs_uri` | Specify an optional gRPCß URI for the target OTel endpoint. | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` | +| `grpc_metrics_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export` | +| `grpc_profiles_uri` | Specify an optional gRPC URI for profiles OTel endpoint. | `/opentelemetry.proto.collector.profiles.v1experimental.ProfilesService/Export` | +| `grpc_traces_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.trace.v1.TraceService/Export` | +| `header` | Add a HTTP header key/value pair. Multiple headers can be set. | _none_ | +| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | +| `http2` | Enable, disable or force HTTP/2 usage. Accepted values : `on`, `off`, or `force`. | `off` | +| `http_passwd` | Set HTTP auth password. | _none_ | +| `http_user` | Set HTTP auth user. | _none_ | +| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | +| `logs_attributes_metadata_key` | Specify an `Attributes` key. | `$Attributes` | +| `logs_body_key` | Specify an optional HTTP URI for the target OTel endpoint. | _none_ | +| `logs_body_key_attributes` | If set and it matched a pattern, it includes the remaining fields in the record as attributes. | `false`| +| `logs_instrumentation_scope_metadata_key` | Specify an `InstrumentationScope` key. | `InstrumentationScope` | +| `logs_observed_timestamp_metadata_key` | Specify an `ObservedTimestamp` key. | `$ObservedTimestamp` | +| `logs_max_resources` | Set the maximum number of OTLP log resources per export request (`0` disables the limit). | `0` | +| `logs_max_scopes` | Set the maximum number of OTLP log scopes per resource (`0` disables the limit). | `0` | +| `logs_metadata_key` | Set the key to lookup in the metadata. | `otlp` | +| `logs_resource_metadata_key` | Specify a `Resource` key. | `Resource` | +| `log_response_payload` | Specify if the response payload should be logged or not. | `true` | +| `logs_severity_number_message_key` | Specify a `SeverityNumber` key. | `$severityNumber` | +| `logs_severity_number_metadata_key` | Specify a `SeverityNumber` key. | `$SeverityNumber` | +| `logs_severity_text_message_key` | Specify a `SeverityText` key. | `$SeverityText` | +| `log_severity_text_metadata_key` | Specify a `SeverityText` key. | `$SeverityText` | +| `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | +| `logs_span_id_message_key` | Specify a `SpanId` key. | `$SpanId` | +| `logs_span_id_metadata_key` | Specify a `SpanId` key. | `$SpanId` | +| `logs_timestamp_metadata_key` | Specify a `Timestamp` key. | `$Timestamp` | +| `logs_trace_flags_metadata_key` | Specify a `TraceFlags` key. | `$TraceFlags` | +| `logs_trace_id_message_key` | Specify a `TraceId` key. | `$TraceId` | +| `logs_trace_id_metadata_key` | Specify a `TraceId` key. | `$TraceId` | +| `logs_uri` | Specify an optional HTTP URI for the target OTel endpoint. | `/v1/logs` | +| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | +| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to wildcards. | _none_ | +| `metrics_uri` | Specify an optional HTTP URI for the target OTel endpoint. | `/v1/metrics` | +| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` | +| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | +| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it retries. | `2000` | +| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | +| `net.dns.prefer_ipv4` | Select the primary DNS resolver type (LEGACY or ASYNC). | _none_ | +| `net.dns.prefer_ipv6` | Prioritize IPv6 DNS results when trying to establish a connection. | _none_ | +| `net.io_timeout` | Set maximum time a connection can stay idle while assigned. | `0s` | +| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | +| `net.proxy_env_ignore` | Ignore the environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` when set. | `false`| +| `net.source_address` | Specify network address to bind for data traffic. | _none_ | +| `net.tcp_keepalive` | Enable or disable Keepalive support. | `off` | +| `net.tcp_keepalive_time` | Interval between the last data packet sent and the first TCP keepalive probe. | `-1` | +| `net.tcp_keepalive_interval` | Interval between TCP keepalive probes when no response is received on a `keepidle` probe. | `-1` | +| `net.tcp_keepalive_probes` | Number of unacknowledged probes to consider a connection dead. | `-1` | +| `port` | TCP port of the target HTTP server. | `80` | +| `profiles_uri` | Specify an optional HTTP URI for the profiles OTel endpoint. | `/v1development/profiles` | +| `proxy` | Specify an HTTP Proxy. The expected format of this value is `http://host:port`.| _none_ | +| `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` | +| `tls` | Enable or disable TLS/SSL support. | `off` | +| `tls.ca_file` | Absolute path to CA certificate file. | _none_ | +| `tls.ca_path` | Absolute path to scan for certificate files. | _none_ | +| `tls.cert_file` | Absolute path to Certificate file. | _none_ | +| `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | _none_ | +| `tls.debug` | Set TLS debug level. Accepts `0` (No debug), `1`(Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | +| `tls.key_file` | Absolute path to private Key file. | _none_ | +| `tls.key_passwd` | Optional password for tls.key_file file. | _none_ | +| `tls.max_version` | Specify the maximum version of TLS. | _none_ | +| `tls.min_version` | Specify the minimum version of TLS. | _none_ | +| `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | +| `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ | +| `tls.verify` | Force certificate validation. | `on` | +| `tls.windows.certstore_name` | Sets the `certstore` name on an output (Windows). | _none_ | +| `tls.windows.use_enterprise_store` | Sets whether using enterprise `certstore` or not on an output (Windows). | _none_ | +| `traces_uri` | Specify an optional HTTP URI for the target OTel endpoint. | `/v1/traces` | ## Get started diff --git a/vale-styles/FluentBit/Acronyms.yml b/vale-styles/FluentBit/Acronyms.yml index 6577a763a..7fb044554 100644 --- a/vale-styles/FluentBit/Acronyms.yml +++ b/vale-styles/FluentBit/Acronyms.yml @@ -91,6 +91,7 @@ exceptions: - OEM - OOM - OSS + - OTLP - PATH - PEM - PDF From 81317b99caad25f775be9c81caf672eb74fa8756 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 11:07:48 -0800 Subject: [PATCH 08/20] Vale: reducing suggestions Signed-off-by: Lynette Miles --- pipeline/outputs/cloudwatch.md | 54 +++++++++++++++++----------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/pipeline/outputs/cloudwatch.md b/pipeline/outputs/cloudwatch.md index f6a681756..4990dbee4 100644 --- a/pipeline/outputs/cloudwatch.md +++ b/pipeline/outputs/cloudwatch.md @@ -14,37 +14,37 @@ See [AWS credentials](https://docs.fluentbit.io/manual/administration/aws-creden ## Configuration parameters -| Key | Description | Default | +| Key | Description | Default | |------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------| -| `add_enity` | Add entity to `PutLogEvent` calls. | `false` | -| `alias` | Sets an alias, useful for multiple instances of the same output plugin. | _none_ | -| `auto_create_group` | Automatically create the log group. Allowed values: `true`, `false` (case insensitive). | `false` | +| `add_enity` | Add entity to `PutLogEvent` calls. | `false` | +| `alias` | Sets an alias, use for multiple instances of the same output plugin. | _none_ | +| `auto_create_group` | Automatically create the log group. Allowed values: `true`, `false` (case insensitive). | `false` | | `auto_retry_requests` | Immediately retry failed requests to AWS services once. This option doesn't affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput when there are transient/random networking issues. | `true` | -| `endpoint` | Specify a custom endpoint for the CloudWatch Logs API. | _none_ | -| `external_id` | Specify an external ID for the STS API, can be used with the `role_arn` parameter if your role requires an external ID. | _none_ | -| `extra_user_agent` | This option appends a string to the default user agent. AWS asks that you not manually set this field yourself, it is reserved for use in our vended configurations, for example, EKS Container Insights. | _none_ | -| `log_format` | An optional parameter that can be used to tell CloudWatch the format of the data. A value of `json/emf` enables CloudWatch to extract custom metrics embedded in a `JSON` payload. | _none_ | -| `log_group_class` | Specifies the log storage class for new log groups when `auto_create_group` is set to `true`. You can't modify the storage class of existing log groups. Allowed values: `STANDARD`, `INFREQUENT_ACCESS`. | `STANDARD` | -| `log_group_name` | The name of the CloudWatch log group that you want log records sent to. | _none_ | -| `log_group_template` | OTemplate for CW Log Group name using record accessor syntax. Plugin falls back to the log_group_name configured if needed. | _none_ | +| `endpoint` | Specify a custom endpoint for the CloudWatch Logs API. | _none_ | +| `external_id` | Specify an external ID for the STS API, can be used with the `role_arn` parameter if your role requires an external ID. | _none_ | +| `extra_user_agent` | This option appends a string to the default user agent. AWS asks that you not manually set this field yourself, it's reserved for use in vended configurations, for example, EKS Container Insights.| _none_ | +| `log_format` | An optional parameter that can be used to tell CloudWatch the format of the data. A value of `json/emf` enables CloudWatch to extract custom metrics embedded in a `JSON` payload. | _none_ | +| `log_group_class` | Specifies the log storage class for new log groups when `auto_create_group` is set to `true`. You can't modify the storage class of existing log groups. Allowed values: `STANDARD`, `INFREQUENT_ACCESS`.| `STANDARD` | +| `log_group_name` | The name of the CloudWatch log group that you want log records sent to. | _none_ | +| `log_group_template` | Template for CW Log Group name using record accessor syntax. Plugin falls back to the `log_group_name` configured if needed. | _none_ | | `log_key` | By default, the whole log record will be sent to CloudWatch. If you specify a key name with this option, then only the value of that key will be sent to CloudWatch. For example, if you are using the Fluentd Docker log driver, you can specify `log_key log` and only the log message will be sent to CloudWatch. | _none_ | -| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | +| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | | `log_retention_days` | If set to a number greater than zero, and newly create log group's retention policy is set to this many days. Allowed values: [`1`, `3`, `5`, `7`, `14`, `30`, `60`, `90`, `120`, `150`, `180`, `365`, `400`, `545`, `731`, `1827`, `3653`] | `0` | -| `log_stream_name` | The name of the CloudWatch log stream that you want log records sent to. | _none_ | -| `log_stream_prefix` | Prefix for the log stream name. The tag is appended to the prefix to construct the full log stream name. Not compatible with the `log_stream_name` option. | _none_ | -| `log_stream_template` | Template for CloudWatch Log Stream name using record accessor syntax. Plugin falls back to the `log_stream_name` or `log_stream_prefix` configured if needed. | _none_ | -| `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | -| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | -| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to simple wildcards. | _none_ | -| `metric_dimensions` | Metric dimensions is a list of lists. If you have only one list of dimensions, put the values as a comma seperated string. If you want to put list of lists, use the list as semicolon seperated strings. If your value is `d1,d2;d3`, it will consider it as `[[d1, d2],[d3]]`. | _none_ | -| `metric_namespace` | An optional string representing the CloudWatch namespace for the metrics. See the [Metrics tutorial](#metrics-tutorial) section for a full configuration. | _none_ | -| `profile` | Option to specify an AWS Profile for credentials. | _none_ | -| `region` | The AWS region to send logs to. | _none_ | -| `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` | -| `role_arn` | ARN of an IAM role to assume for cross account access. | _none_ | -| `sts_endpoint` | Specify a custom STS endpoint for the AWS STS API. | _none_ | -| `tls.windows.certstore_name` | Sets the certificate store name on an output (Windows). | _none_ | -| `tls.windows.use_enterprise_store` | Sets whether using enterprise certificate store or not on an output (Windows). | _none_ | +| `log_stream_name` | The name of the CloudWatch log stream that you want log records sent to. | _none_ | +| `log_stream_prefix` | Prefix for the log stream name. The tag is appended to the prefix to construct the full log stream name. Not compatible with the `log_stream_name` option. | _none_ | +| `log_stream_template` | Template for CloudWatch Log Stream name using record accessor syntax. Plugin falls back to the `log_stream_name` or `log_stream_prefix` configured if needed. | _none_ | +| `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | +| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | +| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to wildcards. | _none_ | +| `metric_dimensions` | Metric dimensions is a list of lists. If you have only one list of dimensions, put the values as a comma separated string. If you want to put list of lists, use the list as semicolon separated strings. If your value is `d1,d2;d3`, it will consider it as `[[d1, d2],[d3]]`. | _none_ | +| `metric_namespace` | An optional string representing the CloudWatch namespace for the metrics. See the [Metrics tutorial](#metrics-tutorial) section for a full configuration. | _none_ | +| `profile` | Option to specify an AWS Profile for credentials. | _none_ | +| `region` | The AWS region to send logs to. | _none_ | +| `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` | +| `role_arn` | ARN of an IAM role to assume for cross account access. | _none_ | +| `sts_endpoint` | Specify a custom STS endpoint for the AWS STS API. | _none_ | +| `tls.windows.certstore_name` | Sets the certificate store name on an output (Windows). | _none_ | +| `tls.windows.use_enterprise_store` | Sets whether using enterprise certificate store or not on an output (Windows). | _none_ | ## Get started From 2a4a7315351b5a906a88c667dac14b2da2bf545c Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 11:11:47 -0800 Subject: [PATCH 09/20] Vale: reducing suggestions Signed-off-by: Lynette Miles --- pipeline/outputs/s3.md | 148 ++++++++++++++++++++--------------------- 1 file changed, 74 insertions(+), 74 deletions(-) diff --git a/pipeline/outputs/s3.md b/pipeline/outputs/s3.md index 582d8ed95..f8415c0dc 100644 --- a/pipeline/outputs/s3.md +++ b/pipeline/outputs/s3.md @@ -25,81 +25,81 @@ The [Prometheus success/retry/error metrics values](../../administration/monitor ## Configuration parameters -| Key | Description | Default | -|---------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------| -| `alias` | Sets an alias, useful for multiple instances of the same output plugin. | _none_ | -| `authorization_endpoint_bearer_token` | Authorization endpoint bearer token. | _none_ | -| `authorization_endpoint_password` | Authorization endpoint basic authentication password. | _none_ | -| `authorization_endpoint_url` | Authorization endpoint URL. | _none_ | -| `authorization_endpoint_username` | Authorization endpoint basic authentication username. | _none_ | -| `auto_retry_requests` | Immediately retry failed requests to AWS services once. This option doesn't affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput during transient network issues. | `true` | | _none_ | -| `blob_database_file` | Absolute path to a database file to be used to store blob files contexts. | _none_ | -| `bucket` | S3 Bucket name | _none_ | -| `canned_acl` | [Predefined Canned ACL policy](https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl) for S3 objects. | _none_ | -| `compression` | Compression type for S3 objects. `gzip`, `arrow`, `parquet` and `zstd` are the supported values, `arrow` and `parquet` are only available if Apache Arrow was enabled at compile time. Defaults to no compression. | _none_ | -| `content_type` | A standard MIME type for the S3 object, set as the Content-Type HTTP header. | _none_ | -| `endpoint` | Custom endpoint for the S3 API. Endpoints can contain scheme and port. | _none_ | -| `external_id` | Specify an external ID for the STS API. Can be used with the `role_arn` parameter if your role requires an external ID. | _none_ | -| `file_delivery_attempt_limit` | File delivery attempt limit. | `1` | -| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | -| `json_date_format` | Specify the format of the date. Accepted values: `double`, `epoch`, `epoch_ms`, `iso8601` (2018-05-30T09:39:52.000681Z), `_java_sql_timestamp_` (2018-05-30 09:39:52.000681). | _none_ | -| `json_date_key` | Specify the name of the date key in the output record. To disable the time key, set the value to `false`. | `date` | -| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | -| `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | -| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | -| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to simple wildcards. | _none_ | -| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` | -| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | -| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it is retried. | `2000` | -| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | -| `net.dns.prefer_ipv4` | Select the primary DNS resolver type (LEGACY or ASYNC). | _none_ | -| `net.dns.prefer_ipv6` | Prioritize IPv6 DNS results when trying to establish a connection. | _none_ | -| `net.io_timeout` | Set maximum time a connection can stay idle while assigned. | `0s` | -| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | -| `net.proxy_env_ignore` | Ignore the environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` when set. | `false` | -| `net.source_address` | Specify network address to bind for data traffic. | _none_ | -| `net.tcp_keepalive` | Enable or disable Keepalive support. | `off` | -| `net.tcp_keepalive_time` | Interval between the last data packet sent and the first TCP keepalive probe. | `-1` | -| `net.tcp_keepalive_interval` | Interval between TCP keepalive probes when no response is received on a keepidle probe. | `-1` | -| `net.tcp_keepalive_probes` | Number of unacknowledged probes to consider a connection dead. | `-1` | -| `log_key` | By default, the whole log record will be sent to S3. When specifying a key name with this option, only the value of that key sends to S3. For example, when using Docker you can specify `log_key log` and only the log message sends to S3. | _none_ | -| `part_delivery_attempt_limit` | File part delivery attempt limit. | `1` | -| `part_size` | Size of each part when uploading blob files. | `25M` | -| `port` | TCP port of the target HTTP server. | `80` | -| `preserve_data_ordering` | When an upload request fails, the last received chunk might swap with a later chunk, resulting in data shuffling. This feature prevents shuffling by using a queue logic for uploads. | `true` | -| `profile` | Option to specify an AWS Profile for credentials. | _none_ | -| `region` | The AWS region of your S3 bucket. | `us-east-1` | -| `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` | -| `role_arn` | ARN of an IAM role to assume (for example, for cross account access.) | _none_ | +| Key | Description | Default | +|-----|-------------|----------| +| `alias` | Sets an alias, use for multiple instances of the same output plugin. | _none_ | +| `authorization_endpoint_bearer_token` | Authorization endpoint bearer token. | _none_ | +| `authorization_endpoint_password` | Authorization endpoint basic authentication password.| _none_ | +| `authorization_endpoint_url` | Authorization endpoint URL. | _none_ | +| `authorization_endpoint_username` | Authorization endpoint basic authentication username.| _none_ | +| `auto_retry_requests` | Immediately retry failed requests to AWS services once. This option doesn't affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput during transient network issues. | `true` | +| `blob_database_file` | Absolute path to a database file to be used to store blob files contexts. | _none_ | +| `bucket` | S3 Bucket name | _none_ | +| `canned_acl` | [Predefined Canned ACL policy](https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl) for S3 objects. | _none_ | +| `compression` | Compression type for S3 objects. `gzip`, `arrow`, `parquet` and `zstd` are the supported values, `arrow` and `parquet` are only available if Apache Arrow was enabled at compile time. Defaults to no compression. | _none_ | +| `content_type` | A standard MIME type for the S3 object, set as the Content-Type HTTP header. | _none_ | +| `endpoint` | Custom endpoint for the S3 API. Endpoints can contain scheme and port. | _none_ | +| `external_id` | Specify an external ID for the STS API. Can be used with the `role_arn` parameter if your role requires an external ID. | _none_ | +| `file_delivery_attempt_limit` | File delivery attempt limit. | `1` | +| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | +| `json_date_format` | Specify the format of the date. Accepted values: `double`, `epoch`, `epoch_ms`, `iso8601` (2018-05-30T09:39:52.000681Z), `_java_sql_timestamp_` (2018-05-30 09:39:52.000681). | _none_ | +| `json_date_key` | Specify the name of the date key in the output record. To disable the time key, set the value to `false`. | `date` | +| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | +| `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | +| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | +| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to wildcards. | _none_ | +| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` | +| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | +| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it retries. | `2000` | +| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | +| `net.dns.prefer_ipv4` | Select the primary DNS resolver type (LEGACY or ASYNC). | _none_ | +| `net.dns.prefer_ipv6` | Prioritize IPv6 DNS results when trying to establish a connection. | _none_ | +| `net.io_timeout` | Set maximum time a connection can stay idle while assigned. | `0s` | +| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | +| `net.proxy_env_ignore` | Ignore the environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` when set. | `false` | +| `net.source_address` | Specify network address to bind for data traffic. | _none_ | +| `net.tcp_keepalive` | Enable or disable Keepalive support. | `off` | +| `net.tcp_keepalive_time` | Interval between the last data packet sent and the first TCP keepalive probe. | `-1` | +| `net.tcp_keepalive_interval` | Interval between TCP keepalive probes when no response is received on a `keepidle` probe. | `-1` | +| `net.tcp_keepalive_probes` | Number of unacknowledged probes to consider a connection dead. | `-1` | +| `log_key` | By default, the whole log record will be sent to S3. When specifying a key name with this option, only the value of that key sends to S3. For example, when using Docker you can specify `log_key log` and only the log message sends to S3. | _none_ | +| `part_delivery_attempt_limit` | File part delivery attempt limit. | `1` | +| `part_size` | Size of each part when uploading blob files. | `25M` | +| `port` | TCP port of the target HTTP server. | `80` | +| `preserve_data_ordering` | When an upload request fails, the last received chunk might swap with a later chunk, resulting in data shuffling. This feature prevents shuffling by using a queue logic for uploads. | `true` | +| `profile` | Option to specify an AWS Profile for credentials. | _none_ | +| `region` | The AWS region of your S3 bucket. | `us-east-1` | +| `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` | +| `role_arn` | ARN of an IAM role to assume (for example, for cross account access.) | _none_ | | `s3_key_format` | Format string for keys in S3. This option supports a UUID, strftime time formatters, a syntax for selecting parts of the Fluent log tag using a syntax inspired by the `rewrite_tag` filter. Add `$UUID` in the format string to insert a random string. Add `$INDEX` in the format string to insert an integer that increments each upload. The `$INDEX` value saves in the `store_dir`. Add `$TAG` in the format string to insert the full log tag. Add `$TAG[0]` to insert the first part of the tag in theS3 key. The tag is split into parts using the characters specified with the `s3_key_format_tag_delimiters` option. Add the extension directly after the last piece of the format string to insert a key suffix. To specify a key suffix in `use_put_object` mode, you must specify `$UUID`. See [S3 Key Format](#s3-key-format-and-tag-delimiters). Time in `s3_key` is the timestamp of the first record in the S3 file. | `/fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S` | -| `s3_key_format_tag_delimiters` | A series of characters which will be used to split the tag into `parts` for use with the s3_key_format option. | `.` | -| `send_content_md5` | Send the Content-MD5 header with `PutObject` and UploadPart requests, as is required when Object Lock is enabled. | `false` | -| `static_file_path` | Disables behavior where UUID string appends to the end of the S3 key name when `$UUID` isn't provided in `s3_key_format`. `$UUID`, time formatters, `$TAG`, and other dynamic key formatters all work as expected while this feature is set to true. | `false` | -| `store_dir` | Directory to locally buffer data before sending. Plugin uses the S3 Multipart upload API to send data in chunks of 5 MB at a time. | `/tmp/fluent-bit/s3` | -| `store_dir_limit_size` | S3 plugin has its own buffering system with files in the `store_dir`. Use the `store_dir_limit_size` to limit the amount of data S3 buffers in the `store_dir` to limit disk usage. If the limit is reached, data will be discarded. Default is 0 which means unlimited. | `0` | -| `storage_class` | Specify the [storage class](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#AmazonS3-PutObject-request-header-StorageClass) for S3 objects. If this option isn't specified, objects store with the default `STANDARD` storage class. | _none_ | -| `sts_endpoint` | Custom endpoint for the STS API. | _none_ | -| `tls` | Enable or disable TLS/SSL support. | `off` | -| `tls.ca_file` | Absolute path to CA certificate file. | _none_ | -| `tls.ca_path` | Absolute path to scan for certificate files. | _none_ | -| `tls.cert_file` | Absolute path to Certificate file. | _none_ | -| `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | _none_ | -| `tls.debug` | Set TLS debug level. Accepts `0` (No debug), `1`(Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | -| `tls.key_file` | Absolute path to private Key file. | _none_ | -| `tls.key_passwd` | Optional password for tls.key_file file. | _none_ | -| `tls.max_version` | Specify the maximum version of TLS. | _none_ | -| `tls.min_version` | Specify the minimum version of TLS. | _none_ | -| `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | -| `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ | -| `tls.verify` | Force certificate validation. | `on` | -| `tls.windows.certstore_name` | Sets the certstore name on an output (Windows). | _none_ | -| `tls.windows.use_enterprise_store` | Sets whether using enterprise certstore or not on an output (Windows). | _none_ | -| `total_file_size` | Specify file size in S3. Minimum size is `1M`. With `use_put_object On` the maximum size is `1G`. With multipart uploads, the maximum size is `50G`. | `100000000` | -| `upload_chunk_size` | The size of each part for multipart uploads. Default: 5M, Max: 50M, Min: 5M. | `5242880` | -| `upload_part_freshness_timeout` | Maximum lifespan of an uncommitted file part. | `6D` | -| `upload_parts_timeout` | Timeout to upload parts of a blob file. | `10m` | -| `upload_timeout` | When this amount of time elapses, Fluent Bit uploads and creates a new file in S3. Set to `60m` to upload a new file every hour. | `10m` | -| `use_put_object` | Use the S3 `PutObject` API instead of the multipart upload API. When enabled, the key extension is only available when `$UUID` is specified in `s3_key_format`. If `$UUID` isn't included, a random string appends format string and the key extension can't be customized. | `false` | +| `s3_key_format_tag_delimiters` | A series of characters which will be used to split the tag into `parts` for use with the s3_key_format option. | `.` | +| `send_content_md5` | Send the Content-MD5 header with `PutObject` and UploadPart requests, as is required when Object Lock is enabled. | `false` | +| `static_file_path` | Disables behavior where UUID string appends to the end of the S3 key name when `$UUID` isn't provided in `s3_key_format`. `$UUID`, time formatters, `$TAG`, and other dynamic key formatters all work as expected while this feature is set to true. | `false` | +| `store_dir` | Directory to locally buffer data before sending. Plugin uses the S3 Multipart upload API to send data in chunks of 5 MB at a time. | `/tmp/fluent-bit/s3` | +| `store_dir_limit_size` | S3 plugin has its own buffering system with files in the `store_dir`. Use the `store_dir_limit_size` to limit the amount of data S3 buffers in the `store_dir` to limit disk usage. If the limit is reached, data will be discarded. Default is 0 which means unlimited. | `0` | +| `storage_class` | Specify the [storage class](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#AmazonS3-PutObject-request-header-StorageClass) for S3 objects. If this option isn't specified, objects store with the default `STANDARD` storage class. | _none_ | +| `sts_endpoint` | Custom endpoint for the STS API. | _none_ | +| `tls` | Enable or disable TLS/SSL support. | `off` | +| `tls.ca_file` | Absolute path to CA certificate file. | _none_ | +| `tls.ca_path` | Absolute path to scan for certificate files. | _none_ | +| `tls.cert_file` | Absolute path to Certificate file. | _none_ | +| `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | _none_ | +| `tls.debug` | Set TLS debug level. Accepts `0` (No debug), `1`(Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | +| `tls.key_file` | Absolute path to private Key file. | _none_ | +| `tls.key_passwd` | Optional password for tls.key_file file.| _none_ | +| `tls.max_version` | Specify the maximum version of TLS. | _none_ | +| `tls.min_version` | Specify the minimum version of TLS. | _none_ | +| `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | +| `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ | +| `tls.verify` | Force certificate validation. | `on` | +| `tls.windows.certstore_name` | Sets the `certstore` name on an output (Windows). | _none_ | +| `tls.windows.use_enterprise_store` | Sets whether using enterprise `certstore` or not on an output (Windows). | _none_ | +| `total_file_size` | Specify file size in S3. Minimum size is `1M`. With `use_put_object On` the maximum size is `1G`. With multipart uploads, the maximum size is `50G`.| `100000000` | +| `upload_chunk_size` | The size of each part for multipart uploads. Default: 5M, Max: 50M, Min: 5M. | `5242880` | +| `upload_part_freshness_timeout` | Maximum lifespan of an uncommitted file part. | `6D` | +| `upload_parts_timeout` | Timeout to upload parts of a blob file. | `10m` | +| `upload_timeout` | When this amount of time elapses, Fluent Bit uploads and creates a new file in S3. Set to `60m` to upload a new file every hour. | `10m` | +| `use_put_object` | Use the S3 `PutObject` API instead of the multipart upload API. When enabled, the key extension is only available when `$UUID` is specified in `s3_key_format`. If `$UUID` isn't included, a random string appends format string and the key extension can't be customized. | `false` | ## TLS / SSL From c51255adfd706bd36a8ef96c8e9548cfbb75c6d1 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 11:14:24 -0800 Subject: [PATCH 10/20] Vale: reducing suggestions Signed-off-by: Lynette Miles --- pipeline/outputs/parseable.md | 6 +++--- vale-styles/FluentBit/Spelling-exceptions.txt | 1 + 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/pipeline/outputs/parseable.md b/pipeline/outputs/parseable.md index a71a15a08..7fe636af0 100644 --- a/pipeline/outputs/parseable.md +++ b/pipeline/outputs/parseable.md @@ -4,7 +4,7 @@ description: Send logs, metrics, and traces to Parseable # Parseable -Stream logs, metrics, and traces to [Parseable](https://www.parseable.com) by utilizing the [OpenTelemetry plugin](opentelemetry.md) to send telemetry data to Parseable's OpenTelemetry-compatible Ingestor endpoints. +Stream logs, metrics, and traces to [Parseable](https://www.parseable.com) by utilizing the [OpenTelemetry plugin](opentelemetry.md) to send telemetry data to the Parseable OpenTelemetry-compatible Ingestor endpoints. ## Configuration parameters @@ -136,12 +136,12 @@ pipeline: {% endtab %} {% endtabs %} -## Stream Configuration +## Stream configuration Parseable uses streams to organize your telemetry data. The `X-P-Stream` header specifies which stream the data should be sent to: - `otellogs`: Stream for log data -- `otelmetrics`: Stream for metrics data +- `otelmetrics`: Stream for metrics data - `oteltraces`: Stream for trace data The `X-P-Log-Source` header helps identify the source of the telemetry data for better organization and filtering. The header must be set to `otel-logs`, `otel-metrics`, or `otel-traces` based on the telemetry type. diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt index 63e49815a..c587f2d49 100644 --- a/vale-styles/FluentBit/Spelling-exceptions.txt +++ b/vale-styles/FluentBit/Spelling-exceptions.txt @@ -100,6 +100,7 @@ Homebrew hostname Hostname Ingester +Ingestor inotify Keepalive Istio From 45334e5c44e7f110da9d249c72bd45bf066c68bf Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 11:16:35 -0800 Subject: [PATCH 11/20] Vale: reducing suggestions Signed-off-by: Lynette Miles --- vale-styles/FluentBit/Acronyms.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/vale-styles/FluentBit/Acronyms.yml b/vale-styles/FluentBit/Acronyms.yml index 7fb044554..2e6bf5026 100644 --- a/vale-styles/FluentBit/Acronyms.yml +++ b/vale-styles/FluentBit/Acronyms.yml @@ -15,9 +15,11 @@ exceptions: - API - ARN - ASC + - ASCII - ASP - ASYNC - AWS + - BMP - BSD - CIDR - CLI From dbd3980970f552969e88df143b517f508e36fa2c Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 11:28:41 -0800 Subject: [PATCH 12/20] Vale: reducing suggestions Signed-off-by: Lynette Miles --- pipeline/inputs/gpu-metrics.md | 32 +++++++++---------- vale-styles/FluentBit/Acronyms.yml | 2 ++ vale-styles/FluentBit/Headings.yml | 1 + vale-styles/FluentBit/Spelling-exceptions.txt | 2 ++ 4 files changed, 20 insertions(+), 17 deletions(-) diff --git a/pipeline/inputs/gpu-metrics.md b/pipeline/inputs/gpu-metrics.md index ed2ba6810..98451c6d8 100644 --- a/pipeline/inputs/gpu-metrics.md +++ b/pipeline/inputs/gpu-metrics.md @@ -1,23 +1,23 @@ # GPU metrics -The **gpu_metrics** input plugin collects graphics processing unit (GPU) performance metrics from graphics cards on Linux systems. It provides real-time monitoring of GPU utilization, memory usage (VRAM), clock frequencies, power consumption, temperature, and fan speeds. +The _gpu_metrics_ input plugin collects graphics processing unit (GPU) performance metrics from graphics cards on Linux systems. It provides real-time monitoring of GPU utilization, memory usage (VRAM), clock frequencies, power consumption, temperature, and fan speeds. -The plugin reads metrics directly from the Linux sysfs filesystem (`/sys/class/drm/`) without requiring external tools or libraries. Currently, **only AMD GPUs are supported** through the amdgpu kernel driver. NVIDIA and Intel GPUs aren't supported at this time. +The plugin reads metrics directly from the Linux `sysfs` filesystem (`/sys/class/drm/`) without requiring external tools or libraries. Only AMD GPUs are supported through the `amdgpu` kernel driver. NVIDIA and Intel GPUs aren't supported at this time. ## Metrics collected The plugin collects the following metrics for each detected GPU: -| Key | Description | -|---------------------------|------------------------------------------------------------------------------------------------------------------------------------------| -| `gpu_utilization_percent` | GPU core utilization as a percentage (0-100). Indicates how busy the GPU is processing workloads. | -| `gpu_memory_used_bytes` | Amount of video RAM (VRAM) currently in use, measured in bytes. | -| `gpu_memory_total_bytes` | Total video RAM (VRAM) capacity available on the GPU, measured in bytes. | +| Key | Description | +|---------------------------|------------------| +| `gpu_utilization_percent` | GPU core utilization as a percentage (0-100). Indicates how busy the GPU is processing workloads. | +| `gpu_memory_used_bytes` | Amount of video RAM (VRAM) currently in use, measured in bytes. | +| `gpu_memory_total_bytes` | Total video RAM (VRAM) capacity available on the GPU, measured in bytes. | | `gpu_clock_mhz` | Current GPU clock frequency in MHz. This metric has multiple instances with different type labels (see [Clock metrics](#clock-metrics)). | -| `gpu_power_watts` | Current power consumption in watts. Can be disabled with enable_power false. | -| `gpu_temperature_celsius` | GPU die temperature in degrees Celsius. Can be disabled with enable_temperature false. | -| `gpu_fan_speed_rpm` | Fan rotation speed in revolutions per minute (RPM). | -| `gpu_fan_pwm_percent` | Fan PWM duty cycle as a percentage (0-100). Indicates fan intensity. | +| `gpu_power_watts` | Current power consumption in watts. Can be disabled with `enable_power` set to `false`.| +| `gpu_temperature_celsius` | GPU die temperature in degrees Celsius. Can be disabled with `enable_temperature`set to `false`. | +| `gpu_fan_speed_rpm` | Fan rotation speed in Revolutions per Minute (RPM). | +| `gpu_fan_pwm_percent` | Fan PWM duty cycle as a percentage (0-100). Indicates fan intensity. | ### Clock metrics @@ -33,18 +33,18 @@ The `gpu_clock_mhz` metric is reported separately for three clock domains: The plugin supports the following configuration parameters: -| Key | Description | Default | +| Key | Description | Default | |----------------------|-------------------------------------------------------------------------------------------------------------------------|-----------| | `scrape_interval` | Interval in seconds between metric collection cycles. | `5` | -| `path_sysfs` | Path to the sysfs root directory. Typically used for testing or non-standard systems. | `/sys` | +| `path_sysfs` | Path to the `sysfs` root directory. Typically used for testing or non-standard systems. | `/sys` | | `cards_include` | Pattern specifying which GPU cards to monitor. Supports wildcards (*), ranges (0-3), and comma-separated lists (0,2,4). | `*` | -| `cards_exclude` | Pattern specifying which GPU cards to exclude from monitoring. Uses the same syntax as cards_include. | _none_ | +| `cards_exclude` | Pattern specifying which GPU cards to exclude from monitoring. Uses the same syntax as `cards_include`. | _none_ | | `enable_power` | Enable collection of power consumption metrics (`gpu_power_watts`). | `true` | | `enable_temperature` | Enable collection of temperature metrics (`gpu_temperature_celsius`). | `true` | ## GPU detection -The GPU metrics plugin will automatically scan for supported **AMD GPUs** that are using the `amdgpu` kernel driver. GPUs using legacy drivers will be ignored. +The GPU metrics plugin will automatically scan for any supported AMD GPU are using the `amdgpu` kernel driver. A GPU using legacy drivers will be ignored. To check if your AMD GPU will be detected run: @@ -153,5 +153,3 @@ pipeline: {% endtab %} {% endtabs %} - - diff --git a/vale-styles/FluentBit/Acronyms.yml b/vale-styles/FluentBit/Acronyms.yml index 2e6bf5026..f2f90b116 100644 --- a/vale-styles/FluentBit/Acronyms.yml +++ b/vale-styles/FluentBit/Acronyms.yml @@ -100,6 +100,7 @@ exceptions: - PHP - POSIX - POST + - PWM - RAM - REPL - REST @@ -139,6 +140,7 @@ exceptions: - UTF - UUID - VPC + - VRAM - WASI - WMI - XML diff --git a/vale-styles/FluentBit/Headings.yml b/vale-styles/FluentBit/Headings.yml index 2f48456d5..ecae99e8b 100644 --- a/vale-styles/FluentBit/Headings.yml +++ b/vale-styles/FluentBit/Headings.yml @@ -74,6 +74,7 @@ exceptions: - Google Cloud BigQuery - Google Cloud Platform - GPG + - GPU - Grafana - Grafana Cloud - Graylog Extended Log Format diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt index c587f2d49..e7ad803d2 100644 --- a/vale-styles/FluentBit/Spelling-exceptions.txt +++ b/vale-styles/FluentBit/Spelling-exceptions.txt @@ -86,6 +86,7 @@ github glibc Golang golib +GPUs Grafana Graphite Graylog @@ -152,6 +153,7 @@ Parquet Parseable performant persistable +pluggable Podman popen Postgres From 28406cfa09399be94f26fb4a338c4cd381acdca6 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 11:38:25 -0800 Subject: [PATCH 13/20] Vale: reducing suggestions Signed-off-by: Lynette Miles --- installation/downloads/source/build-and-install.md | 14 +++++++------- vale-styles/FluentBit/Acronyms.yml | 4 ++++ vale-styles/FluentBit/Headings.yml | 2 ++ vale-styles/FluentBit/Spelling-exceptions.txt | 3 +++ 4 files changed, 16 insertions(+), 7 deletions(-) diff --git a/installation/downloads/source/build-and-install.md b/installation/downloads/source/build-and-install.md index 602be209a..392b03780 100644 --- a/installation/downloads/source/build-and-install.md +++ b/installation/downloads/source/build-and-install.md @@ -95,7 +95,7 @@ Fluent Bit provides configurable options to CMake that can be enabled or disable | `FLB_ARROW` | Build with Apache Arrow support | `No` | | `FLB_AVRO_ENCODER` | Build with Avro encoding support | `No` | | `FLB_AWS` | Enable AWS support | `Yes` | -| `FLB_AWS_ERROR_REPORTER` | Build with aws error reporting support | `No` | +| `FLB_AWS_ERROR_REPORTER` | Build with AWS error reporting support | `No` | | `FLB_BENCHMARKS` | Enable benchmarks | `No` | | `FLB_BINARY` | Build executable | `Yes` | | `FLB_CHUNK_TRACE` | Enable chunk traces | `Yes` | @@ -138,7 +138,7 @@ Fluent Bit provides configurable options to CMake that can be enabled or disable | Option | Description | Default | |:--------------------------|:-----------------------------|:--------| -| `FLB_BACKTRACE` | Enable stacktrace support | `Yes` | +| `FLB_BACKTRACE` | Enable stack trace support | `Yes` | | `FLB_DEBUG` | Build with debug mode (`-g`) | `No` | | `FLB_SMALL` | Optimize for small size | `No` | | `FLB_TESTS_INTERNAL` | Enable internal tests | `No` | @@ -157,7 +157,7 @@ Fluent Bit provides configurable options to CMake that can be enabled or disable ### Input plugins -Input plugins gather information from a specific source type like network interfaces, some built-in metrics, or through a specific input device. +Input plugins gather information from a specific source type like network interfaces, some built-in metrics, or through a specific input device. The following input plugins are available: @@ -199,7 +199,7 @@ The following input plugins are available: | [`FLB_IN_RANDOM`](../../../pipeline/inputs/random.md) | Enable Random input plugin | `On` | | [`FLB_IN_SERIAL`](../../../pipeline/inputs/serial-interface.md) | Enable Serial input plugin | `On` | | [`FLB_IN_SPLUNK`](../../../pipeline/inputs/splunk.md) | Enable Serial input plugin | `On` | -| [`FLB_IN_STATSD`](../../../pipeline/inputs/statsd.md) | Enable Statsd input plugin | `On` | +| [`FLB_IN_STATSD`](../../../pipeline/inputs/statsd.md) | Enable StatsD input plugin | `On` | | [`FLB_IN_STDIN`](../../../pipeline/inputs/standard-input.md) | Enable Standard input plugin | `On` | | [`FLB_IN_SYSLOG`](../../../pipeline/inputs/syslog.md) | Enable Syslog input plugin | `On` | | [`FLB_IN_SYSTEMD`](../../../pipeline/inputs/systemd.md) | Enable Systemd input plugin | `On` | @@ -230,7 +230,7 @@ The following table describes the processors available: ### Filter plugins -Filter plugins let you modify, enrich or drop records. +Filter plugins let you modify, enrich or drop records. The following table describes the filters available on this version: @@ -246,7 +246,7 @@ The following table describes the filters available on this version: | [`FLB_FILTER_LOG_TO_METRICS`](../../../pipeline/filters/log_to_metrics.md) | Enable Log derived metrics filter | `On` | | [`FLB_FILTER_LUA`](../../../pipeline/filters/lua.md) | Enable Lua scripting filter | `On` | | [`FLB_FILTER_MODIFY`](../../../pipeline/filters/modify.md) | Enable Modify filter | `On` | -| [`FLB_FILTER_MULTILINE`](../../../pipeline/filters/multiline-stacktrace.md) | Enable Multiline stacktrace filter | `On` | +| [`FLB_FILTER_MULTILINE`](../../../pipeline/filters/multiline-stacktrace.md) | Enable Multiline stack trace filter | `On` | | [`FLB_FILTER_NEST`](../../../pipeline/filters/nest.md) | Enable Nest filter | `On` | | [`FLB_FILTER_NIGHTFALL`](../../../pipeline/filters/nightfall.md) | Enable Nightfall filter | `On` | | [`FLB_FILTER_PARSER`](../../../pipeline/filters/parser.md) | Enable Parser filter | `On` | @@ -261,7 +261,7 @@ The following table describes the filters available on this version: ### Output plugins -Output plugins let you flush the information to some external interface, service, or terminal. +Output plugins let you flush the information to some external interface, service, or terminal. The following table describes the output plugins available: diff --git a/vale-styles/FluentBit/Acronyms.yml b/vale-styles/FluentBit/Acronyms.yml index f2f90b116..b5f87d425 100644 --- a/vale-styles/FluentBit/Acronyms.yml +++ b/vale-styles/FluentBit/Acronyms.yml @@ -9,6 +9,8 @@ second: '(?:\b[A-Z][a-z]+ )+\(([A-Z]{3,5})\)' # ... with the exception of these: exceptions: - AAD + - ABI + - APT - ACL - AMD - ANSI @@ -102,6 +104,7 @@ exceptions: - POST - PWM - RAM + - RBAC - REPL - REST - RHEL @@ -142,6 +145,7 @@ exceptions: - VPC - VRAM - WASI + - WASM - WMI - XML - XSS diff --git a/vale-styles/FluentBit/Headings.yml b/vale-styles/FluentBit/Headings.yml index ecae99e8b..127df3273 100644 --- a/vale-styles/FluentBit/Headings.yml +++ b/vale-styles/FluentBit/Headings.yml @@ -50,6 +50,7 @@ exceptions: - Datadog - DNS - Docker + - Docker Compose - Docker Hub - DogStatsD - DOS @@ -60,6 +61,7 @@ exceptions: - Elastic Cloud - Emmet - EventBridge + - FAQ - FlowCounter - Fluent Bit - Fluentd diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt index e7ad803d2..03c1abdf4 100644 --- a/vale-styles/FluentBit/Spelling-exceptions.txt +++ b/vale-styles/FluentBit/Spelling-exceptions.txt @@ -2,6 +2,7 @@ accessor accessors allowlist Ansible +Apache Skywalking API APIs Appname @@ -198,6 +199,7 @@ sharding SignalFx Signup Sigstore +Skywalking sparkline sparklines Sparklines @@ -242,6 +244,7 @@ upsert upserts untrusted UUIDs +Valgrind Vectra Vercel VictoriaMetrics From 4d1a218bbd600f4450e3508391e024a10361ecc1 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 11:40:00 -0800 Subject: [PATCH 14/20] Vale: reducing suggestions Signed-off-by: Lynette Miles --- administration/configuring-fluent-bit/yaml/plugins-section.md | 2 +- pipeline/inputs/prometheus-textfile.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/administration/configuring-fluent-bit/yaml/plugins-section.md b/administration/configuring-fluent-bit/yaml/plugins-section.md index c446ada1d..68fbc7271 100644 --- a/administration/configuring-fluent-bit/yaml/plugins-section.md +++ b/administration/configuring-fluent-bit/yaml/plugins-section.md @@ -1,6 +1,6 @@ # Plugins -Fluent Bit comes with a variety of built-in plugins, and also supports loading external plugins at runtime. This feature is especially useful for loading Go or WebAssembly (Wasm) plugins that are built as shared object files (`.so`). Fluent Bit YAML configuration provides the following ways to load these external plugins: +Fluent Bit comes with a variety of built-in plugins, and also supports loading external plugins at runtime. Use this feature for loading Go or WebAssembly (Wasm) plugins that are built as shared object files (`.so`). Fluent Bit YAML configuration provides the following ways to load these external plugins: ## Inline YAML diff --git a/pipeline/inputs/prometheus-textfile.md b/pipeline/inputs/prometheus-textfile.md index 507e38086..82c6aa579 100644 --- a/pipeline/inputs/prometheus-textfile.md +++ b/pipeline/inputs/prometheus-textfile.md @@ -6,7 +6,7 @@ The _Prometheus text file_ input plugin allows Fluent Bit to read metrics from P | Key | Description | Default | |-------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------| -| `alias` | Sets an alias, useful for multiple instances of the same input plugin. If no alias is specified, a default name will be assigned using the plugin name followed by a dot and a sequence number. | _none_ | +| `alias` | Sets an alias, use for multiple instances of the same input plugin. If no alias is specified, a default name will be assigned using the plugin name followed by a dot and a sequence number. | _none_ | | `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | | `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | | `mem_buf_limit` | Set a memory buffer limit for the input plugin. If the limit is reached, the plugin will pause until the buffer is drained. The value is in bytes. If set to 0, the buffer limit is disabled. | `0` | From 93f6742d8d927235c75ad6132fcb62f2bc087c66 Mon Sep 17 00:00:00 2001 From: Lynette Miles <6818907+esmerel@users.noreply.github.com> Date: Mon, 10 Nov 2025 12:06:44 -0800 Subject: [PATCH 15/20] Apply suggestions from code review Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com> --- pipeline/inputs/gpu-metrics.md | 2 +- pipeline/outputs/opentelemetry.md | 4 ++-- pipeline/outputs/s3.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/pipeline/inputs/gpu-metrics.md b/pipeline/inputs/gpu-metrics.md index 98451c6d8..eddba144f 100644 --- a/pipeline/inputs/gpu-metrics.md +++ b/pipeline/inputs/gpu-metrics.md @@ -15,7 +15,7 @@ The plugin collects the following metrics for each detected GPU: | `gpu_memory_total_bytes` | Total video RAM (VRAM) capacity available on the GPU, measured in bytes. | | `gpu_clock_mhz` | Current GPU clock frequency in MHz. This metric has multiple instances with different type labels (see [Clock metrics](#clock-metrics)). | | `gpu_power_watts` | Current power consumption in watts. Can be disabled with `enable_power` set to `false`.| -| `gpu_temperature_celsius` | GPU die temperature in degrees Celsius. Can be disabled with `enable_temperature`set to `false`. | +| `gpu_temperature_celsius` | GPU die temperature in degrees Celsius. Can be disabled with `enable_temperature` set to `false`. | | `gpu_fan_speed_rpm` | Fan rotation speed in Revolutions per Minute (RPM). | | `gpu_fan_pwm_percent` | Fan PWM duty cycle as a percentage (0-100). Indicates fan intensity. | diff --git a/pipeline/outputs/opentelemetry.md b/pipeline/outputs/opentelemetry.md index 918fa5457..e4d538df4 100644 --- a/pipeline/outputs/opentelemetry.md +++ b/pipeline/outputs/opentelemetry.md @@ -19,7 +19,7 @@ Only HTTP endpoints are supported. | `grpc` | Enable, disable or force gRPC usage. Accepted values : `on`, `off`, `auto`. | `off` | | `batch_size` | Set the maximum number of log records to be flushed at a time. | `1000` | | `compress` | Set payload compression mechanism. Options available are `gzip` and `zstd`. | _none_ | -| `grpc_logs_uri` | Specify an optional gRPCß URI for the target OTel endpoint. | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` | +| `grpc_logs_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` | | `grpc_metrics_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export` | | `grpc_profiles_uri` | Specify an optional gRPC URI for profiles OTel endpoint. | `/opentelemetry.proto.collector.profiles.v1experimental.ProfilesService/Export` | | `grpc_traces_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.trace.v1.TraceService/Export` | @@ -36,7 +36,7 @@ Only HTTP endpoints are supported. | `logs_observed_timestamp_metadata_key` | Specify an `ObservedTimestamp` key. | `$ObservedTimestamp` | | `logs_max_resources` | Set the maximum number of OTLP log resources per export request (`0` disables the limit). | `0` | | `logs_max_scopes` | Set the maximum number of OTLP log scopes per resource (`0` disables the limit). | `0` | -| `logs_metadata_key` | Set the key to lookup in the metadata. | `otlp` | +| `logs_metadata_key` | Set the key to look up in the metadata. | `otlp` | | `logs_resource_metadata_key` | Specify a `Resource` key. | `Resource` | | `log_response_payload` | Specify if the response payload should be logged or not. | `true` | | `logs_severity_number_message_key` | Specify a `SeverityNumber` key. | `$severityNumber` | diff --git a/pipeline/outputs/s3.md b/pipeline/outputs/s3.md index f8415c0dc..4e83d8344 100644 --- a/pipeline/outputs/s3.md +++ b/pipeline/outputs/s3.md @@ -71,7 +71,7 @@ The [Prometheus success/retry/error metrics values](../../administration/monitor | `region` | The AWS region of your S3 bucket. | `us-east-1` | | `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` | | `role_arn` | ARN of an IAM role to assume (for example, for cross account access.) | _none_ | -| `s3_key_format` | Format string for keys in S3. This option supports a UUID, strftime time formatters, a syntax for selecting parts of the Fluent log tag using a syntax inspired by the `rewrite_tag` filter. Add `$UUID` in the format string to insert a random string. Add `$INDEX` in the format string to insert an integer that increments each upload. The `$INDEX` value saves in the `store_dir`. Add `$TAG` in the format string to insert the full log tag. Add `$TAG[0]` to insert the first part of the tag in theS3 key. The tag is split into parts using the characters specified with the `s3_key_format_tag_delimiters` option. Add the extension directly after the last piece of the format string to insert a key suffix. To specify a key suffix in `use_put_object` mode, you must specify `$UUID`. See [S3 Key Format](#s3-key-format-and-tag-delimiters). Time in `s3_key` is the timestamp of the first record in the S3 file. | `/fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S` | +| `s3_key_format` | Format string for keys in S3. This option supports a UUID, strftime time formatters, a syntax for selecting parts of the Fluent log tag using a syntax inspired by the `rewrite_tag` filter. Add `$UUID` in the format string to insert a random string. Add `$INDEX` in the format string to insert an integer that increments each upload. The `$INDEX` value saves in the `store_dir`. Add `$TAG` in the format string to insert the full log tag. Add `$TAG[0]` to insert the first part of the tag in the S3 key. The tag is split into parts using the characters specified with the `s3_key_format_tag_delimiters` option. Add the extension directly after the last piece of the format string to insert a key suffix. To specify a key suffix in `use_put_object` mode, you must specify `$UUID`. See [S3 Key Format](#s3-key-format-and-tag-delimiters). Time in `s3_key` is the timestamp of the first record in the S3 file. | `/fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S` | | `s3_key_format_tag_delimiters` | A series of characters which will be used to split the tag into `parts` for use with the s3_key_format option. | `.` | | `send_content_md5` | Send the Content-MD5 header with `PutObject` and UploadPart requests, as is required when Object Lock is enabled. | `false` | | `static_file_path` | Disables behavior where UUID string appends to the end of the S3 key name when `$UUID` isn't provided in `s3_key_format`. `$UUID`, time formatters, `$TAG`, and other dynamic key formatters all work as expected while this feature is set to true. | `false` | From 5cbf0ae2a9f57ac65576a0ced4af756b06b21e54 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 12:07:24 -0800 Subject: [PATCH 16/20] Missed a few updates Signed-off-by: Lynette Miles --- installation/downloads/docker.md | 5 +++++ vale-styles/FluentBit/Units.yml | 2 ++ 2 files changed, 7 insertions(+) diff --git a/installation/downloads/docker.md b/installation/downloads/docker.md index 01025b7c1..7900d6b67 100644 --- a/installation/downloads/docker.md +++ b/installation/downloads/docker.md @@ -269,8 +269,13 @@ Alpine Linux uses Musl C library instead of Glibc. Musl isn't fully compatible w ### Why use distroless containers? The reasons for using distroless are well covered in + + + [Why should I use Distroless images?](https://github.com/GoogleContainerTools/distroless#why-should-i-use-distroless-images). + + - Include only what you need, reduce the attack surface available. - Reduces size and improves performance. - Reduces false positives on scans (and reduces resources required for scanning). diff --git a/vale-styles/FluentBit/Units.yml b/vale-styles/FluentBit/Units.yml index be5ac6589..46c5d55fb 100644 --- a/vale-styles/FluentBit/Units.yml +++ b/vale-styles/FluentBit/Units.yml @@ -9,3 +9,5 @@ tokens: exceptions: - k3s + - k8s + - K8s From 9e4c8bcf152b3add158287f6ba99046f4b777fb3 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Mon, 10 Nov 2025 12:10:41 -0800 Subject: [PATCH 17/20] Vale: correcting order of entries for acronyms Signed-off-by: Lynette Miles --- vale-styles/FluentBit/Acronyms.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/vale-styles/FluentBit/Acronyms.yml b/vale-styles/FluentBit/Acronyms.yml index b5f87d425..40f37b2f1 100644 --- a/vale-styles/FluentBit/Acronyms.yml +++ b/vale-styles/FluentBit/Acronyms.yml @@ -10,11 +10,11 @@ second: '(?:\b[A-Z][a-z]+ )+\(([A-Z]{3,5})\)' exceptions: - AAD - ABI - - APT - ACL - AMD - ANSI - API + - APT - ARN - ASC - ASCII @@ -36,8 +36,8 @@ exceptions: - DEBUG - DESC - DOM - - DOS - DNS + - DOS - DPI - DPPS - DST From a84843dc7bcceee362616f8ccd9c2657d551a798 Mon Sep 17 00:00:00 2001 From: Lynette Miles <6818907+esmerel@users.noreply.github.com> Date: Mon, 10 Nov 2025 12:13:01 -0800 Subject: [PATCH 18/20] Update pipeline/inputs/gpu-metrics.md Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com> --- pipeline/inputs/gpu-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/inputs/gpu-metrics.md b/pipeline/inputs/gpu-metrics.md index eddba144f..84df84d5b 100644 --- a/pipeline/inputs/gpu-metrics.md +++ b/pipeline/inputs/gpu-metrics.md @@ -44,7 +44,7 @@ The plugin supports the following configuration parameters: ## GPU detection -The GPU metrics plugin will automatically scan for any supported AMD GPU are using the `amdgpu` kernel driver. A GPU using legacy drivers will be ignored. +The GPU metrics plugin will automatically scan for any supported AMD GPU using the `amdgpu` kernel driver. A GPU using legacy drivers will be ignored. To check if your AMD GPU will be detected run: From d28da02ef7e60bbbcea719fe9b827941c17bb309 Mon Sep 17 00:00:00 2001 From: Lynette Miles <6818907+esmerel@users.noreply.github.com> Date: Mon, 10 Nov 2025 12:13:33 -0800 Subject: [PATCH 19/20] Update pipeline/outputs/opentelemetry.md Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com> --- pipeline/outputs/opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/outputs/opentelemetry.md b/pipeline/outputs/opentelemetry.md index e4d538df4..b5b17745f 100644 --- a/pipeline/outputs/opentelemetry.md +++ b/pipeline/outputs/opentelemetry.md @@ -79,7 +79,7 @@ Only HTTP endpoints are supported. | `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | _none_ | | `tls.debug` | Set TLS debug level. Accepts `0` (No debug), `1`(Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | | `tls.key_file` | Absolute path to private Key file. | _none_ | -| `tls.key_passwd` | Optional password for tls.key_file file. | _none_ | +| `tls.key_passwd` | Optional password for tls.key_file. | _none_ | | `tls.max_version` | Specify the maximum version of TLS. | _none_ | | `tls.min_version` | Specify the minimum version of TLS. | _none_ | | `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | From be76e117e977e5b0fdb14a4727d582c376238f04 Mon Sep 17 00:00:00 2001 From: Lynette Miles <6818907+esmerel@users.noreply.github.com> Date: Mon, 10 Nov 2025 13:27:20 -0800 Subject: [PATCH 20/20] Apply suggestions from code review Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com> --- pipeline/inputs/gpu-metrics.md | 6 +++--- pipeline/inputs/prometheus-textfile.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/pipeline/inputs/gpu-metrics.md b/pipeline/inputs/gpu-metrics.md index 84df84d5b..92402d3b0 100644 --- a/pipeline/inputs/gpu-metrics.md +++ b/pipeline/inputs/gpu-metrics.md @@ -2,7 +2,7 @@ The _gpu_metrics_ input plugin collects graphics processing unit (GPU) performance metrics from graphics cards on Linux systems. It provides real-time monitoring of GPU utilization, memory usage (VRAM), clock frequencies, power consumption, temperature, and fan speeds. -The plugin reads metrics directly from the Linux `sysfs` filesystem (`/sys/class/drm/`) without requiring external tools or libraries. Only AMD GPUs are supported through the `amdgpu` kernel driver. NVIDIA and Intel GPUs aren't supported at this time. +The plugin reads metrics directly from the Linux `sysfs` filesystem (`/sys/class/drm/`) without requiring external tools or libraries. Only AMD GPUs are supported through the `amdgpu` kernel driver. NVIDIA and Intel GPUs aren't supported. ## Metrics collected @@ -10,7 +10,7 @@ The plugin collects the following metrics for each detected GPU: | Key | Description | |---------------------------|------------------| -| `gpu_utilization_percent` | GPU core utilization as a percentage (0-100). Indicates how busy the GPU is processing workloads. | +| `gpu_utilization_percent` | GPU core utilization as a percentage (`0` to `100`). Indicates how busy the GPU is when processing workloads. | | `gpu_memory_used_bytes` | Amount of video RAM (VRAM) currently in use, measured in bytes. | | `gpu_memory_total_bytes` | Total video RAM (VRAM) capacity available on the GPU, measured in bytes. | | `gpu_clock_mhz` | Current GPU clock frequency in MHz. This metric has multiple instances with different type labels (see [Clock metrics](#clock-metrics)). | @@ -44,7 +44,7 @@ The plugin supports the following configuration parameters: ## GPU detection -The GPU metrics plugin will automatically scan for any supported AMD GPU using the `amdgpu` kernel driver. A GPU using legacy drivers will be ignored. +The GPU metrics plugin scans for any supported AMD GPU using the `amdgpu` kernel driver. Any GPU using legacy drivers is ignored. To check if your AMD GPU will be detected run: diff --git a/pipeline/inputs/prometheus-textfile.md b/pipeline/inputs/prometheus-textfile.md index 82c6aa579..4808686f1 100644 --- a/pipeline/inputs/prometheus-textfile.md +++ b/pipeline/inputs/prometheus-textfile.md @@ -6,7 +6,7 @@ The _Prometheus text file_ input plugin allows Fluent Bit to read metrics from P | Key | Description | Default | |-------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------| -| `alias` | Sets an alias, use for multiple instances of the same input plugin. If no alias is specified, a default name will be assigned using the plugin name followed by a dot and a sequence number. | _none_ | +| `alias` | Sets an alias. Use for multiple instances of the same input plugin. If no alias is specified, a default name is assigned using the plugin name followed by a dot and a sequence number. | _none_ | | `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | | `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | | `mem_buf_limit` | Set a memory buffer limit for the input plugin. If the limit is reached, the plugin will pause until the buffer is drained. The value is in bytes. If set to 0, the buffer limit is disabled. | `0` |