From 63cb8109dc3d378bc9fbfdc994784e6f2951ccdb Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Wed, 15 May 2024 15:14:10 +0900 Subject: [PATCH 1/9] processor_labels: Add a documentation for processor_labels Signed-off-by: Hiroshi Hatake --- SUMMARY.md | 1 + installation/sources/build-and-install.md | 1 + pipeline/processors/labels.md | 104 ++++++++++++++++++++++ 3 files changed, 106 insertions(+) create mode 100644 pipeline/processors/labels.md diff --git a/SUMMARY.md b/SUMMARY.md index 11c377b7f..fd5f9f56f 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -130,6 +130,7 @@ * [Logfmt](pipeline/parsers/logfmt.md) * [Decoders](pipeline/parsers/decoders.md) * [Processors](pipeline/processors/README.md) + * [Labels](pipeline/processors/labels.md) * [Content Modifier](pipeline/processors/content-modifier.md) * [Metrics Selector](pipeline/processors/metrics-selector.md) * [SQL](pipeline/processors/sql.md) diff --git a/installation/sources/build-and-install.md b/installation/sources/build-and-install.md index 568130cb4..99674abf8 100644 --- a/installation/sources/build-and-install.md +++ b/installation/sources/build-and-install.md @@ -220,3 +220,4 @@ The following table describes the processors available on this version: | option | description | default | | :--- | :--- | :--- | | [FLB\_PROCESSOR\_METRICS\_SELECTOR](../../pipeline/processors/metrics-selector.md) | Enable metrics selector processor | On | +| [FLB\_PROCESSOR\_LABELS](../../pipeline/processors/labels.md) | Enable metrics label manipulation processor | On | diff --git a/pipeline/processors/labels.md b/pipeline/processors/labels.md new file mode 100644 index 000000000..0b84b0b82 --- /dev/null +++ b/pipeline/processors/labels.md @@ -0,0 +1,104 @@ +# Labels + +The **labels** processor allows you to manipulate the labels of Metrics. + +Similar to the functionality exposed by filters, this processor presents a enriching/modifying mechanism to perform such operations for labels manipulation. +The most significant difference is that processors perform better than filters, and when chaining them, there are no encoding/decoding performance penalties. + +Note that processors and this specific component can only be enabled using the new YAML configuration format. Classic mode configuration format doesn't support processors. + +## Configuration Parameters + +| Key | Description | +| :---------- | :--- | +| update | Update an existing key with a value into metrics. The key-value pair is required. | +| insert | Insert a new key with a value into metrics. The key-value pair is required. | +| upsert | Upsert a specific key with a value, the `upsert` operation will try to update the value of the key. If the key does not exist, the key will be created. The key-value pair is required. | +| delete | Delete a key from the labels of metrics. The key-value pair is required. | +| hash | Replace the key value with a hash generated by the SHA-1 algorithm from the specified label name, the binary value generated is finally set as an hex string representation. | + + +#### Update example + +Change the value of the `name` to `fluentbit`: + +```yaml +pipeline: + inputs: + - name: fluentbit_metrics + processors: + metrics: + - name: labels + update: name fluentbit + outputs: + - name : stdout + match: '*' +``` + +#### Insert example + +The following example appends the key `agent` with the value `fluentbit` as the label of metrics. + +```yaml +pipeline: + inputs: + - name: fluentbit_metrics + processors: + metrics: + - name: labels + insert: agent fluentbit + outputs: + - name : stdout + match: '*' +``` + +#### Upsert example + +Upsert the value of `name` and insert `fluentbit`: + +```yaml +pipeline: + inputs: + - name: fluentbit_metrics + processors: + metrics: + - name: labels + upsert: name fluentbit + outputs: + - name : stdout + match: '*' +``` + +#### Delete example + +Delete containing `name` key from metrics: + +```yaml +pipeline: + inputs: + - name: fluentbit_metrics + processors: + metrics: + - name: labels + delete: name + outputs: + - name : stdout + match: '*' +``` + +#### Hash example + +Apply the SHA-1 algorithm for the value of the key `hostname`: + +```yaml +pipeline: + inputs: + - name: fluentbit_metrics + processors: + metrics: + - name: labels + hash: hostname + outputs: + - name : stdout + match: '*' +``` From be15335ef79a1127446f598c25eeb00cdf3795b8 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Wed, 15 May 2024 15:26:49 +0900 Subject: [PATCH 2/9] processor_labels: Use SHA-256 instead of SHA-1 This is because using hash algorithm is actually SHA-256. Signed-off-by: Hiroshi Hatake --- pipeline/processors/labels.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/processors/labels.md b/pipeline/processors/labels.md index 0b84b0b82..ea06895eb 100644 --- a/pipeline/processors/labels.md +++ b/pipeline/processors/labels.md @@ -15,7 +15,7 @@ Note that processors and this specific component can only be enabled using the n | insert | Insert a new key with a value into metrics. The key-value pair is required. | | upsert | Upsert a specific key with a value, the `upsert` operation will try to update the value of the key. If the key does not exist, the key will be created. The key-value pair is required. | | delete | Delete a key from the labels of metrics. The key-value pair is required. | -| hash | Replace the key value with a hash generated by the SHA-1 algorithm from the specified label name, the binary value generated is finally set as an hex string representation. | +| hash | Replace the key value with a hash generated by the SHA-256 algorithm from the specified label name, the binary value generated is finally set as an hex string representation. | #### Update example From 3325513faae537894a7bc3cb4b998ada80698498 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Wed, 17 Jul 2024 16:58:03 +0900 Subject: [PATCH 3/9] Update labels.md Signed-off-by: Hiroshi Hatake Co-authored-by: Pat --- pipeline/processors/labels.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pipeline/processors/labels.md b/pipeline/processors/labels.md index ea06895eb..e87785b42 100644 --- a/pipeline/processors/labels.md +++ b/pipeline/processors/labels.md @@ -5,7 +5,8 @@ The **labels** processor allows you to manipulate the labels of Metrics. Similar to the functionality exposed by filters, this processor presents a enriching/modifying mechanism to perform such operations for labels manipulation. The most significant difference is that processors perform better than filters, and when chaining them, there are no encoding/decoding performance penalties. -Note that processors and this specific component can only be enabled using the new YAML configuration format. Classic mode configuration format doesn't support processors. +Note that processors and this specific component can only be enabled using the new YAML configuration format. +Classic mode configuration format doesn't support processors. ## Configuration Parameters From f290a655659fbb4257d69716c1ae5f279f422275 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Wed, 17 Jul 2024 16:54:54 +0900 Subject: [PATCH 4/9] processor_labels: Clarify an effect of absence of the target key when insert and delete operations Signed-off-by: Hiroshi Hatake --- pipeline/processors/labels.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pipeline/processors/labels.md b/pipeline/processors/labels.md index e87785b42..d24b5da8d 100644 --- a/pipeline/processors/labels.md +++ b/pipeline/processors/labels.md @@ -12,10 +12,10 @@ Classic mode configuration format doesn't support processors. | Key | Description | | :---------- | :--- | -| update | Update an existing key with a value into metrics. The key-value pair is required. | +| update | Update an existing key with a value into metrics. The key-value pair is required. If the specified key does not exist, the operation is silently failed and no effect. | | insert | Insert a new key with a value into metrics. The key-value pair is required. | | upsert | Upsert a specific key with a value, the `upsert` operation will try to update the value of the key. If the key does not exist, the key will be created. The key-value pair is required. | -| delete | Delete a key from the labels of metrics. The key-value pair is required. | +| delete | Delete a key from the labels of metrics. The key-value pair is required. If the specified key does not exist, the operation is silently failed and no effect. | | hash | Replace the key value with a hash generated by the SHA-256 algorithm from the specified label name, the binary value generated is finally set as an hex string representation. | From 47b25017847bcf607ec892f4774d593182eacb91 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 26 Jul 2024 11:36:25 +0900 Subject: [PATCH 5/9] Update pipeline/processors/labels.md Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: Hiroshi Hatake --- pipeline/processors/labels.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/pipeline/processors/labels.md b/pipeline/processors/labels.md index d24b5da8d..2627f724f 100644 --- a/pipeline/processors/labels.md +++ b/pipeline/processors/labels.md @@ -1,12 +1,17 @@ # Labels -The **labels** processor allows you to manipulate the labels of Metrics. - -Similar to the functionality exposed by filters, this processor presents a enriching/modifying mechanism to perform such operations for labels manipulation. -The most significant difference is that processors perform better than filters, and when chaining them, there are no encoding/decoding performance penalties. - -Note that processors and this specific component can only be enabled using the new YAML configuration format. -Classic mode configuration format doesn't support processors. +The **labels** processor lets you manipulate the labels of metrics. + +Similar to filters, this processor presents a enriching/modifying mechanism to +perform operations for labels manipulation. The most significant difference is +that processors perform better than filters, and when chaining them there are no +encoding or decoding performance penalties. + +{% hint style="info" %} +**Note:** Both processors and this specific component can be enabled only by using +the YAML configuration format. Classic mode configuration format doesn't support +processors. +{% endhint %} ## Configuration Parameters From 53d5d37a5662783f7bfea6c96f77769f79587a82 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 26 Jul 2024 11:36:43 +0900 Subject: [PATCH 6/9] Update pipeline/processors/labels.md Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: Hiroshi Hatake --- pipeline/processors/labels.md | 1 - 1 file changed, 1 deletion(-) diff --git a/pipeline/processors/labels.md b/pipeline/processors/labels.md index 2627f724f..1aab37438 100644 --- a/pipeline/processors/labels.md +++ b/pipeline/processors/labels.md @@ -23,7 +23,6 @@ processors. | delete | Delete a key from the labels of metrics. The key-value pair is required. If the specified key does not exist, the operation is silently failed and no effect. | | hash | Replace the key value with a hash generated by the SHA-256 algorithm from the specified label name, the binary value generated is finally set as an hex string representation. | - #### Update example Change the value of the `name` to `fluentbit`: From 0bdb544de9a205750e7f5c484914306e0829a431 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 26 Jul 2024 11:36:49 +0900 Subject: [PATCH 7/9] Update pipeline/processors/labels.md Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: Hiroshi Hatake --- pipeline/processors/labels.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pipeline/processors/labels.md b/pipeline/processors/labels.md index 1aab37438..cc2574542 100644 --- a/pipeline/processors/labels.md +++ b/pipeline/processors/labels.md @@ -15,10 +15,10 @@ processors. ## Configuration Parameters -| Key | Description | -| :---------- | :--- | -| update | Update an existing key with a value into metrics. The key-value pair is required. If the specified key does not exist, the operation is silently failed and no effect. | -| insert | Insert a new key with a value into metrics. The key-value pair is required. | +| Key | Description | +| :----- | :---------- | +| update | Update an existing key with a value into metrics. The key/value pair is required. If the specified key doesn't exist, the operation silently fails and has no effect. | +| insert | Insert a new key with a value into metrics. The key/value pair is required. | | upsert | Upsert a specific key with a value, the `upsert` operation will try to update the value of the key. If the key does not exist, the key will be created. The key-value pair is required. | | delete | Delete a key from the labels of metrics. The key-value pair is required. If the specified key does not exist, the operation is silently failed and no effect. | | hash | Replace the key value with a hash generated by the SHA-256 algorithm from the specified label name, the binary value generated is finally set as an hex string representation. | From 8d08f092b273030818e206c08fd375e17ab784d0 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 26 Jul 2024 11:36:54 +0900 Subject: [PATCH 8/9] Update pipeline/processors/labels.md Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: Hiroshi Hatake --- pipeline/processors/labels.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pipeline/processors/labels.md b/pipeline/processors/labels.md index cc2574542..f5db98a3a 100644 --- a/pipeline/processors/labels.md +++ b/pipeline/processors/labels.md @@ -20,8 +20,8 @@ processors. | update | Update an existing key with a value into metrics. The key/value pair is required. If the specified key doesn't exist, the operation silently fails and has no effect. | | insert | Insert a new key with a value into metrics. The key/value pair is required. | | upsert | Upsert a specific key with a value, the `upsert` operation will try to update the value of the key. If the key does not exist, the key will be created. The key-value pair is required. | -| delete | Delete a key from the labels of metrics. The key-value pair is required. If the specified key does not exist, the operation is silently failed and no effect. | -| hash | Replace the key value with a hash generated by the SHA-256 algorithm from the specified label name, the binary value generated is finally set as an hex string representation. | +| delete | Delete a key from the labels of metrics. The key/value pair is required. If the specified key doesn't exist, the operation silently fails and has no effect. | +| hash | Replace the key value with a hash generated by the SHA-256 algorithm from the specified label name. The generated binary value is set as a hex string. | #### Update example From 394c4d73e10ee428d55e6aafc68430f74f1ad301 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 26 Jul 2024 11:37:01 +0900 Subject: [PATCH 9/9] Update pipeline/processors/labels.md Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: Hiroshi Hatake --- pipeline/processors/labels.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pipeline/processors/labels.md b/pipeline/processors/labels.md index f5db98a3a..3820c6139 100644 --- a/pipeline/processors/labels.md +++ b/pipeline/processors/labels.md @@ -42,7 +42,8 @@ pipeline: #### Insert example -The following example appends the key `agent` with the value `fluentbit` as the label of metrics. +The following example appends the key `agent` with the value `fluentbit` as the label +of metrics: ```yaml pipeline: