From b7752eaeef74336dc67ba302680c83e597ccecd0 Mon Sep 17 00:00:00 2001 From: Kennedy Onyia Date: Wed, 15 Oct 2025 12:02:03 -0500 Subject: [PATCH 01/17] Update delete_entries processor to add new features Signed-off-by: Kennedy Onyia --- .../processors/delete-entries.md | 31 +++++++++++++------ 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete-entries.md index 6e12db2103c..f101c82ccd5 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete-entries.md @@ -8,7 +8,11 @@ nav_order: 110 # Delete entries processor -The `delete_entries` processor deletes entries, such as key-value pairs, from an event. You can define the keys you want to delete in the `with-keys` field following `delete_entries` in the YAML configuration file. Those keys and their values are deleted. +The `delete_entries` processor removes entries, such as key-value pairs, from an event. Use the `with_keys` field to specify +the exact keys to delete. To delete keys that match a regular expression pattern, use the `with_keys_regex` field. You can +prevent deletion of specific events when using regular expressions by configuring the `exclude_from_regex` field. + +To use both `with_keys` and `with_keys_regex` in the same configuration, define them under the `entries` field. ## Configuration @@ -21,9 +25,15 @@ This table is autogenerated. Do not edit it. - source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/mutate-event-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutateevent/DeleteEntryProcessorConfig.java --> -| Option | Required | Description | -:--- | :--- | :--- -| `with_keys` | Yes | An array of keys for the entries to be deleted. | +| Option | Required | Description | +:--- |:---------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ +| `with_keys` | No | An array that specifies the keys of the entries to delete. | +| `with_keys_regex` | No | An array of regular expression (regex) patterns used to match the keys of entries to delete. | +| `exclude_from_delete` | No | A set of entries to exclude from deletion when using the `with_keys_regex` configuration. | +| `entries` | No | A list of entries to delete from the event. | +| `delete_when` | No | Specifies under what conditions the deletion should be perfomed. (Example: value= "/some_key == null" - Deletion runs only if some_key is null or does not exist.) | +|`delete_from_element_when` | No | Specifies the condition needed to delete the key from each element of a list whe using `iterate_on` | +| `iterate_on` | No | A string that specifies the key of the list of objects to iterate over and delete the keys specified in `with_keys` ## Usage @@ -33,10 +43,13 @@ To get started, create the following `pipeline.yaml` file: pipeline: source: ... - .... + .... processor: - delete_entries: - with_keys: ["message"] + entries: + - with_keys: [ "message" ] + - with_keys_regex: [ "^tes.*" ] + exclude_from_delete: [ "test" ] sink: ``` {% include copy.html %} @@ -46,13 +59,13 @@ Next, create a log file named `logs_json.log` and replace the `path` in the file For example, before you run the `delete_entries` processor, if the `logs_json.log` file contains the following event record: ```json -{"message": "hello", "message2": "goodbye"} +{"message": "hello", "message2": "goodbye", "test": "friends", "test2": "are", "test3": "kind"} ``` When you run the `delete_entries` processor, it parses the message into the following output: ```json -{"message2": "goodbye"} +{"message2": "goodbye","test": "friends"} ``` -> If `message` does not exist in the event, then no action occurs. +> If `with_keys`, `with_keys_regex`, or `exclude_from_delete` values do not match any event keys, then no action occurs. From 66e3c60623bdf4964ee6dc6aa72cf823c578af3c Mon Sep 17 00:00:00 2001 From: Kennedy Onyia Date: Fri, 31 Oct 2025 14:38:27 -0500 Subject: [PATCH 02/17] update select_entries processor documentation to account for new include_keys_regex feature Signed-off-by: Kennedy Onyia --- .../processors/select-entries.md | 23 ++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/select-entries.md b/_data-prepper/pipelines/configuration/processors/select-entries.md index 78fb4cdcdbe..c0cc0ce8cc8 100644 --- a/_data-prepper/pipelines/configuration/processors/select-entries.md +++ b/_data-prepper/pipelines/configuration/processors/select-entries.md @@ -16,9 +16,10 @@ Only the selected entries remain in the processed event and while all other entr You can configure the `select_entries` processor using the following options. | Option | Required | Description | -| :--- | :--- | :--- | -| `include_keys` | Yes | A list of keys to be selected from an event. | -| `select_when` | No | A [conditional expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"'`, that will be evaluated to determine whether the processor will be run on the event. If the condition is not met, then the event continues through the pipeline unmodified with all the original fields present. | +| :--- |:---------| :--- | +| `include_keys` | No | A list of keys to be selected from an event. | +| `include_keys_regex` | No | A regex pattern that matches the keys to be selected from an event. | +| `select_when` | No | A [conditional expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"'`, that will be evaluated to determine whether the processor will be run on the event. If the condition is not met, then the event continues through the pipeline unmodified with all the original fields present. | ## Usage @@ -32,6 +33,7 @@ pipeline: processor: - select_entries: include_keys: [ "key1", "key2" ] + include_keys_regex: ["^ran.*"] select_when: '/some_key == "test"' sink: ``` @@ -41,13 +43,22 @@ pipeline: For example, when your source contains the following event record: ```json -{"message": "hello", "key1" : "value1", "key2" : "value2", "some_key" : "test"} +{ + "message": "hello", + "key1" : "value1", + "key2" : "value2", + "some_key" : "test", + "random1": "another", + "random2" : "set", + "random3": "of", + "random4": "values" +} ``` -The `select_entries` processor includes only `key1` and `key2` in the processed output: +The `select_entries` processor will output: ```json -{"key1": "value1", "key2": "value2"} +{"key1": "value1", "key2": "value2", "random1": "another", "random2" : "set", "random3": "of", "random4": "values"} ``` ### Accessing nested fields From 26449fa619eb74fd996a7fe7cfc82ecd6595cd81 Mon Sep 17 00:00:00 2001 From: Kennedy Onyia Date: Tue, 4 Nov 2025 11:34:20 -0600 Subject: [PATCH 03/17] fix style check errors and include additional pipeline configurations to clarify new features. Signed-off-by: Kennedy Onyia --- .../processors/delete-entries.md | 85 ++++++++++++++++--- .../processors/select-entries.md | 37 +++++++- 2 files changed, 106 insertions(+), 16 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete-entries.md index f101c82ccd5..d79fd8fc623 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete-entries.md @@ -5,14 +5,14 @@ parent: Processors grand_parent: Pipelines nav_order: 110 --- - + # Delete entries processor The `delete_entries` processor removes entries, such as key-value pairs, from an event. Use the `with_keys` field to specify the exact keys to delete. To delete keys that match a regular expression pattern, use the `with_keys_regex` field. You can prevent deletion of specific events when using regular expressions by configuring the `exclude_from_regex` field. -To use both `with_keys` and `with_keys_regex` in the same configuration, define them under the `entries` field. +The only way to configure both `with_keys` and `with_keys_regex` in the same `delete_entries` processor is by using the `entries` field. ## Configuration @@ -25,17 +25,76 @@ This table is autogenerated. Do not edit it. - source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/mutate-event-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutateevent/DeleteEntryProcessorConfig.java --> -| Option | Required | Description | -:--- |:---------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ -| `with_keys` | No | An array that specifies the keys of the entries to delete. | -| `with_keys_regex` | No | An array of regular expression (regex) patterns used to match the keys of entries to delete. | -| `exclude_from_delete` | No | A set of entries to exclude from deletion when using the `with_keys_regex` configuration. | -| `entries` | No | A list of entries to delete from the event. | -| `delete_when` | No | Specifies under what conditions the deletion should be perfomed. (Example: value= "/some_key == null" - Deletion runs only if some_key is null or does not exist.) | -|`delete_from_element_when` | No | Specifies the condition needed to delete the key from each element of a list whe using `iterate_on` | -| `iterate_on` | No | A string that specifies the key of the list of objects to iterate over and delete the keys specified in `with_keys` +| Option | Required | Description | +:--- |:---------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +| `with_keys` | No | An array that specifies the keys of the entries to delete. | +| `with_keys_regex` | No | An array of regular expression (regex) patterns used to match the keys of entries to delete. | +| `exclude_from_delete` | No | A set of entries to exclude from deletion when using the `with_keys_regex` configuration. | +| `entries` | No | A list of entries to delete from the event. | +| `delete_when` | No | Specifies under what conditions the deletion should be performed. (Example: value= "/some_key == null" - Deletion runs only if some_key is null or does not exist.) | +|`delete_from_element_when` | No | Specifies the condition needed to delete the key from each element of a list when using `iterate_on` | +| `iterate_on` | No | A string that specifies the key of the list of objects to iterate over and delete the keys specified in `with_keys` ## Usage +To get started, create the following `pipeline.yaml` file: + +```yaml +pipeline: + source: + ... + .... + processor: + - delete_entries: + with_keys: [ "message" ] + sink: +``` +{% include copy.html %} + +Next, create a log file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` file with that filepath. For more information, see [Configuring OpenSearch Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). + +For example, before you run the `delete_entries` processor, if the `logs_json.log` file contains the following event record: + +```json +{"message": "hello", "message2": "goodbye"} +``` + +When you run the `delete_entries` processor, it parses the message into the following output: + +```json +{"message2": "goodbye", "message3": "test"} +``` + +### Additional configuration examples + +#### Using `with_keys_regex` field + +To get started, create the following `pipeline.yaml` file: +```yaml +pipeline: + source: + ... + .... + processor: + - delete_entries: + with_keys_regex: [ "^tes.*" ] + exclude_from_delete: [ "test" ] + sink: +``` +{% include copy.html %} + +If your `logs_json.log` file contains the following event record: + +```json +{"test": "friends", "test2": "are", "test3": "kind"} +``` + +When you run the `delete_entries` processor, it parses the message into the following output: + +```json +{"test": "friends"} +``` + +#### Using `entries` field To get started, create the following `pipeline.yaml` file: @@ -54,9 +113,7 @@ pipeline: ``` {% include copy.html %} -Next, create a log file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` file with that filepath. For more information, see [Configuring OpenSearch Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). - -For example, before you run the `delete_entries` processor, if the `logs_json.log` file contains the following event record: +If your `logs_json.log` file contains the following event record: ```json {"message": "hello", "message2": "goodbye", "test": "friends", "test2": "are", "test3": "kind"} diff --git a/_data-prepper/pipelines/configuration/processors/select-entries.md b/_data-prepper/pipelines/configuration/processors/select-entries.md index c0cc0ce8cc8..0da6b0acb7c 100644 --- a/_data-prepper/pipelines/configuration/processors/select-entries.md +++ b/_data-prepper/pipelines/configuration/processors/select-entries.md @@ -23,7 +23,7 @@ You can configure the `select_entries` processor using the following options. ## Usage -The following example shows how to configure the `select_entries` processor in the `pipeline.yaml` file: +The get started, create the following `pipeline.yaml` file: ```yaml pipeline: @@ -33,13 +33,46 @@ pipeline: processor: - select_entries: include_keys: [ "key1", "key2" ] - include_keys_regex: ["^ran.*"] select_when: '/some_key == "test"' sink: ``` {% include copy.html %} +For example, when your source contains the following event record: + +```json +{ + "message": "hello", + "key1" : "value1", + "key2" : "value2", + "some_key" : "test" +} +``` + +The `select_entries` processor will output: + +```json +{"key1": "value1", "key2": "value2"} +``` + +### Additional configuration example +The following example shows how to configure the `include_keys_regex` field in the `pipeline.yaml` file: + +```yaml +pipeline: + source: + ... + .... + processor: + - select_entries: + include_keys: [ "key1", "key2" ] + include_keys_regex: ["^ran.*"] + select_when: '/some_key == "test"' + sink: +``` +{% include copy.html %} + For example, when your source contains the following event record: ```json From db8e06f79ced5e9b44293c66ea16da295cd33a8e Mon Sep 17 00:00:00 2001 From: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> Date: Thu, 6 Nov 2025 08:09:32 -0600 Subject: [PATCH 04/17] Update _data-prepper/pipelines/configuration/processors/delete-entries.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> --- .../pipelines/configuration/processors/delete-entries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete-entries.md index d79fd8fc623..cc36117c416 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete-entries.md @@ -31,7 +31,7 @@ This table is autogenerated. Do not edit it. | `with_keys_regex` | No | An array of regular expression (regex) patterns used to match the keys of entries to delete. | | `exclude_from_delete` | No | A set of entries to exclude from deletion when using the `with_keys_regex` configuration. | | `entries` | No | A list of entries to delete from the event. | -| `delete_when` | No | Specifies under what conditions the deletion should be performed. (Example: value= "/some_key == null" - Deletion runs only if some_key is null or does not exist.) | +| `delete_when` | No | Defines the condition under which the deletion is performed. For example, `value="/some_key == null"` deletes the key only if `/some_key` is null or does not exist. | |`delete_from_element_when` | No | Specifies the condition needed to delete the key from each element of a list when using `iterate_on` | | `iterate_on` | No | A string that specifies the key of the list of objects to iterate over and delete the keys specified in `with_keys` From 9f015590f26be134f807df4564aa18b46a305a06 Mon Sep 17 00:00:00 2001 From: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> Date: Thu, 6 Nov 2025 08:10:09 -0600 Subject: [PATCH 05/17] Update _data-prepper/pipelines/configuration/processors/select-entries.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> --- .../pipelines/configuration/processors/select-entries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/select-entries.md b/_data-prepper/pipelines/configuration/processors/select-entries.md index 0da6b0acb7c..c6abb5e0c78 100644 --- a/_data-prepper/pipelines/configuration/processors/select-entries.md +++ b/_data-prepper/pipelines/configuration/processors/select-entries.md @@ -88,7 +88,7 @@ For example, when your source contains the following event record: } ``` -The `select_entries` processor will output: +The processor retains keys explicitly listed in `include_keys` and any keys matching the` include_keys_regex` pattern, removing all other keys from the event: ```json {"key1": "value1", "key2": "value2", "random1": "another", "random2" : "set", "random3": "of", "random4": "values"} From 912882af7e6ec498e7a1aed5244c21cc29bbd48b Mon Sep 17 00:00:00 2001 From: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> Date: Thu, 6 Nov 2025 08:10:39 -0600 Subject: [PATCH 06/17] Update _data-prepper/pipelines/configuration/processors/delete-entries.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> --- .../pipelines/configuration/processors/delete-entries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete-entries.md index cc36117c416..cddafe4eaf4 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete-entries.md @@ -66,7 +66,7 @@ When you run the `delete_entries` processor, it parses the message into the foll ### Additional configuration examples -#### Using `with_keys_regex` field +### Deleting keys that match a pattern To get started, create the following `pipeline.yaml` file: ```yaml From 80acded0e1d95d0fb74a90c7d62db4d284ebc053 Mon Sep 17 00:00:00 2001 From: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> Date: Thu, 6 Nov 2025 08:12:34 -0600 Subject: [PATCH 07/17] Update _data-prepper/pipelines/configuration/processors/select-entries.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> --- .../pipelines/configuration/processors/select-entries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/select-entries.md b/_data-prepper/pipelines/configuration/processors/select-entries.md index c6abb5e0c78..7b9687e2c80 100644 --- a/_data-prepper/pipelines/configuration/processors/select-entries.md +++ b/_data-prepper/pipelines/configuration/processors/select-entries.md @@ -19,7 +19,7 @@ You can configure the `select_entries` processor using the following options. | :--- |:---------| :--- | | `include_keys` | No | A list of keys to be selected from an event. | | `include_keys_regex` | No | A regex pattern that matches the keys to be selected from an event. | -| `select_when` | No | A [conditional expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"'`, that will be evaluated to determine whether the processor will be run on the event. If the condition is not met, then the event continues through the pipeline unmodified with all the original fields present. | +| `select_when` | No | A [conditional expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"`, that is evaluated to determine whether the processor will be executed on the event. If the condition is not met, then the event continues through the pipeline unmodified, with all the original fields present. | ## Usage From a84322a01c4e453a69412de471e8a551f91921a7 Mon Sep 17 00:00:00 2001 From: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> Date: Thu, 6 Nov 2025 08:13:58 -0600 Subject: [PATCH 08/17] Update _data-prepper/pipelines/configuration/processors/select-entries.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> --- .../pipelines/configuration/processors/select-entries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/select-entries.md b/_data-prepper/pipelines/configuration/processors/select-entries.md index 7b9687e2c80..98afe2294f5 100644 --- a/_data-prepper/pipelines/configuration/processors/select-entries.md +++ b/_data-prepper/pipelines/configuration/processors/select-entries.md @@ -50,7 +50,7 @@ For example, when your source contains the following event record: } ``` -The `select_entries` processor will output: +After processing, only the keys listed in `include_keys` are retained in the event; all other keys are removed: ```json {"key1": "value1", "key2": "value2"} From 9db89bcb6e488c15dca9c8ff56cdeb9a2d01c789 Mon Sep 17 00:00:00 2001 From: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> Date: Thu, 6 Nov 2025 08:14:26 -0600 Subject: [PATCH 09/17] Update _data-prepper/pipelines/configuration/processors/delete-entries.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> --- .../pipelines/configuration/processors/delete-entries.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete-entries.md index cddafe4eaf4..c73aea0e124 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete-entries.md @@ -64,7 +64,6 @@ When you run the `delete_entries` processor, it parses the message into the foll {"message2": "goodbye", "message3": "test"} ``` -### Additional configuration examples ### Deleting keys that match a pattern From b8dba47874d70cba01c853359126bf3e1555b469 Mon Sep 17 00:00:00 2001 From: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> Date: Thu, 6 Nov 2025 08:32:54 -0600 Subject: [PATCH 10/17] Update _data-prepper/pipelines/configuration/processors/select-entries.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> --- .../pipelines/configuration/processors/select-entries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/select-entries.md b/_data-prepper/pipelines/configuration/processors/select-entries.md index 98afe2294f5..0c7bf3c8b79 100644 --- a/_data-prepper/pipelines/configuration/processors/select-entries.md +++ b/_data-prepper/pipelines/configuration/processors/select-entries.md @@ -23,7 +23,7 @@ You can configure the `select_entries` processor using the following options. ## Usage -The get started, create the following `pipeline.yaml` file: +First, create the following `pipeline.yaml` file: ```yaml pipeline: From 2953e1708de0ec12622777edd2b88501ef36894b Mon Sep 17 00:00:00 2001 From: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> Date: Thu, 6 Nov 2025 08:33:09 -0600 Subject: [PATCH 11/17] Update _data-prepper/pipelines/configuration/processors/select-entries.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> --- .../pipelines/configuration/processors/select-entries.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/select-entries.md b/_data-prepper/pipelines/configuration/processors/select-entries.md index 0c7bf3c8b79..368c5960b06 100644 --- a/_data-prepper/pipelines/configuration/processors/select-entries.md +++ b/_data-prepper/pipelines/configuration/processors/select-entries.md @@ -56,7 +56,8 @@ After processing, only the keys listed in `include_keys` are retained in the eve {"key1": "value1", "key2": "value2"} ``` -### Additional configuration example +### Selecting keys using a regex + The following example shows how to configure the `include_keys_regex` field in the `pipeline.yaml` file: ```yaml From 39ca895a310f93bc1c85af91c1022d6750548bda Mon Sep 17 00:00:00 2001 From: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> Date: Thu, 6 Nov 2025 10:31:05 -0600 Subject: [PATCH 12/17] Update _data-prepper/pipelines/configuration/processors/delete-entries.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> --- .../pipelines/configuration/processors/delete-entries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete-entries.md index c73aea0e124..d02d441d142 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete-entries.md @@ -32,7 +32,7 @@ This table is autogenerated. Do not edit it. | `exclude_from_delete` | No | A set of entries to exclude from deletion when using the `with_keys_regex` configuration. | | `entries` | No | A list of entries to delete from the event. | | `delete_when` | No | Defines the condition under which the deletion is performed. For example, `value="/some_key == null"` deletes the key only if `/some_key` is null or does not exist. | -|`delete_from_element_when` | No | Specifies the condition needed to delete the key from each element of a list when using `iterate_on` | +|`delete_from_element_when` | No | Defines the condition that determines whether a key–value pair should be removed from each element in the list specified by `iterate_on`. The condition is evaluated for each element, and deletion occurs only if the element's key matches one defined in `with_keys` or `with_keys_regex` and satisfies the condition. | | `iterate_on` | No | A string that specifies the key of the list of objects to iterate over and delete the keys specified in `with_keys` ## Usage From fea069694b1867d0b119ec35e1d4583dac0704de Mon Sep 17 00:00:00 2001 From: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> Date: Thu, 6 Nov 2025 10:31:14 -0600 Subject: [PATCH 13/17] Update _data-prepper/pipelines/configuration/processors/delete-entries.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Kennedy Onyia <145404406+kennedy-onyia@users.noreply.github.com> --- .../pipelines/configuration/processors/delete-entries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete-entries.md index d02d441d142..1df19933efa 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete-entries.md @@ -33,7 +33,7 @@ This table is autogenerated. Do not edit it. | `entries` | No | A list of entries to delete from the event. | | `delete_when` | No | Defines the condition under which the deletion is performed. For example, `value="/some_key == null"` deletes the key only if `/some_key` is null or does not exist. | |`delete_from_element_when` | No | Defines the condition that determines whether a key–value pair should be removed from each element in the list specified by `iterate_on`. The condition is evaluated for each element, and deletion occurs only if the element's key matches one defined in `with_keys` or `with_keys_regex` and satisfies the condition. | -| `iterate_on` | No | A string that specifies the key of the list of objects to iterate over and delete the keys specified in `with_keys` +| `iterate_on` | No | Specifies the key of the list field that contains objects to iterate over. The processor applies any configured deletion rules, such as `with_keys`, `with_keys_regex`, or `delete_from_element_when`, to each element in the list. | ## Usage To get started, create the following `pipeline.yaml` file: From 3d1252f97d0fa97f5dcf8689591f938f52bd0c02 Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Mon, 10 Nov 2025 15:01:50 -0500 Subject: [PATCH 14/17] Update _data-prepper/pipelines/configuration/processors/delete-entries.md Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- .../pipelines/configuration/processors/delete-entries.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete-entries.md index 1df19933efa..b6153a9ca83 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete-entries.md @@ -67,7 +67,8 @@ When you run the `delete_entries` processor, it parses the message into the foll ### Deleting keys that match a pattern -To get started, create the following `pipeline.yaml` file: +First, create the following `pipeline.yaml` file: + ```yaml pipeline: source: From faea6c190e7ba06f4fdddbf92faf4631733dbdd3 Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Mon, 10 Nov 2025 15:01:58 -0500 Subject: [PATCH 15/17] Update _data-prepper/pipelines/configuration/processors/delete-entries.md Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- .../pipelines/configuration/processors/delete-entries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete-entries.md index b6153a9ca83..de10c7ddff6 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete-entries.md @@ -94,7 +94,7 @@ When you run the `delete_entries` processor, it parses the message into the foll {"test": "friends"} ``` -#### Using `entries` field +### Combining multiple deletion rules To get started, create the following `pipeline.yaml` file: From e495ab05574618a2114cfa53f160d98e3266bc27 Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Mon, 10 Nov 2025 15:02:06 -0500 Subject: [PATCH 16/17] Update _data-prepper/pipelines/configuration/processors/delete-entries.md Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- .../pipelines/configuration/processors/delete-entries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete-entries.md index de10c7ddff6..9ba82cdb22f 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete-entries.md @@ -96,7 +96,7 @@ When you run the `delete_entries` processor, it parses the message into the foll ### Combining multiple deletion rules -To get started, create the following `pipeline.yaml` file: +First, create the following `pipeline.yaml` file: ```yaml pipeline: From 676b41b7b00b71db0afa5227798d7a10789bd5c7 Mon Sep 17 00:00:00 2001 From: Nathan Bower Date: Mon, 10 Nov 2025 15:27:44 -0500 Subject: [PATCH 17/17] Apply suggestions from code review Signed-off-by: Nathan Bower --- .../pipelines/configuration/processors/delete-entries.md | 2 +- .../pipelines/configuration/processors/select-entries.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete-entries.md index 9ba82cdb22f..7bf014c885e 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete-entries.md @@ -125,4 +125,4 @@ When you run the `delete_entries` processor, it parses the message into the foll {"message2": "goodbye","test": "friends"} ``` -> If `with_keys`, `with_keys_regex`, or `exclude_from_delete` values do not match any event keys, then no action occurs. +> If the `with_keys`, `with_keys_regex`, or `exclude_from_delete` values do not match any event keys, then no action occurs. diff --git a/_data-prepper/pipelines/configuration/processors/select-entries.md b/_data-prepper/pipelines/configuration/processors/select-entries.md index 368c5960b06..d9561e856b0 100644 --- a/_data-prepper/pipelines/configuration/processors/select-entries.md +++ b/_data-prepper/pipelines/configuration/processors/select-entries.md @@ -18,7 +18,7 @@ You can configure the `select_entries` processor using the following options. | Option | Required | Description | | :--- |:---------| :--- | | `include_keys` | No | A list of keys to be selected from an event. | -| `include_keys_regex` | No | A regex pattern that matches the keys to be selected from an event. | +| `include_keys_regex` | No | A regular expression (regex) pattern that matches the keys to be selected from an event. | | `select_when` | No | A [conditional expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"`, that is evaluated to determine whether the processor will be executed on the event. If the condition is not met, then the event continues through the pipeline unmodified, with all the original fields present. | ## Usage