Update documentation for delete_entries and select_entries processors

_data-prepper/pipelines/configuration/processors/delete-entries.md

@@ -5,10 +5,14 @@ parent: Processors
 grand_parent: Pipelines
 nav_order: 110
 ---
- 
+
 # Delete entries processor
 
-The `delete_entries` processor deletes entries, such as key-value pairs, from an event. You can define the keys you want to delete in the `with-keys` field following `delete_entries` in the YAML configuration file. Those keys and their values are deleted. 
+The `delete_entries` processor removes entries, such as key-value pairs, from an event. Use the `with_keys` field to specify
+the exact keys to delete. To delete keys that match a regular expression pattern, use the `with_keys_regex` field. You can
+prevent deletion of specific events when using regular expressions by configuring the `exclude_from_regex` field.
+
+The only way to configure both `with_keys` and `with_keys_regex` in the same `delete_entries` processor is by using the `entries` field.
 
 ## Configuration
 
@@ -21,22 +25,27 @@ This table is autogenerated. Do not edit it.
 - source: https://github.com/opensearch-project/data-prepper/blob/c4455a7785bc2da4358067c217be7085e0bc8d0f/data-prepper-plugins/mutate-event-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutateevent/DeleteEntryProcessorConfig.java
 -->
 
-| Option | Required | Description |
-:--- | :--- | :---
-| `with_keys` | Yes | An array of keys for the entries to be deleted. |
+| Option | Required | Description                                                                                                                                                                       |
+:--- |:---------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+| `with_keys` | No       | An array that specifies the keys of the entries to delete.                                                                                                                        |
+| `with_keys_regex` | No       | An array of regular expression (regex) patterns used to match the keys of entries to delete.                                                                                      |
+| `exclude_from_delete` | No | A set of entries to exclude from deletion when using the `with_keys_regex` configuration.                                                                                         |
+| `entries` | No | A list of entries to delete from the event.                                                                                                                                       |
+| `delete_when` | No | Defines the condition under which the deletion is performed. For example, `value="/some_key == null"` deletes the key only if `/some_key` is null or does not exist. |
+|`delete_from_element_when` | No | Defines the condition that determines whether a key–value pair should be removed from each element in the list specified by `iterate_on`. The condition is evaluated for each element, and deletion occurs only if the element's key matches one defined in `with_keys` or `with_keys_regex` and satisfies the condition.                                                 |
+| `iterate_on` | No | Specifies the key of the list field that contains objects to iterate over. The processor applies any configured deletion rules, such as `with_keys`, `with_keys_regex`, or `delete_from_element_when`, to each element in the list. |                                                             
 
 ## Usage
-
 To get started, create the following `pipeline.yaml` file:
 
 ```yaml
 pipeline:
   source:
     ...
-  ....  
+  ....
   processor:
     - delete_entries:
-        with_keys: ["message"]
+        with_keys: [ "message" ]
   sink:
 ```
 {% include copy.html %}
@@ -52,7 +61,68 @@ For example, before you run the `delete_entries` processor, if the `logs_json.lo
 When you run the `delete_entries` processor, it parses the message into the following output:
 
 ```json
-{"message2": "goodbye"}
+{"message2": "goodbye", "message3": "test"}
+```
+
+
+### Deleting keys that match a pattern
+
+First, create the following `pipeline.yaml` file:
+
+```yaml
+pipeline:
+  source:
+    ...
+  ....
+  processor:
+    - delete_entries:
+        with_keys_regex: [ "^tes.*" ]
+        exclude_from_delete: [ "test" ]
+  sink:
+```
+{% include copy.html %}
+
+If your `logs_json.log` file contains the following event record:
+
+```json
+{"test": "friends", "test2": "are", "test3": "kind"}
+```
+
+When you run the `delete_entries` processor, it parses the message into the following output:
+
+```json
+{"test": "friends"}
+```
+
+### Combining multiple deletion rules
+
+First, create the following `pipeline.yaml` file:
+
+```yaml
+pipeline:
+  source:
+    ...
+  ....
+  processor:
+    - delete_entries:
+        entries:
+          - with_keys: [ "message" ]
+          - with_keys_regex: [ "^tes.*" ]
+            exclude_from_delete: [ "test" ]
+  sink:
+```
+{% include copy.html %}
+
+If your `logs_json.log` file contains the following event record:
+
+```json
+{"message": "hello", "message2": "goodbye", "test": "friends", "test2": "are", "test3": "kind"}
+```
+
+When you run the `delete_entries` processor, it parses the message into the following output:
+
+```json
+{"message2": "goodbye","test": "friends"}
 ```
 
-> If `message` does not exist in the event, then no action occurs.
+> If the `with_keys`, `with_keys_regex`, or `exclude_from_delete` values do not match any event keys, then no action occurs.

_data-prepper/pipelines/configuration/processors/select-entries.md

@@ -16,13 +16,14 @@ Only the selected entries remain in the processed event and while all other entr
 You can configure the `select_entries` processor using the following options.
 
 | Option | Required | Description |
-| :--- | :--- | :--- |
-| `include_keys` | Yes | A list of keys to be selected from an event. |
-| `select_when` | No | A [conditional expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"'`, that will be evaluated to determine whether the processor will be run on the event. If the condition is not met, then the event continues through the pipeline unmodified with all the original fields present. |
+| :--- |:---------| :--- |
+| `include_keys` | No       | A list of keys to be selected from an event. |
+| `include_keys_regex` | No | A regular expression (regex) pattern that matches the keys to be selected from an event. |
+| `select_when` | No       | A [conditional expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"`, that is evaluated to determine whether the processor will be executed on the event. If the condition is not met, then the event continues through the pipeline unmodified, with all the original fields present. |
 
 ## Usage
 
-The following example shows how to configure the `select_entries` processor in the `pipeline.yaml` file:
+First, create the following `pipeline.yaml` file:
 
 ```yaml
 pipeline:
@@ -41,15 +42,59 @@ pipeline:
 For example, when your source contains the following event record:
 
 ```json
-{"message": "hello", "key1" : "value1", "key2" : "value2", "some_key" : "test"}
+{
+  "message": "hello",
+  "key1" : "value1",
+  "key2" : "value2",
+  "some_key" : "test"
+}
 ```
 
-The `select_entries` processor includes only `key1` and `key2` in the processed output:
+After processing, only the keys listed in `include_keys` are retained in the event; all other keys are removed:
 
 ```json
 {"key1": "value1", "key2": "value2"}
 ```
 
+### Selecting keys using a regex
+
+The following example shows how to configure the `include_keys_regex` field in the `pipeline.yaml` file:
+
+```yaml
+pipeline:
+  source:
+    ...
+  ....  
+  processor:
+    - select_entries:
+        include_keys: [ "key1", "key2" ]
+        include_keys_regex: ["^ran.*"]
+        select_when: '/some_key == "test"'
+  sink:
+```
+{% include copy.html %}
+
+For example, when your source contains the following event record:
+
+```json
+{
+  "message": "hello",
+  "key1" : "value1",
+  "key2" : "value2",
+  "some_key" : "test",
+  "random1": "another",
+  "random2" : "set",
+  "random3": "of",
+  "random4": "values"
+}
+```
+
+The processor retains keys explicitly listed in `include_keys` and any keys matching the` include_keys_regex` pattern, removing all other keys from the event:
+
+```json
+{"key1": "value1", "key2": "value2", "random1": "another", "random2" : "set", "random3": "of", "random4": "values"}
+```
+
 ### Accessing nested fields
 
 Use `/` to access nested fields.