-
Notifications
You must be signed in to change notification settings - Fork 693
Add documentation for Mutate strings #2950
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 13 commits
Commits
Show all changes
50 commits
Select commit
Hold shift + click to select a range
8077116
Added placeholder files.
carolxob f9dc552
Added source content from Data Prepper repo docs.
carolxob 8b0091a
Edits for consistency and formatting.
carolxob 52bcb93
Minor edits to otel-metrics-string.
carolxob d3a11c3
Content and formatting edits.
carolxob 1da45b8
Minor edits to trace-peer-forwarder
carolxob fbb7aa1
Minor edits.
carolxob 7bf3eec
Updated links to existing Data Prepper documentation as relative URLs.
carolxob c6c6478
Minor updates.
carolxob 0bf8376
Minor edits.
carolxob c70652a
Minor edits.
carolxob cbaa376
Minor edits for consistency and clarity.
carolxob ea845e3
Minor updates.
carolxob c192dc3
Minor update.
carolxob d44a4ec
minor updates.
carolxob d0bef9e
Minor updates.
carolxob 852ef1d
Adjustments based on technical feedback.
carolxob c49f88c
Removed 'Developer guide' section.
carolxob 272a987
Minor adjustments for clarity.
carolxob cba951e
Minor adjustments for consistency.
carolxob 2ca9cf8
Removed markdown comment.
carolxob bb7850e
Minor edit.
carolxob 41ffd68
Updates made based on doc review feedback.
carolxob 300d980
Minor edits.
carolxob f67888c
Removed bullet for single item list.
carolxob b258c0d
Edits made based on doc review feedback.
carolxob 26a4567
Modified file title and doc title to more accurately reflect contents.
carolxob 8d5dd02
Incorporated feedback from tech review.
carolxob 9f4fab0
Minor edit.
carolxob b29bca1
Added note to convert to table.
carolxob b552822
Updated links.
carolxob 00d4af7
Incorporated most doc review feedback.
carolxob e0135e3
Removed unaffected files.
carolxob 54403c3
Minor updates.
carolxob 008e1ee
Minor adjustements.
carolxob c840765
Added some clarification around the substitute string processor.
carolxob f653d79
Adjusted table boundaries.
carolxob 32d6a29
Minor update based on tech review feedback.
carolxob 4719163
Removed Basic from Usage section titles.
carolxob b7f221e
Added link placeholder for config file instructions.
carolxob 2a2a9b8
Minor edit.
carolxob 99234fc
Minor update with sentence structure.
carolxob 13de038
Refined steps around Data Prepper configuration files.
carolxob 7a7b2f1
Renamed file for consistency.
carolxob bc81921
Changed phrasing for consistency.
carolxob e8a975d
Removed redundant processor docs.
carolxob 29966e1
Minor updates from doc review feedback.
carolxob c9ce370
Fixed header justification alignment.
carolxob 03685a6
Made edits based on editorial review.
carolxob 543d9c8
Merging.
carolxob File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
288 changes: 288 additions & 0 deletions
288
_data-prepper/pipelines/configuration/processors/mutate-event.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,288 @@ | ||
| --- | ||
| layout: default | ||
| title: Mutate event | ||
| parent: Processors | ||
| grand_parent: Pipelines | ||
| nav_order: 45 | ||
| --- | ||
|
|
||
| # Mutate event processors | ||
|
|
||
| The following processors allow you to mutate an event. | ||
|
|
||
| <!--- Why would users want to mutate an event? What does it achieve?---> | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| ## AddEntry | ||
|
|
||
| The `AddEntry` processor adds entries to an event. | ||
|
|
||
| ### Basic usage | ||
|
|
||
| To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - add_entries: | ||
| entries: | ||
| - key: "newMessage" | ||
| value: 3 | ||
| overwrite_if_key_exists: true | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
|
|
||
| Create the following file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` with this filepath. | ||
|
|
||
| ```json | ||
| {"message": "value"} | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
| ``` | ||
|
|
||
| When run, the processor parses the message into the following output: | ||
|
|
||
| ```json | ||
| {"message": "value", "newMessage": 3} | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
| ``` | ||
|
|
||
| > If `newMessage` already exists, its existing value is overwritten with a value of `3`. | ||
|
|
||
| ### Configuration | ||
|
|
||
| * `entries` (required): A list of entries to add to an event | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
| * `key` (required): The key of the new entry to be added | ||
| * `value` (required): The value of the new entry to be added. Strings, booleans, numbers, null, nested objects, and arrays containing the aforementioned data types are valid to use | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
| * `overwrite_if_key_exists` (optional): When set to `true`, if `key` already exists in the event, then the existing value is overwritten. The default value is `false`. | ||
|
|
||
| ## Copy value | ||
|
|
||
| The `copy value` processor copies values within an event. | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| ### Basic usage | ||
|
|
||
| To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - copy_values: | ||
| entries: | ||
| - from_key: "message" | ||
| to_key: "newMessage" | ||
| overwrite_if_to_key_exists: true | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
|
|
||
| Create the following file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` with the path of this file: | ||
|
|
||
| ```json | ||
| {"message": "value"} | ||
| ``` | ||
|
|
||
| When run, the processor parses the message into the following output: | ||
|
|
||
| ```json | ||
| {"message": "value", "newMessage": "value"} | ||
| ``` | ||
|
|
||
| > If `newMessage` had already existed, its existing value would have been overwritten with `value` | ||
|
|
||
| ### Configuration | ||
| * `entries` - (required) - A list of entries to be copied in an event | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
| * `from_key` - (required) - The key of the entry to be copied | ||
| * `to_key` - (required) - The key of the new entry to be added | ||
| * `overwrite_if_to_key_exists` - (optional) - When set to `true`, if `to_key` already exists in the event, then the existing value will be overwritten. The default is `false`. | ||
|
|
||
|
|
||
| ## DeleteEntry | ||
|
|
||
| The `DeleteEntry` processor deletes entries in an event. | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| ### Basic usage | ||
|
|
||
| To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - delete_entries: | ||
| with_keys: ["message"] | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
|
|
||
| Create the following file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` with the path of this file. | ||
|
|
||
| ```json | ||
| {"message": "value", "message2": "value2"} | ||
| ``` | ||
|
|
||
| When run, the processor parses the message into the following output: | ||
|
|
||
| ```json | ||
| {"message2": "value2"} | ||
| ``` | ||
|
|
||
| > If `message` had not existed in the event, then nothing would have happened | ||
|
|
||
| ### Configuration | ||
|
|
||
| <!---Need some intro text.---> | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| * `with_keys` - (required) - An array of keys of the entries to be deleted. | ||
|
|
||
|
|
||
| ## RenameKey | ||
|
|
||
| The `rename key` processor renames keys in an event. | ||
|
|
||
| ### Basic usage | ||
|
|
||
| To get started, create the following `pipeline.yaml`. | ||
| ```yaml | ||
| pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - rename_keys: | ||
| entries: | ||
| - from_key: "message" | ||
| to_key: "newMessage" | ||
| overwrite_if_to_key_exists: true | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
|
|
||
| Create the following file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` with the path of this file. | ||
|
|
||
| ```json | ||
| {"message": "value"} | ||
| ``` | ||
|
|
||
| When run, the processor parses the message into the following output: | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| ```json | ||
| {"newMessage": "value"} | ||
| ``` | ||
|
|
||
| > If `newMessage` already exists, its existing value is overwritten with `value`. | ||
|
|
||
| ### Configuration | ||
|
|
||
| <!--- Need some intro text here.---> | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| * `entries` - (required) - A list of entries to rename in an event | ||
| * `from_key` - (required) - The key of the entry to be renamed | ||
| * `to_key` - (required) - The new key of the entry | ||
| * `overwrite_if_to_key_exists` - (optional) - When set to `true`, if `to_key` already exists in the event, then the existing value will be overwritten. The default is `false`. | ||
|
|
||
| ### Special considerations | ||
|
|
||
| The renaming operation occurs in a defined order. <!--- Where is this order defined?---> This means that chaining is implicit with the `RenameKey` processor. See the following `piplines.yaml` file example: | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| ```yaml | ||
| pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - rename_key: | ||
| entries: | ||
| - from_key: "message" | ||
| to_key: "message2" | ||
| - from_key: "message2" | ||
| to_key: "message3" | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
|
|
||
| Add the following contents to the `logs_json.log` file: | ||
|
|
||
| ```json | ||
| {"message": "value"} | ||
| ``` | ||
|
|
||
| After the processor runs, the following output appears: | ||
|
|
||
| ```json | ||
| {"message3": "value"} | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
| ``` | ||
|
|
||
| ## ConvertEntry | ||
|
|
||
| The `ConvertEntry` processor converts the type of value associated with the specified key in a message to the specified type. It is a casting processor that changes the types of some fields in the event or message. Some of inputted data may need to be converted to different types, such as an integer or a double. The data may need to be converted so that it will pass the events through condition-based processors, or to perform conditional routing. | ||
|
|
||
| ## Basic usage | ||
|
|
||
| To get started with type conversion processor using Data Prepper, create the following `pipeline.yaml` file: | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| ```yaml | ||
| type-conv-pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - grok: | ||
| match: | ||
| message: ['%{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] %{NUMBER:response_status}'] | ||
| - convert_entry_type: | ||
| key: "response_status" | ||
| type: "integer" | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
|
|
||
| Create the following file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` with the path of this file. | ||
|
|
||
| ```json | ||
| {"message": "10.10.10.19 [19/Feb/2015:15:50:36 -0500] 200"} | ||
| ``` | ||
|
|
||
| When run, the `Grok` processor parses the message into the following output: | ||
|
|
||
| ```json | ||
| {"message": "10.10.10.10 [19/Feb/2015:15:50:36 -0500] 200", "clientip":"10.10.10.10", "timestamp": "19/Feb/2015:15:50:36 -0500", "response_status": "200"} | ||
| ``` | ||
|
|
||
| The type conversion processor changes the output received into the following output, where the type of `response_status` value changes to an integer: | ||
|
|
||
| ```json | ||
| {"message": "10.10.10.10 [19/Feb/2015:15:50:36 -0500] 200", "clientip":"10.10.10.10", "timestamp": "19/Feb/2015:15:50:36 -0500", "response_status": 200} | ||
| ``` | ||
|
|
||
| ### Configuration | ||
|
|
||
| * `key` (required): Keys whose value needs to be converted to a different type | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
| * `type`: Target type for key value. Possible values are `integer`, `double`, `string`, and `boolean`. Default value is `integer`. | ||
|
|
||
|
|
||
| ## Developer guide | ||
|
|
||
| This plugin is compatible with Java 14. See the following: | ||
|
|
||
| - [Contributing](https://github.com/opensearch-project/data-prepper/blob/main/CONTRIBUTING.md) | ||
| - [Monitoring]({{site.url}}{{site.baseurl}}/data-prepper/monitoring/) | ||
|
|
||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.