generated from amazon-archives/__template_Apache-2.0
-
Notifications
You must be signed in to change notification settings - Fork 642
Add documentation for Mutate strings #2950
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
50 commits
Select commit
Hold shift + click to select a range
8077116
Added placeholder files.
carolxob f9dc552
Added source content from Data Prepper repo docs.
carolxob 8b0091a
Edits for consistency and formatting.
carolxob 52bcb93
Minor edits to otel-metrics-string.
carolxob d3a11c3
Content and formatting edits.
carolxob 1da45b8
Minor edits to trace-peer-forwarder
carolxob fbb7aa1
Minor edits.
carolxob 7bf3eec
Updated links to existing Data Prepper documentation as relative URLs.
carolxob c6c6478
Minor updates.
carolxob 0bf8376
Minor edits.
carolxob c70652a
Minor edits.
carolxob cbaa376
Minor edits for consistency and clarity.
carolxob ea845e3
Minor updates.
carolxob c192dc3
Minor update.
carolxob d44a4ec
minor updates.
carolxob d0bef9e
Minor updates.
carolxob 852ef1d
Adjustments based on technical feedback.
carolxob c49f88c
Removed 'Developer guide' section.
carolxob 272a987
Minor adjustments for clarity.
carolxob cba951e
Minor adjustments for consistency.
carolxob 2ca9cf8
Removed markdown comment.
carolxob bb7850e
Minor edit.
carolxob 41ffd68
Updates made based on doc review feedback.
carolxob 300d980
Minor edits.
carolxob f67888c
Removed bullet for single item list.
carolxob b258c0d
Edits made based on doc review feedback.
carolxob 26a4567
Modified file title and doc title to more accurately reflect contents.
carolxob 8d5dd02
Incorporated feedback from tech review.
carolxob 9f4fab0
Minor edit.
carolxob b29bca1
Added note to convert to table.
carolxob b552822
Updated links.
carolxob 00d4af7
Incorporated most doc review feedback.
carolxob e0135e3
Removed unaffected files.
carolxob 54403c3
Minor updates.
carolxob 008e1ee
Minor adjustements.
carolxob c840765
Added some clarification around the substitute string processor.
carolxob f653d79
Adjusted table boundaries.
carolxob 32d6a29
Minor update based on tech review feedback.
carolxob 4719163
Removed Basic from Usage section titles.
carolxob b7f221e
Added link placeholder for config file instructions.
carolxob 2a2a9b8
Minor edit.
carolxob 99234fc
Minor update with sentence structure.
carolxob 13de038
Refined steps around Data Prepper configuration files.
carolxob 7a7b2f1
Renamed file for consistency.
carolxob bc81921
Changed phrasing for consistency.
carolxob e8a975d
Removed redundant processor docs.
carolxob 29966e1
Minor updates from doc review feedback.
carolxob c9ce370
Fixed header justification alignment.
carolxob 03685a6
Made edits based on editorial review.
carolxob 543d9c8
Merging.
carolxob File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
256 changes: 256 additions & 0 deletions
256
_data-prepper/pipelines/configuration/processors/mutate-string.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,256 @@ | ||
| --- | ||
| layout: default | ||
| title: Mutate string | ||
| parent: Processors | ||
| grand_parent: Pipelines | ||
| nav_order: 45 | ||
| --- | ||
|
|
||
| # Mutate string processors | ||
|
|
||
| You can change the way that a string appears by using a mutate string processesor. For example, you can use the `uppercase_string` processor to convert a string to uppercase, and you can use the `lowercase_string` processor to convert a string to lowercase. The following is a list of processors that allow you to mutate a string: | ||
|
|
||
| * [substitute_string](#substitute_string) | ||
| * [split_string](#split_string) | ||
| * [uppercase_string](#uppercase_string) | ||
| * [lowercase_string](#lowercase_string) | ||
| * [trim_string](#trim_string) | ||
|
|
||
| ## substitute_string | ||
|
|
||
| The `substitute_string` processor matches a key's value against a regular expression (regex) and replaces all returned matches with a replacement string. | ||
|
|
||
| ### Configuration | ||
|
|
||
| You can configure the `substitute_string` processor with the following options. | ||
|
|
||
| Option | Required | Description | ||
| :--- | :--- | :--- | ||
| `entries` | Yes | A list of entries to add to an event. | | ||
| `source` | Yes | The key to be modified. | | ||
| `from` | Yes | The regex string to be replaced. Special regex characters such as `[` and `]` must be escaped using `\\` when using double quotes and `\` when using single quotes. For more information, see [Class Pattern](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/regex/Pattern.html) in the Java documentation. | | ||
| `to` | Yes | The string that replaces each match of `from`. | | ||
|
|
||
| ### Usage | ||
|
|
||
| To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - substitute_string: | ||
| entries: | ||
| - source: "message" | ||
| from: ":" | ||
| to: "-" | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
| {% include copy.html %} | ||
|
|
||
| Next, create a log file named `logs_json.log`. After that, replace the `path` of the file source in your `pipeline.yaml` file with your file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). | ||
|
|
||
| Before you run Data Prepper, the source appears in the following format: | ||
|
|
||
| ```json | ||
| {"message": "ab:cd:ab:cd"} | ||
| ``` | ||
|
|
||
| After you run Data Prepper, the source is converted to the following format: | ||
|
|
||
| ```json | ||
| {"message": "ab-cd-ab-cd"} | ||
| ``` | ||
|
|
||
| `from` defines which string is replaced, and `to` defines the string that replaces the `from` string. In the preceding example, string `ab:cd:ab:cd` becomes `ab-cd-ab-cd`. If the `from` regex string does not return a match, the key is returned without any changes. | ||
|
|
||
| ## split_string | ||
|
|
||
| The `split_string` processor splits a field into an array using a delimiter character. | ||
|
|
||
| ### Configuration | ||
|
|
||
| You can configure the `split_string` processor with the following options. | ||
|
|
||
| Option | Required | Description | ||
| :--- | :--- | :--- | ||
| `entries` | Yes | A list of entries to add to an event. | | ||
| `source` | Yes | The key to be split. | | ||
| `delimiter` | No | The separator character responsible for the split. Cannot be defined at the same time as `delimiter_regex`. At least `delimiter` or `delimiter_regex` must be defined. | | ||
| `delimiter_regex` | No | A regex string responsible for the split. Cannot be defined at the same time as `delimiter`. Either `delimiter` or `delimiter_regex` must be defined. | | ||
|
|
||
| ### Usage | ||
|
|
||
| To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - split_string: | ||
| entries: | ||
| - source: "message" | ||
| delimiter: "," | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
| {% include copy.html %} | ||
|
|
||
| Next, create a log file named `logs_json.log`. After that, replace the `path` in the file source of your `pipeline.yaml` file with your file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). | ||
|
|
||
| Before you run Data Prepper, the source appears in the following format: | ||
|
|
||
| ```json | ||
| {"message": "hello,world"} | ||
| ``` | ||
| After you run Data Prepper, the source is converted to the following format: | ||
|
|
||
| ```json | ||
| {"message":["hello","world"]} | ||
| ``` | ||
|
|
||
| ## uppercase_string | ||
|
|
||
| The `uppercase_string` processor converts the value (a string) of a key from its current case to uppercase. | ||
|
|
||
| ### Configuration | ||
|
|
||
| You can configure the `uppercase_string` processor with the following options. | ||
|
|
||
| Option | Required | Description | ||
| :--- | :--- | :--- | ||
| `with_keys` | Yes | A list of keys to convert to uppercase. | | ||
|
|
||
| ### Usage | ||
|
|
||
| To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - uppercase_string: | ||
| with_keys: | ||
| - "uppercaseField" | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
| {% include copy.html %} | ||
|
|
||
| Next, create a log file named `logs_json.log`. After that, replace the `path` in the file source of your `pipeline.yaml` file with the correct file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). | ||
|
|
||
| Before you run Data Prepper, the source appears in the following format: | ||
|
|
||
| ```json | ||
| {"uppercaseField": "hello"} | ||
| ``` | ||
| After you run Data Prepper, the source is converted to the following format: | ||
|
|
||
| ```json | ||
| {"uppercaseField": "HELLO"} | ||
| ``` | ||
|
|
||
| ## lowercase_string | ||
|
|
||
| The `lowercase string` processor converts a string to lowercase. | ||
|
|
||
| ### Configuration | ||
|
|
||
| You can configure the `lowercase string` processor with the following options. | ||
|
|
||
| Option | Required | Description | ||
| :--- | :--- | :--- | ||
| `with_keys` | Yes | A list of keys to convert to lowercase. | | ||
|
|
||
| ### Usage | ||
|
|
||
| To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - lowercase_string: | ||
| with_keys: | ||
| - "lowercaseField" | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
| {% include copy.html %} | ||
|
|
||
| Next, create a log file named `logs_json.log`. After that, replace the `path` in the file source of your `pipeline.yaml` file with the correct file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). | ||
|
|
||
| Before you run Data Prepper, the source appears in the following format: | ||
|
|
||
| ```json | ||
| {"lowercaseField": "TESTmeSSage"} | ||
| ``` | ||
|
|
||
| After you run Data Prepper, the source is converted to the following format: | ||
|
|
||
| ```json | ||
| {"lowercaseField": "testmessage"} | ||
| ``` | ||
|
|
||
| ## trim_string | ||
|
|
||
| The `trim_string` processor removes whitespace from the beginning and end of a key. | ||
|
|
||
| ### Configuration | ||
|
|
||
| You can configure the `trim_string` processor with the following options. | ||
|
|
||
| Option | Required | Description | ||
| :--- | :--- | :--- | ||
| `with_keys` | Yes | A list of keys from which to trim the whitespace. | | ||
|
|
||
| ### Usage | ||
|
|
||
| To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| pipeline: | ||
| source: | ||
| file: | ||
| path: "/full/path/to/logs_json.log" | ||
| record_type: "event" | ||
| format: "json" | ||
| processor: | ||
| - trim_string: | ||
| with_keys: | ||
| - "trimField" | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
| {% include copy.html %} | ||
|
|
||
| Next, create a log file named `logs_json.log`. After that, replace the `path` in the file source of your `pipeline.yaml` file with the correct file path. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). | ||
|
|
||
| Before you run Data Prepper, the source appears in the following format: | ||
|
|
||
| ```json | ||
| {"trimField": " Space Ship "} | ||
| ``` | ||
|
|
||
| After you run Data Prepper, the source is converted to the following format: | ||
|
|
||
| ```json | ||
| {"trimField": "Space Ship"} | ||
| ``` | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.