-
Notifications
You must be signed in to change notification settings - Fork 693
Add Parse JSON processor doc. #3237
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 19 commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
5d1bbc4
Add Parse JSON processor doc.
carolxob efcdfa9
Minor edit.
carolxob b05f860
Formatted some information as a table instead of a list.
carolxob aaac998
Minor edits.
carolxob 49316ff
Removed smaller Parse JSON file.
carolxob 778e8f9
Minor updates.
carolxob bfe52eb
Minor edits to table.
carolxob f6df00f
Minor updates.
carolxob 7720a99
Minor update based on tech feedback.
carolxob 13b1b42
Minor updates.
carolxob 24f2561
Modified 'indices' to 'indexes'
carolxob 64b8b5f
Minor edits.
carolxob 052c6fc
Minor edits.
carolxob 5ac580a
Minor edits.
carolxob ee92570
Minor edits.
carolxob 4b414c3
Minor edits.
carolxob 1cf75cd
Modified title.
carolxob 77d510c
Content edits from doc and editorial reviews.
carolxob 4ad5e29
Minor edit.
carolxob 8865ea2
Updates.
carolxob df6eeef
Final review comments incorporated.
carolxob File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
78 changes: 65 additions & 13 deletions
78
_data-prepper/pipelines/configuration/processors/parse-json.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,27 +1,79 @@ | ||
| --- | ||
| layout: default | ||
| title: Parse JSON | ||
| title: Parse_JSON | ||
| parent: Processors | ||
| grand_parent: Pipelines | ||
| nav_order: 45 | ||
| --- | ||
|
|
||
| # Parse JSON | ||
| # Parse_JSON | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| ## Overview | ||
| The `Parse_JSON` processor parses JSON data for an event, including any nested fields. The processor extracts the JSON pointer data and adds the input event to the extracted fields. | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| The `parse_json` processor parses JSON data for an event, including any nested fields. The following table describes several optional parameters you can configure in the `parse_json` processor. | ||
|
|
||
| Option | Required | Type | Description | ||
| :--- | :--- | :--- | :--- | ||
| source | No | String | The field in the `Event` that will be parsed. Default value is `message`. | ||
| destination | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. | ||
| pointer | No | String | A JSON Pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON Array indices as well. If the JSON Pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the pointed-to key already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | ||
| ## Configuration | ||
|
|
||
| <!---## Configuration | ||
| You can configure the `Parse_JSON` processor with the following options. | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| Content will be added to this section. | ||
| | Option | Required | Type | Description | | ||
| | :--- | :--- | :--- | :--- | | ||
| | `source` | No | String | The field in the `Event` that will be parsed. Default value is `message`. | | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
| | `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `Event`. Cannot be `""`, `/`, or any whitespace-only `String` because these are not valid `Event` fields. | | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
| | `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `Event`. If the key that is pointed to already exists in the `Event` and the `destination` is the root, then the pointer uses the entire path of the key. | | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| ## Metrics | ||
| ## Usage | ||
|
|
||
| Content will be added to this section.---> | ||
| To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| parse-json-pipeline: | ||
| source: | ||
| stdin: | ||
| processor: | ||
| - parse_json: | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
|
|
||
| ### Basic example | ||
|
|
||
| To test the `Parse JSON` processor with the previous configuration, run the pipeline and paste the following line into your console, then enter `exit` on a new line: | ||
|
carolxob marked this conversation as resolved.
Outdated
|
||
|
|
||
| ``` | ||
| {"outer_key": {"inner_key": "inner_value"}} | ||
| ``` | ||
| {% include copy.html %} | ||
|
|
||
| The processor parses the message into the following format: | ||
|
|
||
| ``` | ||
| {"message": {"outer_key": {"inner_key": "inner_value"}}", "outer_key":{"inner_key":"inner_value"}}} | ||
| ``` | ||
|
|
||
| ### Example with JSON pointer | ||
|
|
||
| Use a JSON pointer to parse a selection of the JSON data by specifying the `pointer` option in the configuration. To get started, create the following `pipeline.yaml` file: | ||
|
|
||
| ```yaml | ||
| parse-json-pipeline: | ||
| source: | ||
| stdin: | ||
| processor: | ||
| - parse_json: | ||
| pointer: "outer_key/inner_key" | ||
| sink: | ||
| - stdout: | ||
| ``` | ||
|
|
||
| To test the JSON processor with the pointer option, run the pipeline, paste the following line into your console, and then enter `exit` on a new line: | ||
|
|
||
| ``` | ||
| {"outer_key": {"inner_key": "inner_value"}} | ||
| ``` | ||
| {% include copy.html %} | ||
|
|
||
| The processor parses the message into the following format: | ||
|
|
||
| ``` | ||
| {"message": {"outer_key": {"inner_key": "inner_value"}}", "inner_key": "inner_value"} | ||
| ``` | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.