Skip to content

Commit 02dac70

Browse files
adding example to write json processor data prepper (#11282) (#11501)
1 parent f57f7db commit 02dac70

File tree

1 file changed

+108
-0
lines changed

1 file changed

+108
-0
lines changed

_data-prepper/pipelines/configuration/processors/write-json.md

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,111 @@ Option | Description | Example
2323
source | Mandatory field that specifies the name of the field in the event containing the message or object to be parsed. | If `source` is set to `"message"` and the input is `{"message": {"key1":"value1", "key2":{"key3":"value3"}}}`, then the `write_json` processor outputs the event as `"{\"key1\":\"value1\",\"key2\":{\"key3\":\"value3\"}}"`.
2424
target | An optional field that specifies the name of the field in which the resulting JSON string should be stored. If `target` is not specified, then the `source` field is used. | `key1`
2525

26+
## Example
27+
28+
The following example uses `write_json` twice, first to copy the `details` object into a new JSON string field named `target` and then to overwrite the original `payload` field when target is omitted:
29+
30+
```yaml
31+
write-json-demo-pipeline:
32+
source:
33+
http:
34+
path: /logs
35+
ssl: false
36+
37+
processor:
38+
# 1) Copy the nested "details" object into a JSON string at "details_json"
39+
- write_json:
40+
source: details
41+
target: details_json
42+
43+
# 2) Overwrite "payload" with its JSON-string representation
44+
- write_json:
45+
# no target -> result stored back into "payload"
46+
source: payload
47+
sink:
48+
- opensearch:
49+
hosts: ["https://opensearch:9200"]
50+
insecure: true
51+
username: admin
52+
password: admin_pass
53+
index_type: custom
54+
index: write-json-demo-%{yyyy.MM.dd}
55+
```
56+
{% include copy.html %}
57+
58+
You can test this pipeline using the following command:
59+
60+
```bash
61+
curl -sS -X POST "http://localhost:2021/logs" \
62+
-H "Content-Type: application/json" \
63+
-d '[
64+
{
65+
"message": "order created",
66+
"details": {"order_id": 123, "items": [{"sku": "A1", "qty": 2}], "expedited": true},
67+
"payload": {"user": {"id": "u-42", "role": "admin"}, "ip": "10.0.0.5"}
68+
},
69+
{
70+
"message": "order updated",
71+
"details": {"order_id": 124, "items": [{"sku": "B9", "qty": 1}], "expedited": false},
72+
"payload": {"user": {"id": "u-77", "role": "viewer"}, "ip": "10.0.0.9"}
73+
}
74+
]'
75+
```
76+
{% include copy.html %}
77+
78+
The documents stored in OpenSearch contain the following information:
79+
80+
```json
81+
{
82+
...
83+
"hits" : {
84+
"total" : {
85+
"value" : 2,
86+
"relation" : "eq"
87+
},
88+
"max_score" : 1.0,
89+
"hits" : [
90+
{
91+
"_index" : "write-json-demo-2025.10.15",
92+
"_id" : "YMQA6ZkB1u9wkbZgz8wu",
93+
"_score" : 1.0,
94+
"_source" : {
95+
"message" : "order created",
96+
"details" : {
97+
"order_id" : 123,
98+
"items" : [
99+
{
100+
"sku" : "A1",
101+
"qty" : 2
102+
}
103+
],
104+
"expedited" : true
105+
},
106+
"payload" : "{\"user\":{\"id\":\"u-42\",\"role\":\"admin\"},\"ip\":\"10.0.0.5\"}",
107+
"details_json" : "{\"order_id\":123,\"items\":[{\"sku\":\"A1\",\"qty\":2}],\"expedited\":true}"
108+
}
109+
},
110+
{
111+
"_index" : "write-json-demo-2025.10.15",
112+
"_id" : "YcQA6ZkB1u9wkbZgz8wu",
113+
"_score" : 1.0,
114+
"_source" : {
115+
"message" : "order updated",
116+
"details" : {
117+
"order_id" : 124,
118+
"items" : [
119+
{
120+
"sku" : "B9",
121+
"qty" : 1
122+
}
123+
],
124+
"expedited" : false
125+
},
126+
"payload" : "{\"user\":{\"id\":\"u-77\",\"role\":\"viewer\"},\"ip\":\"10.0.0.9\"}",
127+
"details_json" : "{\"order_id\":124,\"items\":[{\"sku\":\"B9\",\"qty\":1}],\"expedited\":false}"
128+
}
129+
}
130+
]
131+
}
132+
}
133+
```

0 commit comments

Comments
 (0)