Skip to content

[DOCS] Update ingest node pipelines doc #79187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Oct 13, 2020
Merged

[DOCS] Update ingest node pipelines doc #79187

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4117543
update ingest node pipeline docs
alisonelizabeth Oct 1, 2020
4c0d6a5
Update docs/management/ingest-pipelines/ingest-pipelines.asciidoc
alisonelizabeth Oct 2, 2020
9c9270a
Update docs/management/ingest-pipelines/ingest-pipelines.asciidoc
alisonelizabeth Oct 2, 2020
0478aea
Update docs/management/ingest-pipelines/ingest-pipelines.asciidoc
alisonelizabeth Oct 2, 2020
0674522
address feedback
alisonelizabeth Oct 2, 2020
456e403
address feedback
alisonelizabeth Oct 2, 2020
e186962
Update docs/management/ingest-pipelines/ingest-pipelines.asciidoc
alisonelizabeth Oct 13, 2020
fad893c
Merge branch 'master' into ingest_pipelines/docs
kibanamachine Oct 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified docs/management/ingest-pipelines/images/ingest-pipeline-processor.png
100755 → 100644
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
72 changes: 44 additions & 28 deletions docs/management/ingest-pipelines/ingest-pipelines.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,47 +62,63 @@ You also want to know where the request is coming from.

. In *Ingest Node Pipelines*, click *Create a pipeline*.
. Provide a name and description for the pipeline.
. Define the processors:
. Click *Add a processor* to define your first processor.
. Select the *Grok* processor type from the dropdown menu.
+
Set the field input to `message` and provide the following pattern:
+
[source,js]
----------------------------------
[
{
"grok": {
"field": "message",
"patterns": ["%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}"]
}
},
{
"date": {
"field": "timestamp",
"formats": [ "dd/MMM/YYYY:HH:mm:ss Z" ]
}
},
{
"geoip": {
"field": "clientip"
}
},
{
"user_agent": {
"field": "agent"
}
}
]
%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}
----------------------------------
+
This code defines four {ref}/ingest-processors.html[processors] that run sequentially:
Once complete, click *Update*.
. Add 3 more processors with the following data:
+
*Date*
Copy link
Contributor Author

@alisonelizabeth alisonelizabeth Oct 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There might be a better way to display this data 😄

Copy link
Contributor

@debadair debadair Oct 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, especially the way that the table is rendered. I think I'd split up the steps differently, instead. Something like:

. Add a grok processor to parse the log message:

.. Click *Add a processor* and select the *Grok* processor type.
.. Set the field input to `message` and enter the following grok pattern.
.. Click *Update* to save the processor.

. Add processors to map the Date, GeopIP, and User agent fields:

.. Select the ***Field*** processor type.
.. Map the appropriate parsed value to each field:
+
--
* Date: `timestamp`
* GeoIp: `clientip`
* User agent: `agent`

For the Date field, you also need to specify the date format you want to use.
--

Copy link
Contributor Author

@alisonelizabeth alisonelizabeth Oct 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I like this a lot better. I followed this pattern, but changed the copy in a few places where I felt it wasn't 100% clear.

+
|===
|*Field* |*Value*

|Field
|`timestamp`

|Formats
|`dd/MMM/YYYY:HH:mm:ss Z`

|===
+
*GeoIP*
+
|===
|*Field* |*Value*

|Field
|`clientip`

|===
+
*User agent*
+
|===
|*Field* |*Value*

|Field
|`agent`

|===
+
You now have defined 4 {ref}/ingest-processors.html[processors] that run sequentially:
{ref}/grok-processor.html[grok], {ref}/date-processor.html[date],
{ref}/geoip-processor.html[geoip], and {ref}/user-agent-processor.html[user_agent].
Your form should look similar to this:
Copy link
Contributor

@debadair debadair Oct 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd move this before the part about them running sequentially.

+
[role="screenshot"]
image:management/ingest-pipelines/images/ingest-pipeline-processor.png["Processors for Ingest Node Pipelines"]

. To verify that the pipeline gives the expected outcome, click *Test pipeline*.
. To verify that the pipeline gives the expected outcome, click *Add documents*.

. In the *Document* tab, provide the following sample document for testing:
. In the *Documents* tab, provide the following sample document for testing:
+
[source,js]
----------------------------------
Expand Down