Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
153 changes: 128 additions & 25 deletions docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-apache.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,49 +5,152 @@
<titleabbrev>Apache</titleabbrev>
++++
// tag::apache-jobs[]
These {anomaly-job} wizards appear in {kib} if you use
{filebeat-ref}/index.html[{filebeat}] to ship access logs from your
https://httpd.apache.org/[Apache] HTTP servers to {es} and store it using fields
and data types from the Elastic Common Schema (ECS). For more details, see the
{dfeed} and job definitions in
These {anomaly-job} wizards appear in {kib} if you use the Apache integration in
{fleet} or you use {filebeat} to ship access logs from your
https://httpd.apache.org/[Apache] HTTP servers to {es}. The jobs assume that you
use fields and data types from the Elastic Common Schema (ECS).

[[apache-access-logs]]
== Apache access logs

These {anomaly-jobs} find unusual activity in HTTP access logs.

For more details, see the {dfeed} and job definitions in
https://github.com/elastic/integrations/blob/{branch}/packages/apache/kibana/ml_module/apache-Logs-ml.json[GitHub].
Note that these jobs are available in {kib} only if data exists that matches the
{dfeed} query.

low_request_rate_apache::
Detects low request rates.

Job details:::

* Analyzes request rates (using the <<ml-count,`low_count` function>>).

Required {beats} or {agent} integrations:::

* Apache integration

source_ip_request_rate_apache::
Detects unusual source IPs.

Job details:::

* Analyzes request rates (using the <<ml-count,`high_count` function>>)
relative to all the source IPs (`over_field_name` is `source.address`).

Required {beats} or {agent} integrations:::

* Apache integration

source_ip_url_count_apache::
Detects unusual source IPs.

Job details:::

* Analyzes distinct counts of URLs (using the
<<ml-distinct-count,`high_distinct_count` function>> on the `url.original`
field) relative to all the source IPs (`over_field_name` is `source.address`).

Required {beats} or {agent} integrations:::

* Apache integration

status_code_rate_apache::
Detects unusual status code rates.

Job details:::

* Analyzes request rates (using the <<ml-count,`count` function>>) split by
status code (`partition_field_name` is `http.response.status_code`).

Required {beats} or {agent} integrations:::

* Apache integration

visitor_rate_apache::
Detects unusual visitor rates.

Job details:::

* Analyzes request rates using the <<ml-nonzero-count,`non_zero_count` function>>.

Required {beats} or {agent} integrations:::

* Apache integration

[[apache-access-logs-filebeat]]
== Apache access logs ({filebeat})

These legacy {anomaly-jobs} find unusual activity in HTTP access logs. For the
latest versions, install the Apache integration in {fleet}; see
<<apache-access-logs>>.

For more details, see the {dfeed} and job definitions in
https://github.com/elastic/kibana/tree/{branch}/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/ml[GitHub].

These configurations are only available if data exists that matches the
recognizer query specified in the
https://github.com/elastic/kibana/blob/{branch}/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/manifest.json#L8[manifest file].

low_request_rate_ecs::
Detects low request rates.

Job details:::

* For HTTP web access logs where `event.dataset` is `apache.access`.
* Models the event rate of HTTP requests.
* Detects unusually low counts of HTTP requests compared to the previous event
rate.
* Analyzes request rates (using the <<ml-count,`low_count` function>>).

Required {beats} or {agent} integrations:::

* {filebeat}

source_ip_request_rate_ecs::
Detects unusual source IPs.

Job details:::

* For HTTP web access logs where `event.dataset` is `apache.access`.
* Models the event rate of HTTP requests by source IP.
* Detects source IPs with unusually high request rates in the HTTP access log
compared to the previous rate.
* Analyzes request rates (using the <<ml-count,`high_count` function>>)
relative to all the source IPs (`over_field_name` is `source.address`).

Required {beats} or {agent} integrations:::

* {filebeat}

source_ip_url_count_ecs::
Detects unusal source IPs.

Job details:::

* Analyzes distinct counts of URLs (using the
<<ml-distinct-count,`high_distinct_count` function>> on the `url.original`
field) relative to all the source IPs (`over_field_name` is `source.address`).

* For HTTP web access logs where `event.dataset` is `apache.access`.
* Models the event rate of HTTP requests by source IP.
* Detects source IPs with unusually high distinct count of URLs in the HTTP
access log.
Required {beats} or {agent} integrations:::

* {filebeat}

status_code_rate_ecs::

* For HTTP web access logs where `event.dataset` is `apache.access`.
* Models the occurrences of HTTP response status codes.
* Detects unusual status code rates in the HTTP access log compared to previous
rates.
Detects unusual status code rates.

Job details:::

* Analyzes request rates (using the <<ml-count,`count` function>>) split by
status code (`partition_field_name` is `http.response.status_code`).

Required {beats} or {agent} integrations:::

* {filebeat}

visitor_rate_ecs::
Detects unusual visitor rates.

Job details:::

* Analyzes request rates using the <<ml-nonzero-count,`non_zero_count` function>>.

Required {beats} or {agent} integrations:::

* {filebeat}

* For HTTP web access logs where `event.dataset` is `apache.access`.
* Models visitor rates.
* Detects unusual visitor rates in the HTTP access log compared to previous
rates.
// end::apache-jobs[]
153 changes: 128 additions & 25 deletions docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-nginx.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,89 @@
++++

// tag::nginx-jobs[]
These {anomaly-job} wizards appear in {kib} if you use {filebeat} to ship access
logs from your http://nginx.org/[Nginx] HTTP servers to {es} and store it using
fields and datatypes from the Elastic Common Schema (ECS). For more details, see
the {dfeed} and job definitions in

These {anomaly-job} wizards appear in {kib} if you use the Nginx integration in
{fleet} or you use {filebeat} to ship access logs from your
http://nginx.org/[Nginx] HTTP servers to {es}. The jobs assume that you use
fields and data types from the Elastic Common Schema (ECS).

[[nginx-access-logs]]
== Nginx access logs

These {anomaly-jobs} find unusual activity in HTTP access logs.

For more details, see the {dfeed} and job definitions in
https://github.com/elastic/integrations/blob/{branch}/packages/nginx/kibana/ml_module/nginx-Logs-ml.json[GitHub].
Note that these jobs are available in {kib} only if data exists that matches the
{dfeed} query.

low_request_rate_nginx::
Detects low request rates.

Job details:::

* Analyzes request rates (using the <<ml-count,`low_count` function>>).

Required {beats} or {agent} integrations:::

* Nginx integration

source_ip_request_rate_nginx::
Detects unusual source IPs.

Job details:::

* Analyzes request rates (using the <<ml-count,`high_count` function>>)
relative to all the source IPs (`over_field_name` is `source.address`).

Required {beats} or {agent} integrations:::

* Nginx integration

source_ip_url_count_nginx::
Detects unusual source IPs.

Job details:::

* Analyzes distinct counts of URLs (using the
<<ml-distinct-count,`high_distinct_count` function>> on the `url.original`
field) relative to all the source IPs (`over_field_name` is `source.address`).

Required {beats} or {agent} integrations:::

* Nginx integration

status_code_rate_nginx::
Detects unusual status code rates.

Job details:::

* Analyzes request rates (using the <<ml-count,`count` function>>) split by
status code (`partition_field_name` is `http.response.status_code`).

Required {beats} or {agent} integrations:::

* Nginx integration

visitor_rate_nginx::
Detects unusual visitor rates.

Job details:::

* Analyzes request rates using the <<ml-nonzero-count,`non_zero_count` function>>.

Required {beats} or {agent} integrations:::

* Nginx integration

[[nginx-access-logs-filebeat]]
== Nginx access logs ({filebeat})

These legacy {anomaly-jobs} find unusual activity in HTTP access logs. For the
latest versions, install the Nginx integration in {fleet}; see
<<nginx-access-logs>>.

For more details, see the {dfeed} and job definitions in
https://github.com/elastic/kibana/tree/{branch}/x-pack/plugins/ml/server/models/data_recognizer/modules/nginx_ecs/ml[GitHub].

These configurations are only available if data exists that matches the
Expand All @@ -18,38 +97,62 @@ https://github.com/elastic/kibana/blob/{branch}/x-pack/plugins/ml/server/models/


low_request_rate_ecs::
Detects low request rates.

Job details:::

* For HTTP web access logs where `event.dataset` is `nginx.access`.
* Models the event rate of http requests.
* Detects unusually low counts of HTTP requests compared to the previous event
rate.
* Analyzes request rates (using the <<ml-count,`low_count` function>>).

Required {beats} or {agent} integrations:::

* {filebeat}

source_ip_request_rate_ecs::
Detects unusual source IPs.

Job details:::

* Analyzes request rates (using the <<ml-count,`high_count` function>>)
relative to all the source IPs (`over_field_name` is `source.address`).

* For HTTP web access logs where `event.dataset` is `nginx.access`.
* Models the event rate of HTTP requests by source IP.
* Detects source IPs with unusually high request rates in the HTTP access log
compared to the previous rate.
Required {beats} or {agent} integrations:::

* {filebeat}

source_ip_url_count_ecs::
Detects unusual source IPs.

Job details:::

* Analyzes distinct counts of URLs (using the
<<ml-distinct-count,`high_distinct_count` function>> on the `url.original`
field) relative to all the source IPs (`over_field_name` is `source.address`).

Required {beats} or {agent} integrations:::

* For HTTP web access logs where `event.dataset` is `nginx.access`.
* Models the event rate of HTTP requests by source IP.
* Detects source IPs with unusually high distinct count of URLs in the HTTP
access log.
* {filebeat}

status_code_rate_ecs::
Detects unusual status code rates.

* For HTTP web access logs where `event.dataset` is `nginx.access`.
* Models the occurrences of HTTP response status codes.
* Detects unusual status code rates in the HTTP access log compared to previous
rates.
Job details:::

* Analyzes request rates (using the <<ml-count,`count` function>>) split by
status code (`partition_field_name` is `http.response.status_code`).

Required {beats} or {agent} integrations:::

* {filebeat}

visitor_rate_ecs::
Detects unusual visitor rates.

Job details:::

* Analyzes request rates using the <<ml-nonzero-count,`non_zero_count` function>>.

Required {beats} or {agent} integrations:::

* For HTTP web access logs where `event.dataset` is `nginx.access`.
* Models visitor rates.
* Detects unusual visitor rates in the HTTP access log compared to previous
rates.
* {filebeat}

// end::nginx-jobs[]
// end::nginx-jobs[]