[ML] Adds ML jobs for access logs to Nginx package

[peteharverson]

What does this PR do?

Adds an ML module containing anomaly detection jobs for finding unusual activity in HTTP access logs to the Nginx integration. Requires Kibana 7.13.0 or later.

These are the same five jobs that have previously been stored inside the ML Kibana plugin:

• Detect unusual visitor rates
• Detect unusual status code rates
• Detect unusual source IPs - high distinct count of URLs
• Detect unusual source IPs - high request rate
• Detect low request rates

Some minor edits have been made to the previous job configurations stored in the ML Kibana plugin:

• ID of the module is ngix_data_stream compared to nginx_ecs for the legacy module
• Module and datafeed queries use data_stream.dataset: nginx.access compared to event.dataset: nginx.access for the legacy module
• The ML module no longer adds its own Kibana dashboard, but instead links to the Nginx logs overview dashboard which is already included in the Nginx package.
• The suffix (ECS) has been removed from the module and job descriptions
• _nginx is appended to the IDs of the jobs in the module
• The created_by property used for telemetry is set to ml-module-nginx-access-data-stream compared to ml-module-nginx-access for the legacy module

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.

How to test this PR locally

To test this PR:

• Kibana 7.13 or later is required
• the Nginx package will first need to be added / updated from the Fleet Integrations page, or install the assets from the Settings tab for the Nginx package in Fleet.
• Have an nginx data stream configured in Fleet, so that there is data in an index, such as logs-*, matching the query in the ML module JSON file:

    "query": {
      "bool": {
        "filter": [
          {
            "term": {
              "data_stream.dataset": "nginx.access"
            }
          },
          {
            "exists": {
              "field": "source.address"
            }
          },
          {
            "exists": {
              "field": "url.original"
            }
          },
          {
            "exists": {
              "field": "http.response.status_code"
            }
          }
        ]
      }
    },

• Go to the ML plugin in Kibana, and create a job, selecting the appropriate index (such as logs-*) and select the card for this new Nginx access logs module:
  

• Create and run the jobs from the ML job wizard

• Test that the custom URLs to the Nginx logs overview dashboard and the raw data in Discover work

Related issues

elastic/package-spec#148

Screenshots

ML module is now listed in the Kibana assets section for the Nginx package:

List of Nginx jobs in the ML Job list:

Screenshot showing results of Nginx ML jobs in the ML Anomaly Explorer:

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #912 updated

• Start Time: 2021-04-19T08:47:05.693+0000

• Duration: 14 min 32 sec

• Commit: 758710b

Test stats 🧪

Test	Results
Failed	0
Passed	39
Skipped	0
Total	39

Trends 🧪

[mtojek]

Please rebase it against master as I pushed fix for the missing spec (ML modules).

[ruflin]

I'm really exited to see this. Have been waiting for this moment quite some time: elastic/package-spec#30 !