[Monitoring/Beats] Telemetry Data from Beats

[tsullivan]

This adds anonymous Beats statistics found in the Monitoring data to the telemetry payload.

To test:

1. Collect some Beats Monitoring data

2. x-pack$ node scripts/api_debug.js telemetry --host=https://<kibana_host>:5601 --auth=username:password | jq '.[].stack_stats | { beats: .beats }'
   

3. Example output:

   {
     "beats": {
       "count": 2,
       "versions": {
         "7.0.0-alpha1": 2
       },
       "types": {
         "metricbeat": 1,
         "filebeat": 1
       },
       "outputs": {
         "elasticsearch": 2
       },
       "eventsPublished": 1020,
       "hosts": 1
     }
   }
   

NOTE: module stats are still in-progress on the Beats collection side

[tsullivan]

NOTE: it's possible to filter apm-server out of the search, but I figure it's useful for telemetry

[pickypg]

We should probably filter it out since APM Server isn't really supported by monitoring and it will almost certainly have different metrics. If we change our minds later, it's easier to remove the filter than to add it and adjust everything.

[tsullivan]

6d010fb - I just added the filter to the query, not updated the mock data for the unit test

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[pickypg]

The code looks great, but I think we can optimize for the expensive scenarios.

[pickypg]

We should probably filter it out since APM Server isn't really supported by monitoring and it will almost certainly have different metrics. If we change our minds later, it's easier to remove the filter than to add it and adjust everything.

[pickypg]

When there is a UI involved, the easiest way to determine if you need to get the next page when you cannot trust the total size is to over-request. So in the request, request HITS_SIZE + 1. Then just process HITS_SIZE and, if the + 1 exists, then request the next page and repeat.

But in this case, there's no extra pain because either way we're stuck requesting that next page -- it may come back empty though. We should default this value to 0 and ignore the whole results payload in that case.

[tsullivan]

1673cfd

[pickypg]

We should pre-process each payload before saving it to minimize the amount of data that we maintain in the heap during this process. Otherwise we will have every unique instance sitting in heap until the end.

Perhaps handleBeatsResultSet could be augmented to have the clusters object passed in so that it can just pickup where it left off while also only expecting a single result set instead of the paged array?

[tsullivan]

Sounds good. Are you thinking we pass in results from a single query as well as the clusters object to handleBeatsResultSet, which mutates clusters to have new processed stat data? The object mutation wouldn't worry me since the object's lifecycle exists just in this lib file.

[pickypg]

Exactly!

[tsullivan]

4ba3493

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[pickypg]

LGTM!

[ruflin]

The issue to track the progress on getting these metrics from Beats is here: elastic/beats#7027

[tsullivan]

Thank you!

[tsullivan]

6.x/6.4: #18922