Provide a Kibana Stats API that can be consumed by a Kibana Metricbeat metricset giving 1:1 parity with current Kibana Monitoring data

[tsullivan]

Ingest Team wants to be able to build a Kibana Metricbeat metricset that can call a Kibana HTTP API and receive the same data that gets stored in the .monitoring-kibana-* indices as it exists today.

The data stored in .monitoring-kibana-* comes from multiple sources, and gets compiled into 2 types:

• kibana_stats:
  • Kibana ops stats server uptime, system load, memory, etc *1
  • Kibana usage stats: stats from saved objects: number of Kibana dashboards, saved searches, visualizations, index patterns, Timelion worksheets, Graph Worksheets *2
  • Reporting usage stats: number of generated reports in the .reporting-* history *3
• kibana_settings:
  • Advanced Settings: admin email address, which is needed for Monitoring cluster alerts. This particular setting might not be needed for long, but other settings might come in later *4

The first milestone for the Metricbeat metricset will be to index Kibana-related data in Elasticsearch in the same format as it has been collected internally by Kibana. This will ensure that the current Monitoring UI and Telemetry functionality continue to work as-is without requiring any code changes.

The data includes information about X-Pack plugins (Graph and Reporting) and therefore the API should be implemented in X-Pack Main rather than OSS (core Kibana). The module will gather the same data in the same format that BulkUploader currently produces in the kibana_monitoring module of the Monitoring x-pack plugin.

There will not be a usage or settings API in core Kibana. There will continue to be a stats API in core Kibana featuring the "Kibana ops stats" provided for health checking the Kibana server.

*1 x-pack/plugins/monitoring/server/kibana_monitoring/collectors/get_ops_stats_collector.js
*2 x-pack/plugins/monitoring/server/kibana_monitoring/collectors/get_kibana_usage_collector.js
*3 x-pack/plugins/reporting/server/usage/get_reporting_usage_collector.js
*4 x-pack/plugins/monitoring/server/kibana_monitoring/collectors/get_settings_collector.js

Steps:

6.4.0:

• Move Usage service out of Monitoring into Kibana core: [Monitoring] Move Usage classes for Collector and CollectorSet to OSS #19534
• [ ] Make the “collector types combiner” function (used by the Monitoring Bulk Uploader) re-usable for the new API response
• Ensure that the “new” Stats API represents all kibana_stats data in its response Stats API: implement the "kibana status" spec from the Monitoring data model for stats #20577
• Remove previous (undocumented) HTTP APIs set up for usage collection: Remove X-Pack Usage API endpoint #20800

6.5.0:

• Add a Settings API for kibana_settings documents [Monitoring] Kibana settings api #21100 [Monitoring] Ensure the settings api always return the xpack/default_admin_email #22220
• [ ] Call the collectors’ fetch methods in collectorSet with a savedObjectsClient and use that wherever possible.

[ycombinator]

FYI: @ruflin @chrisronline @pickypg This issue came about after @tsullivan and I had a chat over Zoom about the best path forward to take the first baby step towards Metricbeat monitoring Kibana.

Just to provide a bit more context around the usage of this API:

Once the HTTP API described by this issue is in place, I was thinking of creating an xpack_stats metricset in the Kibana module in Metricbeat that will call this API and ship the resulting data (after some parsing/normalization if necessary) directly into .monitoring-kibana-mb-%{YYYY-MM-DD} indices using the regular bulk API in ES.

At a later point in time, we can talk about and implement things like: no longer shipping settings (or at least ones requires by Cluster Alerts today, once that functionality moves completely into Kibana), changing the structure of documents to match ECS, changing index names, etc.

For now the goal is simply to change the shipping path from custom Kibana code + custom Monitoring bulk endpoint in ES to Metricbeat + regular bulk endpoint in ES.

[ruflin]

@ycombinator I would prefer not to introduce any xpack specific metricsets but use the stats metricset with the xpack.enabled: true config option as we do for Elasticsearch.

[ycombinator]

Sounds good @ruflin.

[tsullivan]

I would prefer not to introduce any xpack specific metricsets but use the stats metricset with the xpack.enabled: true config option as we do for Elasticsearch.

That should be fine and doesn't impact the idea to have a single stats metricset to mirror the current data model for the kibana_stats documents in .monitoring-kibana-* index data.

[ycombinator]

@tsullivan @chrisronline I believe this issue can now be closed (after checking off the rest of the items in the description)?