[Stack Monitoring] Deprecate internal collection topics

[ycombinator]

We now have the ability to monitor Elasticsearch, Kibana, Logstash, Beats, and APM Server with Metricbeat. The previous method of monitoring stack products, using collection code internal to each product that shipped monitoring data to a custom Monitoring Bulk API endpoint, is now deprecated.

We should reflect this deprecation in documentation as well, specifically in these areas:

• Stack docs: https://www.elastic.co/guide/en/elastic-stack-overview/current/xpack-monitoring.html
• Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/monitor-elasticsearch-cluster.html
• Kibana: https://www.elastic.co/guide/en/kibana/current/configuring-monitoring.html
• Beats: https://www.elastic.co/guide/en/beats/heartbeat/current/monitoring.html (Using Heartbeat as a sample here, but applicable to all Beats and APM Server documentation)
• Logstash: TBD; PR introducing Metricbeat collection is currently WIP: [Doc]Add metricbeat as monitoring option logstash#11033

[cachedout]

Hi @lcawl. I've started some of this:

Beats: elastic/beats#14887
Elasticsearch: elastic/elasticsearch#49758

[cachedout]

Elasticsearch done: elastic/elasticsearch#49758

[insukcho]

Just want to make sure that Beat also has the same policy. So, we need to use separate Metricbeat to monitor Metricbeat, right?

If yes, we should change the documentation content because users could be misleading that we recommend using deprecated internal collection:

The benefit of using internal collection instead of Metricbeat is that you have fewer pieces of software to install and maintain.

[consulthys]

Agree with @insukcho as we (Stack Monitoring) start getting some reports (e.g. https://github.com/elastic/sdh-beats/issues/5232) that Beats internal monitoring doesn't show deprecated in the docs.

I see there was an initial attempt here by @cachedout but it was closed. How can we make it happen? Thanks!

[consulthys]

The last comment in that PR intrigues me.

From exchanges I've had with other folks, I don't think we want to recommend Metricbeat for all use cases.

@dedemorton could you provide some context concerning the discussions you had and how we should progress? Thanks a lot

[dedemorton]

@consulthys I'm sorry, it's been too long since I made that comment, and I can't remember the details. Plus Metricbeat has evolved so much over the years that whatever concerns were raised back then are probably no longer valid. I have not worked in that area for quite awhile, so I cannot provide any more info.

[consulthys]

@dedemorton thank you so much for your quick answer!
No worries, we'll figure it out eventually.