Cherry-pick #8336 to 6.x: Adding x-pack monitoring code for elasticsearch/ccr metricset

[ycombinator]

Cherry-pick of PR #8336 to 6.x branch. Original message:

This PR teaches the elasticsearch/ccr metricset to index documents into .monitoring-es-6-* indices when xpack.enabled: true is set in modules.d/elasticsearch.yml.

To test this PR

The idea is that metricbeat (specifically the elasticsearch/ccr metricset with xpack.enabled: true) will create exactly the same documents in .monitoring-es-* indices as Elasticsearch's internal collection and reporting does today.

1. Start up Elasticsearch (using the latest build from master).

2. Start up Kibana.

3. Start a trial license from the Kibana Management UI.

4. Enable Monitoring in Elasticsearch (via the cluster setting xpack.monitoring.collection.enabled: true). You can do this via Elasticsearch's Cluster Update Settings API or by clicking the "Turn on Monitoring" button in the Monitoring UI in Kibana.

5. Set up CCR with a leader and follower index.

6. Let Elasticsearch run for ~20 seconds so a few documents are indexed into .monitoring-es-6-*.

7. From .monitoring-es-6-*, retrieve a document for type = ccr_stats

8. Turn off Elasticsearch's internal collection via the cluster setting xpack.monitoring.elasticsearch.collection.enabled: false.

9. Delete .monitoring-es-6-* indices.

10. Enable the elasticsearch module in metricbeat: ./metricbeat modules enable elasticsearch.

11. In modules.d/elasticsearch.yml, add the ccr metricset and set xpack.enabled: true. Concretely, your modules.d/elasticsearch.yml should look something like this:

    - module: elasticsearch
       metricsets:
       - ccr
       period: 10s
       hosts: ["http://localhost:9200"]
       #username: "user"
       #password: "secret"
       xpack.enabled: true

12. Start metricbeat.

13. Let metricbeat run for ~20 seconds so a few documents are indexed into .monitoring-es-*.

14. Stop metricbeat

15. From .monitoring-es-6-*, retrieve a document for type = ccr_stats

16. Using a tool such as http://www.jsondiff.com, compare the documents indexed by Elasticsearch with those indexed by metricbeat. Verify that their structures are identical (same fields, not necessarily same values), except for these known and expected differences:

    1. Only Metricbeat-indexed documents are expected to contain the fields @timestamp, beat, host, and metricset. These are "standard" fields added by beats and metricbeat and don't have an adverse impact since they are additive.
    2. Only Elasticsearch-indexed documents are expected to contain the field source_node. This field is used for debugging purposes only and not actually consumed by either the Monitoring UI or Telemetry feature in Kibana.

[ycombinator]

jenkins, test this please