[Monitoring] Update /api/stats to use core.metrics

[chrisronline]

Relates to #56675

After migration the Stack Monitoring server to new platform, we had a set of parity tests fails: https://internal-ci.elastic.co/job/elastic+estf-monitoring-snapshots+master+multijob-kibana/157/console

This is because internal collection (of monitoring data) goes through the monitoring code base, which was updated to use the new core.metrics API. Metricbeat collection goes through the /api/stats endpoint, which still exists in legacy.

This PR updates the /api/stats endpoint to use the new core.metrics API.

It also fixes some other issues with parity between Metricbeat collection of kibana_stats and kibana_settings documents.

kibana_stats

Internal

{
  "kibana" : {
    "uuid" : "5b2de169-2785-441b-ae8c-186a1936b17d",
    "name" : "cbr-mbp.lan",
    "index" : ".kibana",
    "host" : "localhost",
    "transport_address" : "localhost:5601",
    "version" : "8.0.0",
    "snapshot" : false,
    "status" : "green"
  },
  "concurrent_connections" : 22,
  "os" : {
    "load" : {
      "1m" : 4.5205078125,
      "5m" : 4.1416015625,
      "15m" : 3.77978515625
    },
    "memory" : {
      "total_in_bytes" : 34359738368,
      "free_in_bytes" : 126353408,
      "used_in_bytes" : 34233384960
    },
    "uptime_in_millis" : 2553488000,
    "platform" : "darwin",
    "platformRelease" : "darwin-19.3.0"
  },
  "process" : {
    "event_loop_delay" : 0.9769420623779297,
    "memory" : {
      "heap" : {
        "total_in_bytes" : 525910016,
        "used_in_bytes" : 326812800,
        "size_limit" : 8564343922
      },
      "resident_set_size_in_bytes" : 1026121728
    },
    "uptime_in_millis" : 31415
  },
  "requests" : {
    "disconnects" : 0,
    "total" : 16
  },
  "response_times" : {
    "average" : 148.5,
    "max" : 517
  },
  "timestamp" : "2020-03-24T01:27:29.100Z"
}

Metricbeat

{
  "process" : {
    "event_loop_delay" : 0.07048416137695312,
    "memory" : {
      "resident_set_size_in_bytes" : 1026928640,
      "heap" : {
        "total_in_bytes" : 526434304,
        "used_in_bytes" : 332255512,
        "size_limit" : 8564343922
      }
    },
    "uptime_in_millis" : 33375
  },
  "requests" : {
    "disconnects" : 0,
    "total" : 13
  },
  "response_times" : {
    "average" : 67,
    "max" : 132
  },
  "kibana" : {
    "snapshot" : false,
    "status" : "green",
    "uuid" : "5b2de169-2785-441b-ae8c-186a1936b17d",
    "name" : "cbr-mbp.lan",
    "index" : ".kibana",
    "host" : "localhost",
    "transport_address" : "localhost:5601",
    "version" : "8.0.0"
  },
  "timestamp" : "2020-03-24T01:27:32.235Z",
  "concurrent_connections" : 5,
  "os" : {
    "platformRelease" : "darwin-19.3.0",
    "load" : {
      "15m" : 3.77978515625,
      "1m" : 4.5205078125,
      "5m" : 4.1416015625
    },
    "memory" : {
      "free_in_bytes" : 114679808,
      "used_in_bytes" : 34245058560,
      "total_in_bytes" : 34359738368
    },
    "uptime_in_millis" : 2553490000,
    "platform" : "darwin"
  }
}

kibana_settings

Internal

{
  "kibana_settings": {
    "kibana": {
      "uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
      "name": "cbr-mbp.lan",
      "index": ".kibana",
      "port" : 5601,
      "host": "localhost",
      "transport_address": "localhost:5601",
      "version": "8.0.0",
      "snapshot": false,
      "status": "green",
      "locale": "en"
    },
    "xpack": {
      "default_admin_email": null
    }
  }
}

Metricbeat

{
  "kibana_settings": {
    "kibana" : {
      "snapshot" : false,
      "name" : "cbr-mbp.lan",
      "host" : "localhost",
      "index" : ".kibana",
      "port" : 5601,
      "locale" : "en",
      "transport_address" : "localhost:5601",
      "version" : "8.0.0",
      "uuid" : "5b2de169-2785-441b-ae8c-186a1936b17d",
      "status" : "green"
    },
    "xpack" : { }
  }
}

(env) cbr-mbp:kibana chris$ python docs_compare.py ../../../ait_workspace/monitoring/kibana/internal ../../../ait_workspace/monitoring/kibana/metricbeat
OK: Metricbeat-indexed doc for type='kibana_stats' has expected parity with internally-indexed doc.
OK: Metricbeat-indexed doc for type='kibana_settings' has expected parity with internally-indexed doc.

I ran the parity tests with a locally built version of Kibana from this PR and they are passing!

OK: Metricbeat-indexed doc for type='kibana_stats' has expected parity with internally-indexed doc.
OK: Metricbeat-indexed doc for type='kibana_settings' has expected parity with internally-indexed doc.

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[igoristic]

nit: I don't think we need to use cloneDeep here (unless metrics is somehow a reference), and looks like _metrics is not used anywhere else. Don't know if Observables hold references though, in which case maybe shallow might be good enough?:
metrics$.subscribe(({ ...metrics }) => {

Only mentioning because I know cloneDeep can be pretty expensive

[chrisronline]

Yea I tried it without it and any delete or re-setting the object caused issues with the other subscriber.

[igoristic]

Looks good 👍

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 3cd0c21

History

• 💚 Build #35901 succeeded 2ce5c74
• 💔 Build #35816 failed a3fff65
• 💚 Build #35812 succeeded 40e7366
• 💛 Build #35678 was flaky 143ce88
• 💚 Build #35575 succeeded 25ef986

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[chrisronline]

Backport:

7.x: e439836