index health in index_stats documents is not aligned with Elasticsearch

[gbanasiak]

Metricbeat reports index health in index_stats documents only based on the availability of primary and replica shards (code). Specifically, if primary shards are not active, index is considered RED.

Elasticsearch is more nuanced, e.g. if primary shards are not active during index creation, snapshot restore or index shrinking, index health can still be declared YELLOW (code).

Here's an example of index_stats document for .monitoring-beats-7-2021.03.14 index where index health was incorrectly reported as RED. The .monitoring-beats-7-2021.03.14 index was being created at the time, so Elasticsearch was reporting YELLOW.

{
  "_index" : ".monitoring-es-7-mb-2021.03.14",
  "_type" : "_doc",
  "_id" : "nXkILngBKUagi_2-XAXb",
  "_score" : null,
  "_source" : {
    "@timestamp" : "2021-03-14T00:00:02.128Z",
    "cluster_uuid" : "j76z5eyGSkSSOkTD0wbq_Q",
    "timestamp" : "2021-03-14T00:00:02.582Z",
    "index_stats" : {
      "uuid" : "c9I2M7s0QLaqtmKjjPfG4g",
      "index" : ".monitoring-beats-7-2021.03.14",
      "created" : 1.615680002054E12,
      "status" : "red",        <---- HERE
      "hidden" : false,
      "shards" : {
        "total" : 2,
        "unassigned_total" : 1,
        "unassigned_replicas" : 1,
        "initializing" : 1,
        "primaries" : 1,
        "replicas" : 1,
        "active_total" : 0,
        "active_primaries" : 0,
        "active_replicas" : 0,
        "unassigned_primaries" : 0,
        "relocating" : 0
      }
    },
[..]

References:

• discussion when this code was introduced into Metricbeat: Adding x-pack code for elasticsearch/index metricset #8260 (comment)
• unmerged PR for cluster API modification: Add index status to cluster state API response (under metadata metric) elasticsearch#33614

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.