Monitoring: Transform script in elasticsearch_cluster_status.json results in ArrayOutOfBoundsException

[jdconrad]

With monitoring enabled, when a cluster becomes yellow/red there is the possibility of the transform script in the monitoring watch elasticsearch_cluster_status.json resulting in an ArrayOutOfBoundsException. Two indices are searched against .monitoring-es-* and .monitoring-alerts-6 for information required to use watcher to send an appropriate email related to changes in cluster status. The condition script ensures that the searches contain results prior to being accessed, but the transform script does not. Guards should be added to ensure the exception does not occur in the transform script. However, this is not necessarily simple because without information from the es index it's not possible to know whether the cluster is red/yellow.

This is being seen in production for a number of customers. But it should be noted that if a cluster is yellow/red this is only a symptom and not the actual problem the cluster may be experiencing.

[elasticmachine]

Pinging @elastic/es-core-infra

[nerophon]

I've experienced this bug in a production deployment recently; so it is still current as of 6.5.1. It persisted until the monitoring data was completely removed. It did not appear to be related to the cluster being yellow/red, but the symptom may have been present and simply missed, with the data removal perhaps accidentally bringing the cluster back to green.

Diagnostics available on request, but probably not necessary given the fix would presumably be to add a guard or two.