Monitoring UI should support data with no cluster UUID

[ycombinator]

Currently, the Monitoring UI is centered around the notion of one or more Elasticsearch production clusters. Therefore, all documents in .monitoring-* indices require a cluster_uuid field in them. If a document has no cluster_uuid field in it, data from that document simply does not show up anywhere in the Monitoring UI.

This implementation is a historical artifact, before we introduced Logstash or Beats monitoring. Concretely, there are two cases where the current Monitoring UI implementation can cause problems:

1. Sometimes Elastic stack products like Beats or Logstash may not touch (be associated with) any Elasticsearch cluster at all. This means the corresponding documents in .monitoring-beats-* or .monitoring-logstash-* will have no cluster_uuid field.
2. Sometimes Logstash pipelines may touch multiple Elasticsearch clusters.
   This means the corresponding documents in .monitoring-logstash-* would have a multi-value cluster_uuid field. This isn't something we do today but it would be how the data would be represented once we implement this change in Logstash.

Case # 2 can already handled by the current Monitoring UI: the data will show up in multiple clusters, which is accurate (under the current cluster-centric model anyway).

However, there is no way to handle case # 1 in the current Monitoring UI. If a monitoring document has no cluster_uuid field in it, data from that document simply does not show up anywhere in the Monitoring UI.

To handle case #1, per a discussion off-PR, it was decided that we create a new area in the Monitoring UI for "no cluster" monitoring data.

[elasticmachine]

Pinging @elastic/stack-monitoring

[chrisronline]

What should we call this no cluster cluster?

Does the phrase orphan apply here? Does that convey intent well enough? Or maybe something else?

[ycombinator]

Ahhh... naming 😄

I'm not sure about orphan because it implies that the default state is to have a cluster, which is a concept I think we want to move away from.

Putting aside the implementation complexity for a bit, and thinking purely from an end-user's POV:

• When more than one cluster is present, we show users the Cluster Listing page. On this page currently we have a table of clusters. If we detect data with no cluster_uuid, I think we might want to show a separate panel/callout altogether for the "no cluster" data. Here we could maybe explain this data a bit and link to the Cluster Overview page for this "cluster"?

• For all other pages, how about "no cluster" or "miscellaneous" (although I suppose someone could actually give their ES cluster these names too)? Or maybe we just don't show a name at all — i.e. leave that space blank?

Maybe @snide @gchaps @yaronp68 @monicasarbu @pickypg have some ideas?

[pickypg]

Perhaps we could end up converging on some terminology that Cloud uses, which monitoring does not: Deployments.

Deployments represent Elasticsearch clusters, as well as whatever else Cloud happens to be running in the same logical, uh, deployment. That matches up pretty well with our current "Cluster Overview" screen where we're really showing entire, linked deployment.

If there's buy in on that idea, then I would suggest "Unlinked Deployments" or "Independent Instances" to bucket everything else (anything without a cluster_uuid).

[ycombinator]

Big +1 to "Deployments" from me. I was going to suggest this as well but I know there was a lot of debate when we came up with "Clusters" originally. Of course, that was before Cloud came up with Deployments 😄.

[chrisronline]

I don't disagree but we should be consistent with the name and start showing Deployments across the entire monitoring UI? I guess we can start with naming this Deployments and make a ticket to update the entire UI?

[ycombinator]

++ to separate issue/PR for the rename (assuming everyone is on board with it). This issue is more about the "no cluster/deployment UI".