[New-Platform][Discuss] Elasticsearch connection availability

[rudolf]

Updated based on discussions: 19 November 2019

The legacy elasticsearch plugin included a health check which periodically checked the following:

• The configured elasticsearch cluster is available
• The configured elasticsearch cluster is compatible with Kibana (Semver check) (Add Elasticsearch version check to New Platform #49785)

If any of these failed the elasticsearch plugin would go into a "red" state signaling to system administrators that Kibana is degraded and that intervention is required.

When there's ES connection or version mismatch problems we can should:

1. Expose errors to plugins directly (i.e. plugins are expected to gracefully handle ES Connection errors)
2. Use the status service to signal a failure in core's elasticsearch subservice (blocked by #Migrate status service and status page to New Platform #41983). This will:
   • inform end-users using the Kibana UI that Kibana is in a degraded state and they need to contact their "system administrators".
   • inform system administrators through an error log on the Kibana server.

[elasticmachine]

Pinging @elastic/kibana-platform

[rudolf]

In the event of (sustained) ES connectivity problems, this is the only course of action that I can think of that either Core or a Plugin author could take:

1. Inform end-users using the Kibana UI that they need to contact their "system administrators".
2. Inform system administrators through an error log on the Kibana server.
3. Don't accept any new requests by blocking the UI and responding with 500 to API requests.
4. Recover with the least amount of disruption to users once connectivity is established again.

Are there instances where a plugin could reasonably continue working or otherwise gracefully handle such a failure?

Unless we can come up with good examples of where plugins would benefit through having control over how they react to connectivity problems, it feels like we should go with option (2) and treat this as a core concern.

[rudolf]

Related to #14163

[joshdover]

I disagree that we should buffer all ES requests. I think individual plugins will need different behaviors and we shouldn't try to anticipate what should happen.

For example, the Task Manager plugin coordinates jobs across Kibana instances. We could potentially run a job more than once if we started buffering ES requests on a node that got disconnected while another healthy node picks up the job.

I think having a global UI, server error, etc. makes sense, but I also think that any plugin that is doing some background work with ES should be handling the possibility that ES becomes unavailable down themselves.

Maybe with the TaskService (#18854) this behavior would change by stopping background tasks from running. But until plugins don't have direct access to Elasticsearch clients in their lifecycle methods, I don't think we should make assumptions about the requests that plugins are making in the background.

[legrego]

I'm working on migrating the spaces server code to a NP plugin (#46181), which led me to this topic.

On startup, the Spaces plugin needs to create the Default space if it doesn't already exist. Before attempting this operation, the plugin waits for the following:

1. xpack_main plugin goes green
2. License information is available

1 is only met when the legacy ES Plugin goes green (valid connection, ES version etc):

kibana/x-pack/legacy/plugins/xpack_main/index.js

Line 87 in 9d69b72

mirrorPluginStatus(server.plugins.elasticsearch, this, 'yellow', 'red');

The Spaces plugin will not "go green" until the Default space has been properly initialized. I'm trying to replicate this behavior in the NP version, but we don't have the notion of ES Availability at this point. I could infer that ES is available if I get a valid license back from the Licensing plugin, but that's not performing a version check as far as I can tell. This is also just a point-in-time connection check too -- if the connection goes down before the default space is created, the Spaces plugin needs to know when to retry its operation before completing its setup (or maybe start) sequence.

[legrego]

I think having a global UI, server error, etc. makes sense, but I also think that any plugin that is doing some background work with ES should be handling the possibility that ES becomes unavailable down themselves.

++ I agree a global UI makes sense, and plugins certainly need to handle error conditions when trying to communicate with ES. I think it'd be helpful for core to be able to inform plugins whenever ES becomes available. We previously had this via the LP healthcheck, and it looks like a number of plugins are currently relying on this behavior (via await waitUntilReady())

I also think core should prevent plugins from making requests to ES if it knows that Kibana is connected to an unsupported version of ES.