Add support for clustering Logstash instances

[suyograo]

Today, each Logstash instance is a full pipeline -- inputs, filters and outputs stages. In large-scale Logstash deployments, users run multiple instances of Logstash in order to horizontally scale event processing. This requires manual management of individual configuration files, or custom/3rd party configuration automation tools such as Puppet or Chef.

We plan to introduce a concept of a Logstash cluster, where instances can be controlled as a whole (on a cluster level), instead of being separate parts. This would entail the following features:

1. Provide an option to centrally store a Logstash config, which is shared across all the instances in the cluster. This would be the single source of truth for all the instances
2. Provide APIs to control the cluster, dynamically, to change configuration. See Provide APIs to manage pipeline  #2612
3. Provide APIs to monitor instances at the cluster level. See Provide APIs to monitor pipeline  #2611

Logstash can still be started in a single-instance, non-clustered mode; file based configuration will continue to work.

Clustering instances will also provide the necessary groundwork for potential long-term enhancements like automatic load balancing, failover, running multiple pipelines and so on.

[bitsofinfo]

For item 1 above, maybe design it so that everything talks through an "ConfigurationStore" abstraction which is implemented via plugins? I.e. so it could be extensible, supporting different implementations of where the "gold copy" of configuration is actually persisted and changes propagated to/from. Have different impls (i.e. ES itself, zookeeper etc)

[wiibaa]

@suyograo should this ticket also mention plan for encryption support or should it be done separately ?
Some background in:
https://logstash.jira.com/browse/LOGSTASH-428
https://logstash.jira.com/browse/LOGSTASH-918

[webmstr]

What about items that are cached in LS instance now, like the data for the elapsed{} filter?

[suyograo]

@bitsofinfo thats exactly our thinking...we'll make it pluggable, so its easy to add in an alternate implementation for a config store. The first implementation will use ES as a config store.