WIP integrate Elasticfence plugins into ELK 5.3.1 #214

jemacom · 2017-12-20T11:03:21Z

NOTE: This is a WIP PR and it should probably merged in another branch like 5.3.1-elasticfence if it's run properly and you accept to add it in the project of course.

I opened the this PR just to reproduce the errors.

I want to integrate Elasticfence into elk-docker. Since I like the project simplicity and ease of use but it was build on earlier version of ELK stack 5.1.2 and it's now unmaintained so I did some work around to run it on ELK 5.3.1.

Long story short I succeeded to install the required Kibana and Elasticsearch plugin into elk-docker. However I'm facing hard time to make it run properly.

Here's some logs:

elasticsearch_1  | [2017-12-20T10:10:37,394][INFO ][o.e.t.TransportService   ] [WVnkU0c] publish_address {172.19.0.2:9300}, bound_addresses {[::]:9300}
elasticsearch_1  | [2017-12-20T10:10:37,408][INFO ][o.e.b.BootstrapChecks    ] [WVnkU0c] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:39Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"Unable to revive connection: http://0.0.0.0:9200/"}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:39Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"}
elasticsearch_1  | [2017-12-20T10:10:40,572][INFO ][o.e.c.s.ClusterService   ] [WVnkU0c] new_master {WVnkU0c}{WVnkU0chSP2zOlMEXDvk0A}{zkldTANLTOeJiwgQs3jOuA}{172.19.0.2}{172.19.0.2:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
elasticsearch_1  | [2017-12-20T10:10:40,676][INFO ][o.e.h.n.Netty4HttpServerTransport] [WVnkU0c] publish_address {172.19.0.2:9200}, bound_addresses {[::]:9200}
elasticsearch_1  | [2017-12-20T10:10:40,699][INFO ][o.e.n.Node               ] [WVnkU0c] started
elasticsearch_1  | [2017-12-20T10:10:40,884][WARN ][o.e.d.i.m.TypeParsers    ] field [include_in_all] is deprecated, as [_all] is deprecated, and will be disallowed in 6.0, use [copy_to] instead.
elasticsearch_1  | [2017-12-20T10:10:40,909][WARN ][o.e.d.i.m.TypeParsers    ] field [include_in_all] is deprecated, as [_all] is deprecated, and will be disallowed in 6.0, use [copy_to] instead.
elasticsearch_1  | [2017-12-20T10:10:41,021][WARN ][o.e.d.i.m.TypeParsers    ] field [include_in_all] is deprecated, as [_all] is deprecated, and will be disallowed in 6.0, use [copy_to] instead.
elasticsearch_1  | [2017-12-20T10:10:41,029][WARN ][o.e.d.i.m.TypeParsers    ] field [include_in_all] is deprecated, as [_all] is deprecated, and will be disallowed in 6.0, use [copy_to] instead.
elasticsearch_1  | [2017-12-20T10:10:41,375][INFO ][o.e.l.LicenseService     ] [WVnkU0c] license [5598dc7d-68ef-4847-8f44-11f5f2fe6403] mode [trial] - valid
elasticsearch_1  | [2017-12-20T10:10:41,376][INFO ][o.e.g.GatewayService     ] [WVnkU0c] recovered [2] indices into cluster_state
elasticsearch_1  | [2017-12-20T10:10:41,406][WARN ][o.e.d.i.m.TypeParsers    ] field [include_in_all] is deprecated, as [_all] is deprecated, and will be disallowed in 6.0, use [copy_to] instead.
elasticsearch_1  | [2017-12-20T10:10:41,407][WARN ][o.e.d.i.m.TypeParsers    ] field [include_in_all] is deprecated, as [_all] is deprecated, and will be disallowed in 6.0, use [copy_to] instead.
elasticsearch_1  | [2017-12-20T10:10:41,410][WARN ][o.e.d.i.m.TypeParsers    ] field [include_in_all] is deprecated, as [_all] is deprecated, and will be disallowed in 6.0, use [copy_to] instead.
elasticsearch_1  | [2017-12-20T10:10:41,411][WARN ][o.e.d.i.m.TypeParsers    ] field [include_in_all] is deprecated, as [_all] is deprecated, and will be disallowed in 6.0, use [copy_to] instead.
elasticsearch_1  | [2017-12-20T10:10:42,205][INFO ][o.e.c.r.a.AllocationService] [WVnkU0c] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:42Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"Unable to revive connection: http://0.0.0.0:9200/"}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:42Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:44Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"Unable to revive connection: http://0.0.0.0:9200/"}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:44Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"}
logstash_1       | Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:47Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"Unable to revive connection: http://0.0.0.0:9200/"}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:47Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"}
logstash_1       | [2017-12-20T10:10:47,464][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
logstash_1       | [2017-12-20T10:10:47,470][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1       | [2017-12-20T10:10:47,677][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x286f237 URL:http://elasticsearch:9200/>}
logstash_1       | [2017-12-20T10:10:47,678][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
logstash_1       | [2017-12-20T10:10:47,756][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
logstash_1       | [2017-12-20T10:10:47,769][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x2e193eee URL://elasticsearch:9200>]}
logstash_1       | [2017-12-20T10:10:47,777][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
logstash_1       | [2017-12-20T10:10:47,802][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:5000"}
logstash_1       | [2017-12-20T10:10:47,820][INFO ][logstash.pipeline        ] Pipeline main started
logstash_1       | [2017-12-20T10:10:47,897][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:49Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"Unable to revive connection: http://0.0.0.0:9200/"}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:49Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:52Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"Unable to revive connection: http://0.0.0.0:9200/"}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:52Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:54Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"Unable to revive connection: http://0.0.0.0:9200/"}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:54Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:57Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"Unable to revive connection: http://0.0.0.0:9200/"}
kibana_1         | {"type":"log","@timestamp":"2017-12-20T10:10:57Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"}

I have Ubuntu 17.04 on my machine.

$ docker version
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.8.3
 Git commit:   092cba3
 Built:        Thu Oct 12 22:34:44 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   092cba3
 Built:        Thu Oct 12 22:34:44 2017
 OS/Arch:      linux/amd64
 Experimental: false

$ docker-compose version
docker-compose version 1.17.0, build ac53b73
docker-py version: 2.5.1
CPython version: 2.7.13
OpenSSL version: OpenSSL 1.0.1t  3 May 2016

I'm not sure if those errors are related to docker-elk or they're specific to the installed plugins.
Any help on this will be really appreciated. 😉

antoineco · 2017-12-20T11:09:27Z

I think we should keep the ELK components totally vanilla and let people extend them as they want.

Why would I, for example, need that plugin if I don't use it? Maybe I missed the point.

jemacom · 2017-12-20T11:27:16Z

Why would I, for example, need that plugin if I don't use it? Maybe I missed the point.

I don't want to merge this into master of course. I wanna add it to another branch like searchguard one to have Elasticfence out of the box. Hope my efforts to integrate it will be useful for others to work with.

antoineco · 2017-12-20T11:40:22Z

Ok got it. Thanks for clarifying.

Do you think the project is abandoned?

jemacom · 2017-12-20T13:27:02Z

Do you think the project is abandoned?

Apparently yes as the project contributor replied here.

antoineco · 2017-12-20T13:31:43Z

Probably not a good fit then :(
Anything stopping you from using Searchguard?

jemacom · 2017-12-21T17:16:47Z

I kinda figure it out. Elasticfence can't run to later versions than 5.1.2 of elasticsearch since they base all the plugin logic on RestFilter which is deprecated by Elasticsearch people now.

Nevertheless I run it on version 5.1.2 and finalizing the integration.

Do you think it's still worth it to submit a PR on this.

Thanks !!

deviantony · 2017-12-21T17:51:23Z

As @antoineco said, I don't think that it would be interesting to add this into the project as it's now not maintained anymore.

jemacom · 2017-12-21T22:30:07Z

Okay I'll close this PR then ;)

antoineco · 2017-12-21T22:39:56Z

@jemacom thanks anyway for the proposal 👍

jemacom added 3 commits December 18, 2017 11:46

Updates for compatibility with 5.3.1

217a57f

Copy config into containers

216043b

Fix URLs

4d9b613

jemacom closed this Dec 21, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP integrate Elasticfence plugins into ELK 5.3.1 #214

WIP integrate Elasticfence plugins into ELK 5.3.1 #214

jemacom commented Dec 20, 2017

antoineco commented Dec 20, 2017

jemacom commented Dec 20, 2017

antoineco commented Dec 20, 2017

jemacom commented Dec 20, 2017

antoineco commented Dec 20, 2017

jemacom commented Dec 21, 2017

deviantony commented Dec 21, 2017

jemacom commented Dec 21, 2017

antoineco commented Dec 21, 2017

WIP integrate Elasticfence plugins into ELK 5.3.1 #214

WIP integrate Elasticfence plugins into ELK 5.3.1 #214

Conversation

jemacom commented Dec 20, 2017

antoineco commented Dec 20, 2017

jemacom commented Dec 20, 2017

antoineco commented Dec 20, 2017

jemacom commented Dec 20, 2017

antoineco commented Dec 20, 2017

jemacom commented Dec 21, 2017

deviantony commented Dec 21, 2017

jemacom commented Dec 21, 2017

antoineco commented Dec 21, 2017