Add support for AWS Elasticsearch

[jicowan]

Update image with the fluent-plugin-aws-elasticsearch-service gem, https://github.com/atomita/fluent-plugin-aws-elasticsearch-service which allows you to specify an IAM access key/secret key for an IAM account to write to ES.

[hixichen]

not a ruby expert, hope someone can do this and make fluntd to aws es be eaiser.

[aerostitch]

I'm currently working on that part for my current setup.
@repeatedly would you prefer that to be a brand new image or just add it as an option to the existing debian-elasticsearch one?

[hixichen]

Actually, I deployed the aws elasticsearch proxy service inside k8s cluster.
And, use fluent-bit/fluent-d to publish data to proxy service.
I checked the atomita's plugin but feel no one maintain that for a long time.

[aerostitch]

Interesting. Are you talking about https://github.com/abutaha/aws-es-proxy ?

I've been using the atomia plugin for a very long time outside of that project and am currently migrating to using this project, just adding the gem from https://github.com/atomita/fluent-plugin-aws-elasticsearch-service . There's been some activity on it lately thanks to @cosmo0920 .

[hixichen]

@aerostitch
Yes.
container image: abutaha/aws-es-proxy:0.9

[nukepuppy]

@hixichen @aerostitch would love to hear more about the aws-es-proxy and any issues you've run into doing so.

Would ideally be best if the stable helm chart https://github.com/kiwigrid/helm-charts/tree/master/charts/fluentd-elasticsearch ? had a way to do this setup or similar.

But still - would rather hear first hand from others who've done it the issues they've faced.. reliability.. scaling.. gotchas.. etc

[Nuru]

I tried the https://github.com/abutaha/aws-es-proxy but it does not seem ready for production. It has 13 open issues at the moment, 2 of which are deal killers for me: it fails to pass along 403 or 302 responses. When it encounters those responses from upstream (Elasticsearch or Kibana), it sends an empty 200 response downstream. That is a big problem, especially for Kibana.

So I tried using the mozilla-services proxy proxy instead, as it seems to be in much better shape than the other one, is very easy to configure, and the whole Docker image is only 11 MB. Unfortunately, while it handles 403 and 302 status codes fine, it drops the Content-Type header, which makes it completely unsuitable for use with Elasticsearch, which requires the Content-Type header (as of version 6.0).

Until one of these proxies get fixed, or a better one comes along, it would be very helpful to have https://github.com/atomita/fluent-plugin-aws-elasticsearch-service installed and available on the elasticsearch Docker images.

[esteban1983cl]

Hi Team.
Is there solution for this ?
Thanks in advance.

[AdrienKuhn]

See #317

[Dolape1983]

See #317

Thanks bro... I'll test and comment any issue

[kenhys]

As of deployment process was improved, there is a room for newer image build pipeline if user helps continuously maintain the image even though still have #293 issue. if you still have interest to do it, please create PR.