Beat exporter setup doesn't work

[ashishvaishno]

Hello,

I wanted to export filebeat metrics via beat exporter.

filebeat has a conf that exposes the metrics https://www.elastic.co/guide/en/beats/filebeat/master/http-endpoint.html

beat exporter's documentation also tells me to do the same https://github.com/trustpilot/beat-exporter/blob/master/readme.md#setup

Filebeat and beat-exporter are running on the same host in the same docker bridge network

Out of the box this config doesn't work. To make it work I had to change

http: 
  enabled: true 
  host: localhost 
  port: 5066

to

http: 
  enabled: true 
  host: 0.0.0.0 
  port: 5066

Why localhost config doesn't work : If the process inside the container is bound to localhost, localhost uses a special loopback interface that is distinct from regular networking. I am able to access the service from within the container (host) but you won't be able to route its traffic externally

In doing so I make the setup work for filebeat and beat-exporter, but cause issues between logstash and filebeat.(filebeat and logstash running on different server) filebeat logs

2020-06-23T10:50:06.823+0200	ERROR	logstash/async.go:256	Failed to publish events caused by: read tcp X.X.X.X:40590->X.X.X.X:5050: read: connection reset by peer 
2020-06-23T10:50:06.894+0200	ERROR	logstash/async.go:256	Failed to publish events caused by: client is not connected 2020-06-23T10:50:08.708+0200 ERROR	pipeline/output.go:121	Failed to publish events: client is not connected

[24th June 2020 14:36 PM] Noticed something :
As soon as I stop beat exporter and restart the filebeat instance, the above mentioned error do not appear and filebeat/logstash work as expected

So the setup has a container of filebeat and beat-exporter running on application-server and logstash running on a my log-server
With the below config in filebeat. filebeat and logstash work (so i will rule out the possibility that 0.0.0.0 doesn't work )

http:
  enabled: true
  host: 0.0.0.0
  port: 5066

Now when i start beat-exporter, I start to see the above errors (not sure what could cause it)
But even after stopping beat-exporter the above errors persist and it only gets resolved once i restart filebeat
Regards
Ashish

[shivas]

I would say this is a networking setup/issue. There is a lot of different ways of how people can run beat-exporter (sidecars, deamon set, and etc.)
In a nutshell beat-exporter is just proxy, converting filebeat JSON response to Prometheus format.
Just to be clear, the readme is just readme on what options are available, actual values you need to set depends on the way you run it.

So in general make sure your filebeat->logstash part is working correctly, then enable stats endpoint in filebeat, point beat-exporter to that stats endpoint

[leonfibal]

Hi,

I faced the same problem. When I run beat-exporter as a sidecar it looks good, I can get metrics through /metrics endpoint, but my filebeat starts logging the same errors as @ashishvaishno mentioned above:

2020-11-04T14:42:12.652Z ERROR logstash/sync.go:155 Failed to publish events caused by: read tcp #pod-ip#:36758->#logtsash-ip#:5044: read: connection reset by peer

As soos as I remove beat-exporter from k8s deployment filebeat starts working correctly.

To be sure I created a simple application that scrapes filebeat stats and exposes prometheus metrics (dummy version of your library) and added it as a sidecar to k8s deployment. Nothing changed in filebeat logs.

On the other hand I deployed it in my local cluster and everything worked good 😕.

No idea what is the problem.