Running Packetbeat and Logstash on the same Elasticsearch instance

[CharlesJUDITH]

Is it possible to update documentation for people who already have Elasticsearch and Kibana installed.

In this case you have just to install packetbeat and import dashboards. You may think this is logic but in my opinion, you should be interesting for newbies and so on.

In documentation there is "We have extended Kibana to support new panels specialised in visualising network data. So it is best to download it from packetbeat GitHub account: " but I make a diff between my Kibana version and this version so it look like the same except config.js file.

Thanks,
Charles

[packetb-old]

Hi Charles. Indeed, it is a common question if one can use packetbeat and logstash with the same elasticsearch+kibana installation. We'll add something to the documentation about his.

Note, however, that our kibana version includes at the moment two new modules hive and dashboard_dropdown. The hive one does the Hosts panel that you can see on packetbeat.com. But it's still entirely possible to run packetbeat with stock kibana.

We also want to propose these panel types to kibana up stream, we just want to do some improvements on them first.

[CharlesJUDITH]

Hi,

Thanks for your all those details.

If you provide a patched Kibana, why not use an init script to launch it instead of python -m SimpleHTTPServer

What do you think about that ?

[packetb-old]

Hi Charles, our new Packetbeat Deploy uses nginx for serving Kibana, so you get a more standard init script coming with that. Also Deploy installs both dashboards for both Logstash and Packetbeat in Kibana, so that part is also more useful now we hope. Thanks for your suggestions.

[ghost]

Has there been any progress on this? I am trying to feed packetbeat into a Nagios Log Server, which is basically the ELK stack, security requirements prevent me from feeding directly to ES and I do not have the space and hardware for a redis server at the moment.

[packetb-old]

Hi, you can use the file output plugin which writes one transaction per line in JSON format. The configuration for it looks something like this:

[output.file]
   # Uncomment the following lines if you want to output to flat files.
  enabled = true
  path="/tmp/packetbeat"
  filename="packetbeat"
  rotate_every_kb=10000
  number_of_files=7

[ghost]

Hi, I have been testing this setup and it uses quite a bit of CPU. With almost no traffic going to my application server the CPU usage was around 25-30%. Once I stopped the packet be at service usage dropped to 10% on the box. Any suggestions on how to get better CPU usage? If I deployed this to our production servers it could crash them.

[tsg]

One thing you could try is to disable the process mapping. I've seen instances where this was taking a lot of CPU. You can do that by adding the don't_read_from_proc option under the [procs] section, so soothing like this:

 [procs]
 dont_read_from_proc = true

Also, I'd love to see a CPU profile of that. You can create one by starting packet beat like this:

 packetbeat -e -c /etc/packetbeat/packetbeat.conf -v -cpuprofile prof.dat

and sending us prof.dat (either by email to [email protected] or by attaching it here).

[ghost]

I ran packetbeat with the pf_ring type set. It started running at about 14% of one cpu and ran pretty well. After about 3 days it is now using 100% cpu on one core.

I tried adding the [procs] option but got the following error:
TOML config parsing failed on /etc/packetbeat/packetbeat.conf: Near line 72, key '': Key 'procs' has already been defined.. Exiting.

I am working on getting the cpuprofile for you. Nothing is printing to the file.

Also I am getting this error: pgsql.go:723: WARN Response from unknown transaction. Ignoring I do not know what this is erroring on.

[tsg]

I tried adding the [procs] option but got the following error:
TOML config parsing failed on /etc/packetbeat/packetbeat.conf: Near line 72, key '': Key 'procs' has > already been defined.. Exiting.

You probably added [procs] the second time. Try adding the dont_read_from_proc line under the existing [procs] section, without adding a new one.

I am working on getting the cpuprofile for you. Nothing is printing to the file.

I think it's only written when you stop the process. Simply use ^C to stop it.

Also I am getting this error: pgsql.go:723: WARN Response from unknown transaction. Ignoring I do not know what this is erroring on.

You can ignore this one for now.

HTH. Let me know if you'd like to schedule a Skype/teamviewer session. I can check what is going on wrong for you.