Crash parsing Redis traffic

[hiteshmahajan]

Too many logs getting generated inside /var/log/messages

Nov 18 12:51:47 ip-10-168-0-218 /usr/bin/packetbeat[27009]: log.go:111: 
Stacktrace:
/go/src/github.com/elastic/packetbeat/Godeps/_workspace/src/github.com/elastic/libbeat/logp/log.go:111 
(0x48bfb6)#012/usr/local/go/src/runtime/asm_amd64.s:437 
(0x47d72e)#012/usr/local/go/src/runtime/panic.go:423 
(0x44d329)#012/usr/local/go/src/runtime/panic.go:42 
(0x44b9e9)#012/usr/local/go/src/runtime/sigpanic_unix.go:24 
(0x46228a)#012/go/src/github.com/elastic/packetbeat/protos/redis/redis.go:579
(0x51af62)#012/go/src/github.com/elastic/packetbeat/protos/tcp/tcp.go:87 
(0x5202f3)#012/go/src/github.com/elastic/packetbeat/protos/tcp/tcp.go:173 
(0x52142d)#012/go/src/github.com/elastic/packetbeat/decoder/decoder.go:136 
(0x6c6041)#012/go/src/github.com/elastic/packetbeat/sniffer/sniffer.go:352 
(0x532a09)#012/go/src/github.com/elastic/packetbeat/packetbeat.go:212 
(0x422f2b)#012/usr/local/go/src/runtime/asm_amd64.s:1696 
(0x47fa71)
Nov 18 12:51:47 ip-10-168-0-218 /usr/bin/packetbeat[27009]: log.go:110: ParseRedis exception. Recovering, but please report this: runtime error: invalid memory address or nil pointer dereference.
Nov 18 12:51:47 ip-10-168-0-218 /usr/bin/packetbeat[27009]: log.go:111: 
Stacktrace:
/go/src/github.com/elastic/packetbeat/Godeps/_workspace/src/github.com/elastic/libbeat/logp/log.go:111 
(0x48bfb6)#012/usr/local/go/src/runtime/asm_amd64.s:437 
(0x47d72e)#012/usr/local/go/src/runtime/panic.go:423 
(0x44d329)#012/usr/local/go/src/runtime/panic.go:42 
(0x44b9e9)#012/usr/local/go/src/runtime/sigpanic_unix.go:24 
(0x46228a)#012/go/src/github.com/elastic/packetbeat/protos/redis/redis.go:579 
(0x51af62)#012/go/src/github.com/elastic/packetbeat/protos/tcp/tcp.go:87 
(0x5202f3)#012/go/src/github.com/elastic/packetbeat/protos/tcp/tcp.go:173 
(0x52142d)#012/go/src/github.com/elastic/packetbeat/decoder/decoder.go:136 
(0x6c6041)#012/go/src/github.com/elastic/packetbeat/sniffer/sniffer.go:352 
(0x532a09)#012/go/src/github.com/elastic/packetbeat/packetbeat.go:212 
(0x422f2b)#012/usr/local/go/src/runtime/asm_amd64.s:1696 
(0x47fa71)
Nov 18 12:51:47 ip-10-168-0-218 /usr/bin/packetbeat[27009]: log.go:110: ParseRedis exception. Recovering, but please report this: runtime error: invalid memory address or nil pointer dereference.

[ruflin]

Based on your log files I assume you are sniffing Redis traffic with Packetbeat. Which version of packetbeat are you using? Would it be possible for you to send us some example pcap messages that cause this issue?

[hiteshmahajan]

Yes we are using redis to store data

output:
  ### Redis as output
  redis:
    enabled: true
    host: x.x.x.x
    port: 6379

packetbeat version 1.0.0-rc1 (amd64)

$5
RPUSH
$10
packetbeat
$490
{"@timestamp":"2015-11-18T09:42:51.282Z","bytes_in":1270,"bytes_out":917,"client_ip":"10.148.128.180","client_port":58972,"client_proc":"","client_server":"","count":1,"direction":"out","http":{"code":200,"content_length":214,"phrase":"OK"},"ip":"10.168.0.218","method":"POST","params":"","path":"/notification/notifyme","port":80,"proc":"","query":"POST /notification/notifyme","responsetime":25,"server":"","shipper
15:12:51.555048 IP 10.168.0.218.55037 > 10.168.231.212.6379: tcp 75
E....8@[email protected]
...
..........4..#C...s.o.....
.x..-.".":"...warpspeed...","status":"OK","tags":["warp-frontend"],"type":"http"}

15:12:51.594480 IP 10.168.231.212.6379 > 10.168.0.218.55037: tcp 0
@.9.@.
...
.........#C.........5.....
-.#..x..
15:12:51.939459 IP 10.168.231.212.6379 > 10.168.0.218.55037: tcp 1083
[email protected].<}
...
.........#C...............
-.$[.x..-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'

15:12:51.939480 IP 10.168.0.218.55037 > 10.168.231.212.6379: tcp 0
E..4.9@.@...
...
.............'~...........
.x.e-.$[
15:12:51.939725 IP 10.168.0.218.55044 > 10.168.231.212.6379: tcp 0
E..<.K@[email protected]
...
.........}.......9............
.x.e........

[andrewkroh]

It looks like it is crashing here. Would it be possible to run packetbeat with the -dump some/dir/captured-data.pcap command line option and send me the capture?

I will then replay the capture file through packetbeat to recreate crash, develop a test case and a fix. I run something like this to replay the file through: packetbeat -c packetbeat.yml -e -v -d "redis" -t -I some/dir/captured-data.pcap -waitstop 30

[jalberto]

I have same error in rc2, @andrewkroh do you want one pcap from my system too?

[andrewkroh]

@jalberto Yes, that would be great because I still need a pcap for this issue. Thanks

[jalberto]

@andrewkroh check your email :)

[voelzmo]

Can we rename this to have 'redis' in the issue title? It is somehow difficult to find using github's issue search right now. Thanks.

[tsg]

@voelzmo good point, I changed it.

[jalberto]

@andrewkroh @tsg Thanks!

Has this been released?

[ruflin]

@jalberto This has not been release yet as it was just merged an hour ago. If you like I can trigger the nightly builds so you will have a binary to test? Alternative you can build it yourself directly from the source code.

[jalberto]

thanks @ruflin,

I saw a new version on the elastic page, so I was wondering if this was included there, but now I can see the 1.1.0 tag

I can wait, but if you trigger the nightly build I will test it

[ruflin]

@jalberto Nightly builds triggered. It normally takes between 10 and 60 minutes to build them (depending on travis load). They will show up with the most recent timestamp soonish here: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/

[jalberto]

Thanks!

On Thu, 26 Nov 2015 at 12:01 Nicolas Ruflin [email protected]
wrote:

@jalberto https://github.com/jalberto Nightly builds triggered. It
normally takes between 10 and 60 minutes to build them (depending on travis
load). They will show up with the most recent timestamp soonish here:
https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/

—
Reply to this email directly or view it on GitHub
#384 (comment).