Random lumberjack input causing logstash to crash

[dsolsona]

I was wondering if someone could point me in the right direction with a weird error I'm seeing with logstash and lumberjack input.

I have a logstash doing some event filtering and sending logs to another logstash server using the lumberjack output/input.

It's been working great with my nginx and syslog data, but today I've enabled a new application that sends json data to logstash.

It looks like it's sending some events that are crashing one of my logstash instances.

I have tried running logstash in debug mode but I can't see anything useful in there.

Zlib::BufError: buffer error
              finish at org/jruby/ext/zlib/ZStream.java:134
             inflate at org/jruby/ext/zlib/JZlibInflate.java:72
  compressed_payload at /opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb:194
                feed at /opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb:93
                 run at /opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb:220
                 run at /opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb:59

Zlib::DataError: incorrect data check
              finish at org/jruby/ext/zlib/ZStream.java:134
             inflate at org/jruby/ext/zlib/JZlibInflate.java:72
  compressed_payload at /opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb:194
                feed at /opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb:93
                 run at /opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb:220
                 run at /opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb:59

[jordansissel]

Sounds like something corrupt is coming in to lumberjack input. Can you show the configuration of whatever is sending to this?

[dsolsona]

Yes, this only happens when we enable this app to send json events to logstash. So it's definitely some data that makes logstash very unhappy, the thing is how to find out which event is causing this.

The way I have this configured is the app sends json events to logstash over a tcp input

input {
  tcp {
  codec => json {}
    port => "5045"
    type => "json_events"
  }
}

Then this logstash instances will send all to another logstash instances using the lumberjack output/input.

output {
      lumberjack {
        codec => json { }
        hosts => ["1.1.1.1","2.2.2.2"]
        port => "5555"
        ssl_certificate => "/opt/logstash/ssl/logstash-forwarder.crt"
        workers => "4"
      }
}

And the input in the other logstash is

input {
  lumberjack {
    codec => json {}
    port => "5555"
    ssl_certificate => "/opt/logstash/ssl/logstash-forwarder.crt"
    ssl_key => "/opt/logstash/ssl/logstash-forwarder.key"
  }
}

[hiir]

our logstash service terminated with same exception:

Zlib::BufError: buffer error
          finish at org/jruby/ext/zlib/ZStream.java:134
         inflate at org/jruby/ext/zlib/JZlibInflate.java:72
compressed_payload at /opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb:194
            feed at /opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb:93
             run at /opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb:220
             run at /opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb:59

version:
logstash 1.4.2-modified

logstash input is only for lumberjack protocol:

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

logs are sent by logstash-forwarder version 0.3.1

... many files / paths configuration ...
"network": {
    "ssl ca": "/etc/ssl/certs/logstash-forwarder.crt",
    "timeout": 15,
    "servers": [
        "x.x.x.x:5000"
    ]
}

[jordansissel]

I tried to reproduce it like so:

• tcp->lumberjack: bin/logstash -e 'input { tcp { port => 3333 codec => json } } output { lumberjack { hosts => "localhost" ssl_certificate => "./lumberjack.crt" port => 3334 codec => json workers => 4 } }'
• lumberjack -> stdout: bin/logstash -e 'input { lumberjack { port => 3334 ssl_key => "lumberjack.key" ssl_certificate => "lumberjack.crt" codec => json } }'
• feeder: seq 4 | xargs -P4 -n1 sh -c 'yes "{\"hello\": \"💚 fancy world\", \"pid\": $$}" | nc localhost 3333' -

This sends data over 4 tcp connections: tcp -> lumberjack -> lumberjack -> stdout.

I waited for a few minutes and wasn't able to see this error.

[jordansissel]

Could be related: #1576

[hiir]

I haven't seen it anymore over the week, so I can't reproduce it reliably neither.

[jordansissel]

For folks hitting this problem, can you report what OS and version you are using?

[jordansissel]

We've only been able to reproduce this on Solaris 11

[hiir]

We saw it on AWS EC2 x64 instance (both logstash and logstash-forwarder)

[jordansissel]

@hiir EC2 tells me the virutalization platform. What operating system and version are you seeing it on?

[hiir]

logstash on Amazon Linux 2014.09.1

Linux version 3.14.27-25.47.amzn1.x86_64 (mockbuild@gobi-build-60 001) (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Wed Dec 17 18 :36:15 UTC 2014

logstash-forwarder on Debian 7.8

Linux version 3.2.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.65-1+deb7u1

[jeremymturner]

I'm seeing this issue as well. Logstash 1.4.2 on Ubuntu Precise.

My hunch is that it's related to the data being sent in, but yet to confirm that. In my case, it causes Logstash to restart about every 90 seconds.

Actually, scratch that. I was being sent data out of a .gz file. So it was invalid data.

[LordMike]

I'm currently writing a Lumberjack client for .Net, and every time I fail at writing frames - logstash stops working due to an exception in Lumberjack. This could indicate that the lumberjack input needs more work to harden it against errors.

[jsvd]

The fix for this zlib error has to do with large messages (over 16k), and a proposal for fix is here: elastic/logstash-forwarder#384