Logstash beat plugin not working

[kds-jprat]

Hello,

We are using logstash 6 but in the latest versions we have been unable to use the plugin beats in logstash. See the error in logs below.
The configuration works in 6.8.6 flawlessly but when the updating or fully clean-reinstalling from 6.8.13, the error may appear definitely (randomly on some version but it repeats indefinitely at each start once it occurs): cannot link Java class io.netty.handler.ssl.OpenSsl, probable missing dependency

I believe this is bug is introduced with the change made to go with the native OpenSSL pluging in ELK or something like that but we are not even using SSL in our configuration.

Any help/suggestion appreciated.

Logstash information:

• version: 6.8.13 to 6.8.18 (the problem does not seem to exist on 6.8.6)
• installation source: docker oss image
• run method: docker

Plugins installed:
Unmodified from the docker image

JVM:
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
JVM installation source: see original image
LS_JAVA_HOME/JAVA_HOME does not seem to be set.

OS:
docker container 6.8.18 container: Linux ***** 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
docker host: Linux ***** 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Configuration used:
input {
beats {
port => 5005
add_field => { "[@metadata][origin]" => "beats_neo" }
}
}

Logs:
[2021-09-09T09:32:17,375][ERROR][logstash.pipeline ] Error registering plugin
{
:pipeline_id=>"neo",
:plugin=>"<LogStash::Inputs::Beats port=>5005, add_field=>{"[@metadata][origin]"=>"beats_neo"}, id=>"139f2aae82e3d69ea5114dcc94ca78ed2c7b58303875be079b5e4ce195cc33e3", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_52d5a005-e3ad-4f47-97b3-deea8ab5c4c6", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, add_hostname=>true, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>2>",
:error=>"cannot link Java class io.netty.handler.ssl.OpenSsl, probable missing dependency: Could not initialize class io.netty.handler.ssl.OpenSsl",
:thread=>"#<Thread:0x16a34cb7@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:173 run>"
}
[2021-09-09T09:32:17,808][ERROR][logstash.pipeline ] Pipeline aborted due to error
{
:pipeline_id=>"neo",
:exception=>#<NameError: cannot link Java class io.netty.handler.ssl.OpenSsl, probable missing dependency: Could not initialize class io.netty.handler.ssl.OpenSsl>,
:backtrace=>[ "org/jruby/javasupport/JavaClass.java:292:in for_name'", "org/jruby/javasupport/JavaClass.java:288:in for_name'",
"org/jruby/javasupport/JavaUtilities.java:38:in get_proxy_class'", "uri:classloader:/jruby/java/core_ext/object.rb:44:in block in java_import'",
"org/jruby/RubyArray.java:2577:in map'", "uri:classloader:/jruby/java/core_ext/object.rb:37:in java_import'",
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-5.1.9-java/lib/logstash/inputs/beats.rb:146:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:259:in register_plugin'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in block in register_plugins'", "org/jruby/RubyArray.java:1792:in each'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:413:in start_inputs'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:311:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:217:in run'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:176:in `block in start'"],
:thread=>"#<Thread:0x16a34cb7@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:173 run>"
}
[2021-09-09T09:32:17,814][ERROR][logstash.agent ] Failed to execute action
{
:id=>:neo,
:action_type=>LogStash::ConvergeResult::FailedAction,
:message=>"Could not execute action: PipelineAction::Create, action_result: false",
:backtrace=>nil
}

configuration:
startup.options.txt
jvm.options.txt
pipeline.contenthub.conf.txt

[andsel]

Hi @kds-jprat
thank's for your reporting. Please could you provide all the information requested by the bug report template https://github.com/elastic/logstash/blob/master/.github/ISSUE_TEMPLATE/bug.md ?

Are you able to provide a simple and minimal sequence of paths to reproduce it?

[kds-jprat]

Alright, i updated the bug.
Unfortunately I don't have a server or an url to give you because they are all private servers but I reproduce this all the time recently.
As for a minimal sequence, I provided some settings if u need it but the problem arise straight from initialization of the pipelines preventing their usage.

[andsel]

I used this pipeline file:

input {
  beats {
    port => 5005
    add_field => { "[@metadata][origin]" => "beats_neo" }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

inserted it in the folder ~/pipeline/ and run the command:

docker run --rm -it -v ~/pipeline/:/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash:6.8.13

and not encountered the problem.

Could you:

• share the docker command line that you use
• test with the pipelines I used to check if the problem is consistent
• share the full log of Logstash (in a file)

When you do the test, please don't start also the Filebeat, keep only the Logstash process up and running, from what you wrote above it should fail.

[kds-jprat]

Hello, sorry for the time I took to reply.

Unfortunately, I did not have time to try your solution yet. I still tried to restart logstash with filebeat instances turned off and it doesn't seem to change a thing.
I'd like to add that we have two pipelines using filebeat actually. They either all fails or work at the same time.
As for the command line we use, it's the default one in the docker provided image. It isn't customised.

As we install the application via ansible in repository, it'll take time before I can try it but I'll try as soon as I get a little time.
Also, I'm thinking of adding rabbitmq between filebeat and logstash. I've noticed another bug in another platform with the current configuration: filebeat or logstash somehow manage to duplicate some logs 2 to 5 times in elasticsearch.

[kds-jprat]

Hello, following the log4j CVE 2021-44228, we had to update to 6.8.21 and we reran into this problem which we need fixed by now. I've been investigating and I've noticed a difference where it works and where it does not. the stack on which does not work actually has 2 beats pipeline whereas the stack where it works has only 1. Could it be the reason. I'll be trying to make a simpler exemple i can share today

[kds-jprat]

Alright, This seem impossible to reproduce when there are only beats input plugins so I wasn't able to make a simplified shareable exemple. We've tried several things such as disabling pipelines, reordering them. Same for the inputs and we've noticed the frequency of the bug change with the number of different input plugins.
We've even tried setting the ssl parameter to false and giving an empty array to cipher and this changed nothing.
Any advice ?

[kds-jprat]

Ok, we've fixed this by patching the beats plugin as follow:
file https://github.com/logstash-plugins/logstash-input-beats/blob/6766a75dce4b3a28a6e969cd9b8957f7cec19731/lib/logstash/inputs/beats.rb:
replace line 148:
java_import "io.netty.handler.ssl.OpenSsl"
by:
if @ssl
java_import "io.netty.handler.ssl.OpenSsl"
end

[andsel]

In a previous comment you wrote that:

the stack on which does not work actually has 2 beats pipeline whereas the stack where it works has only 1

• do you run multipipelines in same Docker image?
• if yes, could you share your pipelines and the docker launch line?