Logstash cannot load org.jruby.ext.openssl.OpenSSL

[jdcc2]

OS: Debian 9.9
Logstash version: logstash-oss/stable 1:7.3.1-1
Java version: openjdk 11.0.4 2019-07-16
OpenJDK Runtime Environment (build 11.0.4+11-post-Debian-1bpo91)
OpenJDK 64-Bit Server VM (build 11.0.4+11-post-Debian-1bpo91, mixed mode, sharing)

Running logstash without any parameters gives the following output:

/usr/share/logstash/bin/logstash
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[ERROR] 2019-09-01 20:40:20.045 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (NameError) cannot load (ext) (org.jruby.ext.openssl.OpenSSL)

The same happens when running logstash-plugin:

/usr/share/logstash/bin/logstash-plugin --help
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.util.SecurityHelper to field java.lang.reflect.Field.modifiers
WARNING: Please consider reporting this to the maintainers of org.jruby.util.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
NameError: cannot load (ext) (org.jruby.ext.openssl.OpenSSL)
  load_ext at org/jruby/ext/jruby/JRubyUtilLibrary.java:201
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/jopenssl/load.rb:31
   require at org/jruby/RubyKernel.java:987
    (root) at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
   require at org/jruby/RubyKernel.java:987
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/openssl.rb:1
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
   require at org/jruby/RubyKernel.java:987
   require at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/security.rb:12
   require at org/jruby/RubyKernel.java:987
    (root) at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
   require at org/jruby/RubyKernel.java:987
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/package.rb:44
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1

Others commented on having similar problems in a different issue -> #10593 (comment)

[pitchakorni]

I have same problem.
Then Can you solve that?

[daks]

I fixed this error with rm /etc/ssl/certs/java/cacerts && update-ca-certificates -f see #9564

[jdcc2]

I fixed this error with rm /etc/ssl/certs/java/cacerts && update-ca-certificates -f see #9564

I tried this, but it did not work. Also the error is slightly different in the issue you referenced. It seems to be very specific to Debian 9.x, as using the same version with Ubuntu had no problems at all.

[flover97]

Same problem...
Linux elk-01 4.9.0-7-amd64 #1 SMP Debian 4.9.110-1 (2018-07-05) x86_64 GNU/Linux
openjdk 11.0.5 2019-10-15
OpenJDK Runtime Environment (build 11.0.5+10-post-Debian-1bpo91)
OpenJDK 64-Bit Server VM (build 11.0.5+10-post-Debian-1bpo91, mixed mode, sharing)
logstash 7.4.2

[CODE-REaD]

With CentOS 7, I found that logstash was failing to start here:

24877 open("/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = -1 EACCES (Permission denied)

So I found the package this file came from (yum provides), then verified this package:

# rpm -V ca-certificates
.M.......  g /etc/pki/ca-trust/extracted/java/cacerts
.M....G..  g /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
.M.......  g /etc/pki/ca-trust/extracted/pem/email-ca-bundle.pem
.M.......  g /etc/pki/ca-trust/extracted/pem/objsign-ca-bundle.pem
.M....G..  g /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

/etc/pki/tls/certs/ca-bundle.crt is a symbolic link to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem, and sure enough rpm -V reports both Mode and Group of this file do not match the installation settings. In my case the file's settings were:

-r--r----- 1 root adm 222148 Dec 30 16:23 /etc/pki/tls/certs/ca-bundle.crt

I therefore reinstalled ca-certificates using yum reinstall. This had the effect of changing this file's directory listing to:

-r--r--r-- 1 root root 222148 Apr 8 10:27 /etc/pki/tls/certs/ca-bundle.crt

After performing this reinstall, my logstash service returned to normal operation.

How did the file's group and mode get changed? I do not have direct evidence, but the last modification time of ca-bundle.crt matches the time when I ran a yum update command and it updated this package.

[yejingchen]

Not sure where the real problem is, but I managed to get logstash running by changing Java version. Our production environment uses an old Oracle JDK 1.8.0_60, which produced this very issue. With JAVA_HOME pointing to the OpenJDK 13.0.2 bundle with elasticsearch (/usr/share/elasticsearch/jdk if you're using the RPM package) logstash runs normally. OpenJDK 1.8.0.232 seems also immune to this problem.

[m7x]

I have fixed using OpenJDK 1.8 on Debian 9 (Debian 4.19.67-2+deb10u2~bpo9+1):

$ sudo apt-get remove --purge openjdk* java-common default-jdk
$ sudo apt install openjdk-8-jdk
$ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1~deb9u1-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
$ ./logstash-plugin --help
Usage:
    bin/logstash-plugin [OPTIONS] SUBCOMMAND [ARG] ...

Parameters:
    SUBCOMMAND                    subcommand
    [ARG] ...                     subcommand arguments

Subcommands:
    list                          List all installed Logstash plugins
    install                       Install a Logstash plugin
    remove                        Remove a Logstash plugin
    update                        Update a plugin
    pack                          Package currently installed plugins, Deprecated: Please use prepare-offline-pack instead
    unpack                        Unpack packaged plugins, Deprecated: Please use prepare-offline-pack instead
    generate                      Create the foundation for a new plugin
    uninstall                     Uninstall a plugin. Deprecated: Please use remove instead
    prepare-offline-pack          Create an archive of specified plugins to use for offline installation

Options:
    -h, --help                    print help

[skystebnicki]

Has anyone figured out a fix for this, other than backdating to OpenJDK 8?

[alexnelson-es]

We found that at least one version of the ca-certificates package tickled this bug.
We want back to a previous version of ca-certificates and that fixed the issue.

[rmheilman]

The downgrade to OpenJDK 8 worked, but so did this in java 11:
cd /usr/lib/jvm/java-11-openjdk-amd64/lib/security/
ln -fs /System/Library/Java/Support/CoreDeploy.bundle/Contents/Home/lib/security/cacerts

[RhombusWeasel]

It seems that jruby requires a JKS cert type and it looks like it was using a PKCS12 cert which was causing the error. For us at least, hope this helps someone.
keytool -importkeystore -srckeystore cacerts -destkeystore cacerts.jks -deststoretype jks
rm cacerts
mv cacerts.jks cacerts

[kuldeep-saini1]

How can we fix this issue in windows 10?