Logstash - Missing JFFI Library #33

bendermIBM · 2019-09-26T20:44:14Z

@cwsolee Trying to run through the Logstash build process for a customer and the automated build script does look like it worked and I am able to run logstash -v however once I try to run it I am getting a missing library error for JFFI

[2019-09-26T14:15:59,875][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: java.lang.UnsatisfiedLinkError: could not locate stub library in jar file.  Tried [jni/s390x-Linux/libjffi-1.2.so, /jni/s390x-Linux/libjffi-1.2.so]

strace : https://gist.github.com/bendermIBM/76cddd0fa7b8da087be723fa5c881cde

Could your team take a look? Let me know if you need access to the machine

The text was updated successfully, but these errors were encountered:

bendermIBM · 2019-09-27T15:50:15Z

@cwsolee I just tried with AdoptJDK 8 per our messages

java -version

openjdk version "1.8.0_222"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_222-b10)
OpenJDK 64-Bit Zero VM (AdoptOpenJDK)(build 25.222-b10, interpreted mode)

echo $JAVA_HOME

/opt/ibm/jdk8u222-b10

echo $PATH

/opt/ibm/jdk8u222-b10/bin:/opt/ibm/jdk8u222-b10/bin:/opt/ibm/java-s390x-80/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

echo $LD_LIBRARY_PATH

/usr/local/jffi/build/jni:/usr/local/jffi/build/jni:/usr/local/jffi/build/jni:

We are still getting a JFFI error as with OpenJDK8, IBM Java 8 and OpenJDK11

logstash -f /etc/logstash/conf.d -l /var/log/logstash --log.level=debug --path.settings=/etc/logstash

Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-09-27T11:44:54,424][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2019-09-27T11:44:54,445][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x407bfc49 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2019-09-27T11:44:54,461][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2019-09-27T11:44:54,462][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x543d242e @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2019-09-27T11:44:59,022][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: java.lang.UnsatisfiedLinkError: could not locate stub library in jar file.  Tried [jni/s390x-Linux/libjffi-1.2.so, /jni/s390x-Linux/libjffi-1.2.so]
	at com.kenai.jffi.internal.StubLoader.getStubLibraryStream(StubLoader.java:412)
	at com.kenai.jffi.internal.StubLoader.loadFromJar(StubLoader.java:355)
	at com.kenai.jffi.internal.StubLoader.load(StubLoader.java:258)
	at com.kenai.jffi.internal.StubLoader.<clinit>(StubLoader.java:449)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at com.kenai.jffi.Init.load(Init.java:68)
	at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:49)
	at com.kenai.jffi.Foreign$InstanceHolder.<clinit>(Foreign.java:45)
	at com.kenai.jffi.Foreign.getInstance(Foreign.java:103)
	at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:242)
	at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237)
	at com.kenai.jffi.Type.resolveSize(Type.java:155)
	at com.kenai.jffi.Type.size(Type.java:138)
	at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:178)
	at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:57)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:41)
	at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:53)
	at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:49)
	at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
	at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
	at jnr.ffi.Library.loadLibrary(Library.java:114)
	at jnr.posix.POSIXFactory$DefaultLibCProvider$SingletonHolder.<clinit>(POSIXFactory.java:281)
	at jnr.posix.POSIXFactory$DefaultLibCProvider.getLibC(POSIXFactory.java:310)
	at jnr.posix.BaseNativePOSIX.<init>(BaseNativePOSIX.java:38)
	at jnr.posix.LinuxPOSIX.<init>(LinuxPOSIX.java:19)
	at jnr.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:146)
	at jnr.posix.POSIXFactory.loadNativePOSIX(POSIXFactory.java:124)
	at jnr.posix.POSIXFactory.loadPOSIX(POSIXFactory.java:93)
	at jnr.posix.LazyPOSIX.loadPOSIX(LazyPOSIX.java:38)
	at jnr.posix.LazyPOSIX.posix(LazyPOSIX.java:32)
	at jnr.posix.LazyPOSIX.isNative(LazyPOSIX.java:402)
	at org.jruby.RubyGlobal.initSTDIO(RubyGlobal.java:287)
	at org.jruby.RubyGlobal.createGlobals(RubyGlobal.java:211)
	at org.jruby.Ruby.init(Ruby.java:1260)
	at org.jruby.Ruby.newInstance(Ruby.java:370)
	at org.logstash.Logstash.<init>(Logstash.java:88)
	at org.logstash.Logstash.main(Logstash.java:43)

 See http://jira.codehaus.org/browse/JRUBY-4583

Error accessing temp directory: /tmp
This often occurs because the temp directory has been mounted with NOEXEC or
the Logstash user has insufficient permissions on the directory. Possible
workarounds include setting the -Djava.io.tmpdir property in the jvm.options
file to an alternate directory or correcting the Logstash user's permissions.

bendermIBM · 2019-09-27T16:08:26Z

Also, I'm not sure AdoptJDK 8 has a JIT

printing help takes 156 seconds

logstash --help  156.18s user 0.41s system 98% cpu 2:39.77 total

joransiu · 2019-09-27T16:13:06Z

Not sure about the libjffi issue w.r.t logstash, but can confirm that OpenJDK 8 with HotSpot VM does not have a JIT on Linux on Z. It was only added in JDK9 or newer. You can consider using OpenJDK 8 with OpenJ9 VM, which will be much more performant: https://adoptopenjdk.net/releases.html?variant=openjdk8&jvmVariant=openj9#s390x_linux

bendermIBM · 2019-09-27T16:18:15Z

@joransiu that does look to have the JIT thanks

logstash --help  28.25s user 1.18s system 180% cpu 16.286 total

Still same JFFI error w/ J9 VM

logstash -f /etc/logstash/conf.d -l /var/log/logstash --log.level=debug --path.settings=/etc/logstash

Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-09-27T12:16:07,243][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2019-09-27T12:16:07,246][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0xffffffff975d9559 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2019-09-27T12:16:07,247][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2019-09-27T12:16:07,247][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x4fd3b9dd @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2019-09-27T12:16:07,429][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: java.lang.UnsatisfiedLinkError: could not locate stub library in jar file.  Tried [jni/s390x-Linux/libjffi-1.2.so, /jni/s390x-Linux/libjffi-1.2.so]
	at com.kenai.jffi.internal.StubLoader.getStubLibraryStream(StubLoader.java:412)
	at com.kenai.jffi.internal.StubLoader.loadFromJar(StubLoader.java:355)
	at com.kenai.jffi.internal.StubLoader.load(StubLoader.java:258)
	at com.kenai.jffi.internal.StubLoader.<clinit>(StubLoader.java:449)
	at java.lang.Class.forNameImpl(Native Method)
	at java.lang.Class.forName(Class.java:403)
	at com.kenai.jffi.Init.load(Init.java:68)
	at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:49)
	at com.kenai.jffi.Foreign$InstanceHolder.<clinit>(Foreign.java:45)
	at com.kenai.jffi.Foreign.getInstance(Foreign.java:103)
	at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:242)
	at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237)
	at com.kenai.jffi.Type.resolveSize(Type.java:155)
	at com.kenai.jffi.Type.size(Type.java:138)
	at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:178)
	at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:57)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:41)
	at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:53)
	at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:49)
	at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
	at java.lang.J9VMInternals.newInstanceImpl(Native Method)
	at java.lang.Class.newInstance(Class.java:1848)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
	at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
	at jnr.ffi.Library.loadLibrary(Library.java:114)
	at jnr.posix.POSIXFactory$DefaultLibCProvider$SingletonHolder.<clinit>(POSIXFactory.java:281)
	at jnr.posix.POSIXFactory$DefaultLibCProvider.getLibC(POSIXFactory.java:310)
	at jnr.posix.BaseNativePOSIX.<init>(BaseNativePOSIX.java:38)
	at jnr.posix.LinuxPOSIX.<init>(LinuxPOSIX.java:19)
	at jnr.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:146)
	at jnr.posix.POSIXFactory.loadNativePOSIX(POSIXFactory.java:124)
	at jnr.posix.POSIXFactory.loadPOSIX(POSIXFactory.java:93)
	at jnr.posix.LazyPOSIX.loadPOSIX(LazyPOSIX.java:38)
	at jnr.posix.LazyPOSIX.posix(LazyPOSIX.java:32)
	at jnr.posix.LazyPOSIX.isNative(LazyPOSIX.java:402)
	at org.jruby.RubyGlobal.initSTDIO(RubyGlobal.java:287)
	at org.jruby.RubyGlobal.createGlobals(RubyGlobal.java:211)
	at org.jruby.Ruby.init(Ruby.java:1260)
	at org.jruby.Ruby.newInstance(Ruby.java:370)
	at org.logstash.Logstash.<init>(Logstash.java:88)
	at org.logstash.Logstash.main(Logstash.java:43)

 See http://jira.codehaus.org/browse/JRUBY-4583

Error accessing temp directory: /tmp
This often occurs because the temp directory has been mounted with NOEXEC or
the Logstash user has insufficient permissions on the directory. Possible
workarounds include setting the -Djava.io.tmpdir property in the jvm.options
file to an alternate directory or correcting the Logstash user's permissions.

joransiu · 2019-09-27T16:27:16Z

I found this issue: #18 that sounds related. Could you try the steps there? It suggests the libjffi libraries need to be rolled up into the jar? I'm not sure why...

bendermIBM · 2019-09-28T18:10:07Z

Hey @joransiu - tried it out.

zip -g jruby-complete-9.1.13.0.jar /usr/local/jffi/build/jni/libjffi-1.2.so

	zip warning: jruby-complete-9.1.13.0.jar not found or empty
  adding: usr/local/jffi/build/jni/libjffi-1.2.so (deflated 70%)

Same error

logstash -f /etc/logstash/conf.d -l /var/log/logstash --log.level=debug --path.settings=/etc/logstash

Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-09-28T13:46:09,433][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2019-09-28T13:46:09,440][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0xffffffffc0a9e58e @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2019-09-28T13:46:09,441][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2019-09-28T13:46:09,441][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0xffffffffcaf57fab @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2019-09-28T13:46:09,637][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: java.lang.UnsatisfiedLinkError: could not locate stub library in jar file.  Tried [jni/s390x-Linux/libjffi-1.2.so, /jni/s390x-Linux/libjffi-1.2.so]
	at com.kenai.jffi.internal.StubLoader.getStubLibraryStream(StubLoader.java:412)
	at com.kenai.jffi.internal.StubLoader.loadFromJar(StubLoader.java:355)
	at com.kenai.jffi.internal.StubLoader.load(StubLoader.java:258)
	at com.kenai.jffi.internal.StubLoader.<clinit>(StubLoader.java:449)
	at java.lang.Class.forNameImpl(Native Method)
	at java.lang.Class.forName(Class.java:403)
	at com.kenai.jffi.Init.load(Init.java:68)
	at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:49)
	at com.kenai.jffi.Foreign$InstanceHolder.<clinit>(Foreign.java:45)
	at com.kenai.jffi.Foreign.getInstance(Foreign.java:103)
	at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:242)
	at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237)
	at com.kenai.jffi.Type.resolveSize(Type.java:155)
	at com.kenai.jffi.Type.size(Type.java:138)
	at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:178)
	at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:57)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:41)
	at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:53)
	at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:49)
	at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
	at java.lang.J9VMInternals.newInstanceImpl(Native Method)
	at java.lang.Class.newInstance(Class.java:1848)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
	at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
	at jnr.ffi.Library.loadLibrary(Library.java:114)
	at jnr.posix.POSIXFactory$DefaultLibCProvider$SingletonHolder.<clinit>(POSIXFactory.java:281)
	at jnr.posix.POSIXFactory$DefaultLibCProvider.getLibC(POSIXFactory.java:310)
	at jnr.posix.BaseNativePOSIX.<init>(BaseNativePOSIX.java:38)
	at jnr.posix.LinuxPOSIX.<init>(LinuxPOSIX.java:19)
	at jnr.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:146)
	at jnr.posix.POSIXFactory.loadNativePOSIX(POSIXFactory.java:124)
	at jnr.posix.POSIXFactory.loadPOSIX(POSIXFactory.java:93)
	at jnr.posix.LazyPOSIX.loadPOSIX(LazyPOSIX.java:38)
	at jnr.posix.LazyPOSIX.posix(LazyPOSIX.java:32)
	at jnr.posix.LazyPOSIX.isNative(LazyPOSIX.java:402)
	at org.jruby.RubyGlobal.initSTDIO(RubyGlobal.java:287)
	at org.jruby.RubyGlobal.createGlobals(RubyGlobal.java:211)
	at org.jruby.Ruby.init(Ruby.java:1260)
	at org.jruby.Ruby.newInstance(Ruby.java:370)
	at org.logstash.Logstash.<init>(Logstash.java:88)
	at org.logstash.Logstash.main(Logstash.java:43)

 See http://jira.codehaus.org/browse/JRUBY-4583

Error accessing temp directory: /tmp
This often occurs because the temp directory has been mounted with NOEXEC or
the Logstash user has insufficient permissions on the directory. Possible
workarounds include setting the -Djava.io.tmpdir property in the jvm.options
file to an alternate directory or correcting the Logstash user's permissions.

bendermIBM · 2019-09-28T23:26:13Z

Hey @cwsolee I have pinged you the details of the clean machine. To assist in the recreate the command we are using to run logstash is :

logstash -f /etc/logstash/conf.d -l /var/log/logstash --log.level=debug --path.settings=/etc/logstash

I have also sent you the related config files

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Logstash - Missing JFFI Library #33

Logstash - Missing JFFI Library #33

bendermIBM commented Sep 26, 2019

bendermIBM commented Sep 27, 2019

bendermIBM commented Sep 27, 2019

joransiu commented Sep 27, 2019

bendermIBM commented Sep 27, 2019

joransiu commented Sep 27, 2019

bendermIBM commented Sep 28, 2019

bendermIBM commented Sep 28, 2019

Logstash - Missing JFFI Library #33

Logstash - Missing JFFI Library #33

Comments

bendermIBM commented Sep 26, 2019

bendermIBM commented Sep 27, 2019

bendermIBM commented Sep 27, 2019

joransiu commented Sep 27, 2019

bendermIBM commented Sep 27, 2019

joransiu commented Sep 27, 2019

bendermIBM commented Sep 28, 2019

bendermIBM commented Sep 28, 2019