Logstash 7.17.7 or higher using Java 8 fails with NoSuchMethodError

[davshapi]

Logstash information:

• Logstash 7.17.6 worked fine. Problem started with Logstash 7.17.7, and is still reproducible with 7.17.9
• Logstash installed on SLES12SP5 through zypper using packages mirrored from https://artifacts.elastic.co/packages/7.x/yum
• Logstash started through systemd

Plugins installed:
logstash-codec-avro (3.3.1)
logstash-codec-cef (6.2.6)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.1)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.1)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.2.2)
logstash-codec-netwitness (1.0.0)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.6)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.15)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.1.5)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.11.1)
logstash-filter-fingerprint (3.3.2)
logstash-filter-geoip (7.2.12)
logstash-filter-grok (4.4.3)
logstash-filter-http (1.2.1)
logstash-filter-json (3.2.0)
logstash-filter-kv (4.5.0)
logstash-filter-memcached (1.1.0)
logstash-filter-metrics (4.0.7)
logstash-filter-multiline (3.0.4)
logstash-filter-mutate (3.5.6)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.1.1)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.3.1)
logstash-filter-truncate (1.0.5)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.3)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.1.3)
logstash-input-azure_event_hubs (1.4.4)
logstash-input-beats (6.2.6)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (1.1.12)
logstash-input-elasticsearch (4.12.3)
logstash-input-exec (3.4.0)
logstash-input-file (4.4.4)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.2)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.4.5)
logstash-input-http_poller (5.1.0)
logstash-input-imap (3.2.0)
logstash-input-jms (3.2.2)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-s3 (3.8.3)
logstash-input-snmp (1.3.1)
logstash-input-snmptrap (3.1.0)
logstash-input-sqs (3.1.3)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.6.0)
logstash-input-tcp (6.2.7)
logstash-input-twitter (4.1.0)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.2)
logstash-integration-elastic_enterprise_search (2.1.2)
├── logstash-output-elastic_app_search
└── logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.2.6)
├── logstash-input-jdbc
├── logstash-filter-jdbc_streaming
└── logstash-filter-jdbc_static
logstash-integration-kafka (10.9.0)
├── logstash-input-kafka
└── logstash-output-kafka
logstash-integration-rabbitmq (7.3.1)
├── logstash-input-rabbitmq
└── logstash-output-rabbitmq
logstash-output-cloudwatch (3.0.10)
logstash-output-csv (3.0.8)
logstash-output-elasticsearch (11.4.1)
logstash-output-email (4.1.1)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.2.5)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-s3 (4.3.7)
logstash-output-sns (4.0.8)
logstash-output-sqs (6.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.0.3)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.0.6)
logstash-patterns-core (4.3.4)

JVM:
openjdk version "1.8.0_362"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_362-b09)
OpenJDK 64-Bit Server VM (Temurin)(build 25.362-b09, mixed mode)

Temurin installed through zypper using packages mirrored from https://packages.adoptium.net/artifactory/rpm/sles/12/x86_64

Using LS_JAVA_HOME defined java: /opt/openjdk-8
This is a symlink we setup on all our systems to point to the Temurin installation.

OS version:
Linux hostname 4.12.14-16.120-azure #1 SMP Tue Dec 13 06:06:02 UTC 2022 (e336005) x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Logstash fails at startup with an NoSuchMethodError (see below for full stack trace).

Steps to reproduce:
This happens during startup on all our environments using this Logstash version, regardless of Logstash configuration.

I'm speculating that between 7.17.6 (which did not have this behavior) and 7.17.7, something changed in the build environment for Logstash 7.

Given this error, it looks to me like the issue is Logstash is expecting to find the method:
MappedByteBuffer position(int newPosition)

as defined in Java 9 and higher, but Java 8 does not have this method. Instead MappedByteBuffer inherits the following from its parent class:

Buffer position(int newPosition)

Provide logs (if relevant):
2023-02-09T14:23:13,684][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::NoSuchMethodError", :message=>"java.nio.MappedByteBuffer.position(I)Ljava/nio/MappedByteBuffer;", :backtrace=>["org.logstash.ackedqueue.io.MmapPageIOV2.create(MmapPageIOV2.java:194)", "org.logstash.ackedqueue.Queue.newCheckpointedHeadpage(Queue.java:386)", "org.logstash.ackedqueue.Queue.openPages(Queue.java:209)", "org.logstash.ackedqueue.Queue.open(Queue.java:176)", "org.logstash.ackedqueue.ext.JRubyAckedQueueExt.open(JRubyAckedQueueExt.java:123)", "org.logstash.ackedqueue.ext.JRubyWrappedAckedQueueExt.initialize(JRubyWrappedAckedQueueExt.java:65)", "org.logstash.ackedqueue.QueueFactoryExt.create(QueueFactoryExt.java:62)", "org.logstash.execution.AbstractPipelineExt.openQueue(AbstractPipelineExt.java:207)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$0$0$openQueue.call(AbstractPipelineExt$INVOKER$i$0$0$openQueue.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:355)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:144)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:345)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:191)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:178)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:397)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:206)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:137)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:60)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:750)"]} warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true): LogStash::Error: Don't know how to handle Java::JavaLang::NoSuchMethodErrorforPipelineAction::Create

create at org/logstash/execution/ConvergeResultExt.java:135 add at org/logstash/execution/ConvergeResultExt.java:60 converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:405

[andsel]

Since JDK 1.4 a method Buffer position(int i) existed on Buffer and so inherited in ByteBuffer and MappedBytByffer
With JDK 9 a new method named ByteBuffer position(int i) was added to ByteBuffer, and despite with Logstash 7.17.6 and 7.17.7 we haven't changed the the target and source settings to the compiler, it could be that this happened because we presumably we compiled 7.17.6 with a JDK8.
The problem is that compiling with JDK 11, despite the source and target setting, the compiler selects the "wrong" method, which doesn't exists in JDK 8.
This is explained very well in the blog post: https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/

We need to configure in Gradle the same concept of Maven's maven.compiler.release.
Maybe the release option of the compiler task would help https://docs.gradle.org/current/dsl/org.gradle.api.tasks.compile.CompileOptions.html#org.gradle.api.tasks.compile.CompileOptions:release

[andsel]

It's reproducible downloading a JDK 8 (tested with 8.0.362) and setting the LS_JAVA_HOME

export LS_JAVA_HOME="/home/andrea/.sdkman/candidates/java/8.0.362-tem/"

Enabling the PQ, in config/logstash.yml set queue.type: persisted and run a simple pipeline:

logstash-7.17.6/bin/logstash -e "input{stdin{}} output{stdout{}}"
- vs - 
logstash-7.17.7/bin/logstash -e "input{stdin{}} output{stdout{}}"

[roaksoax]

I'm going to go ahead and close this issue, provided that we believe it has now been fixed and will be available in the next 7.x Logstash release (7.17.12).

Please note Lgostash 8.x doesn't support Java 8.