logstash not starting after update to 8.13.0 from red hat package manager - files from bundled jdk missing after update

[walkerprog]

Logstash information:

Please include the following information:

1. Logstash version: 8.13.0
2. Logstash installation source: dnf package manager, baseurl=https://artifacts.elastic.co/packages/8.x/yum
3. How is Logstash being run: as system service

Plugins installed:
logstash-codec-avro
logstash-codec-cef
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-aggregate
logstash-filter-anonymize
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop
logstash-filter-elasticsearch
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-http
logstash-filter-json
logstash-filter-kv
logstash-filter-memcached
logstash-filter-metrics
logstash-filter-mutate
logstash-filter-prune
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-translate
logstash-filter-truncate
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-azure_event_hubs
logstash-input-beats
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes
logstash-input-dead_letter_queue
logstash-input-elastic_serverless_forwarder
logstash-input-elasticsearch
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-jms
logstash-input-pipe
logstash-input-redis
logstash-input-snmp
logstash-input-snmptrap
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-integration-aws
├── logstash-codec-cloudfront
├── logstash-codec-cloudtrail
├── logstash-input-cloudwatch
├── logstash-input-s3
├── logstash-input-sqs
├── logstash-output-cloudwatch
├── logstash-output-s3
├── logstash-output-sns
└── logstash-output-sqs
logstash-integration-elastic_enterprise_search
├── logstash-output-elastic_app_search
└── logstash-output-elastic_workplace_search
logstash-integration-jdbc
├── logstash-input-jdbc
├── logstash-filter-jdbc_streaming
└── logstash-filter-jdbc_static
logstash-integration-kafka
├── logstash-input-kafka
└── logstash-output-kafka
logstash-integration-logstash
├── logstash-input-logstash
└── logstash-output-logstash
logstash-integration-rabbitmq
├── logstash-input-rabbitmq
└── logstash-output-rabbitmq
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-file
logstash-output-graphite
logstash-output-http
logstash-output-lumberjack
logstash-output-nagios
logstash-output-null
logstash-output-pipe
logstash-output-redis
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-webhdfs
logstash-patterns-core
[root@linux02 adm_linux02]# /usr/share/logstash/bin/logstash-plugin list --verbose
Using bundled JDK: /usr/share/logstash/jdk
logstash-codec-avro (3.4.1)
logstash-codec-cef (6.2.7)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.2)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.1)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.3.2)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.7)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.15)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.2.0)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.16.1)
logstash-filter-fingerprint (3.4.3)
logstash-filter-geoip (7.2.13)
logstash-filter-grok (4.4.3)
logstash-filter-http (1.5.1)
logstash-filter-json (3.2.1)
logstash-filter-kv (4.7.0)
logstash-filter-memcached (1.2.0)
logstash-filter-metrics (4.0.7)
logstash-filter-mutate (3.5.8)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.2.1)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.4.2)
logstash-filter-truncate (1.0.6)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.5)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.2.0)
logstash-input-azure_event_hubs (1.4.5)
logstash-input-beats (6.7.2)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (2.0.0)
logstash-input-elastic_serverless_forwarder (0.1.4)
logstash-input-elasticsearch (4.19.1)
logstash-input-exec (3.6.0)
logstash-input-file (4.4.6)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.2)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.8.0)
logstash-input-http_poller (5.5.1)
logstash-input-imap (3.2.1)
logstash-input-jms (3.2.2)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-snmp (1.3.3)
logstash-input-snmptrap (3.1.0)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.7.0)
logstash-input-tcp (6.4.1)
logstash-input-twitter (4.1.1)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.2)
logstash-integration-aws (7.1.6)
├── logstash-codec-cloudfront
├── logstash-codec-cloudtrail
├── logstash-input-cloudwatch
├── logstash-input-s3
├── logstash-input-sqs
├── logstash-output-cloudwatch
├── logstash-output-s3
├── logstash-output-sns
└── logstash-output-sqs
logstash-integration-elastic_enterprise_search (3.0.0)
├── logstash-output-elastic_app_search
└── logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.4.8)
├── logstash-input-jdbc
├── logstash-filter-jdbc_streaming
└── logstash-filter-jdbc_static
logstash-integration-kafka (11.3.4)
├── logstash-input-kafka
└── logstash-output-kafka
logstash-integration-logstash (1.0.2)
├── logstash-input-logstash
└── logstash-output-logstash
logstash-integration-rabbitmq (7.3.3)
├── logstash-input-rabbitmq
└── logstash-output-rabbitmq
logstash-output-csv (3.0.10)
logstash-output-elasticsearch (11.22.2)
logstash-output-email (4.1.3)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.6.0)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.1.2)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.1.0)
logstash-patterns-core (4.3.4)

JVM (e.g. java -version):

should use the bundled JDK, no other java installed on server

OS version (uname -a if on a Unix-like system):
RHEL 9.3
Linux servername.domain.local 5.14.0-362.24.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Feb 15 07:18:13 EST 2024 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

after update from version 8.12.2 to 8.13.0 service will not start.
logstash will not start when executing /usr/share/logstash/bin/logstash on console, too.
Error message: Could not find java; set LS_JAVA_HOME or ensure java is in PATH.
After a few debugging found the /usr/share/logstash/jdk folder containing the folder as the previos version (round about 87 folders), but NOT A SINGLE file.
No wonder he looks for alternative java installed when there is no bundled anymore.

reverted back to 8.12.2 via dnf: files in /usr/share/logstash/jdk are back, logstash is running again as service AND on console with output "Using bundled JDK: /usr/share/logstash/jdk" at the beginning.

Updating to 8.13.0 again results to the behaviour described above - jdk-folder contains folders only, not a single file, logstash not starting, error-message: Could not find java; set LS_JAVA_HOME or ensure java is in PATH.

downgrading to 8.12.2 again and logstash works fine.
can reproduce this in a circle

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. dnf clean all (be sure to be up to date)
2. dnf upgrade
3. confirm to update logstash from 8.12.2 to 8.13
4. after dnf has finished try to start logstash and verify the content of the bundled jdk-folder

Provide logs (if relevant):

Mar 26 19:31:49 servername logstash[51427]: Could not find java; set LS_JAVA_HOME or ensure java is in PATH.
Mar 26 19:31:49 servername systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 19:31:49 servername systemd[1]: logstash.service: Failed with result 'exit-code'.
Mar 26 19:31:49 servername systemd[1]: logstash.service: Scheduled restart job, restart counter is at 1.
Mar 26 19:31:49 servername systemd[1]: Stopped logstash.
Mar 26 19:31:49 servername systemd[1]: Started logstash.
Mar 26 19:31:49 servername logstash[51438]: Could not find java; set LS_JAVA_HOME or ensure java is in PATH.
Mar 26 19:31:49 servername systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 19:31:49 servername systemd[1]: logstash.service: Failed with result 'exit-code'.
Mar 26 19:31:49 servername systemd[1]: logstash.service: Scheduled restart job, restart counter is at 2.
Mar 26 19:31:49 servername systemd[1]: Stopped logstash.
Mar 26 19:31:49 servername systemd[1]: Started logstash.
Mar 26 19:31:49 servername logstash[51447]: Could not find java; set LS_JAVA_HOME or ensure java is in PATH.
Mar 26 19:31:49 servername systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 19:31:49 servername systemd[1]: logstash.service: Failed with result 'exit-code'.
Mar 26 19:31:50 servername systemd[1]: logstash.service: Scheduled restart job, restart counter is at 3.
Mar 26 19:31:50 servername systemd[1]: Stopped logstash.
Mar 26 19:31:50 servername systemd[1]: Started logstash.
Mar 26 19:31:50 servername logstash[51456]: Could not find java; set LS_JAVA_HOME or ensure java is in PATH.
Mar 26 19:31:50 servername systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 19:31:50 servername systemd[1]: logstash.service: Failed with result 'exit-code'.
Mar 26 19:31:50 servername systemd[1]: logstash.service: Scheduled restart job, restart counter is at 4.
Mar 26 19:31:50 servername systemd[1]: Stopped logstash.
Mar 26 19:31:50 servername systemd[1]: Started logstash.
Mar 26 19:31:50 servername logstash[51465]: Could not find java; set LS_JAVA_HOME or ensure java is in PATH.
Mar 26 19:31:50 servername systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 19:31:50 servername systemd[1]: logstash.service: Failed with result 'exit-code'.
Mar 26 19:31:50 servername systemd[1]: logstash.service: Scheduled restart job, restart counter is at 5.
Mar 26 19:31:50 servername systemd[1]: Stopped logstash.
Mar 26 19:31:50 servername systemd[1]: logstash.service: Start request repeated too quickly.
Mar 26 19:31:50 servername systemd[1]: logstash.service: Failed with result 'exit-code'.
Mar 26 19:31:50 servername systemd[1]: Failed to start logstash.

[ ]# ls -lisha /usr/share/logstash/jdk/bin
total 0
67110709 0 drwxr-xr-x. 2 root root 6 Mar 26 19:19 .
392218 0 drwxr-xr-x. 9 root root 92 Mar 26 19:19 ..
[]#

just as side node
while installing 8.12.2 dnf downloads about 333MB
while installing 8.13.0 dnf downloads about 210MB

either the code base REALLY shrinked with this version or there is missing a lot in the rpm-packages

[localhorst-org]

Same behavior under Ubuntu 22.04

Update from logstash 8.12.2 to 8.13.0

logstash install from https://artifacts.elastic.co/packages/8.x/apt stable main

no /usr/share/logstash/jdk/ folder

[gdsotirov]

Same with CentOS 8 Stream.

[jsvd]

Thanks for the report folks. it seems that the no-jdk package is being returned in the apt-get, as the artifacts for 8.13.0 in the download page here are ok, and include the jdk.

We're investigating what led to the debian package registry offering the no-jdk package by default instead of the one with JDK.

UPDATE: it looks like the no-jdk packages were incorrectly uploaded to the deb repository alongside the bundled jdk packages, and given they have the same priority, the no-jdk "all" architectures deb is being served first.
UPDATE: users will experience the same with RPM packages.

[MakoWish]

Any workaround for now aside from downgrading? Thankfully we only updated our Development Logstash instance and ran into this.

[andsel]

You can download the deb or rpm for you distribution directly from https://www.elastic.co/downloads/logstash, for example https://artifacts.elastic.co/downloads/logstash/logstash-8.13.0-amd64.deb without passing through the package manager's repository, and install using the file you download.

[walkerprog]

can confirm the rpm-package from https://www.elastic.co/de/downloads/past-releases/logstash-8-13-0 works as expected
installed with rpm -Uvh ./logstash-8.13.0-x86_64.rpm without problems, dnf no longer finds an update for logstash, files in jdk-folder are present and logstash-service is running

I really do hope dnf will recognize the next logstash-update and update from 8.13.0 to next version automatically

[ejgh-oe]

UPDATE: it looks like the no-jdk packages were incorrectly uploaded to the deb repository alongside the bundled jdk packages, and given they have the same priority, the no-jdk "all" architectures deb is being served first. UPDATE: users will experience the same with RPM packages.

Wrt. "deb": Also noticed that logstash went from amd64-package to "all" during the upgrade - your post explains alot... At least I now know I'm not alone with this...

[ejgh-oe]

Hi,
Looks like the update is already out: did a
apt update && apt upgrade
on my Ubuntu 22.04 a couple of minutes ago and logstash:amd64 including java was installed again.

Thanks much for the quick fix.

[andsel]

I can confirm, that the deb package that's on repository, is the correct one. Verified with

$ sudo apt-get download --print-uris logstash
'https://artifacts.elastic.co/packages/8.x/apt/pool/main/l/logstash/logstash-8.13.0-amd64.deb' logstash_1%3a8.13.0-1_amd64.deb 412887318 SHA512:d575daa58f0390be7a03931efe93a11c1e00e6ed4e9c5c5f7a9c3aad455d006967b83269ddba531e0fc70e1b140b058ce10bb65831c7fe9aadd63046b9fd526f

[andsel]

@walkerprog does it fix for you?

[walkerprog]

had already fixed it with manually updating with downloaded rpm-file

quick test with downgrading to 8.12.2 and update with dnf to 8.13.0 was successful
now the jdk-folder contains files (not only empty folders) and service starts and logstash is running although no other java is installed on the machine
incoming syslog is processed as usual

Mar 29 15:03:05 servername logstash[2627299]: Using bundled JDK: /usr/share/logstash/jdk

thx for fixing and PLEASE make sure this never happens again :)

[roaksoax]

Closing the issue as it has now been solved.