com.mongodb.MongoTimeoutException causes failure to start Jetty service

[Greatpomps]

I have 4 VMs running with jetty web service for an App that connects to MongdoDB
Hence there is an Issue with connectivity issue with Mongodb.
I looked in the logs i :
servers=[{address=xxx.xxx.xxx.xxx:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.SocketTimeoutException: connect timed out}},

Not convinced this is just a network issue becos
VM 1 jetty is running and trace route to mongodb is fine
but VM2 jetty is running but trace route to mongodb is failing with the error mentioned above
VM3 and VM4 are failed and no trace route to Mongodb with the same failing with the error mentioned above
NB:the route table is upto date and config is fine.
see details below:

VM 4:
● jetty9.service - LSB: Jetty start script.
Loaded: loaded (/etc/init.d/jetty9; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2024-01-18 06:09:52 UTC; 2h 38min ago
Docs: man:systemd-sysv-generator(8)
Process: 3064 ExecStop=/etc/init.d/jetty9 stop (code=exited, status=0/SUCCESS)
Process: 21589 ExecStart=/etc/init.d/jetty9 start (code=exited, status=1/FAILURE)

Jan 18 06:09:16 BR01 systemd[1]: Starting LSB: Jetty start script....
Jan 18 06:09:16 BR01 jetty9[21589]: ** WARNING: JETTY_LOGS is Deprecated. Please configure logging within the jetty base.
Jan 18 06:09:52 BR01 jetty9[21589]: Starting Jetty: . . . . . . . . FAILED Thu Jan 18 06:09:52 UTC 2024
Jan 18 06:09:52 BR01 systemd[1]: jetty9.service: Control process exited, code=exited status=1
Jan 18 06:09:52 BR01 systemd[1]: Failed to start LSB: Jetty start script..
Jan 18 06:09:52 BR01 systemd[1]: jetty9.service: Unit entered failed state.
Jan 18 06:09:52 BR01 systemd[1]: jetty9.service: Failed with result 'exit-code'.

VM3:
● jetty9.service - LSB: Jetty start script.
Loaded: loaded (/etc/init.d/jetty9; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2024-01-18 07:29:02 UTC; 1h 21min ago
Docs: man:systemd-sysv-generator(8)
Process: 20598 ExecStart=/etc/init.d/jetty9 start (code=exited, status=1/FAILURE)

Jan 18 07:28:58 OTR01 systemd[1]: Starting LSB: Jetty start script....
Jan 18 07:28:58 OR01 jetty9[20598]: ** WARNING: JETTY_LOGS is Deprecated. Please configure logging within the jetty base.
Jan 18 07:29:02 OR01 jetty9[20598]: Starting Jetty: FAILED Thu Jan 18 07:29:02 UTC 2024
Jan 18 07:29:02 OR01 systemd[1]: jetty9.service: Control process exited, code=exited status=1
Jan 18 07:29:02 OR01 systemd[1]: Failed to start LSB: Jetty start script..
Jan 18 07:29:02 OR01 systemd[1]: jetty9.service: Unit entered failed state.
Jan 18 07:29:02 OR01 systemd[1]: jetty9.service: Failed with result 'exit-code'.

VM2:●
jetty9.service - LSB: Jetty start script.
Loaded: loaded (/etc/init.d/jetty9; bad; vendor preset: enabled)
Active: active (running) since Tue 2023-08-22 20:20:09 UTC; 4 months 26 days ago
Docs: man:systemd-sysv-generator(8)
Process: 20141 ExecStop=/etc/init.d/jetty9 stop (code=exited, status=0/SUCCESS)
Process: 20904 ExecStart=/etc/init.d/jetty9 start (code=exited, status=0/SUCCESS)
Tasks: 243
Memory: 799.0M
CPU: 16h 46min 28.547s
CGroup: /system.slice/jetty9.service
└─20931 /usr/bin/java -Djetty.home=/opt/jetty9 -Djetty.base=/opt/jetty9 -Djava.io.tmpdir=/tmp -jar /opt/jetty9/start.jar jetty.state=/opt/jetty9/j

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

VM1:●
jetty.service - SYSV: Jetty 9 webserver
Loaded: loaded (/etc/init.d/jetty; bad; vendor preset: enabled)
Active: active (running) since Tue 2023-08-29 22:14:19 UTC; 4 months 19 days ago
Docs: man:systemd-sysv-generator(8)
CGroup: /system.slice/jetty.service
├─ 760 ping -Q 0x28 -c 10 XX.XX.XX.XX
├─ 879 ping -Q 0x28 -c 10 XX.XX.XX.XX
├─ 880 traceroute -m 17 -t 0x28 -q 4 -n XX.XX.XX.XX
└─21060 /usr/bin/java -Djetty.logging.dir=/opt/jetty/logs -Djetty.home=/opt/jetty -Djetty.base=/opt/jetty -Djava.io.tmpdir=/tmp -jar /opt/jetty/st

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Anyone seen this sort of issue before?

[joakime]

Jetty 9 is at end of Community Support

• EoCS (End of Community Support) for Jetty 9 - June 2022 #7958

Note that Jetty 10 and Jetty 11 are also now at End of Community Support

• EoCS (End of Community Support) for Jetty 10 / Jetty 11 - January 2024 #10485

You should be using Jetty 12 at this point in time.
Note: Jetty 12's ee8 environment still supports javax.servlet. namespace.

[joakime]

Note: Jetty 9 uses MongoDB Java Driver 2.14.3 which is at end of life by the MongoDB project.

The MongoDB Java Driver version 2.14.3 only supports MongoDB versions before MongoDB 2.6
All of those versions of MongoDB are also at end of life.

https://www.mongodb.com/docs/drivers/java/sync/current/compatibility/#compatibility-table-legend

If you are using a MongoDB 2.6 or newer you are using a version incompatible with the support provided by Jetty 9.

[Greatpomps]

Hi,
This is the version.
xxxxx@VM-DB-01:~$ mongod -version
db version v3.6.23
git version: d352e6a4764659e0d0350ce77279de3c1f243e5c
OpenSSL version: OpenSSL 1.1.1 11 Sep 2018
allocator: tcmalloc
modules: none
build environment:
distmod: ubuntu1804
distarch: x86_64
target_arch: x86_64

[joakime]

Per mongo, that database version (3.6.23) is incompatible and unsupported for the MongoDB Java Driver 2.14.3 (what Jetty 9 uses).

Again, you have to upgrade.

[Greatpomps]

Thanks @joakime Joakime..Just wondering why onE of the VM s running on Jetty 9 is still working well with Mongo (3.6.23) .
This is the working VM:
[sudo] password for perfadmin:
● jetty.service - SYSV: Jetty 9 webserver
Loaded: loaded (/etc/init.d/jetty; bad; vendor preset: enabled)
Active: active (running) since Tue 2023-08-29 22:14:19 UTC; 4 months 23 days
Docs: man:systemd-sysv-generator(8)
CGroup: /system.slice/jetty.service
├─21060 /usr/bin/java -Djetty.logging.dir=/opt/jetty/logs -Djetty.hom
├─30811 traceroute -m 17 -t 0x28 -q 4 -n XX.XX.XXX
├─30977 ping -Q 0x28 -c 10 XX.XX.XX.XX
├─30978 traceroute -m 17 -t 0x28 -q 4 -n XX.XX.XX.XX
├─30983 ping -Q 0x28 -c 10 10.77.13.138
└─30984 traceroute -m 17 -t 0x28 -q 4 -n XX.XX.XX.XX

Warning: Journal has been rotated since unit was started. Log output is incomple
lines 1-13/13 (END)

Jetty Environment:

jetty.version = 9.3.6.v20151106
jetty.tag.version = master
jetty.home = /opt/jetty
jetty.base = /opt/jetty

Config Search Order:

${jetty.base} -> /opt/jetty ${jetty.home} -> /opt/jetty

Properties:

java.version = 1.8.0_191
java.version.major = 1
java.version.minor = 8
java.version.revision = 0
java.version.update = 191

[joakime]

Thanks @joakime Joakime..Just wondering why onE of the VM s running on Jetty 9 is still working well with Mongo (3.6.23) .

We cannot answer that.
That's for the folks at MongoDB to answer.
Their own documentation states that it is unsupported. That's the limits of what we know.

Reminder, Jetty 9.x is now at End of Community Support.

If you experience any bugs with Jetty 12 (the only community supported version of Jetty at this point in time) we can help you.