glassfish 7.0.20 fails to start

[rlfnb]

Tested today the official image provided by ee4j for glassfish with jmx_exporter.

I tried to add the files but was not allowed, so I paste them here:

cat Containerfile
FROM ghcr.io/eclipse-ee4j/glassfish:7.0.20
COPY config.yaml /opt/
COPY jmx_prometheus_javaagent-1.1.0.jar /opt/ 
RUN true \
  && AS_START_TIMEOUT=120000 asadmin start-domain \
  && curl  -o /dev/null http://localhost:4848 \
  && asadmin create-jvm-options "-javaagent\:/opt/jmx_prometheus_javaagent-1.1.0.jar=9090\:/opt/config.yaml" \
  && asadmin stop-domain --kill

cat config.yaml:
rules:
- pattern: ".*"

I build the image with podman build -t bug and started the image:

2025-01-28 15:44:23.425 | main | INFO | io.prometheus.jmx.JavaAgent | Starting ...
2025-01-28 15:44:23.541 | main | INFO | io.prometheus.jmx.JavaAgent | HTTP enabled [true]
2025-01-28 15:44:23.541 | main | INFO | io.prometheus.jmx.JavaAgent | HTTP host:port [0.0.0.0:9090]
2025-01-28 15:44:23.541 | main | INFO | io.prometheus.jmx.JavaAgent | OpenTelemetry enabled [false]
2025-01-28 15:44:23.558 | main | INFO | io.prometheus.jmx.JavaAgent | Running ...
Launching GlassFish on Felix platform
OSGI framework packages:
org.glassfish.main.jul;uses:="org.glassfish.main.jul.cfg,org.glassfish.main.jul.handler";version="7.0.20",org.glassfish.main.jul.cfg;version="7.0.20",org.glassfish.main.jul.env;version="7.0.20",org.glassfish.main.jul.formatter;uses:="org.glassfish.main.jul.cfg,org.glassfish.main.jul.record";version="7.0.20",org.glassfish.main.jul.handler;uses:="org.glassfish.main.jul.cfg,org.glassfish.main.jul.formatter,org.glassfish.main.jul.record";version="7.0.20",org.glassfish.main.jul.record;version="7.0.20",org.glassfish.main.jul.rotation;version="7.0.20",org.glassfish.main.jul.tracing;version="7.0.20", org.osgi.dto;version="1.1.1",org.osgi.framework;version="1.10",org.osgi.framework.connect;uses:="org.osgi.framework,org.osgi.framework.launch";version="1.0",org.osgi.framework.dto;uses:="org.osgi.dto";version="1.8",org.osgi.framework.hooks.bundle;uses:="org.osgi.framework";version="1.1",org.osgi.framework.hooks.resolver;uses:="org.osgi.framework.wiring";version="1.0",org.osgi.framework.hooks.service;uses:="org.osgi.framework";version="1.1",org.osgi.framework.hooks.weaving;uses:="org.osgi.framework.wiring";version="1.1",org.osgi.framework.launch;uses:="org.osgi.framework";version="1.2",org.osgi.framework.namespace;uses:="org.osgi.resource";version="1.2",org.osgi.framework.startlevel;uses:="org.osgi.framework";version="1.0",org.osgi.framework.startlevel.dto;uses:="org.osgi.dto";version="1.0",org.osgi.framework.wiring;uses:="org.osgi.framework,org.osgi.resource";version="1.2",org.osgi.framework.wiring.dto;uses:="org.osgi.dto,org.osgi.resource.dto";version="1.3",org.osgi.resource;version="1.0.1",org.osgi.resource.dto;uses:="org.osgi.dto";version="1.0.1",org.osgi.service.condition;version="1.0",org.osgi.service.packageadmin;uses:="org.osgi.framework";version="1.2.1",org.osgi.service.resolver;uses:="org.osgi.resource";version="1.1.1",org.osgi.service.startlevel;uses:="org.osgi.framework";version="1.1.1",org.osgi.service.url;version="1.0.1",org.osgi.util.tracker;uses:="org.osgi.framework";version="1.5.3", org.glassfish.embeddable;version="7.0.20",org.glassfish.embeddable.spi;uses:="org.glassfish.embeddable";version="7.0.20", org.glassfish.main.jul;uses:="org.glassfish.main.jul.cfg,org.glassfish.main.jul.handler";version="7.0.20",org.glassfish.main.jul.cfg;version="7.0.20",org.glassfish.main.jul.env;version="7.0.20",org.glassfish.main.jul.formatter;uses:="org.glassfish.main.jul.cfg,org.glassfish.main.jul.record";version="7.0.20",org.glassfish.main.jul.handler;uses:="org.glassfish.main.jul.cfg,org.glassfish.main.jul.formatter,org.glassfish.main.jul.record";version="7.0.20",org.glassfish.main.jul.record;version="7.0.20",org.glassfish.main.jul.rotation;version="7.0.20",org.glassfish.main.jul.tracing;version="7.0.20"
JDK provided packages:
com.sun.jarsigner, com.sun.java.accessibility.util, com.sun.jdi, com.sun.jdi.connect, com.sun.jdi.connect.spi, com.sun.jdi.event, com.sun.jdi.request, com.sun.management, com.sun.net.httpserver, com.sun.net.httpserver.spi, com.sun.nio.file, com.sun.nio.sctp, com.sun.security.auth, com.sun.security.auth.callback, com.sun.security.auth.login, com.sun.security.auth.module, com.sun.security.jgss, com.sun.source.doctree, com.sun.source.tree, com.sun.source.util, com.sun.tools.attach, com.sun.tools.attach.spi, com.sun.tools.javac, com.sun.tools.jconsole, java.applet, java.awt, java.awt.color, java.awt.datatransfer, java.awt.desktop, java.awt.dnd, java.awt.event, java.awt.font, java.awt.geom, java.awt.im, java.awt.im.spi, java.awt.image, java.awt.image.renderable, java.awt.print, java.beans, java.beans.beancontext, java.io, java.lang, java.lang.annotation, java.lang.constant, java.lang.instrument, java.lang.invoke, java.lang.management, java.lang.module, java.lang.ref, java.lang.reflect, java.lang.runtime, java.math, java.net, java.net.http, java.net.spi, java.nio, java.nio.channels, java.nio.channels.spi, java.nio.charset, java.nio.charset.spi, java.nio.file, java.nio.file.attribute, java.nio.file.spi, java.rmi, java.rmi.dgc, java.rmi.registry, java.rmi.server, java.security, java.security.cert, java.security.interfaces, java.security.spec, java.sql, java.text, java.text.spi, java.time, java.time.chrono, java.time.format, java.time.temporal, java.time.zone, java.util, java.util.concurrent, java.util.concurrent.atomic, java.util.concurrent.locks, java.util.function, java.util.jar, java.util.logging, java.util.prefs, java.util.random, java.util.regex, java.util.spi, java.util.stream, java.util.zip, javax.accessibility, javax.annotation.processing, javax.crypto, javax.crypto.interfaces, javax.crypto.spec, javax.imageio, javax.imageio.event, javax.imageio.metadata, javax.imageio.plugins.bmp, javax.imageio.plugins.jpeg, javax.imageio.plugins.tiff, javax.imageio.spi, javax.imageio.stream, javax.lang.model, javax.lang.model.element, javax.lang.model.type, javax.lang.model.util, javax.management, javax.management.loading, javax.management.modelmbean, javax.management.monitor, javax.management.openmbean, javax.management.relation, javax.management.remote, javax.management.remote.rmi, javax.management.timer, javax.naming, javax.naming.directory, javax.naming.event, javax.naming.ldap, javax.naming.ldap.spi, javax.naming.spi, javax.net, javax.net.ssl, javax.print, javax.print.attribute, javax.print.attribute.standard, javax.print.event, javax.rmi.ssl, javax.script, javax.security.auth, javax.security.auth.callback, javax.security.auth.kerberos, javax.security.auth.login, javax.security.auth.spi, javax.security.auth.x500, javax.security.cert, javax.security.sasl, javax.smartcardio, javax.sound.midi, javax.sound.midi.spi, javax.sound.sampled, javax.sound.sampled.spi, javax.sql, javax.sql.rowset, javax.sql.rowset.serial, javax.sql.rowset.spi, javax.swing, javax.swing.border, javax.swing.colorchooser, javax.swing.event, javax.swing.filechooser, javax.swing.plaf, javax.swing.plaf.basic, javax.swing.plaf.metal, javax.swing.plaf.multi, javax.swing.plaf.nimbus, javax.swing.plaf.synth, javax.swing.table, javax.swing.text, javax.swing.text.html, javax.swing.text.html.parser, javax.swing.text.rtf, javax.swing.tree, javax.swing.undo, javax.tools, javax.transaction.xa, javax.xml, javax.xml.catalog, javax.xml.crypto, javax.xml.crypto.dom, javax.xml.crypto.dsig, javax.xml.crypto.dsig.dom, javax.xml.crypto.dsig.keyinfo, javax.xml.crypto.dsig.spec, javax.xml.datatype, javax.xml.namespace, javax.xml.parsers, javax.xml.stream, javax.xml.stream.events, javax.xml.stream.util, javax.xml.transform, javax.xml.transform.dom, javax.xml.transform.sax, javax.xml.transform.stax, javax.xml.transform.stream, javax.xml.validation, javax.xml.xpath, jdk.dynalink, jdk.dynalink.beans, jdk.dynalink.linker, jdk.dynalink.linker.support, jdk.dynalink.support, jdk.javadoc.doclet, jdk.jfr, jdk.jfr.consumer, jdk.jshell, jdk.jshell.execution, jdk.jshell.spi, jdk.jshell.tool, jdk.management.jfr, jdk.net, jdk.nio, jdk.nio.mapmode, jdk.security.jarsigner, jdk.swing.interop, netscape.javascript, org.ietf.jgss, org.w3c.dom, org.w3c.dom.bootstrap, org.w3c.dom.css, org.w3c.dom.events, org.w3c.dom.html, org.w3c.dom.ls, org.w3c.dom.ranges, org.w3c.dom.stylesheets, org.w3c.dom.traversal, org.w3c.dom.views, org.w3c.dom.xpath, org.xml.sax, org.xml.sax.ext, org.xml.sax.helpers, sun.misc, sun.reflect
Jan 28, 2025 3:44:25 PM com.sun.enterprise.server.logging.LogManagerService postConstruct
INFO: Detected other than GlassFishLogManager, the LogManagerService's features may be limited. Used log manager: java.util.logging.LogManager@7acac06a
Jan 28, 2025 3:44:25 PM com.sun.enterprise.server.logging.LogManagerService reconfigure
INFO: Using property file: /opt/glassfish7/glassfish/domains/domain1/config/logging.properties
Can't load log handler "org.glassfish.main.jul.handler.SimpleLogHandler"
java.lang.ClassNotFoundException: org.glassfish.main.jul.handler.SimpleLogHandler
java.lang.ClassNotFoundException: org.glassfish.main.jul.handler.SimpleLogHandler
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	at java.logging/java.util.logging.LogManager.createLoggerHandlers(LogManager.java:1005)
	at java.logging/java.util.logging.LogManager$4.run(LogManager.java:975)
	at java.logging/java.util.logging.LogManager$4.run(LogManager.java:971)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
	at java.logging/java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:971)
	at java.logging/java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:2424)
	at java.logging/java.util.logging.LogManager$RootLogger.accessCheckedHandlers(LogManager.java:2526)
	at java.logging/java.util.logging.Logger.getHandlers(Logger.java:2090)
	at java.logging/java.util.logging.Logger.log(Logger.java:977)
	at java.logging/java.util.logging.Logger.doLog(Logger.java:1007)
	at java.logging/java.util.logging.Logger.log(Logger.java:1073)
	at com.sun.enterprise.server.logging.LogManagerService.postConstruct(LogManagerService.java:127)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
	at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:362)
	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:410)
	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:479)
	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:288)
	at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2111)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:68)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1366)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1294)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Can't load log handler "org.glassfish.main.jul.handler.GlassFishLogHandler"
java.lang.ClassNotFoundException: org.glassfish.main.jul.handler.GlassFishLogHandler
java.lang.ClassNotFoundException: org.glassfish.main.jul.handler.GlassFishLogHandler
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	at java.logging/java.util.logging.LogManager.createLoggerHandlers(LogManager.java:1005)
	at java.logging/java.util.logging.LogManager$4.run(LogManager.java:975)
	at java.logging/java.util.logging.LogManager$4.run(LogManager.java:971)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
	at java.logging/java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:971)
	at java.logging/java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:2424)
	at java.logging/java.util.logging.LogManager$RootLogger.accessCheckedHandlers(LogManager.java:2526)
	at java.logging/java.util.logging.Logger.getHandlers(Logger.java:2090)
	at java.logging/java.util.logging.Logger.log(Logger.java:977)
	at java.logging/java.util.logging.Logger.doLog(Logger.java:1007)
	at java.logging/java.util.logging.Logger.log(Logger.java:1073)
	at com.sun.enterprise.server.logging.LogManagerService.postConstruct(LogManagerService.java:127)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
	at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:362)
	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:410)
	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:479)
	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:288)
	at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2111)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:68)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1366)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1294)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Can't load log handler "org.glassfish.main.jul.handler.SyslogHandler"
java.lang.ClassNotFoundException: org.glassfish.main.jul.handler.SyslogHandler
java.lang.ClassNotFoundException: org.glassfish.main.jul.handler.SyslogHandler
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	at java.logging/java.util.logging.LogManager.createLoggerHandlers(LogManager.java:1005)
	at java.logging/java.util.logging.LogManager$4.run(LogManager.java:975)
	at java.logging/java.util.logging.LogManager$4.run(LogManager.java:971)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
	at java.logging/java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:971)
	at java.logging/java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:2424)
	at java.logging/java.util.logging.LogManager$RootLogger.accessCheckedHandlers(LogManager.java:2526)
	at java.logging/java.util.logging.Logger.getHandlers(Logger.java:2090)
	at java.logging/java.util.logging.Logger.log(Logger.java:977)
	at java.logging/java.util.logging.Logger.doLog(Logger.java:1007)
	at java.logging/java.util.logging.Logger.log(Logger.java:1073)
	at com.sun.enterprise.server.logging.LogManagerService.postConstruct(LogManagerService.java:127)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
	at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:362)
	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:410)
	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:479)
	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:288)
	at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2111)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:68)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1366)
	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1294)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
SSLParams =org.glassfish.admin.mbeanserver.ssl.SSLParams@47c50497
SSLParams =org.glassfish.admin.mbeanserver.ssl.SSLParams@47c50497

Needless to say everything works as expected WITHOUT the java property for the agent. TBH, I have no idea where to start?

[dhoard]

@rlfnb The issue is that the JMX Exporter Java Agent runs in a different classloader than the application and can't find the required glassfish logging jars.

This appears similar to JMX Exporter issues...

• Cannot run JMX Exporter in Wildfly 17 running on Java 11 #455
• JMX agent interfering with logging format of application #976
• Issue in Integrating jmx_prometheus_javaagent with Jboss #1000

... and GlashFish issues...

• JDK loggers can initialize the JDK LogManager before GlassFishLogManager is set eclipse-ee4j/glassfish#25133
• Redesign of GlassFish bootstrap eclipse-ee4j/glassfish#25183

Background

The JMX Exporter Java Agent uses standard Java logging. Because standard Java logging is loaded as part of the Java agent before GlassFish, the wrong Java logging implementation is loaded.

Typically, you would need to find all of the GlassFish jars and add them to the boot classpath, along with defining a Java system property to define the LogManager implementation. (reference: #455)

However, per the comment by @dmatej in GlassFish issue eclipse-ee4j/glassfish#25133, I'm not sure there is a viable configuration workaround.

We may need to revisit logging in the JMX Exporter.

cc @fstab