Auto-reloading .jar fails

[kryger]

Looks like the .jar file auto-reloads which was kind of surprising to me, especially since it doesn't seem to be handled very well - ugly zip-related exception in the log.

Steps to reproduce:

1. Create a Maven project (Jar) with only "Web" style ticked on http://start.spring.io
2. Unpack, do an mvn package, go to target/ and run: java -jar demo-0.0.1-SNAPSHOT.jar
3. Project works and runs on localhost:8080 (by "works" I mean that it serves the error 404, but that's not the point here)
4. With project still running directly from target/ do an mvn package again - the original .jar file will get overwritten
5. Prod localhost:8080 again: you'll get a stacktrace in the log:

Exception in thread "http-nio-8080-exec-10" java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
    at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:121)
    at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:440)
    at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396)
    at ch.qos.logback.classic.Logger.log(Logger.java:788)
    at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:224)
    at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:301)
    at java.util.logging.Logger.log(Logger.java:616)
    at java.util.logging.Logger.doLog(Logger.java:641)
    at java.util.logging.Logger.logp(Logger.java:841)
    at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185)
    at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1771)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:744)
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Tomcat].StandardHost[localhost].StandardContext[]]" java.lang.NoClassDefFoundError: org/apache/tomcat/util/ExceptionUtils
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1532)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1540)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1519)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: org.apache.tomcat.util.ExceptionUtils
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at org.springframework.boot.loader.LaunchedURLClassLoader.doLoadClass(LaunchedURLClassLoader.java:161)
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:131)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 4 more
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
    at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
    at org.springframework.boot.loader.jar.ZipInflaterInputStream.fill(ZipInflaterInputStream.java:62)
    at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
    at org.springframework.boot.loader.jar.ZipInflaterInputStream.read(ZipInflaterInputStream.java:52)
    at sun.misc.Resource.getBytes(Resource.java:124)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:444)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    ... 11 more

In my applicatiton this makes the application stop responding, but this simplified example still happily serves the 404 - hopefully you won't need an example app that shows it. The stack trace is also slightly different:

Exception in thread "http-nio-8080-Acceptor-0" java.lang.NoClassDefFoundError: org/apache/tomcat/util/ExceptionUtils
    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:840)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: org.apache.tomcat.util.ExceptionUtils
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at org.springframework.boot.loader.LaunchedURLClassLoader.doLoadClass(LaunchedURLClassLoader.java:161)
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:131)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 2 more
Caused by: java.util.zip.ZipException: invalid distance too far back
    at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
    at org.springframework.boot.loader.jar.ZipInflaterInputStream.read(ZipInflaterInputStream.java:52)
    at sun.misc.Resource.getBytes(Resource.java:124)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:444)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    ... 9 more

Example app also fails with Jetty, the stacktrace doesn't mention zip though:

Exception in thread "qtp2113070601-15" java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
    at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:121)
    at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:440)
    at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396)
    at ch.qos.logback.classic.Logger.log(Logger.java:788)
    at org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:607)
    at org.eclipse.jetty.util.log.JettyAwareLogger.warn(JettyAwareLogger.java:431)
    at org.eclipse.jetty.util.log.Slf4jLog.warn(Slf4jLog.java:69)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:726)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:744)
Exception in thread "qtp2113070601-16" java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
    at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:121)
    at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:440)
    at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396)
    at ch.qos.logback.classic.Logger.log(Logger.java:788)
    at org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:607)
    at org.eclipse.jetty.util.log.JettyAwareLogger.warn(JettyAwareLogger.java:431)
    at org.eclipse.jetty.util.log.Slf4jLog.warn(Slf4jLog.java:69)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:726)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:744)

[dsyer]

I wouldn't expect this to work really, and it doesn't seem like you did either. Is it a feature request?

[kryger]

Ah, forgot to add the actual question: is there a way to disable this reloading behaviour or is it something that java -jar just does and it's out of Boot's control? Not reloading would make the deployment script a bit easier to set up in Jenkins as there would be no need to stop the app before scp-ing.

[dsyer]

Good question. I doubt there's much we can do. Did you try running a simple (non-Boot launched) jar and see if that would ever work? Also interesting to know would be if it works with a Boot jar that isn't a webapp I guess. Switching it off might be possible if it is an artifact of the Boot classloader (hence the answers to the questions above might be interesting).

[kryger]

I tried Dropwizard's example app and it doesn't seem to even attempt to reload the .jar file, it serves the old contents after the file is overwritten. This suggests it's something that Boot does explicitly, right?

I could try with a non-web Boot app, but I've already used up all my investigation time for this issue for today, will do this later this week as it's not a stopper for me (...yet!). Thanks for quick reaction.

[philwebb]

The exception is most probably caused by the ground moving underneath the feet of the fat jar support. We have a somewhat unique implementation where we scan the contents of the nested JARs and work out where within the actual file the contents of an entry begins.

When you read a file or class from a nested JAR we actually seek to the relevant position in the file and read the contents. This allows us to have fat jars that look nice (i.e. all the classes from the nested aren't repackaged) and that don't need to unpack things to temporary directories.

When the jar is repackaged, the positions of the nested jars changes but the previously calculated index is not updated. Hence the ZLIB input stream exceptions.

One option would be for us to check the timestamp and size of the JAR whenever an entry is read to see if we need to reindex. I'm not too keen to add this additional logic, especially as most users seem to be running using springBoot:run during development time.

[toolforger]

Is it possible to put a write lock on the jar while the app is running?
Does the Spring Boot plugin rewrite the file in place? If so, it shouldn't, it could delete and recreate the jar; on Windows, it will fail if Boot's classloader keeps the file open (as it should), on Linux, the classloader will simply continue serving from the old, deleted file.

[sincang]

Was this issue resolve? Last February, 2014 I attemted to used the maven jar plugin but did not work. Just today, I am thinking I can employ this again in our application because this feature is really neat. Unfortunately, I get same error I had a year ago.

java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy