Gradle multimodule project + quarkus-container-image-jib: OverlappingFileLockException

[andreas-eberle]

Describe the bug
I have a gradle + Kotlin multi-module project and try to build container images with the quarkus-container-image-jib extension. When I run ./gradlew assemble -Dquarkus.container-image.build=true, I get an OverlappingFileLockException (on Windows 10 as well as on the gradle:6.6-jdk11 docker image).

Apart from the issues I described in #11900, I can build both images by manually running ./gradlew :serviceA:assemble -Dquarkus.container-image.build=true and then ./gradlew :serviceB:assemble -Dquarkus.container-image.build=true.

Detailed exception:

17:28:34: Executing task 'assemble -Dquarkus.container-image.build=true'...


> Task :serviceA:quarkusGenerateCode
preparing quarkus application
kotlin scripting plugin: project ':serviceA'.compileQuarkus-generated-sourcesKotlin - configuration not found: quarkus-generated-sourcesKotlinScriptDef, the plugin is probably applied by a mistake

> Task :serviceB:quarkusGenerateCode
preparing quarkus application
kotlin scripting plugin: project ':serviceB'.compileQuarkus-generated-sourcesKotlin - configuration not found: quarkus-generated-sourcesKotlinScriptDef, the plugin is probably applied by a mistake

> Task :serviceA:quarkusGenerateCodeTests
preparing quarkus application
kotlin scripting plugin: project ':serviceA'.compileQuarkus-test-generated-sourcesKotlin - configuration not found: quarkus-test-generated-sourcesKotlinScriptDef, the plugin is probably applied by a mistake

> Task :serviceB:quarkusGenerateCodeTests
preparing quarkus application
kotlin scripting plugin: project ':serviceB'.compileQuarkus-test-generated-sourcesKotlin - configuration not found: quarkus-test-generated-sourcesKotlinScriptDef, the plugin is probably applied by a mistake

> Task :serviceB:compileKotlin
> Task :serviceA:compileKotlin
> Task :serviceA:compileJava NO-SOURCE
> Task :serviceB:compileJava NO-SOURCE
> Task :serviceA:processResources
> Task :serviceB:processResources
> Task :serviceA:classes
> Task :serviceB:classes
> Task :serviceA:inspectClassesForKotlinIC
> Task :serviceB:inspectClassesForKotlinIC
> Task :serviceA:jar
> Task :serviceB:jar

> Task :serviceA:quarkusBuild
building quarkus jar

Base image 'fabric8/java-alpine-openjdk11-jre' does not use a specific image digest - build may not be reproducible
Base image 'fabric8/java-alpine-openjdk11-jre' does not use a specific image digest - build may not be reproducible

> Task :serviceB:quarkusBuild
building quarkus jar

> Task :serviceB:quarkusBuild FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':serviceB:quarkusBuild'.
> io.quarkus.builder.BuildException: Build failure: Build failed due to errors
  	[error]: Build step io.quarkus.container.image.jib.deployment.JibProcessor#buildFromJar threw an exception: java.lang.RuntimeException: Unable to create container image
  	at io.quarkus.container.image.jib.deployment.JibProcessor.containerize(JibProcessor.java:153)
  	at io.quarkus.container.image.jib.deployment.JibProcessor.buildFromJar(JibProcessor.java:102)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
  	at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:932)
  	at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
  	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
  	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
  	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
  	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
  	at java.base/java.lang.Thread.run(Thread.java:830)
  	at org.jboss.threads.JBossThread.run(JBossThread.java:479)
  Caused by: java.util.concurrent.ExecutionException: java.nio.channels.OverlappingFileLockException
  	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552)
  	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:533)
  	at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82)
  	at com.google.cloud.tools.jib.builder.steps.StepsRunner.lambda$buildImage$8(StepsRunner.java:344)
  	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
  	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
  	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:830)
  Caused by: java.nio.channels.OverlappingFileLockException
  	at java.base/sun.nio.ch.FileLockTable.checkList(FileLockTable.java:229)
  	at java.base/sun.nio.ch.FileLockTable.add(FileLockTable.java:123)
  	at java.base/sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1110)
  	at java.base/java.nio.channels.FileChannel.lock(FileChannel.java:1076)
  	at com.google.cloud.tools.jib.filesystem.LockFile.lock(LockFile.java:67)
  	at com.google.cloud.tools.jib.cache.CacheStorageWriter.writeMetadata(CacheStorageWriter.java:315)
  	at com.google.cloud.tools.jib.cache.Cache.writeMetadata(Cache.java:81)
  	at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.pullBaseImage(PullBaseImageStep.java:249)
  	at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call(PullBaseImageStep.java:160)
  	at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call(PullBaseImageStep.java:56)
  	... 6 more


* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 26s
14 actionable tasks: 14 executed
17:29:00: Task execution finished 'assemble -Dquarkus.container-image.build=true'.

To Reproduce
Steps to reproduce the behavior:

1. Dowload reproducer project and unzip it 2020-09-04_jib-multimodule-problems.zip
2. Run ./gradlew assemble -Dquarkus.container-image.build=true
3. See the error shown above.

Environment (please complete the following information):

• Windows 10 (also happens on gradle:6.6-jdk11 docker image in my CI builds)
• Gradle
• Kotlin

[andreas-eberle]

As a workaround, it works when you specify quarkusBuild.mustRunAfter(':serviceA:quarkusBuild') in the build.gradle file of serviceB. This ensures the quarkus tasks and therefore the jib image building is run sequentially.

However, I don't see why they shouldn't be able to run in parallel.

[quarkusbot]

/cc @geoand
/cc @quarkusio/devtools, @glefloch

[geoand]

I was not able to reproduce this on Linux unfortunately.

[rquinio]

I do see the same behavior with a Maven multi-module project that builds 2 Quarkus application in parallel (-T 1C) on Windows 10 & MacOS (Quarkus 1.7.6.Final, OpenJDK 11.0.8)

<quarkus.container-image.build>true</quarkus.container-image.build>
<quarkus.container-image.push>false</quarkus.container-image.push>

[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:1.7.6.Final:build (augmentation) on project (...): Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR]         [error]: Build step io.quarkus.container.image.jib.deployment.JibProcessor#buildFromJar threw an exception: java.lang.RuntimeException: Unable to create container image
[ERROR]         at io.quarkus.container.image.jib.deployment.JibProcessor.containerize(JibProcessor.java:153)
[ERROR]         at io.quarkus.container.image.jib.deployment.JibProcessor.buildFromJar(JibProcessor.java:102)
[ERROR]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[ERROR]         at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:932)
[ERROR]         at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
[ERROR]         at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[ERROR]         at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
[ERROR]         at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
[ERROR]         at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
[ERROR]         at java.base/java.lang.Thread.run(Thread.java:834)
[ERROR]         at org.jboss.threads.JBossThread.run(JBossThread.java:479)
[ERROR] Caused by: java.util.concurrent.ExecutionException: java.nio.channels.OverlappingFileLockException
[ERROR]         at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552)
[ERROR]         at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:533)
[ERROR]         at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82)
[ERROR]         at com.google.cloud.tools.jib.builder.steps.StepsRunner.lambda$buildImage$8(StepsRunner.java:344)
[ERROR]         at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
[ERROR]         at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
[ERROR]         at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
[ERROR]         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[ERROR]         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[ERROR]         at java.base/java.lang.Thread.run(Thread.java:834)
[ERROR] Caused by: java.nio.channels.OverlappingFileLockException
[ERROR]         at java.base/sun.nio.ch.FileLockTable.checkList(FileLockTable.java:229)
[ERROR]         at java.base/sun.nio.ch.FileLockTable.add(FileLockTable.java:123)
[ERROR]         at java.base/sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1109)
[ERROR]         at java.base/java.nio.channels.FileChannel.lock(FileChannel.java:1063)
[ERROR]         at com.google.cloud.tools.jib.filesystem.LockFile.lock(LockFile.java:67)
[ERROR]         at com.google.cloud.tools.jib.cache.CacheStorageWriter.writeMetadata(CacheStorageWriter.java:315)
[ERROR]         at com.google.cloud.tools.jib.cache.Cache.writeMetadata(Cache.java:81)
[ERROR]         at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.pullBaseImage(PullBaseImageStep.java:249)
[ERROR]         at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call(PullBaseImageStep.java:160)
[ERROR]         at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call(PullBaseImageStep.java:56)
[ERROR]         ... 6 more

I see an old issue on Jib GoogleContainerTools/jib#922, but it's supposed to already be fixed with 0.10.0

[tobad357]

Chime in to say I see the same on MacOS X 11.3.1 with AdoptOpenJDK 16
Multi module Gradle build which ends up in OverlappingFileLockException

[bademux]

problem exists, not only in quarkus
temporary fix:

build.gradle

['jibDockerBuild', 'jibBuildTar', 'jib'].each { taskName ->
    getTasksByName(taskName, true).with {
        withIndex(1).collectEntries({ task, idxNext -> [(task): it[idxNext]] }).findAll({ it.value != null }).each(Task::mustRunAfter)
    }
}

[geoand]

Is this still an issue? I am still not able to reproduce it

[bademux]

@geoand yep GoogleContainerTools/jib#3347 (comment)

[geoand]

Thanks

[rquinio]

The cause appears to be that Quarkus Maven/Gradle plugins do not share the classloaders across sub-modules (different QuarkusClassLoader:Deployment Class Loader and QuarkusClassLoader:Augmentation Class Loader ).

Jib uses a static ConcurrentHashMap to store the multi-threading locks, which assumes LockFile.class will be loaded only once across all sub-modules:
https://github.com/GoogleContainerTools/jib/blob/9027c8dafb8ee12b9d37e1521caf62dfbcc3d374/jib-core/src/main/java/com/google/cloud/tools/jib/filesystem/LockFile.java#L34

I'm seeing a new instance of the map for each thread, which bypasses the thread lock.
This randomly causes OverlappingFileLockException as the FileChannel locking mechanism is designed to lock files against concurrent access by two separate processes, and the JVM complains when it detects the access is from the same process.

[geoand]

Thanks a lot for the analysis @rquinio!

We could probably get around this by loading Jib "parentFirst" (which would result in Jib code being loaded from the System ClassLoader) but that could potentially cause other problems.

[andreas-eberle]

Is there any news on this topic? If I understand the comment from @rquinio on the linked topic correctly (GoogleContainerTools/jib#3347 (comment)), it seems that jib offers an example repo with a multimodule setup that seems to work. For us, this is still a pain as it causes random pipeline crashes :(

[geoand]

Can folks that still encounter this problem give #34873 a try please?

[famod]

We came up with another approach:
@andreas-eberle @rquinio @bademux @tobad357 can anyone of you give #35308 a try please?

[aloubyansky]

Please re-open in case #35308 did not fix the issue, thanks everyone.