Concurrent image builds cause error deleting builder image

[scottfrederick]

When the Maven spring-boot:build-image goal or Gradle bootBuildImage task is run more than once concurrently against the same Docker daemon, builds can fail with an error like

Docker API call to 'localhost/v1.24/images/pack.local/builder/zuslfxipuv:latest?force=1' failed with status code 500 "Internal Server Error" and message "unrecognized image ID sha256:xxxxx"`

[scottfrederick]

When the Spring Boot plugins CNB integration builds an image, it follows a series of steps:

• pull the builder image (by default paketobuildpacks/builder:base)
• clone the builder image to create an ephemeral builder image with a name like pack.local/builder/[random string]
• embellish the ephemeral builder image with any environment variables or buildpacks specified in the build configuration along with metadata like the Spring Boot plugin version
• create a container from the ephemeral builder image and run the CNB lifecycle in the container
• delete the ephemeral builder container and image

If Spring Boot runs two builds in parallel, then two ephemeral builder images get created. If the base image, build parameters, and image metadata are identical then internally the Docker daemon creates one image with two tags instead of creating two unique images. Listing images in the Docker daemon while the build is running might look like this (note that both ephemeral images have the same IMAGE ID:

$ docker images
REPOSITORY                          TAG        IMAGE ID       CREATED        SIZE
paketobuildpacks/builder            base       06d50806f1c2   41 years ago   756MB
pack.local/builder/pcspywwker       latest     316c12b7dd8d   41 years ago   756MB
pack.local/builder/sstditlopy       latest     316c12b7dd8d   41 years ago   756MB

This can result in a race condition inside the Docker daemon when Spring Boot asks Docker to delete the two ephemeral images at exactly the same time. Under just the right conditions the daemon will successfully delete the single image referred to by one tag and throw an unrecognized image ID sha256:316c12b7dd8d... error when it tries to delete the second tag because the underlying image has already been deleted.

[dokaspar]

Hi @scottfrederick
Does this imply that the only solution to the problem is upgrading to at least version 2.4.11?
Or are there any other workarounds, such as running Gradle with the --no-parallel flag?
Regards
Dominik

[scottfrederick]

Does this imply that the only solution to the problem is upgrading to at least version 2.4.11?

Yes, that's the only way to be sure you won't encounter this problem.

Also note that Spring Boot 2.4.x and earlier are out of open-source support. You should upgrade to a supported version of Spring Boot.

Or are there any other workarounds, such as running Gradle with the --no-parallel flag?

Gradle's --no-parallel flag might work if you have multiple images being built from the same Gradle build. It won't help if you are running multiple Maven or Gradle builds against the same Docker daemon in separate processes, as is common in CI systems.

[dokaspar]

Thanks a lot for the quick response and the clarifications!