Certain Native image artifacts are not copied to run image #308
Comments
|
Native Image also provides an experimental option |
@fniephaus Brief clarification -- are you suggesting adding this parameter will fix the issue I'm experiencing? |
|
For me this does not solve the issue I'm using paketobuildpacks/builder-jammy-tiny:latest with the gradle bootBuildImage task, and the libraries that use awt are com.google.zxing:javase and com.google.zxing:core I'm passing the args |
|
Sorry for the delay, just saw this issue as it was opened originally under a repo that the Java team does not monitor. We'll need to look into this more. Off-hand, I wasn't aware that this could happen. I thought that the produced native image binary would include everything it needs to run minus glibc and libz. @fniephaus Do you have a doc link on how this works in GraalVM? Are there certain build flags that will result in this build output? I just want to brush up on it and make sure I understand how this all works before we make changes in the buildpack. Thanks |
Not really. I just wanted to point out that there is
I'm not 100% sure this really is documented (need to double check), but it can certainly be the case that Native Image outputs more than just a binary.
This behavior does not necessarily depend on certain flags, it depends on the app fed into Native Image. If the static analysis finds certain JDK libraries reachable, it will output them together with the executable. A good example is any app that somehow uses AWT (like the one from @tabiStein). AWT is dynamically linked against the shared libs of the JDK and some shims that Native Image produces for libjava and libjvm. For more info on this, take a look at oracle/graal#4921. A simple way to deal with this is to extend the buildpack infra that copies the native executable to also copy any additional |
|
I've added documentation for this, see here: https://github.com/oracle/graal/pull/8424/files#diff-9e46ede5ec9729b1cf39fdd6ad204bf093e84441d2ff5419fb20cca77b7070b2R344 |
|
The same problem happened to me, my project uses ImageIO in awt. My temporary solution is to use the This will work fine. But I think this is just a temporary solution and hope the problem can be solved soon. |
|
Hi ! Just been it by this. You can easily reproduce by cloning this repo, and run |
|
Maybe out of topic, but I have thought of working around this by trying to go away from mostly static image by adding :
But it also seems that those arguments append to the existing one Anyway, musl is neither present, nor configured This works mostly locally here even if I think I have been hit by a (maybe) bug : oracle/graal#8911 Happy to help testing things if needed! Documentation for reference : https://www.graalvm.org/22.0/reference-manual/native-image/StaticImages/#preparation |
|
I had to switch to Quarkus to make it work, with Graal's mandrel version |
|
@dmikusa any way you could take a look how this can be fixed? Copying any additional *.so files may be the easiest way, but you could also generate the artifact list. |
Yes, that's an easy way to do it, just copying all the .so files next to executable is just fine. If this could be fixed that would be awesome, because it blocks a lot of apps IMO (awt is pretty common in transitive libs for example). |
@0xyk3r: How would I do that? |
@khauser You need to copy all the |
|
@0xyk3r Thanks! Do you know which folder to copy it to. From |
|
Nice one @0xyk3r ! for file in ./build/native/nativeCompile/*.so; do
docker cp "$file" mycontainer:/workspace
donedocker commit containerid mytestimagetagAnd now I have a working image! |
|
@dmikusa Hi, I hate being that guy, but is there an opportunity that this could be looked at ? |
|
Sorry, I do have this on my list, but there's just a lot of stuff going on and only a finite amount of time to work on things. Including @paketo-buildpacks/java-maintainers for a wider audience, maybe one of them can jump in faster than me. |
|
@fniephaus Follow up question for you. How does the native image binary locate the shared libraries? I've been doing some testing and copying the "*.so" files to be in
Files all in 
When it runs, it'll execute I've tried setting |
The additional libs are supposed to stay in the same directory with the native executable. The lookup can be adjusted, for example via LD_LIBRARY_PATH, but that's not necessary by default. The problem you are running into is in fact unrelated. Just because the shared libs are in the right location does not mean things will just work unfortunately. AWT heavily uses JNI, which requires registrations. To make that work, you need to run the tracing agent on the app, as hopefully mentioned in the recommendation section of your Native Image build. Hope this helps! |
|
Ah ha, ok. I was so focused on where they should be I missed that. Perfect! I'm making progress then, I'll keep plugging away at it, thanks! |
|
I'm still stuck on the I think the buildpack is doing the right things now. I've published a testing image to https://hub.docker.com/r/dmikusa/paketo-native-image/tags. If you use the following buildpack/builder arguments with pack build, you can try it out... For me, this builds and the image now has Thanks |
|
I ended up getting the native-image build to work on a Linux VM. There was a slight difference when I ran the trace agent on the Linux VM, and after that, the built binary ran OK. I tried building with buildpacks using that same configuration (JVM/native-image vendor+version, code commit), but it still didn't work. In the buildpacks produced app image, I still get the JNI failure. Interested to see if it works for other folks or if there maybe something else we need to do with the buildpack build here. |
|
Many thanks for giving this a try. I have changed my maven configuration like this, as I understand the <plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<image>
<runImage>dmikusa/paketo-native-image:latest</runImage>
<env>
<BP_NATIVE_IMAGE>true</BP_NATIVE_IMAGE>
</env>
</image>
</configuration>
<executions>
<execution>
<goals>
<goal>build-image-no-fork</goal>
</goals>
</execution>
</executions>
</plugin>I can see that it's indeed used But a |
|
Try this: The image I made is a buildpack image, not a runtime image. |
|
Oops ! Sorry for confusion. I have tried your maven configuration, here is the error during the maven build process Looks like as standard configuration, the plugin uses But even with this, the error remains the same as above. Thanks again ! EDIT : forget this, there was a typo, ie |
|
So this still fails for me with
But :
I have made a branch here if you want to give it a shot. |
|
@mpalourdio Sorry about that. I had the wrong thing in there for native image, I just changed the comment above so it should be correct now. |
You can run |
I was not aware of this tool. So there is no way to override the entrypoint with docker run ? |
If you use the If you use the base Paketo image, then you can switch the entry point to bash and do that. I used tiny in the example because it works on my M1 MBP. If you've got a amd64 system, you could use base easily. If you're on an M-series MBP like me then base will be a pain because it'll use amd64 emulation and take a really long time to build, if it ever finishes. |
Wow, it just works now. Many thanks !!!!! And thanks for your patience ! |
Trying now with the base image. I'm on linux, amd64. |
|
I was able to build your sample and run it. I tried it with an audio file and got: Not sure if that's expected, but it didn't crash. |
|
Mhh, the Also, i have just tried the base image, all good for the entrypoint override : $ docker run -it --rm --entrypoint=/bin/bash flhacker:0.0.1-SNAPSHOT
cnb@5f1d10b24a74:/workspace$ ls -alF
total 81164
drwxr-xr-x 2 1001 cnb 4096 Jan 1 1980 ./
drwxr-xr-x 1 root root 4096 Jul 14 21:40 ../
-rwxr-xr-x 1 1001 cnb 78068928 Jan 1 1980 com.mpalourdio.projects.flhacker.FlhackerApplicationKt*
-rw-r--r-- 1 1001 cnb 809536 Jan 1 1980 libawt.so
-rw-r--r-- 1 1001 cnb 38488 Jan 1 1980 libawt_headless.so
-rw-r--r-- 1 1001 cnb 575400 Jan 1 1980 libawt_xawt.so
-rw-r--r-- 1 1001 cnb 2737336 Jan 1 1980 libfontmanager.so
-rwxr-xr-x 1 1001 cnb 14976 Jan 1 1980 libjava.so*
-rw-r--r-- 1 1001 cnb 239976 Jan 1 1980 libjavajpeg.so
-rwxr-xr-x 1 1001 cnb 14976 Jan 1 1980 libjvm.so*
-rw-r--r-- 1 1001 cnb 588888 Jan 1 1980 liblcms.so |
|
I used this file for testing: https://filesamples.com/samples/audio/flac/Symphony%20No.6%20(1st%20movement).flac |
|
Indeed, this file does not contain any embedded artwork. But the "null" is not very graceful. End of out-of-topic. All works here, and many thanks again for your awesome work and availability! |
|
Ok, great. I pushed up a PR, I'll try to wrangle someone to review that this week & hopefully get things published. I'll keep this open until we've got the fix released. |
|
Version 5.14.2 has the fix that I made. @mpalourdio you can use that by changing This should get picked up into the next release, which is tentatively scheduled for Friday so after Friday you can remove all the custom buildpack configuration, i.e. |
|
@dmikusa I can confirm that Many thanks, and I will wait next week for the next release. |
|
Back from holidays. I can confirm all works as expected with the default spring-boot-maven-plugin configuration now! Thanks again. |
I am using the builder-jammy-base buildpack with the spring-boot-maven-plugin to build a native image. After some troubleshooting with the Graal team (oracle/graal#8273) it seems that certain required artifacts (jdk_library and jdk_liberary_shim .so files) are not being included in the workspace/ directory of containers using the run image generated by this buildpack. We do see them listed as "produced artifacts" in the build log, however.
Expected Behavior
The following .so files (in bold) from the log snippet below should be included in the workspace/ directory alongside the native image executable (org.example.App):
INFO] [creator] Produced artifacts:
[INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libawt.so (jdk_library)
[INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libawt_headless.so (jdk_library)
[INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libawt_xawt.so (jdk_library)
[INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libfontmanager.so (jdk_library)
[INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libfreetype.so (jdk_library)
[INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libjava.so (jdk_library_shim)
[INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libjavajpeg.so (jdk_library)
[INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/libjvm.so (jdk_library_shim)
[INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/liblcms.so (jdk_library)
[INFO] [creator] /layers/paketo-buildpacks_native-image/native-image/org.example.App (executable)
Current Behavior
The files are not included, the directory in the run container only contains the native executable:
Steps to Reproduce
Repro:
packageOutput.txt
Environment:
GraalVM version 23.0.2 (as determined by buildpack)
JDK major version: 17
OS: MacOS Sonoma 14.2.1
Architecture: 6-Core Intel Core i9
Maven version: 3.9.6
Docker version: 24.0.7
Motivations
My company is trying to convert one of our microservices to a native executable, using your buildpack. The service uses the Apache PDDocument class, which relies on the Java awt library. Since the library is not available to the native image at runtime, the executable crashes.
The text was updated successfully, but these errors were encountered: