Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate executable via GraalVM #12

Merged
merged 1 commit into from
Jul 4, 2019
Merged

Generate executable via GraalVM #12

merged 1 commit into from
Jul 4, 2019

Conversation

nicokosi
Copy link
Owner

@nicokosi nicokosi commented May 5, 2018

This interesting blog post made me want to try using GraalVM in order to generate hubstats' executable.

@nicokosi nicokosi added the wip label May 5, 2018
@nicokosi
Copy link
Owner Author

Attempt #1

From fat JAR, with Dockerized GraaVM 1.0.0-rc1: visibly hit similar issue than oracle/graal#385 / oracle/graal#375, related to "static initializers" limitation? 😭 :

Step 4/15 : RUN native-image   -jar hubstats-0.1.0-SNAPSHOT-standalone.jar   -H:+ReportUnsupportedElementsAtRuntime   hubstats.core
 ---> Running in e7f911774bd4
Build on Server(pid: 11, port: 26681)*
   classlist:   3,159.26 ms
       (cap):   1,485.02 ms
       setup:   2,563.80 ms
    analysis:  10,109.06 ms
fatal error: java.lang.NullPointerException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
	at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
	at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:398)
	at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:240)
	at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:337)
	at com.oracle.svm.hosted.server.NativeImageBuildServer.executeCompilation(NativeImageBuildServer.java:378)
	at com.oracle.svm.hosted.server.NativeImageBuildServer.lambda$processCommand$8(NativeImageBuildServer.java:315)
	at com.oracle.svm.hosted.server.NativeImageBuildServer.withJVMContext(NativeImageBuildServer.java:396)
	at com.oracle.svm.hosted.server.NativeImageBuildServer.processCommand(NativeImageBuildServer.java:312)
	at com.oracle.svm.hosted.server.NativeImageBuildServer.processRequest(NativeImageBuildServer.java:256)
	at com.oracle.svm.hosted.server.NativeImageBuildServer.lambda$serve$7(NativeImageBuildServer.java:216)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at com.oracle.graal.pointsto.ObjectScanner.scanField(ObjectScanner.java:113)
	at com.oracle.graal.pointsto.ObjectScanner.doScan(ObjectScanner.java:263)
	at com.oracle.graal.pointsto.ObjectScanner.finish(ObjectScanner.java:307)
	at com.oracle.graal.pointsto.ObjectScanner.scanBootImageHeapRoots(ObjectScanner.java:78)
	at com.oracle.graal.pointsto.ObjectScanner.scanBootImageHeapRoots(ObjectScanner.java:60)
	at com.oracle.graal.pointsto.BigBang.checkObjectGraph(BigBang.java:581)
	at com.oracle.graal.pointsto.BigBang.finish(BigBang.java:552)
	at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:653)
	at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:381)
	at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Error: Processing image build request failed

@nicokosi
Copy link
Owner Author

Attempt #2

With locally built substratevm from commit oracle/graal@c101571, native-image hangs:

$> JAVA_HOME=~/Downloads/labsjdk1.8.0_161-jvmci-0.42/Contents/Home ../graal/substratevm/native-image -jar target/hubstats-0.1.0-SNAPSHOT-standalone.jar hubstats.core
Build on Server(pid: 18933, port: 55103)*
   classlist:   2,744.32 ms
       (cap):   1,531.16 ms
       setup:   2,401.40 ms

Similar issue: oracle/graal#411.

@nicokosi
Copy link
Owner Author

Give a try using arguments like:

native-image -H:+ReportUnsupportedElementsAtRuntime -J-Xmx3G -J-Xms3G -jar ./target/hubstats-0.1.0-SNAPSHOT-standalone.jar hubstats.core

Cf. https://www.astrecipes.net/blog/2018/07/20/cmd-line-apps-with-clojure-and-graalvm.

@nicokosi
Copy link
Owner Author

See the related GraalVM issue I have opened: oracle/graal#420.

@nicokosi
Copy link
Owner Author

nicokosi commented Oct 17, 2018

Related resources:

@nicokosi
Copy link
Owner Author

@nicokosi
Copy link
Owner Author

Two big news:

@nicokosi
Copy link
Owner Author

nicokosi commented Jan 21, 2019

Oops, hubstats's executable still cannot be generated with GraalVM 1.0.0 RC 11, visibly. 💥

Compiling hubstats.options
Created /home/nkosinski/perso/hubstats/target/hubstats-0.1.0-SNAPSHOT.jar
Created /home/nkosinski/perso/hubstats/target/hubstats-0.1.0-SNAPSHOT-standalone.jar
Build nicokosi/hubstats's executable from JAR via GraalVM:
1.0.0-rc11: Pulling from oracle/graalvm-ce
Digest: sha256:ab6b50e27cab6bca72e1d35c0b31f9f30cc03f815884d8f18f35ab47626ccc83
Status: Image is up to date for oracle/graalvm-ce:1.0.0-rc11
Build on Server(pid: 10, port: 32789)*
[hubstats.core:10]    classlist:   2,757.55 ms
[hubstats.core:10]        (cap):     627.82 ms
[hubstats.core:10]        setup:   1,675.92 ms

ls
[hubstats.core:10]     analysis: 484,784.87 ms
11 fatal errors detected:
Exception in thread "ForkJoinPool-3-worker-5" fatal error: java.lang.OutOfMemoryError: GC overhead limit exceeded

I had forgotten that all the important code was commented out. 😭

@nicokosi
Copy link
Owner Author

I am now able to report unsupported features while generating native image :feelsgood: :

Error: unsupported features in 5 methods
Detailed message:
Error: Detected a started Thread in the image heap. Threads running in the image generator are no longer running at image run time. The object was probably created by a class initializer and is reachable from a static field. By default, all class initialization is done during native image building.You can manually delay class initialization to image run time by using the option --delay-class-initialization-to-runtime=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Trace: 	object clojure.lang.Var
	object java.lang.Object[]
	object clojure.lang.PersistentHashMap$BitmapIndexedNode
	object clojure.lang.PersistentHashMap$INode[]
	object clojure.lang.PersistentHashMap$ArrayNode
	object clojure.lang.PersistentHashMap$INode[]
	object clojure.lang.PersistentHashMap$ArrayNode
	object clojure.lang.PersistentHashMap
	object java.util.concurrent.atomic.AtomicReference
	object clojure.lang.Namespace
	object java.lang.Object[]
	object clojure.lang.PersistentHashMap$BitmapIndexedNode
	object clojure.lang.PersistentHashMap
	object java.util.concurrent.atomic.AtomicReference
	object clojure.lang.Namespace
	object clojure.lang.Var
	method hubstats.core$_main.invokeStatic(ISeq)
Call path from entry point to hubstats.core$_main.invokeStatic(ISeq): 
	at hubstats.core$_main.invokeStatic(core.clj:47)
	at hubstats.core$_main.doInvoke(core.clj:47)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at hubstats.core.main(Unknown Source)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:152)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)
Error: Detected a started Thread in the image heap. Threads running in the image generator are no longer running at image run time. The object was probably created by a class initializer and is reachable from a static field. By default, all class initialization is done during native image building.You can manually delay class initialization to image run time by using the option --delay-class-initialization-to-runtime=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Trace: 	object java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
	object java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
	object java.util.concurrent.DelayQueue
	object clojure.lang.Var
	method clojure.core.async.impl.timers$timeout.invokeStatic(long)
Call path from entry point to clojure.core.async.impl.timers$timeout.invokeStatic(long): 
	at clojure.core.async.impl.timers$timeout.invokeStatic(timers.clj:43)
	at clojure.core.async.impl.timers$timeout.invoke(timers.clj:43)
	at clojure.lang.AFn.applyToHelper(AFn.java:154)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at hubstats.core.main(Unknown Source)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:152)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)
Error: No instances are allowed in the image heap for a class that is initialized or reinitialized at image runtime: javax.net.ssl.SSLContext
Trace: 	object java.lang.Object[]
	object clojure.lang.PersistentArrayMap
	object clojure.lang.Var
	object java.lang.Object[]
	object clojure.lang.PersistentHashMap$BitmapIndexedNode
	object clojure.lang.PersistentHashMap$INode[]
	object clojure.lang.PersistentHashMap$ArrayNode
	object clojure.lang.PersistentHashMap$INode[]
	object clojure.lang.PersistentHashMap$ArrayNode
	object clojure.lang.PersistentHashMap
	object java.util.concurrent.atomic.AtomicReference
	object clojure.lang.Namespace
	object java.lang.Object[]
	object clojure.lang.PersistentArrayMap
	object java.util.concurrent.atomic.AtomicReference
	object clojure.lang.Namespace
	object clojure.lang.Var
	method clj_http.client$wrap_accept$fn__3338.invoke(Object, Object, Object)
Call path from entry point to clj_http.client$wrap_accept$fn__3338.invoke(Object, Object, Object): 
	at clj_http.client$wrap_accept$fn__3338.invoke(client.clj:722)
	at clojure.lang.AFn.applyToHelper(AFn.java:160)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at hubstats.core.main(Unknown Source)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:152)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)
Error: No instances are allowed in the image heap for a class that is initialized or reinitialized at image runtime: sun.security.provider.NativePRNG
Trace: 	object java.security.SecureRandom
	object sun.security.ssl.SSLContextImpl$TLSContext
	object sun.security.ssl.SSLSocketFactoryImpl
	object org.apache.http.conn.ssl.SSLConnectionSocketFactory
	object clojure.lang.Var
	object java.lang.Object[]
	object clojure.lang.PersistentHashMap$BitmapIndexedNode
	object clojure.lang.PersistentHashMap$INode[]
	object clojure.lang.PersistentHashMap$ArrayNode
	object clojure.lang.PersistentHashMap$INode[]
	object clojure.lang.PersistentHashMap$ArrayNode
	object clojure.lang.PersistentHashMap
	object java.util.concurrent.atomic.AtomicReference
	object clojure.lang.Namespace
	object java.lang.Object[]
	object clojure.lang.PersistentArrayMap
	object java.util.concurrent.atomic.AtomicReference
	object clojure.lang.Namespace
	object clojure.lang.Var
	method clj_http.client$wrap_accept$fn__3338.invoke(Object, Object, Object)
Call path from entry point to clj_http.client$wrap_accept$fn__3338.invoke(Object, Object, Object): 
	at clj_http.client$wrap_accept$fn__3338.invoke(client.clj:722)
	at clojure.lang.AFn.applyToHelper(AFn.java:160)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at hubstats.core.main(Unknown Source)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:152)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)
Error: unbalanced monitors: mismatch at monitorexit, 96|LoadField#lockee__5436__auto__ != 3|LoadField#lockee__5436__auto__
Call path from entry point to clojure.spec.gen.alpha$dynaload$fn__2628.invoke(): 
	at clojure.spec.gen.alpha$dynaload$fn__2628.invoke(alpha.clj:21)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.lang.Thread.run(Thread.java:748)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:473)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
	at com.oracle.svm.core.code.IsolateEnterStub.PosixJavaThreads_pthreadStartRoutine_e1f4a8c0039f8337338252cd8734f63a79b5e3df(generated:0)
Original exception that caused the problem: org.graalvm.compiler.code.SourceStackTraceBailoutException$1: unbalanced monitors: mismatch at monitorexit, 96|LoadField#lockee__5436__auto__ != 3|LoadField#lockee__5436__auto__
	at clojure.spec.gen.alpha$dynaload$fn__2628.invoke(alpha.clj:22)
Caused by: org.graalvm.compiler.core.common.PermanentBailoutException: unbalanced monitors: mismatch at monitorexit, 96|LoadField#lockee__5436__auto__ != 3|LoadField#lockee__5436__auto__
	at org.graalvm.compiler.java.BytecodeParser.bailout(BytecodeParser.java:3673)
	at org.graalvm.compiler.java.BytecodeParser.genMonitorExit(BytecodeParser.java:2654)
	at org.graalvm.compiler.java.BytecodeParser.processBytecode(BytecodeParser.java:5031)
	at org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3184)
	at org.graalvm.compiler.java.BytecodeParser.processBlock(BytecodeParser.java:2993)
	at org.graalvm.compiler.java.BytecodeParser.build(BytecodeParser.java:891)
	at org.graalvm.compiler.java.BytecodeParser.buildRootMethod(BytecodeParser.java:785)
	at org.graalvm.compiler.java.GraphBuilderPhase$Instance.run(GraphBuilderPhase.java:95)
	at org.graalvm.compiler.phases.Phase.run(Phase.java:49)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:197)
	at org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
	at org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
	at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:213)
	at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:332)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.doParse(MethodTypeFlow.java:310)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureParsed(MethodTypeFlow.java:300)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.addContext(MethodTypeFlow.java:107)
	at com.oracle.graal.pointsto.flow.SpecialInvokeTypeFlow.onObservedUpdate(InvokeTypeFlow.java:421)
	at com.oracle.graal.pointsto.flow.TypeFlow.notifyObservers(TypeFlow.java:352)
	at com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:394)
	at com.oracle.graal.pointsto.flow.SourceTypeFlowBase.update(SourceTypeFlowBase.java:121)
	at com.oracle.graal.pointsto.BigBang$2.run(BigBang.java:508)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:169)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

@nicokosi
Copy link
Owner Author

Attempt #7897123 😜

With graalvm-ce:19.0.0:

./generate-executable.sh

# ✂ snip!
Warning: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Build on Server(pid: 167, port: 42445)*
[hubstats.core:167]    classlist:  10,654.27 ms
[hubstats.core:167]        (cap):   1,121.64 ms
[hubstats.core:167]        setup:   2,439.03 ms
[hubstats.core:167]   (typeflow):   3,412.47 ms
[hubstats.core:167]    (objects):     945.15 ms
[hubstats.core:167]   (features):     265.40 ms
[hubstats.core:167]     analysis:   4,722.17 ms
[hubstats.core:167]     (clinit):      96.55 ms
[hubstats.core:167]     universe:     410.29 ms
[hubstats.core:167]      (parse):     624.01 ms
[hubstats.core:167]     (inline):   1,425.12 ms
[hubstats.core:167]    (compile):   4,951.31 ms
[hubstats.core:167]      compile:   7,264.01 ms
[hubstats.core:167]        image:     408.76 ms
[hubstats.core:167]        write:      97.27 ms
[hubstats.core:167]      [total]:  26,167.22 ms
Warning: Image 'hubstats.core' is a fallback image that requires a JDK for execution (use --no-fallback to suppress fallback image generation).
 => Check the executable:
Display statistics for GitHub pull requests.
Mandatory parameters:
	--organization		GitHub organization
	-o			GitHub organization (shorthand)
	--repository		GitHub repository
	-r			GitHub repository (shorthand)
	--token			GitHub access token (optional)
	-t			GitHub access token (shorthand, optional)
Optional parameters:
	--repositories		Comma-separated list of repositories (optional)
	--since-weeks		output events that occcured since this number of weeks (optional, default: 1)
	-w			output events that occcured since this number of weeks (shorthand, optional, default: 1)
	--since-days		output events that occcured since this number of days (optional)
	-d			output events that occcured since this number of days (shorthand, optional)
	--since			output events that occcured since a date with format '"yyyy-MM-dd'T'HH:mm:ssZ' (optional)
	-s			output events that occcured since a date with format '"yyyy-MM-dd'T'HH:mm:ssZ' (shorthand, optional)

Examples:
	lein run --organization docker --repository containerd --token $token
	lein run --organization docker --repository containerd --since "2017-01-17T00:00:00Z"
	lein run --organization docker --repository containerd --since-days 10
	lein run --organization docker --repositories docker,containerd
 => Copy it to current directory:

@nicokosi nicokosi force-pushed the graalvm branch 2 times, most recently from 6f6f75d to 2e42434 Compare July 1, 2019 05:01
nicokosi added a commit to nicokosi/strava-activity-graphs that referenced this pull request Jul 4, 2019
Copy/pasted script from nicokosi/hubstats#12. 😇

Note that the "--no-fallback" native-image option cannot be used for the
moment (the compilation to native code fails), so the application runs in the normal JVM.
See https://www.graalvm.org/docs/reference-manual/aot-compilation/
nicokosi added a commit to nicokosi/strava-activity-graphs that referenced this pull request Jul 4, 2019
Copy/pasted script from nicokosi/hubstats#12. 😇

Note that the "--no-fallback" native-image option cannot be used for the
moment (the compilation to native code fails), so the application runs in the normal JVM.
See https://www.graalvm.org/docs/reference-manual/aot-compilation/
Note that the "--no-fallback" native-image option cannot be used for the
moment (the compilation to native code fails), so the application runs in the normal JVM.
See https://www.graalvm.org/docs/reference-manual/aot-compilation/
@nicokosi
Copy link
Owner Author

nicokosi commented Jul 4, 2019

Merging this pull request with the fallback mode. I'll see later how to fix native-imagecompilation errors and use the --no-fallback mode.

@nicokosi nicokosi merged commit eae7c5b into master Jul 4, 2019
@nicokosi nicokosi deleted the graalvm branch July 4, 2019 05:30
@nicokosi nicokosi removed the wip label Jul 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant