"Non-reducible loop requires too much duplication" error in call to Kotlin suspend function #4265

sschuberth · 2022-01-26T17:10:06Z

Describe the issue

I'm using Graal 22.0.0.2 to build a native image for ORT - the OSS Review Toolkit, and the ./gradlew :cli:nativeImage task fails with:

Error: Non-reducible loop requires too much duplication
------------------------------------------------------------------------------------------------------------------------
Call path from entry point to org.ossreviewtoolkit.scanner.scanners.fossid.FossId.scanPackages(Set, Map, Continuation): 
        at org.ossreviewtoolkit.scanner.scanners.fossid.FossId.scanPackages(FossId.kt)
        at org.ossreviewtoolkit.scanner.scanners.fossid.FossId$scanPackage$1.invokeSuspend(FossId.kt:722)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlin.sequences.SequenceBuilderIterator.hasNext(SequenceBuilder.kt:140)
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:906)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:597)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:194)
        at com.oracle.svm.core.code.IsolateEnterStub.PosixJavaThreads_pthreadStartRoutine_e1f4a8c0039f8337338252cd8734f63a79b5e3df(generated:0)
Original exception that caused the problem: org.graalvm.compiler.core.common.PermanentBailoutException: Non-reducible loop requires too much duplication
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BciBlockMapping.computeBlockOrder(BciBlockMapping.java:1716)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BciBlockMapping.computeBlockOrder(BciBlockMapping.java:1422)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BciBlockMapping.build(BciBlockMapping.java:803)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BciBlockMapping.buildMap(BciBlockMapping.java:1765)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BciBlockMapping.create(BciBlockMapping.java:1758)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.generateBlockMap(BytecodeParser.java:1048)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.build(BytecodeParser.java:1074)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.buildRootMethod(BytecodeParser.java:1041)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.GraphBuilderPhase$Instance.run(GraphBuilderPhase.java:84)
        at com.oracle.svm.hosted.phases.SharedGraphBuilderPhase.run(SharedGraphBuilderPhase.java:80)
        at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.run(Phase.java:49)
        at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:214)
        at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
        at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
        at com.oracle.graal.pointsto.flow.AnalysisParsedGraph.parseBytecode(AnalysisParsedGraph.java:132)
        at com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsed(AnalysisMethod.java:556)
        at com.oracle.graal.pointsto.phases.InlineBeforeAnalysisGraphDecoder.lookupEncodedGraph(InlineBeforeAnalysis.java:182)
        at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.doInline(PEGraphDecoder.java:1155)
        at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.tryInline(PEGraphDecoder.java:1138)
        at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.trySimplifyInvoke(PEGraphDecoder.java:992)
        at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.handleInvoke(PEGraphDecoder.java:946)
        at jdk.internal.vm.compiler/org.graalvm.compiler.nodes.GraphDecoder.processNextNode(GraphDecoder.java:788)
        at com.oracle.graal.pointsto.phases.InlineBeforeAnalysisGraphDecoder.processNextNode(InlineBeforeAnalysis.java:242)
        at jdk.internal.vm.compiler/org.graalvm.compiler.nodes.GraphDecoder.decode(GraphDecoder.java:529)
        at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.decode(PEGraphDecoder.java:822)
        at com.oracle.graal.pointsto.phases.InlineBeforeAnalysis.decodeGraph(InlineBeforeAnalysis.java:99)
        at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:175)
        at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:358)
        at com.oracle.graal.pointsto.flow.MethodTypeFlow.createTypeFlow(MethodTypeFlow.java:297)
        at com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureTypeFlowCreated(MethodTypeFlow.java:286)
        at com.oracle.graal.pointsto.flow.MethodTypeFlow.addContext(MethodTypeFlow.java:107)
        at com.oracle.graal.pointsto.DefaultAnalysisPolicy$DefaultVirtualInvokeTypeFlow.onObservedUpdate(DefaultAnalysisPolicy.java:223)
        at com.oracle.graal.pointsto.flow.TypeFlow.notifyObservers(TypeFlow.java:487)
                      120.6s (19.4% of total time) in 78 GCs    at com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:556)
        at com.oracle.graal.pointsto.PointsToAnalysis$2.run(PointsToAnalysis.java:598)
        at com.oracle.graal.pointsto.util.CompletionExecutor.executeCommand(CompletionExecutor.java:195)
        at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$executeService$0(CompletionExecutor.java:179)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
        at java.base/java.util.concurrent.ForkJoinPool.externalHelpQuiescePool(ForkJoinPool.java:2104)
        at java.base/java.util.concurrent.ForkJoinPool.awaitQuiescence(ForkJoinPool.java:3321)
        at com.oracle.graal.pointsto.util.CompletionExecutor.complete(CompletionExecutor.java:245)
        at com.oracle.graal.pointsto.PointsToAnalysis.doTypeflow(PointsToAnalysis.java:662)
        at com.oracle.graal.pointsto.PointsToAnalysis.finish(PointsToAnalysis.java:637)
        at com.oracle.graal.pointsto.PointsToAnalysis.runAnalysis(PointsToAnalysis.java:734)
        at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:702)
        at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:537)
        at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:494)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:426)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:587)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:126)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner$JDK9Plus.main(NativeImageGeneratorRunner.java:617)

Steps to reproduce the issue

git clone --depth 1 https://github.com/oss-review-toolkit/ort.git
./gradlew :cli:nativeImage

Describe GraalVM and your environment:

GraalVM version: 22.0.0.2 / 21.3.0 (tried both)
JDK major version: 17 / 11 (tried both)
OS: Ubuntu 18.04
Architecture: AMD64

More details

The issue might be related to #3342, but that's already closed.

This is the code location from the stack trace.

Output with --native-image-info and --verbose flags will ~~follow~~ below:

Building image for target platform: org.graalvm.nativeimage.Platform$LINUX_AMD64
Using native toolchain:
   Name: GNU project C and C++ compiler (gcc)
   Vendor: linux
   Version: 7.5.0
   Target architecture: x86_64
   Path: /usr/bin/gcc
Using CLibrary: com.oracle.svm.core.posix.linux.libc.GLibC

Static libraries:
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/static/linux-amd64/glibc/libnet.a
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/static/linux-amd64/glibc/libjaas.a
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/static/linux-amd64/glibc/libextnet.a
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/static/linux-amd64/glibc/libnio.a
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/svm/clibraries/linux-amd64/liblibchelper.a
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/static/linux-amd64/glibc/libjava.a
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/static/linux-amd64/glibc/libprefs.a
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/static/linux-amd64/glibc/libfontmanager.a
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/static/linux-amd64/glibc/libawt_headless.a
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/static/linux-amd64/glibc/libawt.a
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/static/linux-amd64/glibc/libfdlibm.a
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/static/linux-amd64/glibc/libzip.a
   ../../../../../.gradle/caches/com.palantir.graal/22.0.0.2/17/graalvm-ce-java17-22.0.0.2/lib/svm/clibraries/linux-amd64/libjvm.a
Other libraries: stdc++,m,freetype,pthread,dl,z,rt

The text was updated successfully, but these errors were encountered:

oubidar-Abderrahim · 2022-01-31T14:59:02Z

Hi, thank you for reporting this, we will take a look into it and get back to you

dougxc · 2022-02-07T06:55:45Z

I see there is an option for limiting the amount of work done by the compiler to support non-reducible loops. The help text is a little ambiguous on what the value of that option means but please try increasing it to see if it helps. I think the syntax for this option is native-image -H:MaxDuplicationFactor=4 ....

@gilles-duboscq maybe 2.0 is too low a default value for MaxDuplicationFactor?

gilles-duboscq · 2022-02-07T09:09:58Z

2.0 was chosen because it could accommodate all the examples we saw in the original reports while still not allowing large explosions of code size.
Maybe a larger default value is the right answer, but i suspect there will always be "practical" cases that require a higher limit than what we set as default.

It would be a good data point to know the factor needed to this specific case.
Also maybe AOT compilations should have a higher default than the JIT? Native-image can afford a bit more time/memory for compilation and the outcome of a bailout is much more severe with native-image than with the JIT.

The error message should mention the option and its current value so that it's more obvious how to locally solve the issue.

sschuberth · 2022-02-07T09:45:21Z

It would be a good data point to know the factor needed to this specific case.

I'll try to come back to you on that.

The error message should mention the option and its current value so that it's more obvious how to locally solve the issue.

That would have helped absolutely 👍🏻

sschuberth · 2022-02-08T08:36:00Z

It would be a good data point to know the factor needed to this specific case.

I'll try to come back to you on that.

So, I got it working with -H:MaxDuplicationFactor=4.0. I'll bisect between 2.0 and 4.0, but it will take a while as the native image build takes a whopping 18 minutes for me...

sschuberth · 2022-02-08T10:45:45Z

A factor of 3.0 works, 2.5 is too low, and so is 2.75, so I'll stick to 3.0.

See [1] for some background information. [1]: oracle/graal#4265 Signed-off-by: Sebastian Schuberth <[email protected]>

dougxc · 2022-02-09T03:14:38Z

With 3.0, what was the native image build time? And how much does that deviate from similar native image builds that you make?

See [1] for some background information. [1]: oracle/graal#4265 Signed-off-by: Sebastian Schuberth <[email protected]>

sschuberth · 2022-02-09T15:04:28Z

With 3.0, what was the native image build time?

It's a bit hard to tell reliably, as I was running into multiple issues for which I had to find work-arounds to make the build actually complete. But I believe the build time of 18 minutes did not vary significantly when increasing the factor from 2.0 to 3.0 in my case.

And how much does that deviate from similar native image builds that you make?

This is actually the first native image build I've ever tried, so I don't have any comparisons.

dougxc · 2022-02-15T13:16:32Z

This issue should be resolved by 2f72ff6.

sschuberth added bug native-image labels Jan 26, 2022

oubidar-Abderrahim self-assigned this Jan 31, 2022

sschuberth mentioned this issue Feb 5, 2022

Experiment compiling ORT to WASM oss-review-toolkit/ort#5011

Open

sschuberth added a commit to oss-review-toolkit/ort that referenced this issue Feb 8, 2022

Gradle: Add more Graal options to work-around native image build issues

f5c5b84

See [1] for some background information. [1]: oracle/graal#4265 Signed-off-by: Sebastian Schuberth <[email protected]>

sschuberth mentioned this issue Feb 8, 2022

Gradle: Add more Graal options to work-around native image build issues oss-review-toolkit/ort#5028

Merged

sschuberth added a commit to oss-review-toolkit/ort that referenced this issue Feb 9, 2022

Gradle: Add more Graal options to work-around native image build issues

7a4dc6f

See [1] for some background information. [1]: oracle/graal#4265 Signed-off-by: Sebastian Schuberth <[email protected]>

dougxc closed this as completed Feb 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Non-reducible loop requires too much duplication" error in call to Kotlin suspend function #4265

"Non-reducible loop requires too much duplication" error in call to Kotlin suspend function #4265

sschuberth commented Jan 26, 2022 •

edited

Loading

oubidar-Abderrahim commented Jan 31, 2022

dougxc commented Feb 7, 2022

gilles-duboscq commented Feb 7, 2022

sschuberth commented Feb 7, 2022

sschuberth commented Feb 8, 2022

sschuberth commented Feb 8, 2022 •

edited

Loading

dougxc commented Feb 9, 2022

sschuberth commented Feb 9, 2022

dougxc commented Feb 15, 2022

"Non-reducible loop requires too much duplication" error in call to Kotlin suspend function #4265

"Non-reducible loop requires too much duplication" error in call to Kotlin suspend function #4265

Comments

sschuberth commented Jan 26, 2022 • edited Loading

oubidar-Abderrahim commented Jan 31, 2022

dougxc commented Feb 7, 2022

gilles-duboscq commented Feb 7, 2022

sschuberth commented Feb 7, 2022

sschuberth commented Feb 8, 2022

sschuberth commented Feb 8, 2022 • edited Loading

dougxc commented Feb 9, 2022

sschuberth commented Feb 9, 2022

dougxc commented Feb 15, 2022

sschuberth commented Jan 26, 2022 •

edited

Loading

sschuberth commented Feb 8, 2022 •

edited

Loading