Aarch64/TemurinJdk 17.0.9+9: C1 CompilerThread0 crashing with Field too big for insn

[bbeaudreault]

Please provide a brief summary of the bug

See below crash report. Note that the version below says 17.0.6+10, but I've also seen it on 17.0.9+9. I upgraded hoping it would fix it, but it didn't.

This exact code has worked perfectly on temurinjdk11.0.12+7 for a long time. We recently upgraded to 17, and started seeing this. The code in question is opensource -- apache hbase. Here's a link to our fork of the failing code: https://github.com/HubSpot/hbase/blob/hubspot-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java#L778

Unfortunately the crash does not happen consistently, instead intermittently. I haven't been able to figure out what confluence of events lead to the crash, or get a core dump.

Reading around, it seems the @ 568 is a bytecode index. I tried decompiling the RegionServerMetricsWrapperRunnable class with javap -p -c -verbose to see if i could align the bci to a line number. I'm not sure if this is reliable, but it aligns to this line.

Please provide steps to reproduce where possible

I can't really provide this. It seems to happen semi-randomly after the jvm has been running for a while, and is probably workload dependent because we run thousands of servers with this code (some aarch64, some x86) across 10s of clusters, and have only seen it happen on 1 cluster. But it happens pretty periodically there.

Expected Results

Don't crash

Actual Results

Crashes

What Java Version are you using?

openjdk version "17.0.9" 2023-10-17 OpenJDK Runtime Environment Temurin-17.0.9+9 (build 17.0.9+9) OpenJDK 64-Bit Server VM Temurin-17.0.9+9 (build 17.0.9+9, mixed mode, sharing)

What is your operating system and platform?

Host: AArch64, 8 cores, 46G, CentOS Stream release 8

How did you install Java?

No response

Did it work before?

Used to work fine with temurinjdk11.0.12+7

Did you test with the latest update version?

Yes

Did you test with other Java versions?

openjdk version "17.0.6" 2023-01-17
OpenJDK Runtime Environment Temurin-17.0.6+10 (build 17.0.6+10)
OpenJDK 64-Bit Server VM Temurin-17.0.6+10 (build 17.0.6+10, mixed mode, sharing)

openjdk version "17.0.9" 2023-10-17
OpenJDK Runtime Environment Temurin-17.0.9+9 (build 17.0.9+9)
OpenJDK 64-Bit Server VM Temurin-17.0.9+9 (build 17.0.9+9, mixed mode, sharing)

Relevant log output

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (assembler_aarch64.hpp:267), pid=2824384, tid=2824465
#  guarantee(chk == -1 || chk == 0) failed: Field too big for insn
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.6+10 (17.0.6+10) (build 17.0.6+10)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.6+10 (17.0.6+10, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# V  [libjvm.so+0x479360]  LIR_Assembler::osr_entry()+0x200
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to //core.2824384)
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
#

---------------  S U M M A R Y ------------

Command Line: -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx16390m -XX:MaxDirectMemorySize=20825m -Dhbase.default.appender=main -Dlog4j2.configurationFile=/etc/hbase/conf/daemon-log4j2.xml -Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j2.asyncQueueFullPolicy=Discard -Dlog4j2.discardThreshold=FATAL -Dasync.profiler.home=/opt/async-profiler -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/hbase -Djava.net.preferIPv4Stack=true -XX:-OmitStackTraceInFastThrow -Dorg.apache.hbase.thirdparty.io.netty.maxDirectMemory=3221225472 -Dorg.apache.hbase.thirdparty.io.netty.tryReflectionSetAccessible=true -Dorg.apache.hbase.thirdparty.io.netty.leakDetection.level=disabled -Djdk.nio.maxCachedBufferSize=262144 -Djava.util.logging.config.class=org.apache.hadoop.hbase.logging.JulToSlf4jInitializer -Xlog:gc*=info,ergo*=trace,age*=trace,safepoint:file=/var/tmp-gc-logs/hbase/gc-regionserver.log:tags,time,uptime,level:filecount=5,filesize=20M -XX:+UseG1GC -XX:+ResizePLAB -XX:MaxGCPauseMillis=50 -XX:G1HeapRegionSize=16M -XX:ParallelGCThreads=8 -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=5 -XX:G1MaxNewSizePercent=60 -XX:G1HeapWastePercent=12 -XX:MaxTenuringThreshold=1 -XX:G1MixedGCCountTarget=64 -XX:G1ReservePercent=10 -XX:+ParallelRefProcEnabled -XX:G1OldCSetRegionThresholdPercent=5 -XX:-G1UseAdaptiveIHOP -XX:InitiatingHeapOccupancyPercent=85 -XX:+DisableExplicitGC -XX:+PerfDisableSharedMem -XX:-UseBiasedLocking -XX:+SafepointTimeout -XX:SafepointTimeoutDelay=500 -Xms16390m -XX:+AlwaysPreTouch -Dio.netty.tryReflectionSetAccessible=true --illegal-access=permit --add-modules=jdk.unsupported --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED -Dhbase.log.dir=/var/log/hbase -Dhbase.home.dir=/usr/lib/hbase/bin/.. -Dhbase.id.str=hbase -Dhbase.root.logger=INFO,RFA -Djava.library.path=:/usr/lib/hbase/lib/native/Linux-amd64-64/:/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/:/usr/lib/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.regionserver.HRegionServer start

Host: AArch64, 8 cores, 46G, CentOS Stream release 8
Time: Tue Nov 14 13:20:10 2023 UTC elapsed time: 4353.617495 seconds (0d 1h 12m 33s)

---------------  T H R E A D  ---------------

Current thread (0x0000ffff90984e50):  JavaThread "C1 CompilerThread0" daemon [_thread_in_native, id=2824465, stack(0x0000ffff493fe000,0x0000ffff495fe000)]


Current CompileTask:
C1:4353617 24783 %s!   3       org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl$RegionServerMetricsWrapperRunnable::run @ 568 (2086 bytes)

Stack: [0x0000ffff493fe000,0x0000ffff495fe000],  sp=0x0000ffff495fb440,  free space=2037k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x479360]  LIR_Assembler::osr_entry()+0x200
V  [libjvm.so+0x476c24]  LIR_Assembler::emit_lir_list(LIR_List*)+0x90
V  [libjvm.so+0x476e1c]  LIR_Assembler::emit_code(BlockList*)+0x5c
V  [libjvm.so+0x4478d0]  Compilation::emit_code_body()+0x110
V  [libjvm.so+0x447cd8]  Compilation::compile_java_method()+0x2e4
V  [libjvm.so+0x447eec]  Compilation::compile_method()+0x10c
V  [libjvm.so+0x44825c]  Compilation::Compilation(AbstractCompiler*, ciEnv*, ciMethod*, int, BufferBlob*, bool, DirectiveSet*)+0x1a8
V  [libjvm.so+0x448d74]  Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x74
V  [libjvm.so+0x5a34a4]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x900
V  [libjvm.so+0x5a4248]  CompileBroker::compiler_thread_loop()+0x458
V  [libjvm.so+0xd7abc8]  JavaThread::thread_main_inner()+0x144
V  [libjvm.so+0xd7f94c]  Thread::call_run()+0x8c
V  [libjvm.so+0xb5fa7c]  thread_native_entry(Thread*)+0xdc
C  [libpthread.so.0+0x78b8]  start_thread+0x188

[karianna]

@bbeaudreault Is HBase actually supported on Java 17? https://issues.apache.org/jira/browse/HBASE-26038

[bbeaudreault]

I'm one of the hbase developers, and am working on this as part of modernizing my company's jdk install. Despite what that old jira says, a bunch of community members are running hbase in production on jdk17. We also are running it successfully on hundreds of machines for the past 6 months, i think we are nearly ready to say it works in our docs. But as part of that rollout, we hit this snag. Since this bug is in the compiler, I'm not sure I can tackle it on the hbase side. If you gave some guidance on what about the method in question is bad, I could work on that.

[bbeaudreault]

@karianna (sorry forgot to ping back in my last comment)

[karianna]

@bbeaudreault I assume this is on Linux Aarch64? Did you also see this crash on Linux x86-64? Also are you on the latest HBase (2.5.6)

[bbeaudreault]

@karianna Yes, this is on Aarch64. We also run this same software on x86-64 servers, and have not seen an issue there. We are on a modified version of 2.5.2, with most patches backported and some from unreleased 2.6.0. We're effectively on the latest version.

[karianna]

Hmm, I just noticed - this only occurs on one cluster is that correct? Is that cluster running on the same underlying physical host each time?

[bbeaudreault]

That's correct, just one cluster has shown this problem. The cluster is 57 hosts, and we've seen it on 10 of them. We also have other aarch64 clusters, which have not seen issues, so i do think this is somehow workload related. Unfortunately I don't know enough about the JIT compiler to make a guess at what. The code in question (RegionServerMetricsWrapperRunnable) runs every minute or so, but only sometimes seems to fail. It's a long method with a few loops, but nothing about it seems too "dangerous" (for lack of a better word)

[karianna]

OK 10 different hosts eliminates a hardware problem. Are you able to capture the core dump processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to //core.2824384)?

[bbeaudreault]

Unfortunately ulimits were not set properly, so a core dump was not created. I'm trying to remedy that now so that when it happens again we can capture one. Anything else I should also do?

[bbeaudreault]

The other thing is, im not sure what's included in a core dump. If it includes the full memory of the jvm, I may not be able to share it because it may contain some customer data. Are there any specific things you'd want me to look for in them?

[bbeaudreault]

Side note here: We are running on AWS, and the aarch64 servers are graviton instances (is4gen.2xlarge). One of my colleagues noticed that for Correto 17 they recommend the following for graviton: Corretto 17 needs two additional flags: -XX:CICompilerCount=2 -XX:CompilationMode=high-only. Of course, we're on temurin not correto. But wasn't sure if that'd be interesting to you.

[bbeaudreault]

We also have other aarch64 clusters, which have not seen issues, so i do think this is somehow workload related

I discovered that we're seeing it on more than just this cluster, we've seen it on at least 10 clusters. All aarch64.

[karianna]

The heap dump would contain memory yes, can you provide the full hs_err.pid file? Please do also try the Corretto recommended flags. The 2nd flag in particular would disable the C1 compiler and probably eliminate this crash for you. Cheers, Martijn

…

On Thu, 16 Nov 2023 at 11:05, Bryan Beaudreault ***@***.***> wrote: Side note here: We are running on AWS, and the aarch64 servers are graviton instances (is4gen.2xlarge). One of my colleagues noticed that for Correto 17 they recommend the following for graviton: Corretto 17 needs two additional flags: -XX:CICompilerCount=2 -XX:CompilationMode=high-only. Of course, we're on temurin not correto. But wasn't sure if that'd be interesting to you. — Reply to this email directly, view it on GitHub <#951 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABME2B7JL2XRO5BZWK6BATYEU377BFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVIYTOMJXGE3TGMBSHCSG4YLNMWUWQYLTL5WGCYTFNSBKK5TBNR2WLKRRG4YTQNRTGI4TAMVENZQW2ZNJNBQXGX3MMFRGK3FMON2WE2TFMN2F65DZOBS2YSLTON2WKQ3PNVWWK3TUUZ2G64DJMNZZJAVEOR4XAZNKOJSXA33TNF2G64TZUV3GC3DVMWUTEMRWGE3DOMJXG2BKI5DZOBS2K2LTON2WLJLWMFWHKZNKGE4TSNBYGU4DINRXQKSHI6LQMWSWYYLCMVWKK5TBNR2WLKRRG4YTOMJXGMYDEOECUR2HS4DFUVWGCYTFNSSXMYLMOVS2UMJXGE4DMMZSHEYDFJ3UOJUWOZ3FOKTGG4TFMF2GK> . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .