Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] JVM agent crash intermittently in CI integration test #8194

Closed
pxLi opened this issue Apr 28, 2023 · 3 comments
Closed

[BUG] JVM agent crash intermittently in CI integration test #8194

pxLi opened this issue Apr 28, 2023 · 3 comments
Labels
bug Something isn't working test Only impacts tests

Comments

@pxLi
Copy link
Member

pxLi commented Apr 28, 2023 •
edited
Loading

Describe the bug
We found intermittent jvm agent crashes in different pipelines (pre-merge and nightly), e.g.
rapids_premerge-github: 7058, 7050, etc

seems some issue from cudf or JNI side, java gateway core dump below
(the failed cudf submodule ref was at rapidsai/cudf@5234278, JNI ref was at NVIDIA/spark-rapids-jni@9bd78ca)

hs_err_pid1912.log

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f3e873921fe, pid=1912, tid=0x00007f3cad9cf700
#
# JRE version: OpenJDK Runtime Environment (8.0_362-b09) (build 1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.362-b09 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x9a1fe]  __libc_malloc+0x11e
#
# Core dump written. Default location: /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-7058-ci-2/integration_tests/target/run_dir-20230427214600-485H/core or core.1912
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f3dbc016800):  JavaThread "Executor task launch worker for task 2.0 in stage 5997.0 (TID 20531)" daemon [_thread_in_vm, id=13036, stack(0x00007f3cad8cf000,0x00007f3cad9d0000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000

Registers:
RAX=0x00007f3c100623c8, RBX=0x0000000000000005, RCX=0x00007f3c100623aa, RDX=0x0000000000000000
RSP=0x00007f3cad9cad40, RBP=0x0000000000000060, RSI=0x0000000000000009, RDI=0x0000000000000060
R8 =0x5c65b6c2695baec3, R9 =0x00007f3c111cbd70, R10=0x00007f3c100623a0, R11=0x00007f3c10000080
R12=0xffffffffffffffb0, R13=0x0000000000000009, R14=0x00007f3cad9cadb0, R15=0x00007f3c1154450a
RIP=0x00007f3e873921fe, EFLAGS=0x0000000000010246, CSGSFS=0x002b000000000033, ERR=0x0000000000000000
  TRAPNO=0x000000000000000d

Top of Stack: (sp=0x00007f3cad9cad40)
0x00007f3cad9cad40:   0000000000000000 00007f3cad9cad90
0x00007f3cad9cad50:   0000000000000060 00007f3e86e21cb5
0x00007f3cad9cad60:   3f727d5ac4e75769 0000000000000053
0x00007f3cad9cad70:   0000000000000060 00007f3cad9cadb0
0x00007f3cad9cad80:   00007f3c1154450a 00007f3c1154450a
0x00007f3cad9cad90:   00007f3cad9cadf0 00007f3e86f43193
0x00007f3cad9cada0:   e18577ba7e88c61c 66d7687bd51b57f5
0x00007f3cad9cadb0:   0000000000000000 0000000000000000
0x00007f3cad9cadc0:   0000000000000000 0000000000000000
0x00007f3cad9cadd0:   0000000000000000 f24dd0c26f5c6c00
0x00007f3cad9cade0:   00000000c1719ae1 0000000000000053
0x00007f3cad9cadf0:   00007f3cad9cae20 00007f3e86f44570
0x00007f3cad9cae00:   00007f3cad9cae20 00000000c1719ae1
0x00007f3cad9cae10:   00007f3e8004ec60 0000000000000003
0x00007f3cad9cae20:   00007f3cad9caeb0 00007f3e86f45cd2
0x00007f3cad9cae30:   1d0cc14741015531 00007f3e81820e80
0x00007f3cad9cae40:   0000000000000007 00007f3cad9caf20
0x00007f3cad9cae50:   00000000000079a0 00007f3e8004ec60
0x00007f3cad9cae60:   0000000000000003 00007f3cad9cb020
0x00007f3cad9cae70:   00007f3cad9cb080 00007f3c111cbc70
0x00007f3cad9cae80:   51cfa72ba08fe8bb 00007f3dbc016800
0x00007f3cad9cae90:   00007f3e8004ec60 00007f3cad9cb020
0x00007f3cad9caea0:   0000000000000008 00007f3dbc012010
0x00007f3cad9caeb0:   00007f3cad9caf70 00007f3e86f45e66
0x00007f3cad9caec0:   00007f3cad9cb040 00007f3cad9cb060
0x00007f3cad9caed0:   00007f3dbc016800 00007f3e86f45df5
0x00007f3cad9caee0:   1ad7f0005b49df4d 00007f3e8001bab0
0x00007f3cad9caef0:   00007f3cad9cb080 ccb8dba5cdd92e7e
0x00007f3cad9caf00:   00007f3cad9cafe0 00007f3cad9cb040
0x00007f3cad9caf10:   00007f3cad9cb060 00007f3e81820e80
0x00007f3cad9caf20:   00007f3af59defb0 00007f3dbc016800
0x00007f3cad9caf30:   0000000000000007 f24dd0c26f5c6c00

Instructions: (pc=0x00007f3e873921fe)
0x00007f3e873921de:   00 00 48 8d 0c 58 0f b7 11 66 85 d2 0f 84 45 ff
0x00007f3e873921ee:   ff ff 48 8d 04 d8 83 ea 01 4c 8b 80 80 00 00 00
0x00007f3e873921fe:   49 8b 30 48 89 b0 80 00 00 00 66 89 11 49 c7 40
0x00007f3e8739220e:   08 00 00 00 00 e9 6e ff ff ff 0f 1f 84 00 00 00

Register to memory mapping:

RAX=0x00007f3c100623c8 is an unknown value
RBX=0x0000000000000005 is an unknown value
RCX=0x00007f3c100623aa is an unknown value
RDX=0x0000000000000000 is an unknown value
RSP=0x00007f3cad9cad40 is pointing into the stack for thread: 0x00007f3dbc016800
RBP=0x0000000000000060 is an unknown value
RSI=0x0000000000000009 is an unknown value
RDI=0x0000000000000060 is an unknown value
R8 =0x5c65b6c2695baec3 is an unknown value
R9 =0x00007f3c111cbd70 is an unknown value
R10=0x00007f3c100623a0 is an unknown value
R11=0x00007f3c10000080 is an unknown value
R12=0xffffffffffffffb0 is an unknown value
R13=0x0000000000000009 is an unknown value
R14=0x00007f3cad9cadb0 is pointing into the stack for thread: 0x00007f3dbc016800
R15=0x00007f3c1154450a is an unknown value


Stack: [0x00007f3cad8cf000,0x00007f3cad9d0000],  sp=0x00007f3cad9cad40,  free space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x9a1fe]  __libc_malloc+0x11e

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 724  java.lang.ClassLoader.defineClass1(Ljava/lang/String;[BIILjava/security/ProtectionDomain;Ljava/lang/String;)Ljava/lang/Class; (0 bytes) @ 0x00007f3e71478d11 [0x00007f3e71478c80+0x91]
J 5821 C2 java.security.SecureClassLoader.defineClass(Ljava/lang/String;[BIILjava/security/CodeSource;)Ljava/lang/Class; (16 bytes) @ 0x00007f3e720fd874 [0x00007f3e720fd140+0x734]
J 53471 C2 java.net.URLClassLoader$1.run()Ljava/lang/Class; (81 bytes) @ 0x00007f3e77dc1558 [0x00007f3e77dc0500+0x1058]
J 54198 C2 java.net.URLClassLoader$1.run()Ljava/lang/Object; (5 bytes) @ 0x00007f3e73467594 [0x00007f3e73467560+0x34]
v  ~StubRoutines::call_stub
J 178  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (0 bytes) @ 0x00007f3e7131b223 [0x00007f3e7131b1c0+0x63]
J 2321 C2 java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class; (47 bytes) @ 0x00007f3e718ed430 [0x00007f3e718ed3a0+0x90]
J 54944 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122 bytes) @ 0x00007f3e77e22c98 [0x00007f3e77e22820+0x478]
J 3669 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7 bytes) @ 0x00007f3e71412960 [0x00007f3e71412920+0x40]
v  ~StubRoutines::call_stub
j  com.nvidia.spark.rapids.GpuParquetWriter.deepTransformColumn(Lai/rapids/cudf/ColumnVector;Lorg/apache/spark/sql/types/DataType;)Lai/rapids/cudf/ColumnVector;+0
j  com.nvidia.spark.rapids.GpuParquetWriter.$anonfun$writeAndClose$2(Lcom/nvidia/spark/rapids/GpuParquetWriter;Lcom/nvidia/spark/rapids/GpuColumnVector;)Lorg/apache/spark/sql/vectorized/ColumnVector;+17
j  com.nvidia.spark.rapids.GpuParquetWriter$$Lambda$6668.apply(Ljava/lang/Object;)Ljava/lang/Object;+8
J 38938 C2 com.nvidia.spark.rapids.RapidsPluginImplicits$MapsSafely$$Lambda$3208.apply(Ljava/lang/Object;)Ljava/lang/Object; (13 bytes) @ 0x00007f3e760f1528 [0x00007f3e760f14e0+0x48]
J 14465 C2 scala.collection.mutable.ArrayOps$ofRef.foreach(Lscala/Function1;)V (6 bytes) @ 0x00007f3e736b33d0 [0x00007f3e736b3320+0xb0]
J 44007 C2 com.nvidia.spark.rapids.RapidsPluginImplicits$AutoCloseableProducingArray.safeMap(Lscala/Function1;Lscala/reflect/ClassTag;)[Ljava/lang/AutoCloseable; (26 bytes) @ 0x00007f3e792c1484 [0x00007f3e792c12a0+0x1e4]
j  com.nvidia.spark.rapids.GpuParquetWriter.$anonfun$writeAndClose$1(Lcom/nvidia/spark/rapids/GpuParquetWriter;Lorg/apache/spark/sql/vectorized/ColumnarBatch;)Lorg/apache/spark/sql/vectorized/ColumnarBatch;+25
j  com.nvidia.spark.rapids.GpuParquetWriter$$Lambda$6667.apply(Ljava/lang/Object;)Ljava/lang/Object;+8
J 17055 C2 com.nvidia.spark.rapids.Arm$.withResource(Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object; (60 bytes) @ 0x00007f3e740ad94c [0x00007f3e740ad900+0x4c]
j  com.nvidia.spark.rapids.GpuParquetWriter.writeAndClose(Lorg/apache/spark/sql/vectorized/ColumnarBatch;Lscala/collection/Seq;)V+10
j  org.apache.spark.sql.rapids.GpuSingleDirectoryDataWriter.write(Lorg/apache/spark/sql/vectorized/ColumnarBatch;)V+49
j  org.apache.spark.sql.rapids.GpuFileFormatDataWriter.writeWithIterator(Lscala/collection/Iterator;)V+19
j  org.apache.spark.sql.rapids.GpuFileFormatWriter$.$anonfun$executeTask$1(Lorg/apache/spark/sql/rapids/GpuFileFormatDataWriter;Lscala/collection/Iterator;)Lorg/apache/spark/sql/execution/datasources/WriteTaskResult;+2
j  org.apache.spark.sql.rapids.GpuFileFormatWriter$$$Lambda$6134.apply()Ljava/lang/Object;+8
j  org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Lscala/Function0;Lscala/Function0;Lscala/Function0;)Ljava/lang/Object;+4
j  org.apache.spark.sql.rapids.GpuFileFormatWriter$.executeTask(Lorg/apache/spark/sql/rapids/GpuWriteJobDescription;JIIILorg/apache/spark/internal/io/FileCommitProtocol;Lscala/collection/Iterator;Lscala/Option;)Lorg/apache/spark/sql/execution/datasources/WriteTaskResult;+304
j  org.apache.spark.sql.rapids.GpuFileFormatWriter$.$anonfun$write$15(Lorg/apache/spark/sql/rapids/GpuWriteJobDescription;JLorg/apache/spark/internal/io/FileCommitProtocol;Lscala/Option;Lorg/apache/spark/TaskContext;Lscala/collection/Iterator;)Lorg/apache/spark/sql/execution/datasources/WriteTaskResult;+30
j  org.apache.spark.sql.rapids.GpuFileFormatWriter$$$Lambda$6131.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+24
J 61949 C2 org.apache.spark.scheduler.Task.run(JILorg/apache/spark/metrics/MetricsSystem;Lscala/collection/immutable/Map;Lscala/Option;)Ljava/lang/Object; (495 bytes) @ 0x00007f3e75309944 [0x00007f3e753055a0+0x43a4]
J 32899 C2 org.apache.spark.executor.Executor$TaskRunner$$Lambda$2572.apply()Ljava/lang/Object; (12 bytes) @ 0x00007f3e72f2746c [0x00007f3e72f27400+0x6c]
J 32687 C2 org.apache.spark.util.Utils$.tryWithSafeFinally(Lscala/Function0;Lscala/Function0;)Ljava/lang/Object; (207 bytes) @ 0x00007f3e76044508 [0x00007f3e760444c0+0x48]
J 65584 C1 org.apache.spark.executor.Executor$TaskRunner.run()V (2942 bytes) @ 0x00007f3e7a8a3d2c [0x00007f3e7a89f640+0x46ec]
J 63599 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f3e78e93028 [0x00007f3e78e92d80+0x2a8]
J 32991 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x00007f3e7478c584 [0x00007f3e7478c480+0x104]
J 16324 C1 java.lang.Thread.run()V (17 bytes) @ 0x00007f3e71557544 [0x00007f3e71557400+0x144]
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00007f3d1c080000 JavaThread "Worker Monitor for /usr/bin/python" daemon [_thread_blocked, id=15146, stack(0x00007f3af8ce1000,0x00007f3af8de2000)]
  0x00007f3c0c0bb000 JavaThread "Worker Monitor for /usr/bin/python" daemon [_thread_blocked, id=15144, stack(0x00007f3c68aeb000,0x00007f3c68bec000)]
  0x00007f3c114d8000 JavaThread "Worker Monitor for /usr/bin/python" daemon [_thread_blocked, id=15142, stack(0x00007f3c783e6000,0x00007f3c784e7000)]
  0x00007f3b600ba800 JavaThread "Worker Monitor for /usr/bin/python" daemon [_thread_blocked, id=15140, stack(0x00007f3c801e2000,0x00007f3c802e3000)]
  0x00007f3d1c034800 JavaThread "Worker Monitor for /usr/bin/python" daemon [_thread_blocked, id=14997, stack(0x00007f3af97ec000,0x00007f3af98ed000)]
  0x00007f3c100ca800 JavaThread "Worker Monitor for /usr/bin/python" daemon [_thread_blocked, id=14994, stack(0x00007f3c696f7000,0x00007f3c697f8000)]
  0x00007f3c0d24b800 JavaThread "Worker Monitor for /usr/bin/python" daemon [_thread_blocked, id=14991, stack(0x00007f3c994f5000,0x00007f3c995f6000)]
  0x00007f3b6002f000 JavaThread "Worker Monitor for /usr/bin/python" daemon [_thread_blocked, id=14988, stack(0x00007f3cacac9000,0x00007f3cacbca000)]
  0x00007f3d8c056800 JavaThread "block-manager-storage-async-thread-pool-940" daemon [_thread_blocked, id=13101, stack(0x00007f3c99bfc000,0x00007f3c99cfd000)]
  0x00007f3d78003800 JavaThread "process reaper" daemon [_thread_blocked, id=13086, stack(0x00007f3d300bc000,0x00007f3d300f5000)]
  0x00007f3c0d245800 JavaThread "process reaper" daemon [_thread_blocked, id=13041, stack(0x00007f3e48007000,0x00007f3e48040000)]
  0x00007f3dbc017800 JavaThread "Executor task launch worker for task 0.0 in stage 5997.0 (TID 20529)" daemon [_thread_blocked, id=13037, stack(0x00007f3d10042000,0x00007f3d10143000)]
=>0x00007f3dbc016800 JavaThread "Executor task launch worker for task 2.0 in stage 5997.0 (TID 20531)" daemon [_thread_in_vm, id=13036, stack(0x00007f3cad8cf000,0x00007f3cad9d0000)]
  0x00007f3dbc015800 JavaThread "Executor task launch worker for task 1.0 in stage 5997.0 (TID 20530)" daemon [_thread_blocked, id=13035, stack(0x00007f3c701e2000,0x00007f3c702e3000)]
  0x00007f3d8c04d000 JavaThread "block-manager-storage-async-thread-pool-939" daemon [_thread_blocked, id=11653, stack(0x00007f3cad7ce000,0x00007f3cad8cf000)]
  0x00007f3be8009000 JavaThread "block-manager-storage-async-thread-pool-938" daemon [_thread_blocked, id=9257, stack(0x00007f3c796f7000,0x00007f3c797f8000)]
  0x00007f3d8c02c800 JavaThread "block-manager-storage-async-thread-pool-937" daemon [_thread_blocked, id=9256, stack(0x00007f3c788eb000,0x00007f3c789ec000)]
  0x00007f3d8c02a800 JavaThread "block-manager-storage-async-thread-pool-936" daemon [_thread_blocked, id=8172, stack(0x00007f3c809ea000,0x00007f3c80aeb000)]
  0x00007f3d8c056000 JavaThread "block-manager-storage-async-thread-pool-935" daemon [_thread_blocked, id=5447, stack(0x00007f3afa2f5000,0x00007f3afa3f6000)]
  0x00007f3d8c055000 JavaThread "block-manager-storage-async-thread-pool-934" daemon [_thread_blocked, id=4209, stack(0x00007f3c980e3000,0x00007f3c981e4000)]
  0x00007f3b0800d000 JavaThread "block-manager-storage-async-thread-pool-933" daemon [_thread_blocked, id=1430, stack(0x00007f3c983e6000,0x00007f3c984e7000)]
  0x00007f3d8c054800 JavaThread "block-manager-storage-async-thread-pool-932" daemon [_thread_blocked, id=1429, stack(0x00007f3c98bee000,0x00007f3c98cef000)]
  0x00007f3d8c047000 JavaThread "block-manager-storage-async-thread-pool-931" daemon [_thread_blocked, id=29932, stack(0x00007f3c905e6000,0x00007f3c906e7000)]
  0x00007f3d1c009000 JavaThread "shuffle-server-8-1" daemon [_thread_in_native, id=29929, stack(0x00007f3c680e3000,0x00007f3c681e4000)]
  0x00007f3be4025800 JavaThread "shuffle-client-6-1" daemon [_thread_in_native, id=29928, stack(0x00007f3c685e6000,0x00007f3c686e7000)]
  0x00007f3d8c012800 JavaThread "block-manager-storage-async-thread-pool-930" daemon [_thread_blocked, id=25631, stack(0x00007f3c200f2000,0x00007f3c201f3000)]
  0x00007f3af0a2b800 JavaThread "BoneCP-pool-watch-thread" daemon [_thread_blocked, id=18147, stack(0x00007f3c684e5000,0x00007f3c685e6000)]
  0x00007f3af0a2b000 JavaThread "BoneCP-keep-alive-scheduler" daemon [_thread_blocked, id=18146, stack(0x00007f3c718f9000,0x00007f3c719fa000)]
  0x00007f3af0a22000 JavaThread "com.google.common.base.internal.Finalizer" daemon [_thread_blocked, id=18145, stack(0x00007f3c901e2000,0x00007f3c902e3000)]
  0x00007f3c67f00800 JavaThread "BoneCP-pool-watch-thread" daemon [_thread_blocked, id=18011, stack(0x00007f3af8de2000,0x00007f3af8ee3000)]
  0x00007f3c67f04800 JavaThread "BoneCP-keep-alive-scheduler" daemon [_thread_blocked, id=18010, stack(0x00007f3c99cfd000,0x00007f3c99dfe000)]
  0x00007f3c67f01800 JavaThread "com.google.common.base.internal.Finalizer" daemon [_thread_blocked, id=18009, stack(0x00007f3af8adf000,0x00007f3af8be0000)]
  0x00007f3af06d2000 JavaThread "derby.rawStoreDaemon" daemon [_thread_blocked, id=17866, stack(0x00007f3ca1afd000,0x00007f3ca1bfe000)]
  0x00007f3af0680800 JavaThread "Timer-2" daemon [_thread_blocked, id=17848, stack(0x00007f3af89de000,0x00007f3af8adf000)]
  0x00007f3dd0589000 JavaThread "process reaper" daemon [_thread_blocked, id=17709, stack(0x00007f3e48040000,0x00007f3e48079000)]
  0x00007f3e4400d800 JavaThread "block-manager-ask-thread-pool-496" daemon [_thread_blocked, id=15883, stack(0x00007f3c904e5000,0x00007f3c905e6000)]
  0x00007f3bb0004000 JavaThread "block-manager-ask-thread-pool-495" daemon [_thread_blocked, id=15882, stack(0x00007f3c69cfd000,0x00007f3c69dfe000)]
  0x00007f3d8c037800 JavaThread "block-manager-storage-async-thread-pool-928" daemon [_thread_blocked, id=15880, stack(0x00007f3c889ea000,0x00007f3c88aeb000)]
  0x00007f3e100e7800 JavaThread "block-manager-ask-thread-pool-494" daemon [_thread_blocked, id=15784, stack(0x00007f3af99ee000,0x00007f3af9aef000)]
  0x00007f3da4002000 JavaThread "block-manager-ask-thread-pool-493" daemon [_thread_blocked, id=15783, stack(0x00007f3af9ef1000,0x00007f3af9ff2000)]
  0x00007f3d9c005000 JavaThread "block-manager-ask-thread-pool-492" daemon [_thread_blocked, id=15782, stack(0x00007f3af9aef000,0x00007f3af9bf0000)]
  0x00007f3d9c004800 JavaThread "block-manager-storage-async-thread-pool-927" daemon [_thread_blocked, id=15781, stack(0x00007f3c71afb000,0x00007f3c71bfc000)]
  0x00007f3d8c046000 JavaThread "block-manager-storage-async-thread-pool-926" daemon [_thread_blocked, id=15780, stack(0x00007f3c694f5000,0x00007f3c695f6000)]
  0x00007f3d94003000 JavaThread "block-manager-ask-thread-pool-491" daemon [_thread_blocked, id=15779, stack(0x00007f3c985e8000,0x00007f3c986e9000)]
  0x00007f3d98013000 JavaThread "block-manager-ask-thread-pool-490" daemon [_thread_blocked, id=15778, stack(0x00007f3c215f7000,0x00007f3c216f8000)]
  0x00007f3bd0006000 JavaThread "block-manager-storage-async-thread-pool-925" daemon [_thread_blocked, id=15777, stack(0x00007f3c803e4000,0x00007f3c804e5000)]
  0x00007f3bd0005000 JavaThread "block-manager-storage-async-thread-pool-924" daemon [_thread_blocked, id=15776, stack(0x00007f3c20cf4000,0x00007f3c20df5000)]
  0x00007f3d8c035800 JavaThread "block-manager-storage-async-thread-pool-923" daemon [_thread_blocked, id=15775, stack(0x00007f3ca04eb000,0x00007f3ca05ec000)]
  0x00007f3b94004000 JavaThread "block-manager-ask-thread-pool-489" daemon [_thread_blocked, id=15669, stack(0x00007f3afa3f6000,0x00007f3afa4f7000)]
  0x00007f3bbc003800 JavaThread "block-manager-ask-thread-pool-488" daemon [_thread_blocked, id=15664, stack(0x00007f3c918f9000,0x00007f3c919fa000)]
  0x00007f3bc0002000 JavaThread "block-manager-ask-thread-pool-487" daemon [_thread_blocked, id=15663, stack(0x00007f3c882e3000,0x00007f3c883e4000)]
  0x00007f3bb8004800 JavaThread "block-manager-storage-async-thread-pool-922" daemon [_thread_blocked, id=15662, stack(0x00007f3c69bfc000,0x00007f3c69cfd000)]
  0x00007f3bb8003800 JavaThread "block-manager-storage-async-thread-pool-921" daemon [_thread_blocked, id=15661, stack(0x00007f3c79dfe000,0x00007f3c79eff000)]
  0x00007f3d8c053800 JavaThread "block-manager-storage-async-thread-pool-920" daemon [_thread_blocked, id=15660, stack(0x00007f3c709ea000,0x00007f3c70aeb000)]
  0x00007f3e4c00b000 JavaThread "block-manager-ask-thread-pool-486" daemon [_thread_blocked, id=15457, stack(0x00007f3c80ff0000,0x00007f3c810f1000)]
  0x00007f3e4c006800 JavaThread "block-manager-storage-async-thread-pool-919" daemon [_thread_blocked, id=15456, stack(0x00007f3c902e3000,0x00007f3c903e4000)]
  0x00007f3d8c016800 JavaThread "block-manager-storage-async-thread-pool-918" daemon [_thread_blocked, id=15455, stack(0x00007f3c81eff000,0x00007f3c82000000)]
  0x00007f3e3c018800 JavaThread "block-manager-ask-thread-pool-485" daemon [_thread_blocked, id=15454, stack(0x00007f3c993f4000,0x00007f3c994f5000)]
  0x00007f3e5c01f000 JavaThread "block-manager-ask-thread-pool-484" daemon [_thread_blocked, id=15453, stack(0x00007f3c883e6000,0x00007f3c884e7000)]
  0x00007f3e68004000 JavaThread "block-manager-storage-async-thread-pool-917" daemon [_thread_blocked, id=15452, stack(0x00007f3c987ea000,0x00007f3c988eb000)]
  0x00007f3d8c050000 JavaThread "block-manager-storage-async-thread-pool-915" daemon [_thread_blocked, id=15450, stack(0x00007f3c687ea000,0x00007f3c688eb000)]
  0x00007f3d8c024000 JavaThread "block-manager-storage-async-thread-pool-913" daemon [_thread_blocked, id=15383, stack(0x00007f3afbbfc000,0x00007f3afbcfd000)]
  0x00007f3dc4012000 JavaThread "block-manager-storage-async-thread-pool-912" daemon [_thread_blocked, id=15382, stack(0x00007f3afb9fa000,0x00007f3afbafb000)]
  0x00007f3dc400e800 JavaThread "block-manager-storage-async-thread-pool-911" daemon [_thread_blocked, id=15381, stack(0x00007f3afb5f8000,0x00007f3afb6f9000)]
  0x00007f3d8c04e800 JavaThread "block-manager-storage-async-thread-pool-910" daemon [_thread_blocked, id=15380, stack(0x00007f3afa4f7000,0x00007f3afa5f8000)]
  0x00007f3d3402d800 JavaThread "block-manager-ask-thread-pool-483" daemon [_thread_blocked, id=15313, stack(0x00007f3ca1bfe000,0x00007f3ca1cff000)]
  0x00007f3d74005800 JavaThread "block-manager-ask-thread-pool-482" daemon [_thread_blocked, id=15312, stack(0x00007f3c802e3000,0x00007f3c803e4000)]
  0x00007f3d70001800 JavaThread "block-manager-ask-thread-pool-481" daemon [_thread_blocked, id=15311, stack(0x00007f3c906e7000,0x00007f3c907e8000)]
  0x00007f3d70004000 JavaThread "block-manager-storage-async-thread-pool-909" daemon [_thread_blocked, id=15310, stack(0x00007f3c68ff0000,0x00007f3c690f1000)]
  0x00007f3d8c04a000 JavaThread "block-manager-storage-async-thread-pool-908" daemon [_thread_blocked, id=15307, stack(0x00007f3c717f8000,0x00007f3c718f9000)]
  0x00007f3d28002000 JavaThread "block-manager-ask-thread-pool-480" daemon [_thread_blocked, id=15236, stack(0x00007f3c896f7000,0x00007f3c897f8000)]
  0x00007f3d44007000 JavaThread "block-manager-ask-thread-pool-479" daemon [_thread_blocked, id=15235, stack(0x00007f3c899fa000,0x00007f3c89afb000)]
  0x00007f3d3c00c800 JavaThread "block-manager-ask-thread-pool-478" daemon [_thread_blocked, id=15234, stack(0x00007f3c88bec000,0x00007f3c88ced000)]
  0x00007f3d3c00b800 JavaThread "block-manager-storage-async-thread-pool-907" daemon [_thread_blocked, id=15233, stack(0x00007f3c897f8000,0x00007f3c898f9000)]
  0x00007f3d8c01e800 JavaThread "block-manager-storage-async-thread-pool-906" daemon [_thread_blocked, id=15232, stack(0x00007f3c913f4000,0x00007f3c914f5000)]
  0x00007f3b9c005000 JavaThread "block-manager-ask-thread-pool-477" daemon [_thread_blocked, id=15111, stack(0x00007f3ca1cff000,0x00007f3ca1e00000)]
  0x00007f3be8008000 JavaThread "block-manager-ask-thread-pool-476" daemon [_thread_blocked, id=15110, stack(0x00007f3c792f3000,0x00007f3c793f4000)]
  0x00007f3bb4003000 JavaThread "block-manager-ask-thread-pool-475" daemon [_thread_blocked, id=15109, stack(0x00007f3c982e5000,0x00007f3c983e6000)]
  0x00007f3bb8001800 JavaThread "block-manager-ask-thread-pool-474" daemon [_thread_blocked, id=15108, stack(0x00007f3c686e7000,0x00007f3c687e8000)]
  0x00007f3d8c014800 JavaThread "block-manager-storage-async-thread-pool-905" daemon [_thread_blocked, id=15107, stack(0x00007f3c790f1000,0x00007f3c791f2000)]
  0x00007f3b38006800 JavaThread "block-manager-ask-thread-pool-473" daemon [_thread_blocked, id=15033, stack(0x00007f3c710f1000,0x00007f3c711f2000)]
  0x00007f3b60003000 JavaThread "block-manager-ask-thread-pool-472" daemon [_thread_blocked, id=15032, stack(0x00007f3c79eff000,0x00007f3c7a000000)]
  0x00007f3b30004800 JavaThread "block-manager-ask-thread-pool-471" daemon [_thread_blocked, id=15031, stack(0x00007f3c986e9000,0x00007f3c987ea000)]
  0x00007f3df406d800 JavaThread "block-manager-ask-thread-pool-470" daemon [_thread_blocked, id=15030, stack(0x00007f3c69dfe000,0x00007f3c69eff000)]
  0x00007f3b28007800 JavaThread "block-manager-ask-thread-pool-469" daemon [_thread_blocked, id=15029, stack(0x00007f3c68dee000,0x00007f3c68eef000)]
  0x00007f3b20001000 JavaThread "block-manager-ask-thread-pool-468" daemon [_thread_blocked, id=15028, stack(0x00007f3c90ff0000,0x00007f3c910f1000)]
  0x00007f3b20005000 JavaThread "block-manager-storage-async-thread-pool-904" daemon [_thread_blocked, id=15027, stack(0x00007f3c697f8000,0x00007f3c698f9000)]
  0x00007f3d8c00c000 JavaThread "block-manager-storage-async-thread-pool-903" daemon [_thread_blocked, id=15026, stack(0x00007f3ca17fc000,0x00007f3ca18fd000)]
  0x00007f3e6001a000 JavaThread "block-manager-ask-thread-pool-467" daemon [_thread_blocked, id=14963, stack(0x00007f3c91bfc000,0x00007f3c91cfd000)]
  0x00007f3d6c001800 JavaThread "block-manager-ask-thread-pool-466" daemon [_thread_blocked, id=14962, stack(0x00007f3ca16fb000,0x00007f3ca17fc000)]
  0x00007f3e34004000 JavaThread "block-manager-ask-thread-pool-465" daemon [_thread_blocked, id=14961, stack(0x00007f3c89afb000,0x00007f3c89bfc000)]
  0x00007f3d60006800 JavaThread "block-manager-ask-thread-pool-464" daemon [_thread_blocked, id=14960, stack(0x00007f3c21efe000,0x00007f3c21fff000)]
  0x00007f3e3000c800 JavaThread "block-manager-ask-thread-pool-463" daemon [_thread_blocked, id=14959, stack(0x00007f3c21dfd000,0x00007f3c21efe000)]
  0x00007f3e20003000 JavaThread "block-manager-ask-thread-pool-462" daemon [_thread_blocked, id=14958, stack(0x00007f3c70bec000,0x00007f3c70ced000)]
  0x00007f3e2806c000 JavaThread "block-manager-ask-thread-pool-461" daemon [_thread_blocked, id=14957, stack(0x00007f3c70dee000,0x00007f3c70eef000)]
  0x00007f3e1c00a800 JavaThread "block-manager-ask-thread-pool-460" daemon [_thread_blocked, id=14956, stack(0x00007f3c892f3000,0x00007f3c893f4000)]
  0x00007f3d8c04b000 JavaThread "block-manager-storage-async-thread-pool-902" daemon [_thread_blocked, id=14955, stack(0x00007f3c71bfc000,0x00007f3c71cfd000)]
  0x00007f3e184a9800 JavaThread "block-manager-ask-thread-pool-459" daemon [_thread_blocked, id=14953, stack(0x00007f3c80bec000,0x00007f3c80ced000)]
  0x00007f3e0c768000 JavaThread "block-manager-ask-thread-pool-458" daemon [_thread_blocked, id=14952, stack(0x00007f3c894f5000,0x00007f3c895f6000)]
  0x00007f3b4c006000 JavaThread "block-manager-storage-async-thread-pool-900" daemon [_thread_blocked, id=14951, stack(0x00007f3c70aeb000,0x00007f3c70bec000)]
  0x00007f3dc8014000 JavaThread "block-manager-ask-thread-pool-457" daemon [_thread_blocked, id=14918, stack(0x00007f3c815f6000,0x00007f3c816f7000)]
  0x00007f3d64004800 JavaThread "block-manager-ask-thread-pool-456" daemon [_thread_blocked, id=14917, stack(0x00007f3afbafb000,0x00007f3afbbfc000)]
  0x00007f3dc000c000 JavaThread "block-manager-ask-thread-pool-455" daemon [_thread_blocked, id=14916, stack(0x00007f3c795f6000,0x00007f3c796f7000)]
  0x00007f3b8c003000 JavaThread "block-manager-ask-thread-pool-454" daemon [_thread_blocked, id=14915, stack(0x00007f3ca03ea000,0x00007f3ca04eb000)]
  0x00007f3db8002800 JavaThread "block-manager-ask-thread-pool-453" daemon [_thread_blocked, id=14914, stack(0x00007f3afbcfd000,0x00007f3afbdfe000)]
  0x00007f3be4009000 JavaThread "block-manager-ask-thread-pool-452" daemon [_thread_blocked, id=14913, stack(0x00007f3c715f6000,0x00007f3c716f7000)]
  0x00007f3db0004800 JavaThread "block-manager-ask-thread-pool-451" daemon [_thread_blocked, id=14912, stack(0x00007f3c216f8000,0x00007f3c217f9000)]
  0x00007f3d3402f800 JavaThread "block-manager-ask-thread-pool-450" daemon [_thread_blocked, id=14911, stack(0x00007f3cac8c7000,0x00007f3cac9c8000)]
  0x00007f3da8004800 JavaThread "block-manager-ask-thread-pool-449" daemon [_thread_blocked, id=14910, stack(0x00007f3ca05ec000,0x00007f3ca06ed000)]
  0x00007f3d9c002800 JavaThread "block-manager-ask-thread-pool-448" daemon [_thread_blocked, id=14909, stack(0x00007f3c700e1000,0x00007f3c701e2000)]
  0x00007f3b10004800 JavaThread "block-manager-storage-async-thread-pool-897" daemon [_thread_blocked, id=14908, stack(0x00007f3c800e1000,0x00007f3c801e2000)]
  0x00007f3b10001800 JavaThread "block-manager-storage-async-thread-pool-896" daemon [_thread_blocked, id=14907, stack(0x00007f3c702e3000,0x00007f3c703e4000)]
  0x00007f3d8c049000 JavaThread "block-manager-storage-async-thread-pool-895" daemon [_thread_blocked, id=14906, stack(0x00007f3ca07ee000,0x00007f3ca08ef000)]
  0x00007f3d5402d000 JavaThread "block-manager-ask-thread-pool-447" daemon [_thread_blocked, id=14857, stack(0x00007f3c787ea000,0x00007f3c788eb000)]
  0x00007f3d50002000 JavaThread "block-manager-ask-thread-pool-446" daemon [_thread_blocked, id=14856, stack(0x00007f3c90dee000,0x00007f3c90eef000)]
  0x00007f3d4c001800 JavaThread "block-manager-ask-thread-pool-445" daemon [_thread_blocked, id=14855, stack(0x00007f3c716f7000,0x00007f3c717f8000)]
  0x00007f3bec007800 JavaThread "block-manager-ask-thread-pool-444" daemon [_thread_blocked, id=14854, stack(0x00007f3c690f1000,0x00007f3c691f2000)]
  0x00007f3d8c027000 JavaThread "block-manager-storage-async-thread-pool-894" daemon [_thread_blocked, id=14853, stack(0x00007f3c789ec000,0x00007f3c78aed000)]
  0x00007f3d4813a800 JavaThread "block-manager-ask-thread-pool-443" daemon [_thread_blocked, id=14852, stack(0x00007f3c80ced000,0x00007f3c80dee000)]
  0x00007f3d3c00e000 JavaThread "block-manager-ask-thread-pool-442" daemon [_thread_blocked, id=14851, stack(0x00007f3ca02e9000,0x00007f3ca03ea000)]
  0x00007f3d3402e800 JavaThread "block-manager-storage-async-thread-pool-893" daemon [_thread_blocked, id=14850, stack(0x00007f3ca0ef5000,0x00007f3ca0ff6000)]
  0x00007f3d34028800 JavaThread "block-manager-storage-async-thread-pool-892" daemon [_thread_blocked, id=14849, stack(0x00007f3c782e5000,0x00007f3c783e6000)]
  0x00007f3d8c048800 JavaThread "block-manager-storage-async-thread-pool-891" daemon [_thread_blocked, id=14848, stack(0x00007f3c91eff000,0x00007f3c92000000)]
  0x00007f3d2c0b8800 JavaThread "block-manager-ask-thread-pool-441" daemon [_thread_blocked, id=14847, stack(0x00007f3c912f3000,0x00007f3c913f4000)]
  0x00007f3d24004000 JavaThread "block-manager-ask-thread-pool-440" daemon [_thread_blocked, id=14846, stack(0x00007f3c713f4000,0x00007f3c714f5000)]
  0x00007f3bc4002000 JavaThread "block-manager-storage-async-thread-pool-890" daemon [_thread_blocked, id=14845, stack(0x00007f3c88ced000,0x00007f3c88dee000)]
  0x00007f3bc4003000 JavaThread "block-manager-storage-async-thread-pool-889" daemon [_thread_blocked, id=14844, stack(0x00007f3c909ea000,0x00007f3c90aeb000)]
  0x00007f3d8c052800 JavaThread "block-manager-storage-async-thread-pool-888" daemon [_thread_blocked, id=14843, stack(0x00007f3c996f7000,0x00007f3c997f8000)]
  0x00007f3b94003000 JavaThread "block-manager-storage-async-thread-pool-887" daemon [_thread_blocked, id=13912, stack(0x00007f3c20ef6000,0x00007f3c20ff7000)]
  0x00007f3d8c01c800 JavaThread "block-manager-storage-async-thread-pool-886" daemon [_thread_blocked, id=13911, stack(0x00007f3c812f3000,0x00007f3c813f4000)]
  0x00007f3b90003000 JavaThread "block-manager-storage-async-thread-pool-885" daemon [_thread_blocked, id=13910, stack(0x00007f3c895f6000,0x00007f3c896f7000)]
  0x00007f3b90001000 JavaThread "block-manager-storage-async-thread-pool-884" daemon [_thread_blocked, id=13909, stack(0x00007f3c890f1000,0x00007f3c891f2000)]
  0x00007f3d8c032000 JavaThread "block-manager-storage-async-thread-pool-883" daemon [_thread_blocked, id=13908, stack(0x00007f3c79cfd000,0x00007f3c79dfe000)]
  0x00007f3b3c004000 JavaThread "block-manager-ask-thread-pool-439" daemon [_thread_blocked, id=13677, stack(0x00007f3ca0cf3000,0x00007f3ca0df4000)]
  0x00007f3b34006800 JavaThread "block-manager-storage-async-thread-pool-882" daemon [_thread_blocked, id=13676, stack(0x00007f3c793f4000,0x00007f3c794f5000)]
  0x00007f3b34004800 JavaThread "block-manager-storage-async-thread-pool-881" daemon [_thread_blocked, id=13675, stack(0x00007f3c791f2000,0x00007f3c792f3000)]
  0x00007f3d8c043000 JavaThread "block-manager-storage-async-thread-pool-880" daemon [_thread_blocked, id=13674, stack(0x00007f3c807e8000,0x00007f3c808e9000)]
  0x00007f3b2c005000 JavaThread "block-manager-ask-thread-pool-438" daemon [_thread_blocked, id=13673, stack(0x00007f3c799fa000,0x00007f3c79afb000)]
  0x00007f3b30003800 JavaThread "block-manager-ask-thread-pool-437" daemon [_thread_blocked, id=13672, stack(0x00007f3c78ff0000,0x00007f3c790f1000)]
  0x00007f3b28006000 JavaThread "block-manager-storage-async-thread-pool-878" daemon [_thread_blocked, id=13670, stack(0x00007f3c999fa000,0x00007f3c99afb000)]
  0x00007f3d8c028800 JavaThread "block-manager-storage-async-thread-pool-877" daemon [_thread_blocked, id=13669, stack(0x00007f3c699fa000,0x00007f3c69afb000)]
  0x00007f3b20003000 JavaThread "block-manager-ask-thread-pool-436" daemon [_thread_blocked, id=13668, stack(0x00007f3c804e5000,0x00007f3c805e6000)]
  0x00007f3b14004000 JavaThread "block-manager-ask-thread-pool-435" daemon [_thread_blocked, id=13667, stack(0x00007f3c705e6000,0x00007f3c706e7000)]
  0x00007f3b74001000 JavaThread "block-manager-storage-async-thread-pool-876" daemon [_thread_blocked, id=13666, stack(0x00007f3c99eff000,0x00007f3c9a000000)]
  0x00007f3b74004800 JavaThread "block-manager-storage-async-thread-pool-875" daemon [_thread_blocked, id=13665, stack(0x00007f3c910f1000,0x00007f3c911f2000)]
  0x00007f3d8c00d000 JavaThread "block-manager-storage-async-thread-pool-874" daemon [_thread_blocked, id=13664, stack(0x00007f3c80eef000,0x00007f3c80ff0000)]
  0x00007f3b9c003800 JavaThread "block-manager-storage-async-thread-pool-873" daemon [_thread_blocked, id=13217, stack(0x00007f3c914f5000,0x00007f3c915f6000)]
  0x00007f3b9c002800 JavaThread "block-manager-storage-async-thread-pool-872" daemon [_thread_blocked, id=13216, stack(0x00007f3c810f1000,0x00007f3c811f2000)]
  0x00007f3d8c006800 JavaThread "block-manager-storage-async-thread-pool-871" daemon [_thread_blocked, id=13215, stack(0x00007f3cac0c1000,0x00007f3cac1c2000)]
  0x00007f3b98004800 JavaThread "block-manager-storage-async-thread-pool-870" daemon [_thread_blocked, id=13213, stack(0x00007f3c784e7000,0x00007f3c785e8000)]
  0x00007f3b98003800 JavaThread "block-manager-storage-async-thread-pool-869" daemon [_thread_blocked, id=13212, stack(0x00007f3c81bfc000,0x00007f3c81cfd000)]
  0x00007f3d8c01b000 JavaThread "block-manager-storage-async-thread-pool-868" daemon [_thread_blocked, id=13211, stack(0x00007f3c898f9000,0x00007f3c899fa000)]
  0x00007f3ba400a800 JavaThread "block-manager-ask-thread-pool-434" daemon [_thread_blocked, id=13210, stack(0x00007f3ca01e8000,0x00007f3ca02e9000)]
  0x00007f3b0c007800 JavaThread "block-manager-storage-async-thread-pool-867" daemon [_thread_blocked, id=13209, stack(0x00007f3c201f3000,0x00007f3c202f4000)]
  0x00007f3b0c003800 JavaThread "block-manager-storage-async-thread-pool-866" daemon [_thread_blocked, id=13208, stack(0x00007f3afbeff000,0x00007f3afc000000)]
  0x00007f3d8c034000 JavaThread "block-manager-storage-async-thread-pool-865" daemon [_thread_blocked, id=13207, stack(0x00007f3c819fa000,0x00007f3c81afb000)]
  0x00007f3e082bc800 JavaThread "block-manager-ask-thread-pool-433" daemon [_thread_blocked, id=12519, stack(0x00007f3c998f9000,0x00007f3c999fa000)]
  0x00007f3b54004800 JavaThread "block-manager-ask-thread-pool-430" daemon [_thread_blocked, id=9115, stack(0x00007f3c992f3000,0x00007f3c993f4000)]
  0x00007f3e3c016800 JavaThread "block-manager-ask-thread-pool-428" daemon [_thread_blocked, id=9113, stack(0x00007f3c888e9000,0x00007f3c889ea000)]
  0x00007f3d8c039000 JavaThread "block-manager-storage-async-thread-pool-858" daemon [_thread_blocked, id=9015, stack(0x00007f3c814f5000,0x00007f3c815f6000)]
  0x00007f3d9c001800 JavaThread "block-manager-ask-thread-pool-424" daemon [_thread_blocked, id=9013, stack(0x00007f3c68eef000,0x00007f3c68ff0000)]
  0x00007f3d94001800 JavaThread "block-manager-storage-async-thread-pool-856" daemon [_thread_blocked, id=9011, stack(0x00007f3c81cfd000,0x00007f3c81dfe000)]
  0x00007f3d14317800 JavaThread "block-manager-ask-thread-pool-422" daemon [_thread_blocked, id=8911, stack(0x00007f3c712f3000,0x00007f3c713f4000)]
  0x00007f3d8c026000 JavaThread "block-manager-storage-async-thread-pool-854" daemon [_thread_blocked, id=8869, stack(0x00007f3c908e9000,0x00007f3c909ea000)]
  0x00007f3d8c00b000 JavaThread "block-manager-storage-async-thread-pool-853" daemon [_thread_blocked, id=8868, stack(0x00007f3c781e4000,0x00007f3c782e5000)]
  0x00007f3e1c007000 JavaThread "block-manager-ask-thread-pool-420" daemon [_thread_blocked, id=8728, stack(0x00007f3c78bee000,0x00007f3c78cef000)]
  0x00007f3de4cef000 JavaThread "block-manager-storage-async-thread-pool-851" daemon [_thread_blocked, id=8666, stack(0x00007f3ca00e7000,0x00007f3ca01e8000)]
  0x00007f3d8c042800 JavaThread "block-manager-storage-async-thread-pool-850" daemon [_thread_blocked, id=8665, stack(0x00007f3ca13f8000,0x00007f3ca14f9000)]
  0x00007f3d8c04e000 JavaThread "block-manager-storage-async-thread-pool-847" daemon [_thread_blocked, id=8661, stack(0x00007f3ca14f9000,0x00007f3ca15fa000)]
  0x00007f3b18007000 JavaThread "block-manager-storage-async-thread-pool-846" daemon [_thread_blocked, id=8659, stack(0x00007f3c903e4000,0x00007f3c904e5000)]
  0x00007f3b18001000 JavaThread "block-manager-storage-async-thread-pool-845" daemon [_thread_blocked, id=8657, stack(0x00007f3c981e4000,0x00007f3c982e5000)]
  0x00007f3dcc010800 JavaThread "block-manager-ask-thread-pool-416" daemon [_thread_blocked, id=8656, stack(0x00007f3c78eef000,0x00007f3c78ff0000)]
  0x00007f3d8c020800 JavaThread "block-manager-storage-async-thread-pool-841" daemon [_thread_blocked, id=8652, stack(0x00007f3c891f2000,0x00007f3c892f3000)]
  0x00007f3d64006800 JavaThread "block-manager-storage-async-thread-pool-839" daemon [_thread_blocked, id=8575, stack(0x00007f3c780e3000,0x00007f3c781e4000)]
  0x00007f3b0c001000 JavaThread "block-manager-storage-async-thread-pool-838" daemon [_thread_blocked, id=8574, stack(0x00007f3c911f2000,0x00007f3c912f3000)]
  0x00007f3d8c008800 JavaThread "block-manager-storage-async-thread-pool-831" daemon [_thread_blocked, id=8544, stack(0x00007f3c991f2000,0x00007f3c992f3000)]
  0x00007f3d8c051000 JavaThread "block-manager-storage-async-thread-pool-827" daemon [_thread_blocked, id=8539, stack(0x00007f3c886e7000,0x00007f3c887e8000)]
  0x00007f3d8c009800 JavaThread "block-manager-storage-async-thread-pool-825" daemon [_thread_blocked, id=8464, stack(0x00007f3c71eff000,0x00007f3c72000000)]
  0x00007f3c194a5000 JavaThread "block-manager-ask-thread-pool-409" daemon [_thread_blocked, id=5863, stack(0x00007f3c99dfe000,0x00007f3c99eff000)]
  0x00007f3b94001800 JavaThread "block-manager-ask-thread-pool-408" daemon [_thread_blocked, id=5832, stack(0x00007f3afb8f9000,0x00007f3afb9fa000)]
  0x00007f3d8c044800 JavaThread "block-manager-storage-async-thread-pool-821" daemon [_thread_blocked, id=5831, stack(0x00007f3c98eef000,0x00007f3c98ff0000)]
  0x00007f3bbc003000 JavaThread "block-manager-ask-thread-pool-407" daemon [_thread_blocked, id=5830, stack(0x00007f3c915f6000,0x00007f3c916f7000)]
  0x00007f3d8c02e000 JavaThread "block-manager-storage-async-thread-pool-818" daemon [_thread_blocked, id=5826, stack(0x00007f3c919fa000,0x00007f3c91afb000)]
  0x00007f3b4c005000 JavaThread "block-manager-ask-thread-pool-403" daemon [_thread_blocked, id=5772, stack(0x00007f3c217f9000,0x00007f3c218fa000)]
  0x00007f3e3c01e800 JavaThread "block-manager-ask-thread-pool-402" daemon [_thread_blocked, id=5733, stack(0x00007f3c21cfc000,0x00007f3c21dfd000)]
  0x00007f3e0814b000 JavaThread "block-manager-ask-thread-pool-400" daemon [_thread_blocked, id=5679, stack(0x00007f3c785e8000,0x00007f3c786e9000)]
  0x00007f3df4134800 JavaThread "block-manager-ask-thread-pool-398" daemon [_thread_blocked, id=5676, stack(0x00007f3c71dfe000,0x00007f3c71eff000)]
  0x00007f3db0002000 JavaThread "block-manager-ask-thread-pool-395" daemon [_thread_blocked, id=5633, stack(0x00007f3c706e7000,0x00007f3c707e8000)]
  0x00007f3da8001800 JavaThread "block-manager-ask-thread-pool-394" daemon [_thread_blocked, id=5632, stack(0x00007f3c708e9000,0x00007f3c709ea000)]
  0x00007f3d3c00d000 JavaThread "block-manager-ask-thread-pool-393" daemon [_thread_blocked, id=5602, stack(0x00007f3c70ced000,0x00007f3c70dee000)]
  0x00007f3d34026800 JavaThread "block-manager-storage-async-thread-pool-806" daemon [_thread_blocked, id=5600, stack(0x00007f3c711f2000,0x00007f3c712f3000)]
  0x00007f3d8c011000 JavaThread "block-manager-storage-async-thread-pool-805" daemon [_thread_blocked, id=5599, stack(0x00007f3c714f5000,0x00007f3c715f6000)]
  0x00007f3be0003800 JavaThread "block-manager-ask-thread-pool-387" daemon [_thread_blocked, id=5520, stack(0x00007f3c880e1000,0x00007f3c881e2000)]
  0x00007f3b70001000 JavaThread "block-manager-ask-thread-pool-385" daemon [_thread_blocked, id=5474, stack(0x00007f3c881e2000,0x00007f3c882e3000)]
  0x00007f3b80001000 JavaThread "block-manager-ask-thread-pool-384" daemon [_thread_blocked, id=5473, stack(0x00007f3ca0df4000,0x00007f3ca0ef5000)]
  0x00007f3b80008000 JavaThread "block-manager-ask-thread-pool-376" daemon [_thread_blocked, id=5426, stack(0x00007f3c81dfe000,0x00007f3c81eff000)]
  0x00007f3b70004000 JavaThread "block-manager-ask-thread-pool-374" daemon [_thread_blocked, id=5424, stack(0x00007f3c98ff0000,0x00007f3c990f1000)]
  0x00007f3d8c018000 JavaThread "block-manager-storage-async-thread-pool-792" daemon [_thread_blocked, id=5338, stack(0x00007f3c89eff000,0x00007f3c8a000000)]
  0x00007f3df417c800 JavaThread "block-manager-ask-thread-pool-364" daemon [_thread_blocked, id=5336, stack(0x00007f3c907e8000,0x00007f3c908e9000)]
  0x00007f3dec5fc800 JavaThread "block-manager-storage-async-thread-pool-791" daemon [_thread_blocked, id=5335, stack(0x00007f3c90bec000,0x00007f3c90ced000)]
  0x00007f3de8002000 JavaThread "block-manager-ask-thread-pool-362" daemon [_thread_blocked, id=5331, stack(0x00007f3c916f7000,0x00007f3c917f8000)]
  0x00007f3dd8005800 JavaThread "block-manager-ask-thread-pool-361" daemon [_thread_blocked, id=5327, stack(0x00007f3c91cfd000,0x00007f3c91dfe000)]
  0x00007f3dbc014800 JavaThread "block-manager-ask-thread-pool-359" daemon [_thread_blocked, id=5322, stack(0x00007f3ca08ef000,0x00007f3ca09f0000)]
  0x00007f3d90003000 JavaThread "block-manager-ask-thread-pool-357" daemon [_thread_blocked, id=5296, stack(0x00007f3c99afb000,0x00007f3c99bfc000)]
  0x00007f3d68001800 JavaThread "block-manager-ask-thread-pool-352" daemon [_thread_blocked, id=5288, stack(0x00007f3cac1c2000,0x00007f3cac2c3000)]
  0x00007f3d54027800 JavaThread "block-manager-storage-async-thread-pool-775" daemon [_thread_blocked, id=5286, stack(0x00007f3c79afb000,0x00007f3c79bfc000)]
  0x00007f3d8c022000 JavaThread "block-manager-storage-async-thread-pool-772" daemon [_thread_blocked, id=5283, stack(0x00007f3c90eef000,0x00007f3c90ff0000)]
  0x00007f3d50005800 JavaThread "block-manager-ask-thread-pool-350" daemon [_thread_blocked, id=5280, stack(0x00007f3c887e8000,0x00007f3c888e9000)]
  0x00007f3d4810e800 JavaThread "block-manager-storage-async-thread-pool-769" daemon [_thread_blocked, id=5278, stack(0x00007f3c806e7000,0x00007f3c807e8000)]
  0x00007f3d8c006000 JavaThread "block-manager-storage-async-thread-pool-765" daemon [_thread_blocked, id=5272, stack(0x00007f3c988eb000,0x00007f3c989ec000)]
  0x00007f3d1c012800 JavaThread "block-manager-ask-thread-pool-346" daemon [_thread_blocked, id=5238, stack(0x00007f3c70ff0000,0x00007f3c710f1000)]
  0x00007f3d14316800 JavaThread "block-manager-storage-async-thread-pool-763" daemon [_thread_blocked, id=5236, stack(0x00007f3c900e1000,0x00007f3c901e2000)]
  0x00007f3d04006800 JavaThread "block-manager-ask-thread-pool-344" daemon [_thread_blocked, id=5233, stack(0x00007f3c79bfc000,0x00007f3c79cfd000)]
  0x00007f3c0d307800 JavaThread "block-manager-ask-thread-pool-343" daemon [_thread_blocked, id=5229, stack(0x00007f3c98aed000,0x00007f3c98bee000)]
  0x00007f3afc006800 JavaThread "block-manager-storage-async-thread-pool-757" daemon [_thread_blocked, id=5226, stack(0x00007f3cac9c8000,0x00007f3cacac9000)]
  0x00007f3b84002800 JavaThread "block-manager-storage-async-thread-pool-754" daemon [_thread_blocked, id=5005, stack(0x00007f3c816f9000,0x00007f3c817fa000)]
  0x00007f3d8c030000 JavaThread "block-manager-storage-async-thread-pool-749" daemon [_thread_blocked, id=4690, stack(0x00007f3c89bfc000,0x00007f3c89cfd000)]
  0x00007f3b80006800 JavaThread "block-manager-storage-async-thread-pool-747" daemon [_thread_blocked, id=4140, stack(0x00007f3c695f6000,0x00007f3c696f7000)]
  0x00007f3b9c001800 JavaThread "block-manager-ask-thread-pool-332" daemon [_thread_blocked, id=3775, stack(0x00007f3c71cfd000,0x00007f3c71dfe000)]
  0x00007f3d2c0b9000 JavaThread "block-manager-storage-async-thread-pool-736" daemon [_thread_blocked, id=3459, stack(0x00007f3c703e4000,0x00007f3c704e5000)]
  0x00007f3bcc003800 JavaThread "block-manager-ask-thread-pool-328" daemon [_thread_blocked, id=2933, stack(0x00007f3c88aeb000,0x00007f3c88bec000)]
  0x00007f3b2c003800 JavaThread "block-manager-ask-thread-pool-310" daemon [_thread_blocked, id=2093, stack(0x00007f3c813f4000,0x00007f3c814f5000)]
  0x00007f3bd0001800 JavaThread "block-manager-ask-thread-pool-298" daemon [_thread_blocked, id=2063, stack(0x00007f3c90ced000,0x00007f3c90dee000)]
  0x00007f3d98011800 JavaThread "block-manager-ask-thread-pool-269" daemon [_thread_blocked, id=1353, stack(0x00007f3afbdfe000,0x00007f3afbeff000)]
  0x00007f3dcc00d000 JavaThread "Thread-27" [_thread_in_native, id=16795, stack(0x00007f3c69afb000,0x00007f3c69bfc000)]
  0x00007f3dc0008000 JavaThread "task-result-getter-3" daemon [_thread_blocked, id=2833, stack(0x00007f3cac4c3000,0x00007f3cac5c4000)]
  0x00007f3dc0007000 JavaThread "task-result-getter-2" daemon [_thread_blocked, id=2832, stack(0x00007f3cacbca000,0x00007f3cacccb000)]
  0x00007f3dc0006000 JavaThread "task-result-getter-1" daemon [_thread_blocked, id=2830, stack(0x00007f3cac5c4000,0x00007f3cac6c5000)]
  0x00007f3dc0002000 JavaThread "task-result-getter-0" daemon [_thread_blocked, id=2829, stack(0x00007f3cac7c6000,0x00007f3cac8c7000)]
  0x00007f3c0c10e000 JavaThread "stderr reader for /usr/bin/python" daemon [_thread_in_native, id=2794, stack(0x00007f3cacccb000,0x00007f3cacdcc000)]
  0x00007f3c0c10b000 JavaThread "stdout reader for /usr/bin/python" daemon [_thread_in_native, id=2793, stack(0x00007f3cacdcc000,0x00007f3cacecd000)]
  0x00007f3c0c109000 JavaThread "Idle Worker Monitor for /usr/bin/python" daemon [_thread_blocked, id=2692, stack(0x00007f3cacecd000,0x00007f3cacfce000)]
  0x00007f3db41d2000 JavaThread "Executor task launch worker for task 3.0 in stage 5997.0 (TID 20532)" daemon [_thread_blocked, id=2673, stack(0x00007f3d08022000,0x00007f3d08123000)]
  0x00007f3dd1a2e000 JavaThread "spark-listener-group-shared" daemon [_thread_blocked, id=2655, stack(0x00007f3d307f6000,0x00007f3d308f7000)]
  0x00007f3d2c02c000 JavaThread "element-tracking-store-worker" daemon [_thread_blocked, id=2650, stack(0x00007f3d306f5000,0x00007f3d307f6000)]
  0x00007f3c679f6000 JavaThread "spark-listener-group-eventLog" daemon [_thread_blocked, id=2627, stack(0x00007f3d32efe000,0x00007f3d32fff000)]
  0x00007f3c679f5800 JavaThread "spark-listener-group-executorManagement" daemon [_thread_blocked, id=2626, stack(0x00007f3d32cfc000,0x00007f3d32dfd000)]
  0x00007f3c679f4000 JavaThread "spark-listener-group-appStatus" daemon [_thread_blocked, id=2625, stack(0x00007f3d328f8000,0x00007f3d329f9000)]
  0x00007f3c679ec000 JavaThread "context-cleaner-periodic-gc" daemon [_thread_blocked, id=2622, stack(0x00007f3d32bfb000,0x00007f3d32cfc000)]
  0x00007f3c679ea800 JavaThread "Spark Context Cleaner" daemon [_thread_blocked, id=2621, stack(0x00007f3d329f9000,0x00007f3d32afa000)]
  0x00007f3c675ba800 JavaThread "shuffle-boss-7-1" daemon [_thread_in_native, id=2602, stack(0x00007f3d32afa000,0x00007f3d32bfb000)]
  0x00007f3dd0431000 JavaThread "Cleaner Thread" daemon [_thread_blocked, id=2588, stack(0x00007f3d32dfd000,0x00007f3d32efe000)]
  0x00007f3dc400c000 JavaThread "rpc-server-4-1" daemon [_thread_in_native, id=2494, stack(0x00007f3e490dd000,0x00007f3e491de000)]
  0x00007f3dd11b3000 JavaThread "files-client-5-1" daemon [_thread_in_native, id=2487, stack(0x00007f3e491de000,0x00007f3e492df000)]
  0x00007f3dd1185000 JavaThread "executor-heartbeater" daemon [_thread_blocked, id=2484, stack(0x00007f3e492df000,0x00007f3e493e0000)]
  0x00007f3dd116a000 JavaThread "driver-heartbeater" daemon [_thread_blocked, id=2478, stack(0x00007f3e493e0000,0x00007f3e494e1000)]
  0x00007f3dd112d000 JavaThread "dag-scheduler-event-loop" daemon [_thread_blocked, id=2474, stack(0x00007f3e496e1000,0x00007f3e497e2000)]
  0x00007f3dd10fb000 JavaThread "Timer-1" daemon [_thread_blocked, id=2465, stack(0x00007f3e497e2000,0x00007f3e498e3000)]
  0x00007f3dd10f9800 JavaThread "Timer-0" daemon [_thread_blocked, id=2462, stack(0x00007f3e498e3000,0x00007f3e499e4000)]
  0x00007f3dd10f4000 JavaThread "dispatcher-org.apache.spark.internal.plugin.PluginEndpoint" daemon [_thread_blocked, id=2453, stack(0x00007f3e499e4000,0x00007f3e49ae5000)]
  0x00007f3d5401d000 JavaThread "netty-rpc-env-timeout" daemon [_thread_blocked, id=2451, stack(0x00007f3e49ee5000,0x00007f3e49fe6000)]
  0x00007f3dc800d000 JavaThread "heartbeat-receiver-event-loop-thread" daemon [_thread_blocked, id=2447, stack(0x00007f3e49fe6000,0x00007f3e4a0e7000)]
  0x00007f3dd09c6800 JavaThread "SparkUI-54-acceptor-1@6ee7915a-ServerConnector@b90636b{HTTP/1.1, (http/1.1)}{0.0.0.0:4042}" daemon [_thread_in_native, id=2436, stack(0x00007f3e4a0e7000,0x00007f3e4a1e8000)]
  0x00007f3dd09c4800 JavaThread "SparkUI-53-acceptor-0@5578db1-ServerConnector@b90636b{HTTP/1.1, (http/1.1)}{0.0.0.0:4042}" daemon [_thread_blocked, id=2435, stack(0x00007f3e4a1e8000,0x00007f3e4a2e9000)]
  0x00007f3dd09c3000 JavaThread "SparkUI-52" daemon [_thread_in_native, id=2427, stack(0x00007f3e4a2e9000,0x00007f3e4a3ea000)]
  0x00007f3dd09a9000 JavaThread "SparkUI-51" daemon [_thread_in_native, id=2426, stack(0x00007f3e4a3ea000,0x00007f3e4a4eb000)]
  0x00007f3dd0894000 JavaThread "SparkUI-50" daemon [_thread_in_native, id=2393, stack(0x00007f3e4a4eb000,0x00007f3e4a5ec000)]
  0x00007f3dd0892000 JavaThread "SparkUI-49" daemon [_thread_in_native, id=2392, stack(0x00007f3e4a5ec000,0x00007f3e4a6ed000)]
  0x00007f3dd0890800 JavaThread "SparkUI-48" daemon [_thread_in_native, id=2391, stack(0x00007f3e4a6ed000,0x00007f3e4a7ee000)]
  0x00007f3dd088e800 JavaThread "SparkUI-47" daemon [_thread_in_native, id=2389, stack(0x00007f3e4a7ee000,0x00007f3e4a8ef000)]
  0x00007f3dd088d000 JavaThread "SparkUI-46" daemon [_thread_in_native, id=2386, stack(0x00007f3e4a8ef000,0x00007f3e4a9f0000)]
  0x00007f3dd088b000 JavaThread "SparkUI-45" daemon [_thread_in_native, id=2385, stack(0x00007f3e4a9f0000,0x00007f3e4aaf1000)]
  0x00007f3dd0889800 JavaThread "SparkUI-44" daemon [_thread_in_native, id=2384, stack(0x00007f3e4aaf1000,0x00007f3e4abf2000)]
  0x00007f3dd0888000 JavaThread "SparkUI-43" daemon [_thread_in_native, id=2383, stack(0x00007f3e4abf2000,0x00007f3e4acf3000)]
  0x00007f3dd075c000 JavaThread "RemoteBlock-temp-file-clean-thread" daemon [_thread_blocked, id=2380, stack(0x00007f3e4b0f3000,0x00007f3e4b1f4000)]
  0x00007f3dd0759000 JavaThread "dispatcher-BlockManagerEndpoint1" daemon [_thread_blocked, id=2374, stack(0x00007f3e4b1f4000,0x00007f3e4b2f5000)]
  0x00007f3dd04bc800 JavaThread "dispatcher-BlockManagerMaster" daemon [_thread_in_Java, id=2372, stack(0x00007f3e4b2f5000,0x00007f3e4b3f6000)]
  0x00007f3dd03da800 JavaThread "map-output-dispatcher-7" daemon [_thread_blocked, id=2351, stack(0x00007f3e4b3f6000,0x00007f3e4b4f7000)]
  0x00007f3dd03d9000 JavaThread "map-output-dispatcher-6" daemon [_thread_blocked, id=2348, stack(0x00007f3e4b4f7000,0x00007f3e4b5f8000)]
  0x00007f3dd03d7800 JavaThread "map-output-dispatcher-5" daemon [_thread_blocked, id=2344, stack(0x00007f3e4b5f8000,0x00007f3e4b6f9000)]
  0x00007f3dd03d6000 JavaThread "map-output-dispatcher-4" daemon [_thread_blocked, id=2342, stack(0x00007f3e4b6f9000,0x00007f3e4b7fa000)]
  0x00007f3dd03db800 JavaThread "map-output-dispatcher-3" daemon [_thread_blocked, id=2338, stack(0x00007f3e4b7fa000,0x00007f3e4b8fb000)]
  0x00007f3dd03cd800 JavaThread "map-output-dispatcher-2" daemon [_thread_blocked, id=2335, stack(0x00007f3e4b8fb000,0x00007f3e4b9fc000)]
  0x00007f3dd03de000 JavaThread "map-output-dispatcher-1" daemon [_thread_blocked, id=2332, stack(0x00007f3e4b9fc000,0x00007f3e4bafd000)]
  0x00007f3dd03d1800 JavaThread "map-output-dispatcher-0" daemon [_thread_blocked, id=2330, stack(0x00007f3e4bafd000,0x00007f3e4bbfe000)]
  0x00007f3dd039d000 JavaThread "dispatcher-event-loop-3" daemon [_thread_blocked, id=2319, stack(0x00007f3e4bdfe000,0x00007f3e4beff000)]
  0x00007f3dd039b000 JavaThread "dispatcher-event-loop-2" daemon [_thread_blocked, id=2318, stack(0x00007f3e4beff000,0x00007f3e4c000000)]
  0x00007f3dd0399800 JavaThread "dispatcher-event-loop-1" daemon [_thread_blocked, id=2317, stack(0x00007f3e50062000,0x00007f3e50163000)]
  0x00007f3dd0398000 JavaThread "dispatcher-event-loop-0" daemon [_thread_blocked, id=2316, stack(0x00007f3e50163000,0x00007f3e50264000)]
  0x00007f3dd0393000 JavaThread "rpc-boss-3-1" daemon [_thread_in_native, id=2303, stack(0x00007f3e50264000,0x00007f3e50365000)]
  0x00007f3dcc00a800 JavaThread "Thread-5" [_thread_blocked, id=2297, stack(0x00007f3e507cb000,0x00007f3e508cc000)]
  0x00007f3e82a33000 JavaThread "Thread-4" [_thread_in_native, id=2294, stack(0x00007f3e508cc000,0x00007f3e509cd000)]
  0x00007f3e82a25000 JavaThread "pool-2-thread-1" [_thread_blocked, id=2289, stack(0x00007f3e509cd000,0x00007f3e50ace000)]
  0x00007f3e829b9000 JavaThread "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" daemon [_thread_blocked, id=2283, stack(0x00007f3e50ee0000,0x00007f3e50fe1000)]
  0x00007f3e8285c800 JavaThread "process reaper" daemon [_thread_in_native, id=2277, stack(0x00007f3e50fe1000,0x00007f3e5101a000)]
  0x00007f3e80102800 JavaThread "Service Thread" daemon [_thread_blocked, id=2233, stack(0x00007f3e52603000,0x00007f3e52704000)]
  0x00007f3e800ff800 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=2223, stack(0x00007f3e52705000,0x00007f3e52805000)]
  0x00007f3e800fd800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=2221, stack(0x00007f3e52806000,0x00007f3e52906000)]
  0x00007f3e800fb800 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=2219, stack(0x00007f3e52907000,0x00007f3e52a07000)]
  0x00007f3e800f9800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=2217, stack(0x00007f3e52a08000,0x00007f3e52b08000)]
  0x00007f3e800f7000 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=2215, stack(0x00007f3e52b09000,0x00007f3e52c09000)]
  0x00007f3e800f5000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=2213, stack(0x00007f3e52c0a000,0x00007f3e52d0a000)]
  0x00007f3e800f3000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=2211, stack(0x00007f3e52d0b000,0x00007f3e52e0b000)]
  0x00007f3e800f0800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=2209, stack(0x00007f3e52e0c000,0x00007f3e52f0c000)]
  0x00007f3e800ee800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=2208, stack(0x00007f3e52f0d000,0x00007f3e5300d000)]
  0x00007f3e800ec800 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=2207, stack(0x00007f3e5300e000,0x00007f3e5310e000)]
  0x00007f3e800ea800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=2205, stack(0x00007f3e5310f000,0x00007f3e5320f000)]
  0x00007f3e800e7800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=2203, stack(0x00007f3e53210000,0x00007f3e53310000)]
  0x00007f3e800e0000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=2202, stack(0x00007f3e53310000,0x00007f3e53411000)]
  0x00007f3e800b2000 JavaThread "Finalizer" daemon [_thread_blocked, id=2183, stack(0x00007f3e536f7000,0x00007f3e537f8000)]
  0x00007f3e800ad800 JavaThread "Reference Handler" daemon [_thread_blocked, id=2181, stack(0x00007f3e537f8000,0x00007f3e538f9000)]
  0x00007f3e8001f000 JavaThread "main" [_thread_in_native, id=2096, stack(0x00007f3e86063000,0x00007f3e86163000)]

Other Threads:
  0x00007f3e800a3800 VMThread [stack: 0x00007f3e538fa000,0x00007f3e539fa000] [id=2176]
  0x00007f3e80105000 WatcherThread [stack: 0x00007f3e52503000,0x00007f3e52603000] [id=2237]

VM state:synchronizing (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00007f3e8001bab0] SymbolTable_lock - owner thread: 0x00007f3dbc016800
[0x00007f3e8001bf30] Safepoint_lock - owner thread: 0x00007f3e800a3800
[0x00007f3e8001bfb0] Threads_lock - owner thread: 0x00007f3e800a3800

heap address: 0x00000000c0000000, size: 1024 MB, Compressed Oops mode: 32-bit
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x0000000100000000

Heap:
 PSYoungGen      total 326656K, used 25587K [0x00000000eab00000, 0x0000000100000000, 0x0000000100000000)
  eden space 313344K, 3% used [0x00000000eab00000,0x00000000eb6fd358,0x00000000fdd00000)
  from space 13312K, 99% used [0x00000000ff280000,0x00000000fff7fc78,0x00000000fff80000)
  to   space 17920K, 0% used [0x00000000fdd00000,0x00000000fdd00000,0x00000000fee80000)
 ParOldGen       total 699392K, used 600017K [0x00000000c0000000, 0x00000000eab00000, 0x00000000eab00000)
  object space 699392K, 85% used [0x00000000c0000000,0x00000000e49f4700,0x00000000eab00000)
 Metaspace       used 216123K, capacity 284599K, committed 298224K, reserved 1269760K
  class space    used 31471K, capacity 73716K, committed 77552K, reserved 1048576K

Card table byte_map: [0x00007f3e85a2d000,0x00007f3e85c2e000] byte_map_base: 0x00007f3e8542d000

Marking Bits: (ParMarkBitMap*) 0x00007f3e87292700
 Begin Bits: [0x00007f3e5a000000, 0x00007f3e5b000000)
 End Bits:   [0x00007f3e5b000000, 0x00007f3e5c000000)

Polling page: 0x00007f3e87532000

CodeCache: size=245760Kb used=158483Kb max_used=158504Kb free=87276Kb
 bounds [0x00007f3e71000000, 0x00007f3e7ac50000, 0x00007f3e80000000]
 total_blobs=45131 nmethods=43706 adapters=1330
 compilation: enabled
...

Steps/Code to reproduce bug
just run integration_tests/run_pyspark_from_build.sh, this is not always reproducible (30%)

Expected behavior
pass the CI
@pxLi pxLi added bug Something isn't working ? - Needs Triage Need team to review and classify test Only impacts tests labels Apr 28, 2023
@pxLi
Copy link
Member Author

pxLi commented Apr 28, 2023

We met in parquet write, should also be related to rapidsai/cudf#13238

@abellina
Copy link
Collaborator

Seeing a related segfault in the multi-threaded shuffle tests:

Stack: [0x00007fd168377000,0x00007fd168478000],  sp=0x00007fd168473da0,  free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x971b7]  __libc_malloc+0x197

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 101  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007fe32cdc1e58 [0x00007fe32cdc1e00+0x58]
J 4549 C2 java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (101 bytes) @ 0x00007fe32d7f165c [0x00007fe32d7f1300+0x35c]
J 17936 C2 sun.net.www.protocol.jar.JarURLConnection.connect()V (163 bytes) @ 0x00007fe32f9bbddc [0x00007fe32f9ba6c0+0x171c]
J 20978 C2 sun.misc.URLClassPath$Loader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (100 bytes) @ 0x00007fe32de16e4c [0x00007fe32de16d20+0x12c]
J 2796 C2 sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (83 bytes) @ 0x00007fe32d57e064 [0x00007fe32d57da40+0x624]
J 8350 C2 java.net.URLClassLoader$1.run()Ljava/lang/Object; (5 bytes) @ 0x00007fe32e032468 [0x00007fe32e031fc0+0x4a8]
v  ~StubRoutines::call_stub
J 175  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (0 bytes) @ 0x00007fe32cdf5823 [0x00007fe32cdf57c0+0x63]
J 3032 C2 java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class; (47 bytes) @ 0x00007fe32cf82b88 [0x00007fe32cf82b00+0x88]
J 8416 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122 bytes) @ 0x00007fe32e087378 [0x00007fe32e086f20+0x458]
J 4348 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7 bytes) @ 0x00007fe32d767ed0 [0x00007fe32d767ea0+0x30]
v  ~StubRoutines::call_stub
j  com.nvidia.spark.rapids.GpuParquetWriter.deepTransformColumn(Lai/rapids/cudf/ColumnVector;Lorg/apache/spark/sql/types/DataType;)Lai/rapids/cudf/ColumnVector;+12
j  com.nvidia.spark.rapids.GpuParquetWriter.$anonfun$writeAndClose$2(Lcom/nvidia/spark/rapids/GpuParquetWriter;Lcom/nvidia/spark/rapids/GpuColumnVector;)Lorg/apache/spark/sql/vectorized/ColumnVector;+17
j  com.nvidia.spark.rapids.GpuParquetWriter$$Lambda$5674.apply(Ljava/lang/Object;)Ljava/lang/Object;+8
J 32518 C2 com.nvidia.spark.rapids.RapidsPluginImplicits$MapsSafely$$Lambda$1719.apply(Ljava/lang/Object;)Ljava/lang/Object; (13 bytes) @ 0x00007fe333839094 [0x00007fe333839060+0x34]
J 15636 C2 scala.collection.mutable.ArrayOps$ofRef.foreach(Lscala/Function1;)V (6 bytes) @ 0x00007fe32f16ea40 [0x00007fe32f16e9a0+0xa0]
J 38757 C2 com.nvidia.spark.rapids.RapidsPluginImplicits$AutoCloseableProducingArray.safeMap(Lscala/Function1;Lscala/reflect/ClassTag;)[Ljava/lang/AutoCloseable; (26 bytes) @ 0x00007fe32d337688 [0x00007fe32d3374e0+0x1a8]
j  com.nvidia.spark.rapids.GpuParquetWriter.$anonfun$writeAndClose$1(Lcom/nvidia/spark/rapids/GpuParquetWriter;Lorg/apache/spark/sql/vectorized/ColumnarBatch;)Lorg/apache/spark/sql/vectorized/ColumnarBatch;+25
j  com.nvidia.spark.rapids.GpuParquetWriter$$Lambda$5673.apply(Ljava/lang/Object;)Ljava/lang/Object;+8
J 32886 C2 com.nvidia.spark.rapids.Arm$.withResource(Ljava/lang/AutoCloseable;Lscala/Function1;)Ljava/lang/Object; (60 bytes) @ 0x00007fe32f3a81dc [0x00007fe32f3a81a0+0x3c]
j  com.nvidia.spark.rapids.GpuParquetWriter.writeAndClose(Lorg/apache/spark/sql/vectorized/ColumnarBatch;Lscala/collection/Seq;)V+10

I will ping on the cuDF issue.

@abellina abellina mentioned this issue Apr 28, 2023
Merged
3 tasks
@abellina
Copy link
Collaborator

abellina commented May 1, 2023

I am going to close this issue for now. I do see signs that the segfaults have gone away now after the fix in rapidsai/cudf#13240.

@abellina abellina closed this as completed May 1, 2023
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label May 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working test Only impacts tests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@abellina @mattahrens @pxLi