Skip to content

Conversation

@elek
Copy link
Member

@elek elek commented Nov 7, 2019

What changes were proposed in this pull request?

I realized multiple JVM crashes in the daily builds:

ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? 
        [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs && /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -jar /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar /workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1 surefire7569723928289175829tmp surefire_947955725320624341206tmp
        [ERROR] Error occurred in starting fork, check output in log
        [ERROR] Process Exit Code: 139
        [ERROR] Crashed tests:
        [ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractRename
        [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
        [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs && /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -jar /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter5429192218879128313.jar /workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1 surefire7227403571189445391tmp surefire_1011197392458143645283tmp
        [ERROR] Error occurred in starting fork, check output in log
        [ERROR] Process Exit Code: 139
        [ERROR] Crashed tests:
        [ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractDistCp
        [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
        [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs && /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -jar /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter1355604543311368443.jar /workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1 surefire3938612864214747736tmp surefire_933162535733309260236tmp
        [ERROR] Error occurred in starting fork, check output in log
        [ERROR] Process Exit Code: 139
        [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
        [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs && /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -jar /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar /workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1 surefire7569723928289175829tmp surefire_947955725320624341206tmp
        [ERROR] Error occurred in starting fork, check output in log
        [ERROR] Process Exit Code: 139 

Based on the crash log (uploaded) it's related to the rocksdb JNI interface.

In the current ozone-build docker image (which provides the environment for build) we use alpine where musl libc is used instead of the main glibc. I think it would be more safe to use the same glibc what is used in production.

I tested with centos based docker image and it seems to be more stable. Didn't see any more JVM crashes.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-2308

How this patch can be tested?

Built and pushed to my own dockerhub account:

docker build -t elek/ozone-build:20191106-1 .
docker puseh elek/ozone-build:20191106-1

And used in the PR builds.

@anuengineer anuengineer merged commit af2a8c7 into apache:build-latest Nov 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants