MarkDuplicatesSpark fails on because of missing inteldeflater

[akiezun]

gcloud dataproc jobs submit spark --cluster markeddupe --properties spark.kryoserializer.buffer.max=512m,spark.driver.maxResultSize=0,spark.driver.userClassPathFirst=true,spark.io.compression.codec=lzf,spark.yarn.executor.memoryOverhead=600,spark.driver.extraJavaOptions=-Dsamjdk.intel_deflater_so_path=libIntelDeflater.so -Dsamjdk.compression_level=1 -DGATK_STACKTRACE_ON_USER_EXCEPTION=true ,spark.executor.extraJavaOptions=-Dsamjdk.intel_deflater_so_path=libIntelDeflater.so -Dsamjdk.compression_level=1 -DGATK_STACKTRACE_ON_USER_EXCEPTION=true ,spark.driver.memory=8G,spark.executor.cores=3,spark.executor.memory=25G,spark.yarn.executor.memoryOverhead=2500 --files /app/build/libIntelDeflater.so --jar /app/build/libs/gatk-all-4.alpha.1-159-gd0db982-SNAPSHOT-spark.jar MarkDuplicatesSpark --shardedOutput true -O /scratch/tmp.md.bam --numReducers 0 --apiKey <API_KEY> -I gs://broad-gatk-test-jenkins/CEUTrio.HiSeq.WEx.b37.NA12892.readnamesort.bam --sparkMaster yarn-client

related to #1903

[akiezun]

@davidbernick for now, the solution is to remove -Dsamjdk.intel_deflater_so_path=libIntelDeflater.so and --files /app/build/libIntelDeflater.so from the commandlines

Closing this one. Will open another one for re-enabling intel deflater on spark

[davidbernick]

I'm running gatk-launch which generates this commandline. Will that be fixed so I can run gatk-launch with spark jobs?

[akiezun]

yes it will be fixed. should be fixed now (via
#1964)

On Fri, Jul 1, 2016 at 3:17 PM, David Bernick [email protected]
wrote:

I'm running gatk-launch which generates this commandline. Will that be
fixed so I can run gatk-launch with spark jobs?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#1960 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AB5rLwJM-Lz14JMdgCM6F8s8LjA7kjJWks5qRWfUgaJpZM4JCjTp
.