# Run-time error when using spark-submit

**URL:** <https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764>\
**Category:** Hail Query & hailctl\
**Created:** [November 13, 2018, 1:34am UTC](https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764 "2018-11-13T01:34:02Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![whu\_wade](https://avatars.discourse-cdn.com/v4/letter/w/f19dbf/32.png) [@whu\_wade](https://discuss.hail.is/u/whu_wade)\
**Post date:** [November 13, 2018, 1:34am UTC](https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764/1 "2018-11-13T01:34:02Z")

</div>

My submit command like below, and get some error. I have test some hail function, import\_vcf() is ok, but output errors when I use count(),show() ,take() or export\_vcf() such action functions.

spark2-submit --master yarn --jars “$HAIL\_HOME/build/libs/hail-all-spark.jar”   
–py-files “$HAIL\_HOME/build/distributions/hail-python.zip”   
–conf spark.driver.extraClassPath="$HAIL\_HOME/build/libs/hail-all-spark.jar"   
–conf spark.executor.extraClassPath=./hail-all-spark.jar   
–conf spark.serializer=org.apache.spark.serializer.KryoSerializer   
–conf spark.kryo.registrator=is.hail.kryo.HailKryoRegistrator   
–num-executors 10 --executor-memory 20g --executor-cores 4 stat\_call\_rate.py

#stat\_call\_rate.py  
import hail as hl  
import hail.expr.aggregators as agg  
hl.init(app\_name=‘hail\_statCallRate’,log=‘cr.log’,tmp\_dir=‘file:///path\_prefix/population’,default\_reference=‘GRCh38’)  
vcf=hl.import\_vcf(“file:///path\_prefix/test.vcf”)  
vcf=hl.sample\_qc(vcf)  
hl.export\_vcf(vcf,“file:///path\_prefix/test.vcf.bgz”)

ERROR:  
[Stage 1:\> (0 + 7) / 7]Traceback (most recent call last):  
File “stat\_call\_rate.py”, line 8, in   
hl.export\_vcf(vcf,“file:///test\_path/test.vcf.bgz”)  
File “”, line 2, in export\_vcf  
File “/path\_prefix/hail/build/distributions/hail-python.zip/hail/typecheck/check.py”, line 560, in wrapper  
File “/path\_prefix/hail/build/distributions/hail-python.zip/hail/methods/impex.py”, line 422, in export\_vcf  
File “/path\_prefix/SPARK2/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java\_gateway.py”, line 1133, in **call**  
File “/path\_prefix/hail/build/distributions/hail-python.zip/hail/utils/java.py”, line 210, in deco  
hail.utils.java.FatalError: SparkException: Job aborted due to stage failure: Task 5 in stage 1.0 failed 4 times, most recent failure: Lost task 5.3 in stage 1.0 (TID 20, sz-hadoop-55-60.local, executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Container marked as failed: container\_1537175360952\_0263\_01\_000012 on host: sz-hadoop-55-60.local. Exit status: 1. Diagnostics: Exception from container-launch.  
Container id: container\_1537175360952\_0263\_01\_000012  
Exit code: 1  
Stack trace: ExitCodeException exitCode=1:  
at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)  
at org.apache.hadoop.util.Shell.run(Shell.java:504)  
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)  
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:373)  
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)  
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)  
at java.util.concurrent.FutureTask.run(FutureTask.java:266)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:744)

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [November 13, 2018, 11:00am UTC](https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764/2 "2018-11-13T11:00:58Z")

</div>

Are you able to run pyspark on your cluster?

---

<div class="post-metadata">

**Author:** ![whu\_wade](https://avatars.discourse-cdn.com/v4/letter/w/f19dbf/32.png) [@whu\_wade](https://discuss.hail.is/u/whu_wade)\
**Post date:** [November 14, 2018, 1:26am UTC](https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764/3 "2018-11-14T01:26:55Z")

</div>

I tried just now:  
pyspark2 --jars $HAIL\_HOME/build/libs/hail-all-spark.jar --conf spark.driver.extraClassPath=$HAIL\_HOME/build/libs/hail-all-spark.jar --conf spark.executor.extraClassPath=./hail-all-spark.jar --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.kryo.registrator=is.hail.kryo.HailKryoRegistrator

#interactive command  
import hail as hl  
hl.init(sc)  
vcf=hl.import\_vcf(“file:///prefix\_path/test.vcf”,reference\_genome=‘GRCh38’) #that’s all OK until here  
vcf.count() # error as below

[Stage 1:\> (0 + 2) / 7]Traceback (most recent call last):  
File “”, line 1, in   
File “/prefix\_path/hail/build/distributions/hail-python.zip/hail/matrixtable.py”, line 2117, in count  
File “/prefix\_path/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java\_gateway.py”, line 1133, in **call**  
File “/prefix\_path/hail/build/distributions/hail-python.zip/hail/utils/java.py”, line 210, in deco  
hail.utils.java.FatalError: SparkException: Job aborted due to stage failure: Task 4 in stage 1.0 failed 4 times, most recent failure: Lost task 4.3 in stage 1.0 (TID 15, seps-hadoop-a02-4.seps.sz.hpc, executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container marked as failed: container\_e26\_1537320290255\_262214\_01\_000024 on host: seps-hadoop-a02-4.seps.sz.hpc. Exit status: 1. Diagnostics: Exception from container-launch.  
Container id: container\_e26\_1537320290255\_262214\_01\_000024  
Exit code: 1  
Stack trace: ExitCodeException exitCode=1:  
at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)  
at org.apache.hadoop.util.Shell.run(Shell.java:504)  
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)  
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:373)  
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)  
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)  
at java.util.concurrent.FutureTask.run(FutureTask.java:266)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:744)

Shell output: main : command provided 1  
main : run as user is wade  
main : requested yarn user is wade  
Writing to tmp file /export/sdh/yarn/nm/nmPrivate/application\_1537320290255\_262214/container\_e26\_1537320290255\_262214\_01\_000024/container\_e26\_1537320290255\_262214\_01\_000024.pid.tmp

Container exited with a non-zero exit code 1

Driver stacktrace:

Java stack trace:  
org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 1.0 failed 4 times, most recent failure: Lost task 4.3 in stage 1.0 (TID 15, seps-hadoop-a02-4.seps.sz.hpc, executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container marked as failed: container\_e26\_1537320290255\_262214\_01\_000024 on host: seps-hadoop-a02-4.seps.sz.hpc. Exit status: 1. Diagnostics: Exception from container-launch.  
Container id: container\_e26\_1537320290255\_262214\_01\_000024  
Exit code: 1  
Stack trace: ExitCodeException exitCode=1:  
at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)  
at org.apache.hadoop.util.Shell.run(Shell.java:504)  
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)  
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:373)  
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)  
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)  
at java.util.concurrent.FutureTask.run(FutureTask.java:266)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:744)

Shell output: main : command provided 1  
main : run as user is wade  
main : requested yarn user is wade  
Writing to tmp file /export/sdh/yarn/nm/nmPrivate/application\_1537320290255\_262214/container\_e26\_1537320290255\_262214\_01\_000024/container\_e26\_1537320290255\_262214\_01\_000024.pid.tmp

Container exited with a non-zero exit code 1

Driver stacktrace:  
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1499)  
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1487)  
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1486)  
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)  
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)  
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1486)  
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)  
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)  
at scala.Option.foreach(Option.scala:257)  
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814)  
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1714)  
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)  
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)  
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)  
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)  
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2022)  
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2043)  
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2062)  
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2087)  
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:936)  
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)  
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)  
at org.apache.spark.rdd.RDD.collect(RDD.scala:935)  
at is.hail.sparkextras.ContextRDD.collect(ContextRDD.scala:139)  
at is.hail.rvd.RVD$.getKeyInfo(RVD.scala:1063)  
at is.hail.rvd.RVD$.makeCoercer(RVD.scala:1127)  
at is.hail.io.vcf.MatrixVCFReader.coercer$lzycompute(LoadVCF.scala:976)  
at is.hail.io.vcf.MatrixVCFReader.coercer(LoadVCF.scala:976)  
at is.hail.io.vcf.MatrixVCFReader.apply(LoadVCF.scala:1010)  
at is.hail.expr.ir.MatrixRead.execute(MatrixIR.scala:427)  
at is.hail.expr.ir.CastMatrixToTable.execute(TableIR.scala:1169)  
at is.hail.expr.ir.Interpret$$anonfun$apply$1.apply$mcJ$sp(Interpret.scala:695)  
at is.hail.expr.ir.Interpret$$anonfun$apply$1.apply(Interpret.scala:695)  
at is.hail.expr.ir.Interpret$$anonfun$apply$1.apply(Interpret.scala:695)  
at scala.Option.getOrElse(Option.scala:121)  
at is.hail.expr.ir.Interpret$.apply(Interpret.scala:695)  
at is.hail.expr.ir.Interpret$.apply(Interpret.scala:107)  
at is.hail.expr.ir.Interpret$.apply(Interpret.scala:77)  
at is.hail.variant.MatrixTable.countRows(MatrixTable.scala:614)  
at is.hail.variant.MatrixTable.count(MatrixTable.scala:612)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:483)  
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)  
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)  
at py4j.Gateway.invoke(Gateway.java:280)  
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)  
at py4j.commands.CallCommand.execute(CallCommand.java:79)  
at py4j.GatewayConnection.run(GatewayConnection.java:214)  
at java.lang.Thread.run(Thread.java:744)

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [November 14, 2018, 2:30am UTC](https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764/4 "2018-11-14T02:30:18Z")

</div>

to confirm that pyspark alone is working, what happens if you do the following?

```auto
pyspark2

```

then

```python
>>> spark.range(10).show()

```

```auto

```

---

<div class="post-metadata">

**Author:** ![whu\_wade](https://avatars.discourse-cdn.com/v4/letter/w/f19dbf/32.png) [@whu\_wade](https://discuss.hail.is/u/whu_wade)\
**Post date:** [November 14, 2018, 5:53am UTC](https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764/5 "2018-11-14T05:53:15Z")

</div>

pyspark2 works well:

> > > spark.range(10).show()  
> > > ±–+  
> > > | id|  
> > > ±–+  
> > > | 0|  
> > > | 1|  
> > > | 2|  
> > > | 3|  
> > > | 4|  
> > > | 5|  
> > > | 6|  
> > > | 7|  
> > > | 8|  
> > > | 9|  
> > > ±–+

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [November 14, 2018, 12:45pm UTC](https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764/6 "2018-11-14T12:45:52Z")

</div>

What system are you running Hail on, and how did you install Hail?

---

<div class="post-metadata">

**Author:** ![whu\_wade](https://avatars.discourse-cdn.com/v4/letter/w/f19dbf/32.png) [@whu\_wade](https://discuss.hail.is/u/whu_wade)\
**Post date:** [November 14, 2018, 2:48pm UTC](https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764/7 "2018-11-14T14:48:08Z")

</div>

operating system information:  
 ![image](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/651bc81bef8783aad77f6b21fbe88ad23e59af95.png)

When run spark-shell, return the following text:

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/a25b44557f60c4a10d5947754598708cc7550bdd.png)

Install steps: (seems successful)  
git clone [https://github.com/hail-is/hail.git](https://github.com/hail-is/hail.git)  
cd hail/hail  
./gradlew -Dspark.version=2.2.0 shadowJar archiveZip

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [November 14, 2018, 4:17pm UTC](https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764/8 "2018-11-14T16:17:53Z")

</div>

@danking @wang do you think the native libs are the problem?

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [November 14, 2018, 7:16pm UTC](https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764/9 "2018-11-14T19:16:57Z")

</div>

Try

```auto
./gradlew nativeLibPrebuilt
./gradlew -Dspark.version=2.2.0 shadowJar archiveZip

```

Apologies, this is a known compilation bug when compiling manually.

---

<div class="post-metadata">

**Author:** ![whu\_wade](https://avatars.discourse-cdn.com/v4/letter/w/f19dbf/32.png) [@whu\_wade](https://discuss.hail.is/u/whu_wade)\
**Post date:** [November 15, 2018, 2:20am UTC](https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764/10 "2018-11-15T02:20:01Z")

</div>

I re-installed Hail with the two commands you supply, but same errors appear again when running hl.export\_vcf() or count() action functions…

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [November 15, 2018, 3:15pm UTC](https://discuss.hail.is/t/run-time-error-when-using-spark-submit/764/11 "2018-11-15T15:15:50Z")

</div>

> [@whu\_wade](#):
>
> vcf=hl.import\_vcf(“file:///prefix\_path/test.vcf”,reference\_genome=‘GRCh38’) #that’s all OK until here

Oh sorry, I totally missed that this is wrong.

`file:///` refers to a local file, `file:///prefix_path/test.vcf` almost certainly does not exist on every worker node. You need to put the VCF in HDFS.